Distributed AI Workloads and the Future of Network Design

For three decades, data center networks were designed around a simple assumption: traffic flows north-south, between clients and servers. AI has shattered that assumption. Training a frontier model requires tens of thousands of GPUs exchanging gradient tensors hundreds of times per second — traffic that is overwhelmingly east-west, bandwidth-intensive, latency-sensitive, and bursty in ways no traditional leaf-spine fabric was built to handle. This is driving a fundamental shift from hierarchical network structures to high-radix, flat topologies optimized for massive machine-to-machine communication. And as AI moves from training clusters into production inference at scale, the demands are spreading beyond centralized hyperscale facilities into distributed edge, metro, and regional sites. The result: the network is no longer plumbing. It is the performance-defining layer of AI infrastructure. Every millisecond of added latency, every dropped packet in a synchronous training collective, every congestion event during an inference burst directly degrades GPU utilization and model performance. Network design has become an AI workload decision — and the industry is only beginning to internalize what that means.

The use-cases driving this shift are concrete and multiplying. Autonomous vehicles demand sub-10ms edge inference for obstacle detection and real-time sensor fusion. Public safety systems run video analytics locally for threat detection and crowd monitoring, where cloud round-trips introduce unacceptable delay and privacy risk. Autonomous delivery robots distribute perception between on-board AI and nearby edge nodes for fleet coordination. Real-time world models and interactive media — gaming, digital twins, generative environments — require sub-35ms latency that only edge-distributed inference can deliver. Personal AI assistants need edge processing for always-on responsiveness, privacy, and offline capability. In healthcare, hospitals deploy ambient monitoring and surgical assistance AI that demand instant local decisions under strict HIPAA and GDPR compliance. Industrial manufacturing runs real-time defect detection and predictive maintenance on factory floors with limited or intermittent connectivity. Retail environments use local inference for personalized promotions, smart checkout, and real-time inventory management. Smart city systems process traffic flow, air quality, and energy grid data across thousands of distributed sensors. Agentic AI workflows for enterprise automation — customer service chains, IT operations, supply chain orchestration — generate up to 25x more network traffic than standard queries, making distributed placement an architectural necessity. And telecom operators themselves are running distributed AI for RAN optimization, energy management, traffic steering, and predictive fault detection at cell sites and regional hubs rather than centralized NOCs.

Training and inference are fundamentally different computational regimes with fundamentally different network requirements — not two points on a continuum. Conflating them is one of the most expensive mistakes an infrastructure team can make. Training is compute-intensive, highly synchronous, and time-bounded. Every GPU in a distributed cluster must synchronize gradient updates — typically via AllReduce — after each forward-backward pass. If one node is slower, the entire cluster waits. The network requirement is unforgiving: ultra-high bandwidth (400G-800G per port), ultra-low latency (sub-5 microseconds), lossless transport, and predictable performance. Traffic is almost entirely east-west within the cluster. Scale matters enormously — a 2,048-GPU training job generates traffic patterns that can saturate even well-designed fabrics.

Inference is latency-sensitive but asynchronous. Individual requests arrive independently; the system processes them and returns results. Traffic is burstier, less predictable, and spread across more endpoints. Inference demands ultra-low end-to-end latency from request to response — the very requirement driving AI toward distributed edge deployment — but its traffic is asynchronous and bursty rather than synchronous. As models scale — mixture-of-experts, multi-step reasoning, agentic chains — inference is becoming more network-intensive, but the pattern is fan-out, not synchronous collective.

The implication is clear: you cannot design one network for both. Organizations that try will either over-provision for inference, wasting cost, or under-provision for training, wasting GPUs. The network architecture must be workload-aware from the ground up. This is also an energy decision: inference now accounts for up to 90% of a model's total lifecycle energy consumption, making where and how you run inference a question of sustainability and cost, not just latency. Scale-Up refers to GPU-to-GPU interconnect within a server or rack — the latest proprietary interconnects now deliver up to 3.6 TB/s per GPU, providing the tightest, lowest-latency communication for synchronous collectives. Scale-Out covers rack-to-rack and pod-to-pod connectivity within a single data center via 400G/800G Ethernet (RoCEv2) or InfiniBand fabrics, using rail-optimized topologies designed for lossless transport. Scale-Across addresses data center-to-data center connectivity as AI clusters outgrow single facilities. Purpose-built AI Ethernet switching platforms now deliver up to 1.9x higher AllReduce bandwidth over long-haul links through distance-aware algorithms, while industry consortia are defining open, interoperable Ethernet-based architectures specifically for inter-datacenter AI connectivity. Increasingly, enterprises are also using multi-cloud networking to bridge private data centers with cloud providers, balancing data compliance requirements with compute availability across distributed AI environments. This three-tier model is becoming the foundational framework for AI network design — and within it, every design decision shapes what AI can and cannot do.

Traditional TCP/IP is fundamentally insufficient for distributed AI workloads — its protocol overhead and latency make it unsuitable for GPU collective communication, which is why the industry has moved to RDMA-based transports. Proprietary high-performance fabrics dominated AI training in 2023 with roughly 80% market share — sub-2 microsecond latency and native RDMA made them the default for hyperscale clusters. But by mid-2025, Ethernet had taken the lead in new AI backend deployments. The drivers: cost (proprietary fabrics carry a 1.5-2.5x per-port premium), vendor lock-in concerns, and rapid maturation of RDMA over Converged Ethernet (RoCEv2) and AI-optimized Ethernet switching platforms.

The industry's answer is Ultra Ethernet — a new open Ethernet transport layer, defined by the Ultra Ethernet Consortium, with built-in RDMA, advanced congestion control, and multipath capabilities designed from the ground up for GPU collective traffic. These specifications reached maturity in 2025, with major silicon vendors building support into next-generation switching ASICs. For network architects, the practical takeaway: proprietary fabrics remain the performance leader for the largest training clusters, but AI-optimized Ethernet is the strategic direction for most new deployments, and the only viable fabric for distributed, multi-site AI infrastructure. This is where distributed AI workloads meet the telecom operator opportunity. In early 2026, six major operators announced 'AI Grids': geographically distributed AI infrastructure that repurposes existing telecom real estate — central offices, regional hubs, mobile switching centers — as edge inference platforms. The numbers behind this are staggering: telecom operators collectively run approximately 100,000 distributed network data centers worldwide, with enough spare power capacity to offer more than 100 gigawatts of new AI compute over time .

The thesis is compelling. Centralized hyperscale data centers cannot serve every AI use case alone — the latency-sensitive applications driving distributed adoption demand inference at the edge, not hundreds of miles away. A major North American operator’s validation tests demonstrated that edge-based inference can be both cheaper and faster than centralized deployments during burst conditions. This is a structural shift in how AI is delivered. Operators are no longer just carrying AI traffic — they are hosting AI compute. The transport layer must now serve not only subscriber traffic but also GPU-to-GPU communication, model distribution, and telemetry aggregation across a distributed AI fabric. The convergence is going further with AI-RAN, where AI and radio workloads run on shared GPU infrastructure at cell sites, allowing operators to monetize the same hardware for both network processing and commercial AI services. For telecom operators, this is the clearest path yet from connectivity provider to compute-and-AI platform.

Distributed AI at the edge is not just faster — it is greener. A joint position paper by Arm and the Special Competitive Studies Project found that edge-based AI inference can reduce power consumption by up to 60% for comparable tasks, by avoiding the energy overhead of long-haul data transmission, centralized cooling at hyperscale, and redundant data movement. Combined with model optimization techniques like quantization and knowledge distillation — which can cut inference energy by 60-90% — distributed architectures turn the energy equation from a liability into an advantage. For operators managing thousands of distributed sites, this means AI compute can be deployed within existing power envelopes rather than requiring massive new grid connections. The network design playbook is being rewritten. East-west traffic dominates. Lossless transport is non-negotiable. Workload-aware fabric design — differentiated for training vs. inference — is replacing one-size-fits-all leaf-spine. Scale-across architectures must connect distributed AI factories as a single logical cluster. Ethernet is winning the fabric war, but only when purpose-built for AI with RDMA, adaptive routing, and congestion control tuned for GPU collectives. Distributing AI data and models across multiple locations also expands the attack surface, requiring secure, private AI infrastructure with encryption, access controls, and data sovereignty compliance built into the network fabric. Meanwhile, facilities themselves are evolving — rack densities are climbing from traditional 10-15 kW to 30-100+ kW per rack to accommodate GPU clusters, forcing fundamental changes in power delivery, cooling, and physical network cabling. And telecom operators have an unprecedented opportunity to position their distributed infrastructure as the edge layer of a global AI compute fabric.

The organizations that treat network design as an AI-first discipline — not as an afterthought to GPU procurement — will be the ones that extract full value from their AI investments. Treating AI workloads like traditional applications will be a fundamental mistake; they demand specialized, highly adaptable networking built for their unique traffic patterns from day one. The network is no longer just the pipe. It is the platform.