UCIe and in-package optical I/O enable two specific use cases—AI scale-up and memory disaggregation. This post will explain both in detail. But first, let’s begin with a brief recap of what’s new in UCIe 2.0.
What is the UCIe 2.0 Specification?
UCIe is an open specification that facilitates die-to-die interconnects between chiplets within a package. Established in March 2022, UCIe introduced version 2.0 in August 2024, with enhancements that streamline the shift from traditional copper to advanced optical interconnects. This evolution is crucial as current AI and HPC systems demand more bandwidth and efficiency beyond copper’s capabilities. By incorporating protocols and a software stack compatible with PCIe, CXL, and Ethernet, UCIe 2.0 enables these next-generation connections and paves the way for universal chiplet communication across vendors.
UCIe is important because semiconductors are moving toward multi-die systems. This creates a need for standardized die-to-die interconnects to ensure chiplets from different vendors can communicate with one another. The goal is to create an ecosystem of universally compatible chiplets that not only reduces manufacturing costs and complexity but also removes the need for bespoke solutions for each customer. Ultimately, this allows smaller companies to access advanced technologies like optical I/O more easily.
UCIe 2.0 also enables the transition from copper to optical interconnects. Current AI and HPC systems are pushing the limits of copper interconnects, and moving to optical has become a necessity. The UCIe standard includes optical I/O, so any chiplet that is compatible with UCIe can seamlessly integrate Ayar Labs’ optical solutions.
AI systems require complex chips to function together as a single unit. However, these chips increasingly cannot fit into a single package or board due to physical limitations (in the case of memory) or thermal limitations (in the case of compute). So, they still need extremely low-latency and high-bandwidth connections between them.
Copper-based solutions limit the range of connectivity between chips to a few inches for packages and printed circuit boards (PCBs) and to a few meters in the case of copper cables. Optical I/O eliminates copper’s physical limitations, allowing compute and memory units to be distributed over many racks that are tens or even hundreds of meters apart, with only the latency caused by the speed of light.
You can read more about UCIe 2.0 and optical versus copper interconnects in our recent blog post, The Future of Chip Connectivity: UCIe and Optical I/O FAQs Explained.
Use Case #1: AI Scale-Up
AI scale-up involves adding more resources to an AI compute cluster without altering existing infrastructure. This requires creating clusters of GPUs that function in unison. This approach delivers better performance and more compute power for tasks like AI model training, tuning or inference. This is akin to boosting a car’s horsepower.
Note that scale-up differs from “scale-out,” which means connecting larger clusters of GPUs through Ethernet or InfiniBand switching networks. Scale-out generally requires lower bandwidth, does not involve latency sensitivity, and uses data center-scale transmission distances (<2km).
What’s Driving AI Scale-Up?
AI models continue to need more and more compute power. In general, training an AI model requires thousands of GPUs, fine-tuning a model requires hundreds, and inference requires tens of GPUs. As generative AI (GenAI) models continue to grow rapidly in size and complexity, these numbers will likely increase by at least an order of magnitude in each category in the next generation. A typical chassis holds at most eight GPUs, and a rack holds 16-32 GPUs, so even inferencing is a rack-scale operation.
In response, organizations are scaling up. In early 2024, NVIDIA launched the GB200 NVL72 system for this purpose. It’s a multi-node, liquid-cooled, rack-scale system offering 720 petaflops of AI training and 1.4 exaflops of AI inference performance in a single rack. NVIDIA CEO Jensen Huang called it “one giant GPU.”
A 72 GPU system is just the beginning—there will be a natural progression to systems with 256, 512, and even 1,024 GPUs working together as one unit.
Requirements
AI scale-up requires very high bandwidth and very low latency data transmission between closely packed GPUs. Optical I/O provides this.
Optical I/O offers link latency of less than 10ns + time of flight (TOF), and Ayar Labs’ TeraPHY chiplet—when combined with the SuperNova™ light source—provides 4 Tbps of bidirectional bandwidth. Future generations of UCIe-compliant optical I/O chiplets are projected to offer more than 100 Tbps of off-package I/O bandwidth while enabling a connectivity radix of up to 128 ports per package.
How It Works
Here is an example of what a scale-up AI compute cluster might look like, using Ayar Labs’ UCIe-compliant optical I/O chiplets:
Figure 1: Items marked “1” in the above diagram refer to a scale-up scenario. Source: Optical Internetworking Forum (OIF), OFC 2024.
Optical interconnects are used between the compute package and the memory package, between the compute package and an L1 switch, and between the L1 switch and the L2 switch. UCIe facilitates the interconnects between the compute ASIC and the optical chiplet.
The UCIe standard is ideal for AI scale-up because it offers lower latency and equal or better bandwidth density than other high-bandwidth transmission technologies like SerDes. This low latency is essential for AI scale-up.
Use Case #2: Memory Disaggregation
Optical I/O can also be used for xPU-to-memory connectivity (also known as memory disaggregation). This allows memory to be located separately from the GPU while still maintaining very low latency. More memory can be added without needing to be physically “next door” to the GPU.
Disaggregation solves the “memory wall” problem, where overall system performance is limited by memory bandwidth. This leads to increased available memory per GPU and improved system performance overall. Again, UCIe facilitates this transition by simplifying the adoption of optical I/O.
What’s Driving Memory Disaggregation?
Hardware FLOPS have exploded by 60,000x over the past two decades, while interconnect bandwidth has only increased 30x. This means the limiting factor of high-powered systems is the delay in getting data from memory to processors. The growing gap between processor speed and memory bandwidth creates this memory wall—a bottleneck on overall system performance—as the processor spends more time waiting for data from memory.
Adding more memory per GPU can remove this bottleneck. However, with copper interconnects, high-bandwidth memory must be right next to the GPU, or high-speed connectivity will be lost. There isn’t enough physical space to fit in more memory and still maintain connectivity.
Requirements
Moving memory away from the GPU requires ultra-high bandwidth density of a few to tens of terabits per GPU in future systems. Here again, optical I/O can provide this. Today, electrical I/O at data rates above 50 Gbps requires forward error correction (FEC) coding that introduces an added latency of ~100 ns.
How It Works
In this scenario, using UCIe-compatible optical I/O chiplets, the memory can be located up to 2 kilometers away from the compute package while maintaining higher bandwidth densities and < 2 x 5ns + TOF latency.
Building a Robust UCIe Ecosystem for Optical I/O Integration
In summary, the adoption of UCIe enables AI scale-up and memory disaggregation by providing:
- Seamless integration of optical I/O solutions
- Reduced latency and increased bandwidth capacities
- Greater interoperability across different vendor chiplets
The more chiplets that adopt UCIe, the easier it will be for organizations to build these clusters without the need for custom solutions to integrate the optical I/O chiplet. More options and more interoperability will make optical I/O more accessible to smaller companies.
To discover the latest in chiplet ecosystem innovations and UCIe advancement, check out the Chiplet Summit 2025 from January 21-23, 2025, at the Santa Clara Convention Center. On Tuesday, January 21, the UCIe Consortium is sponsoring the Applying Die-to-Die Interfaces workshop from 1:00-5:00 PM. As part of this workshop, Optical Retimer Chiplets for the Next Wave of AI Infrastructure will be presented by Lakshmikant (LK) Bhupathi, Ayar Labs’ VP of products, strategy and ecosystem. Other presenting companies include Alphawave Semi, AMD, Arm, Cadence, and Siemens.
