What is PCIe 6.0?

Peripheral Component Interconnect Express (PCIe)® is the standard interface for high-speed component connections to the CPU. The PCIe standard first introduced by the PCI Special Interest Group (PCI-SIG)® in 2003 is now entering its sixth successful generation. PCIe 6.0 brings a host of new features and capabilities designed to support emerging compute-intensive applications like Artificial Intelligence (AI) and Machine Learning (ML).  

Like all past releases, PCIe Gen6 remains backwards compatible with previous generations while doubling the data rate. PCIe 6.0 bandwidth supports the transition to 800G Ethernet for cloud computing applications while providing plenty of headroom for next-generation GPUs and ultra-fast non-volatile memory express (NVMe)® and Compute Express Link (CXL)™ solutions. For the first time, significant modulation (PAM4) and error correction (FEC) updates were necessary to continue the bandwidth doubling tradition. 

Click to enlarge. Image credit: PCI-SIG

Important updates were made to the PCIe protocol to provide increased speed and lower latency: 

• PAM4 Modulation 
  PAM4 modulation supports high-speed data transmission by using four amplitude pulses, each containing two bits, to double the bandwidth of binary non-return-to-zero (NRZ) modulation. The incorporation of PAM4 has made data rates of 64 GT/s possible while reducing channel loss and extending reach. 
• Forward Error Correction (FEC) 
  While PAM4 is a key to higher throughput for PCIe Gen 6, tightly spaced voltage levels lower the signal to noise ratio (SNR) and increase the bit-error rate. Forward Error Correction (FEC) compensates for this performance trade-off. The PCIe 6.0 specification limits additional latency from FEC to under two nanoseconds. FEC and CRC (cycle redundancy checks) combine to keep the link retry probability low. 
• FLIT Mode has turned the addition of FEC into a benefit rather than a hindrance. Flow control unit (FLIT) based encoding was incorporated into PCIe 6.0 to accommodate the fixed packet size requirements of FEC. This change also simplified data management processes, reduced latency, and improved efficiency by eliminating the need to frame packets at the physical layer. 

The Impact of PAM4 Signaling 

The addition of pulse amplitude modulation with 4-levels (PAM4) to PCIe Gen 6 was significant. The upgrade was deemed necessary by PCI-SIG to enhance PCIe 6.0 performance and support future technology markets. PAM4 signaling has continued to expand across high-speed network elements since its addition to the IEEE 802.3 Ethernet standard in 2017. 

Click to enlarge. Image credit: PCI-SIG

• PAM4 vs NRZ: Basic non-return-to zero signaling was sufficient for all previous PCIe generations, with just two signal levels (1 or 0) representing digital information through a positive or negative voltage. PAM4 enables higher bandwidth by utilizing four discrete signal levels (0,1,2,3) with each level carrying two bits of logic.  
• Signal Analysis: The incorporation of PAM4 presents new technical challenges based on higher levels of signal noise, distortion, and dispersion. PAM4 relies on three-eye diagrams to assess signal integrity and lane margining in terms of voltage and time margining. Predistortion and adaptive equalization methods help to mitigate signal impairment issues.  
• Interoperability: Replacing the 128b/130b encoding scheme that had been in place since PCIe 3.0 with PAM4/FLIT encoding also introduced interoperability challenges. To address this issue, PCI-SIG incorporated new circuitry and logic to coexist with the existing NRZ circuitry and logic. This additional overhead was a necessary trade-off to prevent the allocation of separate pins and slots for each mode.  

PAM4 and FEC get most of the attention when the PCI Express base specification revision 6.0 is discussed. These high-profile features improved speed and performance while adding complexity to protocol analysis, error detection, jamming, and trace analysis test processes. Other enhancements in the PCIe6 specification include:      

• Cyclic Redundant Check (CRC): In addition to FEC, cyclic redundant check error detection was added. While FEC works by sending additional data to correct errors at the receiver, CRC coding detects errors. This secondary error correction mode made the “lightweight” version of PCIe 6.0 FEC possible and minimized latency. 
• New L0p Power State: PAM4 signaling draws more energy, so design upgrades that reduced power consumption were a high priority. The new low power state known as L0p acts like a dimmer on a light switch, allowing bandwidth to be scaled up or down without interrupting the data flow. This has resulted in improved power efficiency over PCIe 5.0 
• Component Measurement and Authentication (CMA): Security features were also a priority for the PCIe 6.0 specification. CMA uses firmware to create a cryptographic signature for each PCIe device. Engineers and IT teams verify signature accuracy to detect or prevent security issues. Integrity and data encryption (IDE) is another new security feature that protects PCIe 6.0 FLIT packets from cyberattacks. 

Despite the design and efficiency improvements, public and industry interest remain focused on PCIe Gen 6 speed. By extending the doubling tradition, PCIe 6.0 speed has reached an unprecedented level. This has created new possibilities for the PCIe interface while raising performance levels for existing use cases. 

• Compute Express Link (CXL) has leveraged the support for alternative protocols that began with PCIe 5.0 to establish a common memory pool for connected devices. Initial CXL implementations were deployed using PCIe5 speeds. CXL applications are expected to expand even further with PCIe6.   
• 800G Ethernet requires 100 GB/sec of bandwidth over eight lanes. Hyperscale data centers have quickly made the leap from 400G to 800G for high-density switches and other bandwidth-hungry applications. Rapid data center evolution makes the 128 GB/sec. of a x16 PCI Express 6.0 link mandatory. 
• The Internet of Things (IoT), driverless transportation, and factory automation (Industry 4.0) are among the data-intensive applications that will continue to expand as new 5G networks are deployed. The 64 GT/sec data rate and low latency of PCIe 6.0 supports the requirements of these emerging use cases. 
• Military and Aerospace applications have continually availed PCIe high-speed data transfer for advanced satellite communication, high-resolution sensors, and on-board data processing systems with increasing bandwidth and latency requirements. PCIe 6.0 speed doubling helps to support the continued roll-out of new technology for these industries.  
• Hyperscale Datacenters are early adopters of PCIe 6.0 specifications. While consumer products might lag behind by a generation or more, the hyperscale ecosystem stands ready to incorporate PCIe Gen 6 speed to maintain performance, scalability, and efficiency expectations. PCIe 6.0 SSDs utilizing stacked die technology help to alleviate hyperscale storage capacity constraints.  
• Artificial Intelligence (AI) and Machine Learning (ML) utilize PCIe 6.0 technology for high bandwidth GPU and AI accelerator interfaces. PCIe Gen 6 supports the ultra-fast transfer speeds across multiple processors required for ML training cycles. Disaggregated, high-performance computing (HPC) in hyperscale data centers can accommodate the requirements of AI engines.  
• Quantum Computing has the potential to redefine processing speeds, using quantum mechanical principles to replace traditional computing bits with multi-state qubits. Although quantum computing has yet to hit the mainstream, exponentially faster processing adds little value without the data transfer speeds and error correction capabilities of PCIe 6.0 to support it.
  
  

  
  Click to enlarge. Image credit: pcworld.com

PCI 6.0 and Beyond 

The PCIe 6.0 release date in January of 2022 was the culmination of a three-year development cycle that included significant architectural challenges and improvements. By maintaining backwards compatibility while doubling speed yet again, the developers of PCIe Gen 6 upheld expectations for new PCIe releases and set the bar higher for future generations. As part of the official PCIe Gen 6 spec announcement, PCI-SIG acknowledged the advanced applications for PCIe 6.0 technology that have already outgrown the limitations of PCIe 5.0 interfaces and SSDs.  

PCI-SIG is expected to maintain its three-year release cadence into the foreseeable future with the release of PCIe 7.0 expected in 2025. Along with 512 GB/sec of bi-directional throughput over a 16-lane connection, PAM4 modulation and FEC will remain in place. Additional design enhancements to improve efficiency and reach will help the latest HPC, AI/ML, and quantum computing technologies reach the next performance plateau. PCIe test and troubleshooting processes will be the key to sustaining this wave of improvement into the future.