PCIe 5.0

With the significant increase in data rates, test standards and practices continue to be challenged with each new PCIe release, and PCIe gen-5 is no exception. For example, revised Electrical Idle Exit Ordered Set (EIEOS) and clocking features have impacted test practices at both the hardware and system levels. 
Alternate protocols also require full test support, as this improved versatility now permits other protocols to leverage the proven PCIe physical layer stack. 

• Compute Express Link (CXL) is an industry-supported Cache-Coherent Interconnect for Processors, Memory Expansion and Accelerators
• NVM Express® (NVMe®) defines how host software communicates with non-volatile memory across multiple transports like PCI Express® (PCIe®), RDMA, TCP and more. It is the industry standard for solid state drives (SSDs) in all form factors (U.2, M.2, AIC, EDSFF).

Xgig Analyzers

Fully downward compatible protocol analyzers capable of supporting 32GT/sec data link speed operations, such as the VIAVI Xgig Analyzer, are invaluable for PCIe 5.0 testing and debugging. Superior memory, storage capacity, and segmentation for higher volume upstream and downstream traffic capture allows long sequences to be recorded with specific packets filtered out for robust protocol analysis. See all Protocol Analyzers.

Xgig Exerciser

The VIAVI Solutions Xgig® Exerciser for PCIe 5.0 brings turn-key compliance test capability to the Xgig PCIe product portfolio. 

• Valuable tool for debugging difficult protocol communication problems because it provides full, bit-level, repeatable control over the PCIe data traffic
• Replaces the communication link partner and provides full control over Root Complex or Endpoint testing as shown in the diagram
• Ideal for component level analysis and debugging

 

Exercisers have two modes of operation. Host or Root Complex mode (left) is used to test endpoints.
Endpoint mode (right) is used to test host systems, switch ports and CPUs.

The Xgig Exerciser is approved by the Peripheral Component Interconnect Special Interest Group (PCI-SIG) for use in PCI Express (PCIe) 4.0 protocol compliance certification testing performed at PCI-SIG compliance workshops.  PCIe 4.0 is the highest level of compliance certification currently performed by the PCI-SIG. See all Exercisers. 

Xgig Jammer

Jammers operate in-line between the CPU and endpoints and inject errors in the communication between actual system devices.  See all Jammers.

• Best for system tests and error recovery analysis 
• Simulation of network traffic in real time creates an important litmus test for PCIe 5.0 hardware
• Helps ensure devices perform exactly the way they need to in real world environments
• Essential for mission-critical products 

Interposers

Interposers enable capture of communication between a host system and a device under test (DUT). VIAVI offers a full range of popular PCIe Interposers including:

• Host Test Stand
• CEM
• EDSFF, E1, E3
• U.2, U.3
• U.2 CEM
• M.2
• SFF-8674
• Flying-Lead
• Oculink Slimline

Xgig Tools Suite

The Xgig PCIe 5.0 platform is supported by the Xgig Tool Suite including:

• Xgig Expert™ is a user-friendly interface that aids in rapid data interpretation, performance analysis and troubleshooting.  
• Xgig Serialytics™ builds on Xgig Expert to expose hidden behaviors and events in a trace. It enhances contextual debugging and provides smart analytics for serial protocols
• Xgig Trace Control is the system settings user interface the operator uses to configure the system for testing
•  Medusa Labs Test Tools Suite (MLTT) is an application-based data and signal integrity testing tool enabling developers to identify, locate and resolve errors through stress testing.  
   

Traditionally, PCI Express® protocol analysis is performed with one user conducting a single test on dedicated, single function equipment. Now, multiple users can log into one VIAVI Xgig® 5P16 or 5P8 Analyzer at the same time to perform simultaneous independent tests.

Bifurcation Enables Simultaneous Multi-User (SMU) Flexibility

Users don’t have to be in the same physical location as the Analyzer either. They can log in remotely, from anywhere in the world, if they have access to a network. The graphic below is an example of a 16-lane Analyzer bifurcated into four independent links for separate simultaneous tests by multiple users.

Bifurcation works by dividing a port that is normally one link into two or more links. As shown in the graphic below, a VIAVI Xgig 5P8 can be bifurcated into two links of 4-lanes each—and a 5P16 can be bifurcated into two links of 8-lanes each or four links of 4-lanes each.

The Xgig 5P16 Chassis also Supports Analyzer, Exerciser and Jammer Functions

The Xgig 5P16, our flagship system, supports full-featured Analyzer, Exerciser and Jammer functionality on the same chassis. A flexible, software-defined platform like this simplifies equipment management. Software upgrades can be provided faster and more economically than hardware modules—and can be added at any time enabling managers to conserve funds until the upgrade is needed

Increased device productivity reduces time-to-market and lowers overall TCO

Xgig multi-functional Analyzers create operational efficiencies that are difficult to match with single-function devices. This enables users to streamline configuration and debug processes, so products and components get through testing and to market faster.

Approved for PCI-SIG Certification Testing

The Xgig Exerciser is approved by the Peripheral Component Interconnect Special Interest Group (PCI-SIG) for use in PCI Express (PCIe) 4.0 protocol compliance certification testing performed at PCI-SIG compliance workshops. PCIe 4.0 is the highest level of compliance certification currently performed by the PCI-SIG. VIAVI is actively involved with the PCI-SIG on enabling the PCIe5 compliance tests.

The final release of the PCI Express 5.0 standard in 2019 was the culmination of an accelerated 18-month development cycle deemed necessary to address the escalating performance demands of data-intensive applications.

Like all previous generations, PCIe 5.0 maintains backwards compatibility with past iterations, although the lowest version (speed) between the PCIe slot and connecting card remains the gating factor. In addition to the bandwidth increase, the PCIe 5.0 spec also includes electrical enhancements to improve signal integrity and mechanical updates to improve connector performance.

As has been the case with each successive PCIe release, a coexistence between PCIe Express 5.0 and previous PCIe versions, enabled by the intrinsic backwards compatibility of the PCIe interface, allows the most demanding high-performance applications, such as cloud computing and Artificial Intelligence (AI), to take advantage of the highest available transfer rate, while previous generations of technology remain in use for less demanding applications.

The PCIe 5.0 release date in 2019 was the first of many incremental steps on the path to implementation. Preliminary compliance and interoperability testing must also be completed successfully. These additional milestones may extend introductions of the first certified PCIe 5.0 commercial product offerings well into the future.

With PCIe incorporated into almost every type of computing system in use today, the symbiotic customer and supply chain infrastructure for PCI Express 5.0 includes many of the world’s largest corporations in the electronics, computing, data storage, and E-commerce industries. This includes intellectual property (IP) suppliers enabling PCIe Gen 5 functionality, switch and retimer manufacturers, along with PCIe 5.0 motherboard, storage device, and graphics controller suppliers. These essential building blocks feed into the advanced computer systems and hardware for data centers and other fundamental network deployments.

Market Segments for PCIe 5.0 Early Adoption
Not all enterprise and market segments are expected to adopt PCI Express Gen 5 technology simultaneously. IP markets will require early validation capabilities to ensure functionality in either FPGA or silicon formats. CPU, Ethernet, and select Accelerator segments will also be heavily involved in the early roll-out of PCIe 5.0. Data center servers and high-performance computing (HPC) infrastructure, already struggling to meet the ever-expanding bandwidth and latency demands, will quickly absorb the inherent benefits of PCIe Gen 5 technology as it becomes available.

PCIe 5.0 Protocol Analysis Users
Faster validation and debug cycles are essential for accelerated time to market. This creates a greater demand for advanced PCIe Gen 5 protocol analysis tools for system integration teams performing verification and validation, and debugging teams working to qualify components and resolve interoperability issues. Performance tuning teams for devices, drivers, and application software also reap a strong return on investment from advanced PCIe 5.0 protocol analysis functionality.

The speed doubling convention of PCIe remains consistent with the release of PCIe 5.0. Using the same 128b/130b encoding method that has been standard since PCIe version 3.0, PCIe 5.0 will deliver 64 GB/sec of throughput in each direction. Since PCIe technology allows data to flow full duplex bidirectionally, the total throughput for both directions combined amounts to 128 GB/sec.  

The encoding standard prior to PCIe 3.0 was 8b/10b, meaning eight bits of data were encoded and transmitted as a 10-bit number. This in turn created a 20% performance overhead factor that reduced a raw bit transfer rate of 2.5 GT/sec to a net bandwidth of just 2.0 Gbit/sec. This more efficient 1.5% overhead factor remains in effect with the PCIe 5.0 encoding convention.

The phenomenal speed of PCIe 5.0 enables the equivalent of a typical Blu-ray disc contents to be transferred to non-volatile memory (NVM) on a PCI 5.0 motherboard in under one second. Although this exceptional speed may seem like a luxury, it has been necessitated by enhancements to network architecture in other arenas. For example, 400G Ethernet requires 50 GB/sec of bandwidth in each direction to interface with the CPU at maximum capacity.

With PCIe 4.0, the 32 GB/sec available on a full size x16 slot proved to be insufficient. Using PCIe 5.0 technology, the available bandwidth exceeds the requirements of this interface with room to spare.

In addition to Ethernet, this continuous improvement cycle is driven by the advent of real-time systems with latency-critical performance requirements, such as autonomous driving, defense applications requiring instantaneous response, and critical financial security applications where hacking attempts must be thwarted immediately. Individual users of multi-GPU systems and premium graphics cards also reap tangible benefits from PCIe 5.0 speed and bandwidth enhancement.

The PCIe 5.0 specification might be classified as a natural evolution of the backwards compatible PCIe standard with no inherent link or transaction layer changes included in this iteration. The PCIe 5.0 specification continues to benefit from the scaled flow control and extended tags and credits established with PCIe 4.0.

A new CEM connector designated for add-in cards has also been added. Improvements to signal integrity and connector design features have improved overall performance and reliability. Physical layer enhancements also include an update to the EIEOS, SKP ordered sets, and equalization sequences.

The PCI Express 5.0 specification has been generally lauded by hardware manufacturers and industry insiders. In particular, the enhancements made to improve testability, accelerate link training, and provide alternate protocol support have been recognized as exceptional PCIe 5.0 characteristics. This industry consensus, along with the relatively benign set of implementation prerequisites required to transition from PCIe 4.0 to PCIe 5.0, have spearheaded the aggressive hardware development and commercialization targets.

Advancements and specification changes accompanying the PCIe 5.0 release have further amplified many of the PCIe architectural challenges that complicate test and development activities. This includes a 2x reduction requirement for transmitter (Tx) jitter and a 3x reference clock jitter reduction.

Signal loss remains an ongoing architectural challenge with PCIe 5.0. Retimers, redrivers and alternate printed circuit board (PCB) base materials are some of the cost/benefit options available to hardware designers to mitigate these concerns. A retimer can be used to retransmit the signal or a redriver can be used to amplify it, with either method improving the physical reach of the link.

The PCI Express 5.0 specification defines a channel loss budget of 36db, which is only ~28% greater than the PCIe 4.0 threshold. The insertion loss inherent to PCIe technology continues with the PCIe 5.0 release, so the loss levels experienced with FR4 PCB construction are no longer tenable and alternate materials like MEGTRON, costing up to 2.5 times more than FR4, have become compulsory. New equalization circuit designs for the transmitter (TX) and receiver (RX) have also been necessitated by the channel requirements for PCI Express 5.0, and lane margining at Rx for both voltage and timing has become mandatory.

In making the jump from PCIe 4.0 to PCI Express 5.0, speed doubling, backwards compatibility and an accelerated release cycle were three essential pillars upon which the implementation strategy was constructed.

New features that enabled or supported the speed increase were prioritized over other recommended or requested changes. For example, the EIEOS and data bit rate definition changes were required to enable the speed increase, but fundamental PCIe elements such as the encoding method and target bit error rate (BER) remained constant. The signaling and scrambling schemes also remained consistent with PCIe Gen 4, and existing transmitter (Tx) and receiver (Rx) test methods were leveraged where possible to minimize the implementation impact.

Despite the emphasis on time-to-market and compatibility, other important design changes between PCIe 4.0 and PCIe 5.0 necessarily impacted mating hardware and test practices. These included clock data recovery (CDR) with a second-order response and a CEM connector that is only compatible with a surface mount PCBA footprint, although it remains backwards compatible at the add-in card interface. PCI Express 5.0 also supports alternate protocols, through modified TS1/TS2 sequences.

Consistency between PCIe 4.0 and PCIe 5.0 standards was further necessitated by the unusually long release time for the 4.0 standard, as the network landscape and bandwidth demand continued to unfold in the background. This has virtually assured an overlap period between the two standards, making commonality in design and test practices essential for a smooth transition.

PCIe 5.0 Physical Layer Updates
In addition to the more stringent jitter requirements, channel loss budget constraints, and lane margining requirements for voltage and time that accompanied the PCI Express Gen 5 release, additional physical layer changes were necessitated by the speed increase, while other advancements were included to maintain the requisite backwards compatibility with previous PCIe versions.

Ordered set changes were an important modification accompanying the PCI Express 5.0 specification release. The EIEOS ordered set is used to facilitate an exit from an electrical idle state. The familiar pattern of sixteen zeroes and ones used for each PCIe 4.0 ordered pair became thirty-two zeroes and ones repeated for each lane in the PCIe Gen 5 convention. Back-to-back (repeated) EIEOS signals are an additional PCIe 5.0 protocol change. The Start of Data Stream Ordered Set (SDS) has also been updated, so that PCI Express Gen 5 data stream origin points can be clearly distinguished by the receiver. 

Training sequences (TS1/TS2) have benefitted from innovative new options intended to facilitate the PCIe Gen 5 speed doubling. Training sequences are a necessary precursor for link bring-up and equalization (EQ), but can also cause delays as the ordered sets progress through each speed support increment, starting from 2.5 GT/sec and moving step-wise to the 32.0 GT/sec PCIe Gen 5 speed. To remedy this dilemma, EQ bypass options have been provided to essentially “skip” the intermediate speed equalization levels, or to omit equalization altogether by using the “No EQ” option for immediate transition to the L0 active data transfer state.

New fields for alternate protocol ID and enhanced precoding support have also been added to the modified TS1 and TS2 of PCIe 5.0. Once the negotiation between system and device is successful, the link can immediately move to the L0 state at the highest supported speed and begin to transfer data using the negotiated alternate protocol. Should an alternate protocol negotiation fail, the system can quickly revert to the backbone PCI Express 5 protocol.

A nearly unlimited variety of link conditions and scenarios creates troubleshooting challenges during both the bring-up (pre-L0) phase and fully-up (L0) phases of PCIe 5.0 link activity. The best PCIe Gen 5 test solutions enable a layered, systematic approach that significantly reduces troubleshooting time and effort while improving ongoing system performance metrics.

Link bring-up issues observed during the Link Training and Status State Machine (LTSSM) state include signal integrity and detection issues, incorrect link speeds, and other potential conditions that can be effectively diagnosed using a protocol analyzer. After the L0 state is attained, performance inefficiencies such as excessive replays, recovery issues, and delays at various layers of the protocol stack can also be efficiently detected and mitigated with the assistance of an advanced PCIe protocol analyzer.

One common PHY layer condition requiring precision diagnostic capability is experienced during the exit state from electrical idle (EI), where mismatches between Tx and Rx logic can introduce latency. The VIAVI Xgig Analyzer identifies these conditions reliably by employing short lock times and advanced post-processing capabilities, thereby minimizing the loss of captured data during low power state transitions.  

Real-time metrics for the Physical layer are extremely important for monitoring overall signal health and the recovery data of links. For example, when the link is operating in the L0 state, repeated replays and recoveries, potentially creating a significant downgrade in system performance, often go undetected. Xgig Real-time monitoring, metrics, and post-capture analysis features with per-lane resolution effectively monitor and diagnose non-acknowledgments (NAKs), replays, link errors, and flow control (FC) statistics such as receiver buffer overflows and excessive transaction queue depths.

The breakneck cadence of PCIe release dates seems destined to continue, with the final specification release for PCIe 6.0 expected in 2021. This new iteration will continue the traditional bandwidth doubling and backwards compatibility that have signified the PCIe standard, this time reaching a staggering 256 GB/sec of bidirectional bandwidth. This will effectively put PCIe on par with the VRAM bandwidth of a low-end GPU.

To enable yet another two-fold speed increase and maintain high reliability standards, pulse amplitude modulation (PAM4) and forward error correction (FEC) technology will be incorporated. Artificial Intelligence and Machine Learning are potential beneficiaries of this enhancement, as their performance relies upon a combination of exceptional speed, low latency, and rapid access to multiple peripherals simultaneously.

PCIe Gen 5 is yet another successful leap in I/O bussing technology. It seems that the PCIe 5.0 specification will be keeping pace with Moore’s law while shedding the networking architecture bottleneck mantle for the foreseeable future. With new and improved PCIe 5.0 test tools arriving every day, this progress should continue successfully through the release of PCIe 6.0 and many future generations to come.