PCIe Test Equipment

Peripheral Component Interconnect Express, commonly known as PCI Express and abbreviated as “PCIe”, is a high-performance computer bus architecture which has become the industry standard for nearly all internal computer component connections.

PCIe provides the connections from a computer’s processor(s) and memory to other components and peripherals. This interface is a high-speed serial transport designed to replace older parallel bus architectures including PCI and ISA (Industry Standard Architecture). PCIe uses a point-to-point access bus that provides outstanding data transfer speed improvement over its predecessors. 

In the 1980’s, early computer bus architecture was based on parallel data paths between the CPU and all components.

By the late 1990’s, the predecessors of PCIe had been developed, integrating the familiar form factor of connecting a device card that slides into corresponding locations on the motherboard. The PCI bus came in 32-bit and 64-bit versions, while the AGP (Accelerated Graphics Port) was developed specifically for graphic controller connections with greater bandwidth needs than the PCI bus could provide. Today both interfaces have largely been supplanted by PCI Express.

By the early 2000’s, demands on processing bandwidth by internal hardware and peripherals had led to the next generation of PCI, known as PCI Express, or PCIe. Its biggest advantage over conventional PCI is that it utilizes a serial interface, rather than the parallel interface used in PCI architecture. Another advantage is the use of individual, rather than shared busses, for each connected device.

Data transfer is further enhanced with the utilization of multiple lanes for the connected devices that require them. Defined by the number of data port connections assigned, each increase in lanes, i.e. x1 (by 1) to x16 (by 16), represents a corresponding increase in data transfer volume. Now the passengers headed from point A to point B can ride on multiple busses simultaneously, if the capacity demand requires it.

With the widespread acceptance and popularity of PCIe as the de facto standard, a variety of functional components have adopted this platform, including USB, Bluetooth products and video cards. Video controllers and storage devices are examples of a technology that have been enabled by the advent of PCIe architecture.  This is evidenced by the improved graphics and computer performance we have experienced over the past decade.

The breakthroughs included in the architecture of PCI Express versions have enabled substantially faster transfer rates, as compared to PCI, and each successive version has continued to build upon these improvements. Whereas conventional PCI utilized common address, data and control lines, the point-to-point topology of PCIe provides separate links to each connected device, freeing each from the potential constraints of the others.

PCIe communication utilizes a mode of data transfer known as “packets”. The transaction layer of the PCI Express port performs the task of packetizing and de-packetizing the data. This difference in electrical signaling inherent to PCIe vs. conventional PCI required distinct electronic and connector architecture to be developed, making PCIe slots incompatible with PCI cards.

PCI Express Architecture – Inside Server

---

NVME Over Fabrics – Outside Architecture

A variety of formats have been developed for PCI Express, each with slightly different nomenclature. It is important to learn what these PCIe formats indicate in terms of size and version. These are key resources for understanding what expansion cards will (or will not) work with your system.

PCI Express Versions

Since the introduction of PCIe 1.0 in 2002, successive releases have been introduced to meet the growing demand for bandwidth and speed. Version 1.0 had an available bandwidth of just under 8GB/sec for a combined 16 lanes of traffic while the transfer rate for 1.0 was 2.5 gigatransfers per second (GT/s). With each new release, the total bandwidth has doubled, eventually leading to 128GB/sec of bandwidth for PCIe 5.0, released in 2019. At the same time, the transfer rate has incrementally risen from 2.5 to 32.0 GT/s. Along with performance improvements, each version has introduced new features and increased energy efficiency.

With a lofty standard of doubling bandwidth every few years to avoid becoming the bottleneck as processor speed and memory continues to increase rapidly, each revision presents new challenges for designers. Potential for crosstalk and electrical discontinuity increases along with bandwidth, so new materials and design innovations continue to push the envelope.

PCIe Sizes

The size of any PCI Express card, as well as the number of pins, is dictated by the number of lanes and total connections. The same holds true for all PCIe versions. Available sizes and their corresponding pin counts are as follows.

Unlike PCI cards, a PCI Express card may be installed in any size PCIe slot, as long as the slot is at least as large, if not larger than, the card. This means an x1 expansion card can be installed in an x1, x4, x8 or x16 slot. The same applies in reverse, meaning you can install an x16 card into an x4 slot, for example, but only if the slot is the type configured with its rear side open. In this case, however, the bandwidth would be limited to that of an x4 card.

The bandwidth available to the PCIe card is also driven by the revision of the CPU or motherboard controller, meaning a version 5.0 card will only operate at the bandwidth of a version 4.0 card, if the controller was configured for version 4.0.

While the original PCIe 1.0 had a low available bandwidth of just 8GB/sec, doubling the speed with each iteration has raised the bar for PCI Express 5.0 to 128GB/sec over 16 lanes of traffic. As the bandwidth increases, so does the potential for crosstalk and discontinuity, making recent innovations in PCB trace materials and lane margining a prerequisite for this rapid evolution. The technical versatility of the PCI Express format is another important factor, and the focus on interchangeability has influenced the designers of PCIe test equipment and hardware in equal measure. Backwards compatibility between revisions has remained a hallmark, and slot sizes of 1x through 16x are congruous with any size PCI Express card, with the smaller of the two items dictating the bandwidth availability.

PCIe and NVMe

Non-Volatile Memory Express (NVMe) is the solid-state drive (SSD) memory protocol developed specifically for use with PCI Express. “Non-volatile” in this context refers to memory that does not require power to retain data. NVMe has leveraged the PCIe architectural advantages to obtain read and write speeds more than twice as fast as a 12Gbps SAS interface. Instead of utilizing a traditional I/O controller, NVMe leverages a direct connection to the CPU via PCI Express. In order to test the PCIe/NVMe combination effectively, test equipment with high-end memory capacity and segmentation features is integral to accurate NVMe traffic decoding.

The use of NVMe over Fabrics (NVMe-oF) has introduced additional PCIe test challenges by increasing the range between the NVMe storage device and host, while maintaining a goal of 10µs or less of additional latency introduction. Although the NVMe-oF specification is inherently similar to the NVMe base specification, characterizing the transport mechanisms adds additional complexity. Utilizing the appropriate PCIe/Ethernet/Fibre Channel Analyzer/Jammer test equipment at both the NVMe Enabled Host and NVMe Device ends of the circuit is a recommended best practice.

PCI Express 4.0

Released in October of 2017, PCI Express 4.0 brought improvements such as lane margining and latency reduction to go along with the customary speed and bandwidth improvements inherent to each new generation. Extended tags and credits and enhanced I/O virtualization are additional enhancements included with the PCIe 4.0 release that will continue to be optimized with each successive generation. The adoption of Gen 4 has given added visibility to many of the PCI Express test challenges brought about by increased speed and resultant insertion loss and margin requirements. Traffic flow monitoring, data storage and error detection functions require a higher performance standard from PCI Express tools, and this trend has continued unabated with Gen 5 and beyond.

PCI Express 5.0

The release of PCIe 5.0 in May of 2019 followed an accelerated development cycle focused on achieving the requisite two-fold increase in speed and bandwidth while minimizing new features or significant architectural changes. Gen 5 has continued with the 128b/130b encoding scheme, NRZ signaling and jitter parameters of Gen 4. To complement the 2X speed increase, electrical design emphasis was placed on signal integrity and equalization. The resulting combination of I/O throughput and performance has made Gen 5 a viable option for 400G Ethernet links and demanding cloud-based networking.

Valuable Possibilities with PCIe Test Equipment

Since the release of Gen 1, the challenge of effective PCI Express testing has been addressed through the development of a versatile assortment of PCIe test equipment and software. These tools have been designed to accurately measure performance, rapidly detect issues and simulate use conditions for robust and comprehensive analysis. New and innovative protocol/jammer combinations have proven to be the ideal complement for complex use conditions such as NVMe-oF.

Analyzers with detailed visibility into traffic flow and link performance enable comprehensive verification of the PCIe 5.0 protocol while supporting the burgeoning test solution requirements for transfer speed and memory. Jamming tools with the capability to manipulate live traffic create a robust level of simulation that otherwise would not be possible. The software behind these cutting-edge PCIe tools continues to tie it all together, with ever-improving interface and reporting enhancements that seamlessly unite the operator with the PCI Express test case.

The PCI Special Interest Group (PCI-SIG) defines the specifications and compliance testing required to guarantee interoperability of PCI Express systems. The group was formed in 1992 and now has over 800 member companies developing products based on PCI-SIG released specifications. These specifications are free to their member organizations or individuals. PCI-SIG also offers PCIe testing, technical support, and compliance workshops. The PCI-SIG compliance library includes a comprehensive list of recommended equipment.

The PCI Express test categories included in the PCI-SIG specifications include electrical testing, configuration testing, link protocol testing, transaction protocol testing and platform BIOS testing. Channel topologies have become much more complex as the data rates have increased. Using simulation to optimize power and signal integrity is a recommended practice for PCIe link evaluation. Determining whether data packets are reliably transferred can be performed by utilizing protocol validation at the physical layer. Transaction protocol testing can check and debug unwanted errors and ensure PCI-SIG compliance.

A wide array of PCIe test tools are now available. With each successive generation, PCI Express test equipment has evolved to meet the increasingly stringent demands, enabling new and exciting solutions. Feature-rich tools from industry-leading PCI Express test equipment suppliers are ideally mobile and rugged, with readily-available training and certification. Given the backwards-compatible nature of the interface, interoperability for multiple sizes and versions and rapid upgradability are other overall characteristics of outstanding PCIe test equipment.

Jammers

A jammer can manipulate live traffic to simulate errors in real time. The term “jamming” originated in the early 20th century to describe the deliberate interference introduced to disrupt enemy digital communication lines. For PCIe testing, a jammer is an inline error injection tool that can simulate real-world conditions and shorten test cycles. Often using pre-defined automated test scripts, a jammer can recreate a wide variety of error-testing scenarios.

Jammers such as the Xgig are highly intelligent and protocol aware, and can utilize conditional jamming to maintain control over the test process and ensure comprehensive test coverage. Working in conjunction with a protocol analyzer or other PCIe tester, the jammer produces discernable triggers at the error injection points. By introducing errors into real-world environments, the responsiveness and efficacy of the error recovery process can be accurately discerned.

Analyzers

The protocol analyzer is a versatile PCI Express product tool for bus throughput and link performance measurement as well as packet monitoring and recording. Additional triggering, error reporting and filtering features can enable rapid error identification. Jamming capabilities can artificially create latencies and retransmissions to exercise error-detection capabilities.

Powerful analyzers with advanced trace analysis, traffic flow visibility and memory segmentation features are invaluable. The Xgig 4K16 Protocol Analyzer/Jammer allows simultaneous error injection and protocol analysis. Interoperability features include lane width support for x1, x2, x4, x8 and x16. Users are alerted to errors at every layer of the stack and advanced memory utilization empowers the simultaneous capture of multiple traces.

Software

PCIe test software is the backbone upon which protocol analyzer and jammer technology has continued to meet the increasing speed, functionality and versatility requirements. Well-designed software can help automate repetitive functions, create customized routines based on released specifications and integrate multiple tools for seamless functionality. Advanced reporting software can simplify complex data analysis and facilitate optimized interpretation.

The Xgig Expert software provided with all Xgig analyzers creates a user-friendly interface that aids in rapid data interpretation, debugging and troubleshooting. The software efficiently sorts through captured events to characterize performance and interoperability issues, identify physical layer issues including protocol violations and organize results into comprehensive reports on demand. Additional reporting software can enable efficient data management filtering, analysis and presentation capabilities.

Troubleshooting PCI Express failures can often become a challenging task. Fortunately, many readily available PCIe tools also provide exceptional debugging and troubleshooting capabilities. Keys to effective testing and troubleshooting include increased visibility of traffic flow and insight into the most observed hardware issues. These issues include link speed problems such as equalization failures, traffic issues and quality issues observed after recovery. Error-reporting software can find and log application-specific errors and characterize error types, such as time-outs and data transfer stops.

A protocol analyzer equipped with analytical software can become an important and effective PCIe troubleshooting tool. The analyzer will enable the easy view of traffic flow and historical data necessary to pinpoint issues and chronology. The analyzer also has powerful trigger conditions with the ability to capture extremely long sequences and filtering capabilities for specific packets that further enhance investigative prowess. The logging of time-ordered sequences facilitates troubleshooting by detailing events directly before and after reported errors.

Since the release of Gen 1, the challenge of effective PCIe testing has been addressed through the ongoing development of a versatile assortment of test equipment and software. These tools have been designed to accurately measure performance, rapidly detect issues, and simulate use conditions for robust and comprehensive analysis. New and innovative protocol/jammer combinations have proven to be the ideal complement for complex use conditions such as NVMe-oF.

Analyzers with a high level of visibility into traffic flow and advanced trace analysis will continue to be invaluable test tools with the upcoming release of PCIe 5.0. Jamming tools with the capability to manipulate live traffic create a robust level of simulation that otherwise would not be possible. The software behind these cutting-edge PCIe tools continues to tie it all together, with ever-improving interface and reporting enhancements that seamlessly unite the operator with the test case.

The immediate future of PCI Express has been shaped by the release of Gen 5 in 2019. This revision, like its predecessors, doubles the available bandwidth of Gen 4. In addition to the performance upgrade, physical differences inherent to Gen 5 include shorter keys and integration of improved materials on the printed circuit board. Gen 5remains backwards compatible with Gen 3 and 4. For this reason, motherboards and adapter cards do not require immediate upgrades. As computer hardware designers and manufacturers continue to integrate PCIe into their latest offerings, the entrenched position of a de facto standard becomes more secure.

The demands on computing power and memory continue to evolve at breakneck speed. So far, the PCI Express bus has managed to keep pace, but the gap is narrowing. PCIe 6.0, expected to be released by 2021, will provide an incredible 256 GB/sec of bidirectional bandwidth and data rates of 64 GT/s to meet the demands of machine learning, artificial intelligence (AI) and other emerging, cutting-edge applications. Data transfer performance in the not-so-distant future may seem like a Hyperloop compared to our familiar and reliable bus, and necessarily so. Perhaps no computer technology plays as important a role in meeting the demand for speed and bandwidth, so the impetus on designers for ongoing breakthroughs in PCIe performance will continue into the foreseeable future.