What is Data Center Interconnect?

It refers to the technology used to link together two or more individual data centers to pool resources, balance employee workloads, replicate data, or implement disaster recovery plans – and provide business data closer to the edge. Fiber optic DCIs can be divided in 3 categories, based on the transmission distance:

• Campus DCI: typically within 5km max, interconnecting multiple data centers of the same campus 
• Metro DCI: within 100km, this is the interconnection between data center campus, within a region and/or availability zone. 
• Submarine: ultra long-distance data center interconnection , with hundreds of kilometers, using undersea fiber optic cables   

Click to enlarge

A wide array of optical transport technologies is utilized to maximize bandwidth and minimize latency as cloud computing, and the IoT continues to drive new levels of scale and complexity.

The demand for storage and speed has led to an increase of new locations around the world, including many in the hyperscale category with thousands of interconnects between them. With 256 QAM modulation, 400G can now be achieved on a single wavelength. This innovation provides a much-needed capacity boost, while fault detection, throughput, latency, and stress testing remain essential during installation.  

VIAVI supports the data center interconnect market from end-to-end by providing powerful, innovative data center solutions for the construction, commissioning, expansion, and monitoring of DCIs. This holistic approach encompasses robust performance verification over the DCI as well as the comprehensive testing of the essential equipment populating the hyperscale ecosystem.    

The term “hyperscale” has become synonymous with massive locations housing countless servers within seemingly endless facilities. Cloud service providers , internet content providers (ICP), and big data storage industries have indeed scaled their brick and mortar to unprecedented levels. 

Hyperscale is increasingly used to define not just the scale and size of these new data centers, but also their architecture. Their size and scale are a useful place to start. Hyperscale data centers have a minimum of 5,000 operating servers and at least 10,000 square feet in size. In general, however, hyperscale locations are much larger, frequently numbering tens of thousands of servers. Beyond the footprint and server figures, equally important is what is going on inside, where they’re architected for a homogeneous scale-out greenfield application portfolio using increasingly disaggregated, high-density, and power-optimized infrastructures. 

A hyperscaler can and must provide nearly flawless service to millions of customers. Despite the advantages of customized computing, high speed networks, and advanced virtualization software occupying more real estate than many cities, no one center can accommodate these demands independently. In the new hyperscale paradigm, data center connectivity solutions provide the binding element between all locations and enable a large conglomeration to behave as one entity. A healthcare corporation is just one example of an industry that benefits from this technology.   

Rapid deployment, and use of interconnect solutions, is an efficient method for maintaining scalability and versatility without adding server capacity or square footage. Reliance on this vital DCI technology for metro and long-haul linkage has increased vulnerability to external intrusions, accidental breaks, or disconnections that can lead to costly outages. For this reason, fiber monitoring solutions and performance monitoring solutions are essential tools for protecting DCI in a proactive fashion and minimizing mean time to repair (MTTR). 

As the size and complexity have continued to develop, the architecture has also evolved. Traditional three-tiered architectural models consisting of core routers, aggregate routers, and switches have been replaced by leaf/spine architectures to meet the demands of hyperscale solutions. Top of Rack (TOR) architecture reduces cabling by co-locating connections between switches and servers within the rack. End of Row/Middle of Row (EOR/MOR) architecture consolidates switch ports in designated locations and, in some cases, employs a patch panel as the central location for connectivity. 

With throughput requirements between data centers in the Terabit per second (Tbps) range, architectural solutions must also consider the data center interconnect configuration. Fiber infrastructure is selected based on the relative location, east-west traffic flow volume, and hardware configuration of each center. Extreme-density, high-fiber count cabling, 256 QAM, and dense wave division multiplexing (DWDM) are essential options in this architectural mix. Long-haul DCI transport of 80 km or more also increases latency and may require EDFA or Raman amplification, or other active amplification elements that are not necessary for metro or campus deployments.

Does Data Center Interconnect Extend into the Data Center? 

Although DCI is an integral component of data center architecture, it is also an independent entity with its own unique test and reliability considerations, and therefore requires custom solutions.  

Advanced, rack-mounted interface solutions in the entrance room provide connection and access points for terminations. Interface cards within these devices introduce essential transponding, switching, and multiplexing functions that complete the DCI to data center handoff. 

The compressed, densely populated hardware footprints found within most centers are in stark contrast with delicate solutions that can extend for hundreds of miles. This combination of physical attributes and constraints creates unique test and certification challenges that can only be mitigated by a comprehensive portfolio of test solutions. 

Advanced fiber network certification solutions such as the VIAVI SmartClass Fiber OLTS-85 enable complete Tier 1 fiber network certification with a single compact device and provide test visibility of both the local and remote ends. The MPOLx is a full-function optical loss test set designed specifically for loss, polarity, end-face inspection, and certification over native MPO connection interfaces. 

Ever expanding customer demand has led to exponential growth in new hyperscale data center and DCI installations, carrying more complex workloads. This construction bonanza has mirrored a shift from the traditional stand-alone center to the “mesh” network structure defined by increased leveraging of DCI. Testing strategies have likewise evolved to place more emphasis on automation, dependencies, fiber network security, and the aggregation of connections.  

To achieve 24/7/365 uptime, performance metrics like throughput, latency, and bit error rate (BER) must now be diligently tested and monitored both inside and outside the center. The miles of connecting fiber between centers also introduce countless new opportunities for breakdowns or disruption that can potentially impact millions of users, people, clients, and employees.  

Problems can be hard to identify in the complex web of networks, so solutions must emphasize visibility and scalability. The rapid deployment of proprietary hyperscale locations and DCIs has led to more interoperability protocols and infrastructure based on exotic or proprietary specifications that further complicate test solution development.  

A high volume of connections and pathways coupled with a high demand for uptime reliability makes testing of both the network and services essential in the environment. Hyperscale DCIs will often be expected to run continuously at full capacity. Monitoring solutions with little or no human intervention required are a logical fit for these environments, especially those that can be launched and controlled from virtual access points. The MAP-2100 rack-mounted test unit can execute high-accuracy bandwidth throughput testing (RFC 2544, Y.1564) of DCIs to and from unmanned locations. 

A successful DCI testing strategy will begin with an end-to-end (e2e) visibility strategy in the planning and deployment stages that propagates through the ongoing management and maintenance phases. Interoperability is a key attribute for test solutions in the DCI environment because test tools that support open APIs provide more flexibility and enable hyperscale organizations to establish their own automation code.  

As these new environments advance their symbiotic transformation, VIAVI Solutions continues to work closely with standards bodies on open-source initiatives and best practices that preserve independence and pathways to innovation.  

Data Center Interconnect Solutions 

The challenges associated with DCI installation, activation, and monitoring require the best available fiber test solutions working collectively to ensure ongoing service quality. With the rapid evolution of hyperscale technology, combined with the high utilization factors driving DCI pathways, little margin for error is available. 

Fiber Inspection 
Fiber is the lifeblood of connectivity solutions. Versatile fiber inspection solutions are an important element of verification and can prevent defects or contamination introduced during installation from degrading performance. VIAVI inspection solutions with automated routines, compact form factors, and available multi-fiber connectivityestablish a firm foundation for DCI fiber connection assurance.    

High-Speed Transport of 400G and 800G 
The bandwidth demands placed on extreme networks ripple down to interconnects as they carry voluminous information over vast distances. Terabit Ethernet in the 400G and 800G categories is now among the transport technologies employed for interconnect data systems that demand a more sophisticated level of BER testing and PAM-4 modulation support. Test solutions incorporating these cutting-edge data center technologies are invaluable during the development, deployment, and maintenance phases of the lifecycle. 

Metro 100G and 200G 
DCIs in the metro category typically fall between 5km and 80km and link locations in nearby cities or mid-sized geographical areas. Metro interconnects often utilize 100G or 200G networking topology requiring fiber link qualification, characterization, and performance testing to ensure optimal performance. All-in-one network and fiber testing solutions utilizing automated test scripts add the requisite efficiency to turn-up, throughput, and high-accuracy latency test requirements. 

Fiber Monitoring and Remote Fiber Test On Demand 
DCIs require consistent protection from intrusions, accidental outages, cuts, and breaks to ensure maximum uptime and minimal latency. Even a minor degradation in fiber quality can significantly reduce throughput and increase bit error rates. 

With capacity stretched to the limit, there is no allowance for compromised performance. The FTH-5000 or FTH-9000 rack-mounted OTDR test unit works in conjunction with the ONMSi remote fiber test system to form a proactive, automated solution for continuous monitoring. Fiber defects are located with pinpoint accuracy, triggering instant assessments, and alerts. Using a wavelength ranging between 1625-1650nm, the center can be tested on demand while in service after a repair or routinely monitored with a very low scan time per fiber. Remote, automatic fiber monitoring discovers and located fiber faults, allowing them to be tracked over time for the degradation that can cause latency and bit error rates.  

Preventative maintenance can stop 20% of outages caused by the fiber plant. In the case of leased fiber, both parties have access to objective data that allows both to demarcate the fiber fault, repair it with the right team and to manage MTTR (Mean Time to Restoration/Resolution) key performance indicators.  This technique is critical to managing SLA contract terms for uptime as one-third of data center outages are caused by fiber plant problems. 

Typical DCIs will have two or three redundant, geographically unique fiber paths to provide resiliency and high availability. Both fiber monitoring and ONMSi products can ensure visibility into the health of the data center interconnect, providing rapid notification when there is a threat from a break, bend, bad connector, and a dislodged/disconnected patch cord disrupting transmission. 

Finally, using the high-powered, modular FTH-9000, in high-sensitivity mode, one can automatically identify security threats from optical hackers who seek to take a copy of traffic. This is done by adding an optical coupler to the line and bending the fiber to capture a small amount of light. It is a cost-efficient way to access millions of dollars of data center traffic without detection. A fiber monitoring solution designed just for this use case with an ultra-high performance OTDR, and patented calculations are required to protect against this intrusion. Most DCIs carry regulated sensitive personal information that must be protected against theft, and automated monitoring is the first line of defense.  Even if the information is encrypted, codes can be broken or stolen over time, so eliminating theft is critical. 

Virtual Test and Activation 
Metro and core network mesh testing is greatly simplified through virtualized test agents. The software-based Fusion platform can be used to facilitate virtual service activation and performance monitoring via a combination of virtual test agents and hardware embedded in a network. With the expansion of global data center interconnects, these tools minimize technician dispatches and further enable rapid deployment, verification, and support SLA-protection. 

Fiber Test and Certification 
As bandwidth continues to expand, meeting customer requirements means multiplying fiber capacity. This translates to a higher volume and diversity in the services market, more endpoints/users, and countless monetization opportunities. Fiber networks are an obvious solution to addressing bandwidth constraints.  

OTDR/Fiber Characterization and OSA test support the commissioning, construction, and in-service test of today’s high-capacity fiber optic interconnects. VIAVI offers the most comprehensive range of fiber testers for essential fiber optic test for this phase of install, maintenance, and troubleshooting with built-in automation for the full range of basic to detailed testing needs. 

MPO 
The compact footprint and fiber density inherent to Multi-fiber Push On (MPO) connectors are another option for exceedingly crowded hyperscale centers and interconnects. Inspection and test tools providing a convergence of native MPO connectivity, automated workflows, and fast and reliable reporting allow the scalability and efficiency of multi-fiber connectors to be fully leveraged.