Top of rack vs structured cabling

There is no one-size-fits-all solution for data centre cabling. To make the decision-making process easier for data centre professionals, CIOs and IT managers, the Communications Cable and Connectivity Association details the pros and cons of structured cabling versus top of rack .

Data centre cabling configuration choices are impacted by a number of factors including the need to lower power consumption and ensure efficient cooling of critical equipment, as well as by budget constraints and management structure. Below are some factors to consider.

System design and topology

When the first data centres were built, end user terminals were connected via point-to-point connections. This was a viable option for small computer rooms with no foreseeable need for growth or reconfiguration. As computing needs increased and new equipment was added, these point-to-point connections resulted in cabling chaos with associated complexity and higher cost. In response, data centre standards like TIA-942-A and ISO 24764 recommended a hierarchical structured cabling infrastructure for connecting equipment. Instead of point-to-point connections, structured cabling uses distribution areas that provide flexible, standards-based connections between equipment, such as connections from switches to servers, servers to storage devices and switches to switches.

With today’s high-performance servers and virtualisation, more applications can be delivered from a single rack of servers than ever before. In response, several switch manufacturers recommend a top of rack (ToR) configuration where smaller (1RU to 2RU) edge switches are placed in the top of each server rack (or cabinet) and connect directly to the servers in the rack via short preterminated small form factor pluggable (eg, SFP+ and QSFP) twinaxial cable assemblies, active optical cable assemblies or RJ45 modular patch cords.

ToR significantly increases the number of switches and reduces the initial amount of structured cabling. It is often recommended for its rack-at-a-time deployment, ability to limit the use of copper cabling within racks, support for east-west (ie, server-to-server) traffic and rack-level management capabilities. Both TIA 942-compliant structured cabling and ToR have advantages and disadvantages. When selecting the cabling configuration to best meet the needs of the data centre, it is important to examine the impact that structured cabling and ToR have on overall total cost of operations, as well as other trade-offs.

Manageability considerations

With structured cabling, patch panels that mirror switch ports and server ports connect to corresponding panels in one or more central patching areas or zones via permanent (or fixed) links. Also referred to as distribution areas, these patching areas may be located at the end or in the middle of a row of cabinets. Moves, adds and changes (MACs) are accomplished by repositioning patch cord or fibre jumper connections at the central patching area. The fixed portion of the channel remains unchanged and switches and equipment are left untouched and secure. This creates an ‘any-to-all’ configuration where any switch port can be connected to any equipment port. Furthermore, structured cabling can be field terminated to any length to maintain a clean, slack-free appearance.

In a ToR configuration, switches at the top of each rack connect directly to the servers in the same rack, requiring all changes to be made within each individual rack (or cabinet). This eliminates the use of central patching and reduces the amount of structured cabling in the data centre. MACs in a ToR configuration can be more complicated and time consuming - especially in large data centres with hundreds of cabinets. Changes must be made in individual racks or cabinets, rather than at one convenient central patching area. Identifying the specific rack or cabinet requiring the change can be a complicated process.

ToR can be a solution for data centres where individual racks of servers and their corresponding switches need to be managed as their own entity or segregated by application. ToR does not allow network administrators to keep switches separate from server administrators, which can be problematic when these groups manage switches and servers separately and when switches must remain protected for security purposes.

Scalability and upgrade considerations

A widespread switch upgrade with ToR impacts many more switches than with structured cabling and requires equipment at the network core to have the port densities and bandwidth capacity to support the increased number of switches. An individual switch upgrade with structured cabling can increase connection speeds to multiple servers across several racks in the data centre. An upgrade to an individual ToR switch improves connection speed to only the servers in that rack.

ToR switches using short-distance, small form factor pluggable twinaxial cable assemblies cannot support autonegotiation. Because the twisted-pair cabling used with structured cabling is backwards compatible, it supports autonegotiation where individual ports can switch between a 10 and 1 gigabit operation depending on the connected equipment. Autonegotiation enables partial switch or server upgrades on an as-needed basis, enabling a cost-effective migration over time. Without autonegotiation, a switch upgrade requires all equipment connected to that switch to be upgraded simultaneously, incurring full upgrade costs all at once.

Because ToR small form factor pluggable twinaxial cable assemblies are typically more expensive than copper patch cords in a structured cabling system, costs can escalate even further during upgrades as some equipment vendors require use of their cable assemblies and force cable upgrades with equipment upgrades. The core physical layer infrastructure or the fixed portion of the cabling channel is typically installed once, as long as the minimum standards recommended fibre and copper cabling are used.

In a ToR configuration using small form factor pluggable twinaxial cable assemblies, distance between the switches and the servers is limited to a length of seven metres in passive mode. While this is not a problem if each rack will always be managed as an individual unit, these short lengths can restrict the location of equipment if needs change. Structured cabling lengths can be up to 100 metres, allowing flexible equipment placement.

Interoperability concerns

Interoperability and the open systems concept is the ‘bedrock’ of cabling industry standards. Data centre managers expect and value interoperability to fully leverage their existing cabling investment by ensuring performance and a competitive market regardless of which vendors’ equipment and cable designs are selected.

Unfortunately, some switch vendors now require their proprietary cable assemblies for connecting ToR switches to servers when using small form factor pluggable twinaxial cable assemblies. Some ToR switches are designed to check vendor security IDs on cables and either display errors or prevent ports from functioning when connected to an unsupported vendor ID. While this helps ensure that vendor-approved cable assemblies are used with corresponding electronics, it can limit data centre design options by locking data centre managers into a proprietary solution. This is a substantial change from the industry standards-based fibre connectivity and copper connectivity successfully deployed in data centres for decades.

Third-party independent testing by University of New Hampshire's Interoperability Lab (UNH IOL) proves that passive small form factor pluggable twinaxial cable assemblies from cabling vendors pass interoperability testing with several vendors’ ToR switches that are designed to display errors. These tests demonstrate that proprietary cables are not necessarily required. This may not be the case for switches designed to actually prevent ports from functioning altogether when connected to an unsupported vendor ID.

Maintenance, equipment and cabling cost

With a ToR switch in every cabinet (or two for dual primary and secondary networks), the total number of switch ports depends on the total number of cabinets in the data centre, rather than on the actual number of switch ports needed to support the equipment. This can nearly double the amount of switches and power supplies required, compared to structured cabling. Unlike passive structured cabling, ToR switches require power and ongoing maintenance.