Ethernet has emerged as the dominant network protocol but it has to overcome several key limitations if it is to become the foundation of choice for the converged datacenter.
Ethernet’s most significant limitation is the lack of support for quality of service (QoS), which means it can’t supplant other network fabrics, most notably Fibre Channel (FC). QoS refers to the ability to create and manage networks to deliver different levels of service for different types of traffic.
Currently, Ethernet QoS doesn’t do enough to distinguish between types or classes of traffic. While it’s true that Ethernet can achieve some level of QoS, native Ethernet gives all classes and types of traffic equal access to bandwidth.
Another key Ethernet limitation is the fact that, when over-saturated with traffic, buffers overflow and packets are dropped. Ethernet tries to compensate by resending dropped packets, but that often only exacerbates the problem. Although these resends happen quickly (<25 milliseconds), they contribute to the lack of consistent response times.
There are, however, ways to address these issues and build out converged data centers on an Ethernet base.
Data Center Bridging
Ethernet utilizes upper-level layer protocols (TCP) to manage end-to-end data delivery and integrity. FC provides a buffer-to-buffer credit that ensures packets aren’t dropped due to network congestion, making it lossless. As Ethernet is an 802-based network, we can make Ethernet lossless only by adopting higher-level protocols such as TCP/IP that have adapted to the intent of IEEE 802-based networks by incorporating end-to-end congestion avoidance and flow control algorithms.
Data Center Bridging (DCB) is an architectural extension to Ethernet designed to improve and expand its role in the data center. With DCB, organizations can logically manage networks end-to-end with QoS through:
Quantized Congestion Notification (QCN) — provides end-to-end congestion management mechanisms to enable throttling of traffic during traffic congestion.
Priority-based Flow Control (PFC) — enables control over individual data flows on shared lossless links.
Enhanced Transmission Selection (ETS) — Control/Negotiation protocol to negotiate QoS parameters and queuing and provides discovery exchange protocol to ensure consistent configurations across the network.
With these enhancements, we now only need one type of network adapter, eliminating the redundant infrastructure cost by using one cable infrastructure to support both storage and IP traffic.
It should be noted that Ethernet does support storage, but mainly through iSCSI, the server-to-storage protocol designed to transport SCSI block storage commands over Ethernet using TCP/IP. iSCSI adoption is growing and has found considerable acceptance in small and midsize enterprises.
Data Center Bridging enhancements to Ethernet networks also enable Fibre Channel over Ethernet (FCoE), a specification developed in the INCITS T11 committee. With FCoE, we can transmit FC commands natively over DCB networks. FCoE specifies the mechanism for encapsulating FC frames onto DCB networks, allowing FC and Ethernet to coexist with a converged infrastructure.
By deploying a converged network, data center managers gain broad benefits such as:
Lower costs from a reduced need for dedicated single-use components such as FC-only HBAs and switches.
A common management platform, personnel, knowledge, and training across the data center.
A platform with a future that today looks to provide a performance upgrade path to 100 Gbps.
Though DCB can now provide an adequate transport for storage protocols, it still needs to overcome its shortcomings with the Spanning Tree Protocol (STP). In traditional Layer 2 (L2) Ethernet networks, organizations create highly available networks by designating paths through the network as active or standby using STP. While this provides an alternate path, only one path can be used at a time, which means that network bandwidth isn’t well utilized. Since one of the goals of server virtualization is increased utilization of the physical server, we can also expect increased utilization of network bandwidth.
To increase network utilization, MSTP (Multiple Spanning Tree Protocol) and similar protocols allow for separate spanning trees per VLAN. While this improves bandwidth utilization, it still leaves the STP limit of one active path between switches. Because traffic paths are manually configured with MSTP, complexity increases.
Another challenge with STP is network behavior when links fail. Links do fail, and when that occurs the spanning tree needs to be redefined. This can take anywhere from five seconds with Rapid Spanning Tree (RSTP) to several minutes with STP — and this situation can vary unpredictably even with small topology changes. Finally, when a redefined spanning tree is reconverging, broadcast storms can occur causing more congestion. These STP limitations are why L2 networks typically are kept small in the data center.
These limitations become more important when applications run as virtual machines (VMs) instead of on physical servers. VM mobility now occurs within a cluster of physical servers, and with the limitations of STP, the sphere of VM migration is constrained to L2 subnets. The solution for flexible VM mobility is a more scalable and available L2 network with higher network bandwidth utilization.
What we need is a L2 network that:
Is highly available.
Guarantees high-bandwidth utilization over equal-cost paths.
Doesn’t stall traffic
when links are added or removed due to failure or network reconfiguration.
Makes latency deterministic and is lossless.
DCB provides these features through the use of TRILL (Transparent Connection of Lots of Links). The basic premise of TRILL is to replace STP as a mechanism to find loop-free trees within Layer 2 broadcast domains. It is an IETF standard that allows every switch to know the MAC address and World Wide Name (WWN) of all devices located on all the other switches. The shared view of the network lets you add a new physical link between two switches with no software configuration. The switches see the new link and add it to the network.
Ethernet DCB and TRILL allows data centers to broaden the sphere of application mobility and optimize server resources for applications just as it improves networking for storage. Unfortunately, different vendors are introducing different adaptations of TRILL. Brocade implements Virtual Chassis Switch (VCS) whereas Cisco uses its own FabricPath implementation. Juniper is approaching the problem with a TRILL-less implementation known as QFabric.
With DCB, Ethernet data centers can now:
Logically eliminate the management of multiple switching layers.
Apply policies and manage traffic across many physical switches as if they were one switch.
Scale network bandwidth without manual reconfiguration of switch ports and network policies.
Provide a single, customized view of network status available to server, network, and storage administrators.
With DCB and these other enhancements, Ethernet will enable IT to simplify network architecture, more rapidly scale their networks, adopt higher speed Ethernet standards (40Gbps is already defined with 100Gbps soon to follow) and significantly reduce management overhead in the converged data center.
Exabytes growing faster than expected
Global business has been caught off-guard by the recent explosion in data volumes and is trying to cope with short-term fixes such as buying in data centre capacity, research by Oracle has found.
The headline finding of the company’s Next Generation Data Centre Index Cycle II is that between 2010 and 2011 there was a rise from 40 percent to 60 percent in the number of businesses using external data centers.
Polling 950 senior IT personnel across the globe (but excluding the US), Oracle also found that the number of businesses looking to build new data centers within the next two years has risen from 27 percent to 38 percent.
Data centre capacity and data volumes should be expected to go up — this drives data centre capacity building — but the scale of what is happening is still striking. Oracle’s research indicates the scale of data overload, showing that volumes have soared from an estimated 135 Exabytes in 2005 to 2,720 Exabytes by 2012. The staggering prediction is that this will reach 7,910 Exabytes by 2015.
However, more than 90 percent of those asked saw the need for more data centers, 60 percent within only two years and one in five within 12 months. Most companies would prefer that these data centers were in-house, which raises the question of why they are having to buy the capacity instead.
“Wrestling big data is going to be the single biggest IT challenge facing businesses over the next two years,” said Oracle senior vice president of systems, Luigi Freguia. “By the end of that period they will either have got it right or they will be very seriously adrift.”
Where this data is coming from is no particular mystery, and neither is why the sudden boom should have taken organizations by surprise. Heading the list of offenders is mobile data, which thanks to the take-off of social media has surged in ways that were not predicted even two years ago, ditto on-demand home entertainment growth.
Perhaps the least expected rise has come from machine-to-machine (M2M) systems, including services such as remote smart metering, something that 4G technologies will only accelerate.
All of this is good news for Oracle’s business, which is built on selling database, middleware, and Exadata storage technologies that fit into the data centre environment with the Sparc T4 processor and server hardware as a proprietary centerpiece.
— By John E Dunn, Techworld.com