In a classic Ethernet network, the connections between switches, or Inter-Switch Links are not allowed to form a loop or frames aren’t delivered. Spanning Tree Protocol (STP) prevents loops by creating a tree topology with only one active path between any two switches. This means that ISL bandwidth is limited to a single logical connection, since multiple connections between switches are prohibited. Enhancements to Ethernet tried to overcome this limitation. Link Aggregation Groups (LAGs) were defined so that multiple links between switches were treated as a single connection without forming loops. But, a LAG must be manually configured on each port in the LAG and is not very flexible.
The tree topology requires traffic to move up and down the tree, or "north-south,” to get to an adjacent rack. When most of the access traffic is between servers in a rack, this is not a problem. But server clusters, such as those required for clustered applications and server virtualization, have traffic between servers in multiple racks, travelling "east-west,” so the tree topology increases latency with multiple hops and restricts bandwidth with single links between switches.
STP automatically recovers when a link is lost. However, it halts all traffic through the network and must reconfigure the single path between all switches in the network before allowing traffic to flow again. Halting all traffic for tens of seconds up to minutes on all links limits scalability and constrains traffic to applications that can tolerate data path blocking to achieve link resiliency. In the past, traffic relied on TCP to handle this interruption in service, but today, with almost all data center applications running in a 24 x 7 high availability mode and storage traffic growing on the Ethernet network, loss of connectivity in the data path for even a few seconds is unacceptable.
Classic Ethernet switch architecture presents other limitations. Each switch has its own control and management planes. Each switch has to discover and process the protocol of each frame as it arrives on an ingress port. As more switches are added, protocol processing time increases adding latency. Each switch and each port in the switch has to be configured individually, since there is no sharing of common configuration and policy information between switches. Complexity increases, configuration mistakes increase, and operations and management resources do not scale well.
In an Ethernet fabric, the control path replaces STP with link state routing, while the data path provides equal-cost multipath forwarding at Layer 2 so data always takes the shortest path using multiple ISL connections without loops. Combined with the fabric’s control plane, scaling bandwidth is made simple. For example, it becomes possible to automate the formation of a new trunk when a new switch connects to any other switch in the fabric.
If a trunk link fails or is removed, traffic is rebalanced on the existing links non-disruptively.
Finally, if an ISL is added or removed anywhere in the fabric, traffic on other ISLs continues to flow instead of halting as with STP.
With this architecture in place, a group of switches can be defined as part of a "logical chassis”, similar to port cards in a chassis switch. This simplifies management, monitoring, and operations since policy and security configuration parameters can be easily shared across all switches in the logical chassis. In addition, because information about connections to physical and virtual servers and storage is now known to all switches in the fabric, the fabric can ensure all network policies and security settings continue to be applied to any given virtual machine no matter whether it moves or where it resides.
In modern, virtualized data centers, IT groups need to better scale virtual server environments, provide application mobility, and reduce infrastructure complexity and management overhead.
Scaling virtual Server Environments
When organisations attempt to scale virtual server environments, the network often presents challenges due to Spanning Tree Protocol, the growing number of GbE connections per server, low utilization, and link failure recovery.
Enabling virtualization capabilities, such as Virtual Machine (VM) mobility, requires VMs to migrate within a single Layer 2 network, since non-disruptive migration of VMs across Virtual LANs (VLANs) using Layer 3 protocols is not supported by virtualization hypervisors.
In traditional Layer 2 Ethernet networks, to create a highly available network, organisations designate paths through the network as active or standby using STP. While this provides an alternate path, only one path can be used at a time, which means that network bandwidth is not well utilized. Since one of the goals of server virtualization is to increase utilization of the physical server, increased utilization of network bandwidth should also be expected.
To increase network utilization, Multiple Spanning Tree Protocol (MSTP) and similar protocols allow for separate spanning trees per VLAN. While this improves bandwidth utilization, the STP limit of one active path between switches remains. And, because traffic paths are manually configured with MSTP, complexity increases.
Another challenge with STP is network behavior when links fail. When failures occur, the spanning tree needs to be redefined. This can take anywhere from five seconds with Rapid Spanning Tree (RSTP) up to several minutes with STP—and this convergence can vary unpredictably even with small topology changes. The demands for non-stop traffic flow increases with server virtualization, and consequently network convergence times have to shrink. STP does not provide an adequate solution for these requirements. Finally, when a spanning tree is reconverging, broadcast storms can occur and result in network slowdown. All of these limitations of STP are why Layer 2 networks are typically kept small in the data center.
Compared to classic hierarchical Ethernet architectures, Ethernet fabrics provide higher levels of performance, utilization, availability, and simplicity.
They have the following characteristics at a minimum:
Ethernet fabrics eliminate the need for Spanning Tree Protocol, while still being completely interoperable with existing Ethernet networks.
Can be architected in any topology to best meet the needs of any variety of workloads.
Multiple "least cost” paths are used for high performance and high reliability.
Easily scales up and down at need.
More advanced Ethernet fabrics borrow further from Fibre Channel fabric constructs: