Andy Bechtolsheim (Arista) (May 2016)

Andy Bechtolsheim (Arista Co-Founder) – Moore’s Law and Networking (May 2016)

Chapters

00:01:11 Moore's Law, Networking, and the Future of Computing

00:13:20 Next-Generation Networking Technologies for Cloud Data Centers

Cost-Performance Advancements:
Server systems’ cost performance is expected to improve as silicon content scales, leading to faster data processing.

Server Networking Speed Adoption:
Networking speeds have evolved from gigabit to 10-gigabit and are poised to adopt 40-gigabit and 100-gigabit interfaces in the coming years.

Growing Data Center Spending:
Market research predicts a significant increase in data center spending for 10-gigabit, 40-gigabit, and eventually 100-gigabit network connections.

Faster CPUs and 10-Gigabit Connectivity:
Increasing CPU speeds, like Intel’s Sandy Bridge, drive the adoption of 10-gigabit NICs on server motherboards.

Network Scalability:
MLAG (Multi-Chassis LAG) provides layer two scalability for small clusters, while ECMP (Equal Cost Multipath Load Balancing) enables scalable layer three designs.

Large-Scale Data Centers:
Arista’s 7050 racktop switch and modular chassis switch offer high-performance networking solutions for large data centers.

Optimizing Cluster Design:
Cluster design considerations include pod size, bandwidth per server, and the degree of ECMP redundancy to determine the aggregate throughput.

Transition from 10-Gigabit to 40-Gigabit and 100-Gigabit:
The shift from 10-gigabit to higher speeds depends on cost-effectiveness, with 40-gigabit being more viable than 100-gigabit due to lower cost optics.

Leaf-Spine-Cluster Configuration:
Within racks, twin X copper cables are used, but inter-switch connectivity requires optics, with LR4 and SR4 being common options.

10-Gigabit Optics and Form Factor Standards:
SFPs are widely used for 10-gigabit optics, supporting both Twin X copper cables and various laser types. The 40-gigabit standard is QSFP, supporting both copper cables and lasers like LR4 and SR4.

100-Gigabit Optics and Challenges:
100-gigabit optics face challenges in cost and distance limitations, with SR4 only reaching 100 meters and LR4 being expensive.

New Technology: Silicon Photonics:
Silicon photonics, with on-chip modulators and a single laser, offers low cost, high density, and the potential for longer distances like a kilometer.

Parallel Fiber Technology:
Parallel fiber, using multiple fibers in a single jacket, is cost-effective and enables high-density 100-gigabit connectivity.

00:26:20 The Future of Data Center Networking: Silicon Photonics and Parallel Single-Mode Fiber

Important Developments in Data Center Switching:
Adoption of silicon photonics promises significant cost reductions, lower power consumption, higher reliability, and support for longer distances in data center switching. The industry is transitioning rapidly to 40 gig and 100 gigabit speeds as optics costs become more competitive. Parallel single-mode fiber is a cost-effective alternative to traditional duplex multimode fiber, offering significant savings for large-scale data centers.

Challenges in Data Center Cabling:
Most installed base in data centers consists of multi-mode fiber, which is more expensive than single-mode fiber. The high density of cables and connectors in data centers can lead to increased confusion and complexity.

Benefits of Innovations:
Silicon photonics combined with parallel single-mode fiber is seen as the most promising solution to reducing the cost of high-speed optics while minimizing the number of cables and connectors. The adoption of these technologies is expected to improve data center switching performance and cost-effectiveness significantly.

Power Consumption Considerations:
The power consumption of data center switching equipment varies depending on design and cooling capabilities. High-density data centers may target a maximum power consumption of around 10-12 kilowatts per rack, while high-performance computing data centers may aim for 20-30 kilowatts per rack. Arista designs its switches to be within the typical power envelope of a data center rack, aiming for less than 250 watts per 1U.

Historical Use of ASIC Design:
The prevalence of ASIC design in the past was driven by its lower cost compared to full custom chip design. ASICs offered a cost-effective solution for chips with lower production volumes, such as those used in routers. As volumes for high-speed Ethernet interfaces increased, the industry shifted towards full custom designs to optimize performance and cost-effectiveness.

Cost Considerations for Optics:
While gigabit and 10-gigabit optics and fiber have been similar in pricing for smaller-scale deployments, the cost difference becomes more significant at larger scales. At the 100-gigabit level, the current LR4 CFP modules are much more expensive than equivalent 10-gigabit modules, making them cost-ineffective on a per-bit basis. Silicon photonics is expected to make significant cost reductions in 100-gigabit optics, making them more competitive with 10-gigabit costs.

Ethernet as the Default Interconnect:
Ethernet has become the default interconnect for data centers due to its widespread adoption and interoperability. While other interconnect technologies exist, Ethernet’s dominance has led to a lack of discussion and exploration of alternatives.

00:38:21 Optimizing Buffer Allocation for Switch Performance

Abstract

The Evolution of Data Center Technologies: A Comprehensive Overview

Revolutionizing Data Center Connectivity: The Rise of Moore’s Law, Networking Innovations, and Silicon Photonics

The field of data center technology has undergone a significant transformation, driven by the relentless progress of Moore’s Law, innovations in networking, and the advent of silicon photonics. This article delves into the exponential growth in computing power, with CPUs experiencing a million-fold increase in transistors, and a predicted further 100x rise in the next 12 years. Concurrently, networking performance, though initially lagging, is now catching up, with next-generation silicon poised to double current networking chip performance. This evolution is underpinned by advancements in server networking speeds, Arista’s innovative switches, and cutting-edge technologies in cluster design, fiber optics, and chip design, ultimately leading to a drastic shift in power consumption trends and a dominance of Ethernet in data centers.

Moore’s Law and CPU Evolution

Moore’s Law has been the cornerstone of computing, with the transistor count on CPUs doubling approximately every two years. This has led to a staggering million-fold increase over four decades, and the roadmap suggests a 1000x improvement in speed and cost-efficiency in the next 20 years. Such exponential growth underpins the entire spectrum of data center technologies, from chip design to overall computational capabilities.

Networking Performance: Catching Up with Moore’s Law

Initially, networking performance did not keep pace with Moore’s Law, primarily due to limitations in I/O pins and the ASIC design flow. However, recent trends indicate a shift towards full custom chip designs, yielding higher throughput and reliability. This shift has resulted in the networking sector gradually aligning with Moore’s Law, promising significant improvements in future silicon technologies.

Advancements in Server Networking Speeds

Server networking speeds have seen a progressive increase, moving from gigabit to 10 gigabit Ethernet, and now evolving towards 40 and 100 gigabit interfaces. This transition is reflective of the market’s growing demand and expenditure in data center networking, highlighting the importance of speed in modern data center operations.

Arista’s Pioneering Switch Technologies

Arista has been at the forefront of this evolution with its 7050 racktop switch and modular chassis switches. These devices offer high throughput, low power consumption, and large-scale connectivity, representing a significant leap in networking hardware capabilities.

Cluster Design and Network Scaling

Efficient cluster design is crucial for optimal data center performance. Factors such as bandwidth per server, pod size, and ECMP redundancy play a vital role in determining the total cluster bandwidth and the cost-efficiency of the network.

Transitioning to Higher Gigabit Networks

The industry is gradually moving towards 40 and 100 gigabit Ethernet, despite the current high costs associated with these speeds. The economics of optics, especially for 100 gigabit, remains a critical consideration in this transition.

Innovations in Fiber Optics

Fiber optics technology has seen notable advancements, with parallel fiber and various form factors like SFPs, QSFPs, and CFPs becoming increasingly relevant. The push towards single-mode fiber, supported by silicon photonics, is poised to redefine the cost and performance landscape of data center networking.

Data Center Power Consumption Trends

As data centers pack more chips onto single boards, power consumption per rack is on the rise. This has led to the design of switches that fit within specific power envelopes, emphasizing the need for energy-efficient technologies.

ASIC vs. Custom Chip Design

The transition from ASIC to custom chip design in high-volume products like Ethernet switches demonstrates a market shift towards achieving optimal performance, particularly in the context of high-performance computing.

Single-Mode Fiber and Ethernet Dominance

Single-mode fiber is increasingly being adopted in data centers, given its cost-effectiveness for longer distances. Simultaneously, Ethernet has solidified its position as the default interconnect in data centers, overshadowing other options like SONET. However, SONET interfaces are still available, although they are costlier than Ethernet. Packet-based traffic predominantly utilizes Ethernet infrastructure, making it the default choice in most scenarios.

Ethernet Frame Size Considerations

Most equipment supports jumbo frames, allowing for frame sizes up to 9 kilobytes.

Buffer Sizing in Single-Chip Designs

In single-chip designs, buffer size is limited by the die area. For instance, a 16 megabyte buffer (128 megabits) requires nearly a billion transistors.

Modular Chassis vs. Single-Chip Designs

Modular chassis designs employ external DRAM for buffering, providing virtually infinite buffer capacity. However, external DRAM chips necessitate more pins, reducing port density, and the use of fabric interfaces on these chips results in fewer ports compared to single-chip designs.

Performance Considerations for Buffer Sizing

Predictable performance demands large buffers at the spine layer to prevent packet loss. Resilient networks can utilize single-chip designs with smaller buffers, depending on the application. The TCP stack’s slow start recovery time amplifies the impact of packet drops.

Flow Control and Buffer Management

803 text flow control can be implemented at the leaf layer to prevent buffer overrun. At the spine layer, flow control is less effective due to aggregated flows.

Arista’s Dynamic Buffer Limiting (DBL)

DBL, co-invented by Andy Bechtolsheim, allocates buffer memory per flow hash, preventing collisions and ensuring fairness in buffer allocation. Its successful implementation in the Catalyst 4000s resulted in excellent performance.

Buffer Size Comparison between Catalyst 4948 and Single-Chip Designs

The Catalyst 4948 had a buffer size of 50 milliseconds per gigaport, while single-chip designs typically have a buffer size of approximately one TCP window per port.

Buffering Strategies for Single-Chip Designs

Single-chip designs rely on access emission flow control at the leaf layer, while spine layers employ over-provisioning with DRAM to provide sufficient buffering.

The Future of Data Center Technologies

In conclusion, the data center industry is at a pivotal point, with technologies like silicon photonics and advanced chip designs paving the way for more efficient, high-speed connectivity. While the dominance of Ethernet raises questions about innovation versus historical inertia, the overall trend is clear: data centers are rapidly evolving to meet the demands of modern computing, with an emphasis on speed, efficiency, and cost-effectiveness. This evolution, driven by Moore’s Law and networking innovations, represents a significant leap forward in the capabilities and potential of global data networks.

Notes by: MythicNeutron

Andy Bechtolsheim (Arista Co-Founder) – Moore’s Law and Networking (May 2016)

Chapters

Abstract

Related posts: