|By Marten Terpstra||
|April 18, 2014 09:15 AM EDT||
Earlier this week, Ethan Banks wrote a very nice article about Mellanox’s dual spine and leaf network in support of a large amount 10GbE access ports. After describing the scaled up network design, he reviews 8 observations about the design, not to point out good or bad, but merely to point out specific points to consider. Fully coincidental (Ethan lives close to us, but I am pretty sure he cannot peek through our windows) we had gone through a similar exercise this week, documenting the choices and limitations of spine and leaf networks. And as always, the conclusions are not ones of right or wrong, more of awareness of choices and consequences.
The Mellanox design Ethan describes employes an extra spine layer, we have seen and heard the same from Arista and others, some calling it a spine-spine or similar. Nitpicking perhaps, but adding a spine layer to a spine and leaf network and still calling it a spine and leaf network is like adding a docking station, screen and keyboard to a phone and still calling it a phone. It’s a computer that can make calls. And a spine and leaf network with an extra spine is a fat tree.
Ethan points out in his first few points that the sheer amount of cables and optics is astonishing. Let me try and put some numbers around that statement. If you build an approximate 3:1 oversubscribed spine and leaf network out of generic switches, I would probably use a Trident2 based ToR switch with 48 SFP+/10GbE ports and 6 QSFP/40GbE ports. The 48 ports should serve most rack deployments with single homed servers, only really dense or heavily multi homed servers would need more ports and I will use 4 of the QSFP ports to connect to my spine. That leaves 2 QSFPs on each leaf to be used for extra access ports, at the cost of some oversubscription. Or if I wanted I could use these as extra spine connections, lowering my oversubscription to 2:1. As a spine, and staying away from chassis based systems, I would pick a 32xQSFP/40GbE switch.
The largest spine and leaf I could build out of this combination is one that contains 16 spine switches and 96 leaf switches. In a spine and leaf I need to connect each leaf to each spine and I have the equivalent of 16x10GbE to use out of each leaf, which I can connect to at best 16 spines. With each spine receiving 1x10GbE from each leaf, I can build out to 96 leafs to fill out my 32xQSFP or 96x10GbE equivalent spine switch capacity. 96 Leaf switches give me 4608 10GbE access ports at 3:1 oversubscription, 5376 at a slightly worse oversubscription if I use the extra QSFP ports on my leaf switches. To support these 5376 access ports, I need 3584 10GbE fabric ports: 16 each from 96 leaf switches plus 32×4 from each spine. That means 1792 switch interconnect cables. And 3584 10GbE short reach optics, because the vast majority of connections between spines and leafs is likely to be at a distance that DAC cables cannot cover (not considering the fact that most high density 10GbE switches are designed without PHYs, limiting the use of passive cables to usually 5m or less).
I let you do the math on the cost of that infrastructure. And the installation. And the maintenance. And the sheer complexity of running a 10GbE from each leaf to each spine. If you want to reduce some of this complexity, you can switch to using 40GbE/QSFPs between the leafs and spines, but by doing so you have reduced the maximum size of the network. Each leaf will now contribute 1 QSFP worth of interconnect to each spine, to a 32xQSFP spine can only support 32 leaf switches, or 1536 10GbE access ports (1792 if you use the 2 extra leaf QSFP ports). And you probably noticed that I left no room on the spine switches to actually leave this spine and leaf network to the rest of the network infrastructure. Taking a few QSFPs for that will reduce the size of the network more.
Even in the spine, spine and leaf network (or spine, leaf and ToR in Mellanox terminology) the amount of cabling between the ToR (someone explain why that is not the leaf of the network?) and the aggregation leaf may be reduced, the cabling between their aggregation leaf and spine still follows the same model as above.
We often focus on the cost of switches and the derived cost per port. Of course the cost per port is important, but don’t fool yourself by taking the cost of a switch and dividing it by the number of ports it supports. It makes for great quotes in press releases, but the actual cost for that port is way higher the moment you add in the overhead required to connect that port to the rest of the network. And for spine and leaf networks, the cost of the rest of the network has a very large portion of cables and optics. Even for reasonably priced optics, that piece of the infrastructure is likely going to cost more than the cost of the spine switches together. Even when you create one of these multispined animals…
[Today's fun fact: The human spine contains 120 muscles and approximately 220 individual ligaments.]
The post There is cost per port and then there is cost per port… appeared first on Plexxi.
The Internet of Things is not only adding billions of sensors and billions of terabytes to the Internet. It is also forcing a fundamental change in the way we envision Information Technology. For the first time, more data is being created by devices at the edge of the Internet rather than from centralized systems. What does this mean for today's IT professional? In this Power Panel at @ThingsExpo, moderated by Conference Chair Roger Strukhoff, panelists will addresses this very serious issue o...
May. 26, 2015 10:35 AM EDT
In their general session at 16th Cloud Expo, Michael Piccininni, Global Account Manager – Cloud SP at EMC Corporation, and Mike Dietze, Regional Director at Windstream Hosted Solutions, will review next generation cloud services, including the Windstream-EMC Tier Storage solutions, and discuss how to increase efficiencies, improve service delivery and enhance corporate cloud solution development. Speaker Bios Michael Piccininni is Global Account Manager – Cloud SP at EMC Corporation. He has b...
May. 26, 2015 10:30 AM EDT Reads: 1,013
SYS-CON Events announced today that MetraTech, now part of Ericsson, has been named “Silver Sponsor” of SYS-CON's 16th International Cloud Expo®, which will take place on June 9–11, 2015, at the Javits Center in New York, NY. Ericsson is the driving force behind the Networked Society- a world leader in communications infrastructure, software and services. Some 40% of the world’s mobile traffic runs through networks Ericsson has supplied, serving more than 2.5 billion subscribers.
May. 26, 2015 10:15 AM EDT Reads: 1,093
EMC Corporation on Tuesday announced it has entered into a definitive agreement to acquire privately held Virtustream. When the transaction closes, Virtustream will form EMC’s new managed cloud services business. The acquisition represents a transformational element of EMC’s strategy to help customers move all applications to cloud-based IT environments. With the addition of Virtustream, EMC completes the industry’s most comprehensive hybrid cloud portfolio to support all applications, all workl...
May. 26, 2015 10:15 AM EDT Reads: 202
While great strides have been made relative to the video aspects of remote collaboration, audio technology has basically stagnated. Typically all audio is mixed to a single monaural stream and emanates from a single point, such as a speakerphone or a speaker associated with a video monitor. This leads to confusion and lack of understanding among participants especially regarding who is actually speaking. Spatial teleconferencing introduces the concept of acoustic spatial separation between confe...
May. 26, 2015 10:00 AM EDT Reads: 2,993
Buzzword alert: Microservices and IoT at a DevOps conference? What could possibly go wrong? In this Power Panel at DevOps Summit, moderated by Jason Bloomberg, the leading expert on architecting agility for the enterprise and president of Intellyx, panelists will peel away the buzz and discuss the important architectural principles behind implementing IoT solutions for the enterprise. As remote IoT devices and sensors become increasingly intelligent, they become part of our distributed cloud en...
May. 26, 2015 10:00 AM EDT Reads: 1,744
In a recent research, analyst firm IDC found that the average cost of a critical application failure is $500,000 to $1 million per hour and the average total cost of unplanned application downtime is $1.25 billion to $2.5 billion per year for Fortune 1000 companies. In addition to the findings on the cost of the downtime, the research also highlighted best practices for development, testing, application support, infrastructure, and operations teams.
May. 26, 2015 10:00 AM EDT Reads: 1,000
We are all here because we are sold on the transformative promise of The Cloud. But what good is all of this ephemeral, on-demand infrastructure if your usage doesn't actually improve the agility and speed of your business? How must Operations adapt in order to avoid stifling your Cloud initiative? In his session at DevOps Summit, Damon Edwards, co-founder and managing partner of the DTO Solutions, will highlight the successful organizational, process, and tooling patterns of high-performing c...
May. 26, 2015 10:00 AM EDT Reads: 5,286
Discussions about cloud computing are evolving into discussions about enterprise IT in general. As enterprises increasingly migrate toward their own unique clouds, new issues such as the use of containers and microservices emerge to keep things interesting. In this Power Panel at 16th Cloud Expo, moderated by Conference Chair Roger Strukhoff, panelists will address the state of cloud computing today, and what enterprise IT professionals need to know about how the latest topics and trends affec...
May. 26, 2015 09:45 AM EDT Reads: 1,326
There is no question that the cloud is where businesses want to host data. Until recently hypervisor virtualization was the most widely used method in cloud computing. Recently virtual containers have been gaining in popularity, and for good reason. In the debate between virtual machines and containers, the latter have been seen as the new kid on the block – and like other emerging technology have had some initial shortcomings. However, the container space has evolved drastically since coming on...
May. 26, 2015 09:45 AM EDT Reads: 1,141
T-Mobile has been transforming the wireless industry with its “Uncarrier” initiatives. Today as T-Mobile’s IT organization works to transform itself in a like manner, technical foundations built over the last couple of years are now key to their drive for more Agile delivery practices. In his session at DevOps Summit, Martin Krienke, Sr Development Manager at T-Mobile, will discuss where they started their Continuous Delivery journey, where they are today, and where they are going in an effort ...
May. 26, 2015 09:30 AM EDT Reads: 1,210
SYS-CON Media named Andi Mann editor of DevOps Journal. DevOps Journal is focused on this critical enterprise IT topic in the world of cloud computing. DevOps Journal brings valuable information to DevOps professionals who are transforming the way enterprise IT is done. Andi Mann, Vice President, Strategic Solutions, at CA Technologies, is an accomplished digital business executive with extensive global expertise as a strategist, technologist, innovator, marketer, communicator, and thought lea...
May. 26, 2015 09:00 AM EDT Reads: 1,279
The 4th International Internet of @ThingsExpo, co-located with the 17th International Cloud Expo - to be held November 3-5, 2015, at the Santa Clara Convention Center in Santa Clara, CA - announces that its Call for Papers is open. The Internet of Things (IoT) is the biggest idea since the creation of the Worldwide Web more than 20 years ago.
May. 26, 2015 09:00 AM EDT Reads: 1,535
Even though it’s now Microservices Journal, long-time fans of SOA World Magazine can take comfort in the fact that the URL – soa.sys-con.com – remains unchanged. And that’s no mistake, as microservices are really nothing more than a new and improved take on the Service-Oriented Architecture (SOA) best practices we struggled to hammer out over the last decade. Skeptics, however, might say that this change is nothing more than an exercise in buzzword-hopping. SOA is passé, and now that people are ...
May. 26, 2015 09:00 AM EDT Reads: 3,426
Enterprises are fast realizing the importance of integrating SaaS/Cloud applications, API and on-premises data and processes, to unleash hidden value. This webinar explores how managers can use a Microservice-centric approach to aggressively tackle the unexpected new integration challenges posed by proliferation of cloud, mobile, social and big data projects. Industry analyst and SOA expert Jason Bloomberg will strip away the hype from microservices, and clearly identify their advantages and d...
May. 26, 2015 09:00 AM EDT Reads: 1,343