Network as Instrument: Please Adjust Your Mental Models

This week, ESnet announces the most significant upgrade in its 26-year history.

We’re finishing up construction of the world’s fastest science network, in collaboration with our close partners at Internet2.  In fact, both organizations now share an optical platform that can transmit staggering amounts of data.  To visualize just how much, it may be helpful to think about a piano keyboard, which has 88 keys.  Imagine that ESnet’s new 100 Gigabit-per-second network fits on middle C, and that Internet2’s new 100G network fits on C#.   That leaves 86 keys’ worth of network capacity for us to share in the future.  Remember, each piano key represents one instance of the fastest national-scale network currently in existence.

That’s impressive, but it’s not the whole story.  Networks like ESnet are passing through three inflection points at this moment in time:

  1. We’re entering an era of cheap, hyper-abundant capacity.
  2. We’re developing a new global architecture that will transform research networks into programmable systems capable of virtualization, reservation, customization, and inspection.  Many of these new capabilities simply won’t be available from commercial carriers in a worldwide, multi-domain context.
  3. We’re finally making progress at tackling the end-to-end problem, as more campus networks implement a ScienceDMZ for data mobility.

Taken together, these changes are revolutionary.  They motivate a new way of thinking about research networks, as instruments for discovery, not just infrastructures for service delivery.

The ATLAS Detector (courtesy CERN).

This idea that research networks serve as vital extensions of a discovery instrument isn’t new.  In fact, it’s implicit in the architecture of the Large Hadron Collider (LHC) at CERN.  LHC may have been the first large instrument designed around the premise of a highly-capable research networks, but it won’t be the last.  In fact, dozens of experiments currently on the drawing board – accelerators, supercomputers, photon and neutron sources, and the extraordinary Square Kilometer Array – share exactly the same premise.

Here’s another respect in which ESnet departs from the model of simple infrastructure.  The amount of traffic we carry is growing roughly twice as fast as the commercial Internet, and increasingly that traffic is very different from traffic in the Internet core: it’s dominated by a relatively small number of truly massive flows.   You’ve heard this before, but modern science has entered an age of extreme-scale data.  In more and more fields, scientific discovery depends on data mobility, as huge data sets from experiments or simulations move to computational and analysis facilities around the globe.

Large-scale scientific collaborations simply require advanced networks, pure and simple.

One last thing about the historical upgrade we’re finishing up.  It would not have been possible without heroic efforts of four people who have moved on from the DOE family: Steve Cotter (now CEO of REANNZ in New Zealand), and Dan Hitchcock, Joe Burrescia, and Jim Gagliardi (all happily retired).   Steve, Dan, Joe and Jim: we’ll drink a toast in your honor at Supercomputing this week!

– Greg

Vote for ESnet5 – a good steward of Petabytes

It has almost been a year since we turned 25, and transferred a “whole universe of data” at Supercomputing 2011 – and that was over a single 100G link between NERSC and Seattle. Now we are close to the end of building out the fifth generation of our network, ESnet5.

In order to minimize the downtime for the sites, we are building ESnet5 parallel to ESnet4, with just a configuration-driven switch of traffic from one network to the other. Since the scientific community we serve depends on the network to be up, it’s important to have assurance that the transition is not disruptive in anyway. The question we have heard over and over again from some of our users – when you switch the ESnet4 production traffic to ESnet5, how confident are you that the whole network will work, and not collapse?

In this blog post, I’d like to introduce an innovative testing concept the ESnet network engineering team (with special kudos to Chris Tracy) developed and implemented to address this very problem.

The goal of our testing was to ensure that the entire set of backbone network ports would perform solidly at full 100 Gbps saturation with no packet loss, over a 24 hour period. However we had some limitations. With only one Ixia test-set with 100 GE cards at hand to generate and receive packets and not enough time to ship that equipment to every PoP and test each link, we had to create a test scenario that would generate confidence that all the deployed routers and optical hardware, optics, the fiber connections, and the underlying fiber would performing flawlessly in production.

This implied creating a scenario where the 100 Gbps traffic stream being generated by the Ixia would be carried bi-directionally over every router interface deployed in ESnet5, traverse it only once and cover the entire ESnet5 topology before being directed back to the test hardware. A creative traffic loop was created that traversed the entire footprint, and we called it the ‘Snake Test’. Even though the first possible solution was used to create the ‘snake’, I am wondering if this could be framed as a NP-hard theoretical computer science and optimization approach known as the traveling salesman problem for more complex topologies?

The diagram below illustrates the test topology:

So after sending around 1.2 petabytes of data in 24 hours, and accounting for surprise fiber maintenance events that caused the link to flap, the engineering team was happy to see a zero loss situation.

Here’s a sample portion of the data collected:

Automation is key – utility scripts had been built to do things like load/unload the config from the routers, poll the firewall counters (to check for loss ingress/egress at every interface), clear stats, parse the resulting log files and turn them into CSV (a snapshot you see in the picture) for analysis.

Phew! – the transition from ESnet4 to ESnet5 continues without a hitch. Watch out for the completion news, it may come quicker than you think…..

On behalf of the ESnet Engineering team

Science Data Weathers the Storm

Hurricane Sandy took a terrible toll this week, and our thoughts remain with the millions of people whose lives were impacted.

From the nightly news, we learned about critical systems that temporarily failed during the extreme weather, and others that continued to function.  I’d like to share an example in the second category.

Here’s a screenshot from our public web portal,, showing traffic to and from Brookhaven National Laboratory on Long Island, during the worst of the storm:

Network traffic to and from Long Island’s Brookhaven National Laboratory was not impacted by the devastating storm that struck the New York area this week.

Why does it matter that Brookhaven’s connection to ESnet, and to the broader Internet, remained functional during the disaster? It matters because modern science has entered an age of extreme-scale data. In more and more fields, scientific discovery depends on data mobility, as huge data sets from experiments or simulations move to computational and analysis facilities around the globe. Large-scale science instruments are now designed around the premise that high-speed research networks exist. In this kind of architecture, loss of network connectivity impairs scientific productivity. In the worst case, unique and vital data can be lost.

Much of the network traffic in these graphs can be attributed to ATLAS, one of the extraordinary experiments at CERN’s Large Hadron Collider, outside of Geneva. It’s been a great year at CERN, with announcement of a Higgs-like particle this summer. At ESnet, we were very happy that productivity of the ATLAS experiment was not affected by the massive storm.

Although the research networking community may have benefited from good fortune during storm, it’s important to recognize that a lot of careful planning and sound engineering – conducted over the course of many years – contributed to this outcome. Research networks like ESnet and Internet2, regional networks like NYSERNet, campus networking groups at Brookhaven and elsewhere, and exchange points like MAN LAN all work very hard to harden themselves against individual points of failure. As with all engineering, the devil is in the details: fiber diversity, elimination of shared fate in single conduits and data centers, location and specification of generators, availability of fuel, optical protection schemes, failover for dynamic circuits… the list goes on and on.

We won’t always be so fortunate, but the screenshot above is something our research networking community can take pride in. My thanks to the hundreds of people whose sound decisions made this good outcome possible.