Gregory Bell

This week, ESnet announces the most significant upgrade in its 26-year history.

We’re finishing up construction of the world’s fastest science network, in collaboration with our close partners at Internet2. In fact, both organizations now share an optical platform that can transmit staggering amounts of data. To visualize just how much, it may be helpful to think about a piano keyboard, which has 88 keys. Imagine that ESnet’s new 100 Gigabit-per-second network fits on middle C, and that Internet2’s new 100G network fits on C#. That leaves 86 keys’ worth of network capacity for us to share in the future. Remember, each piano key represents one instance of the fastest national-scale network currently in existence.

That’s impressive, but it’s not the whole story. Networks like ESnet are passing through three inflection points at this moment in time:

We’re entering an era of cheap, hyper-abundant capacity.
We’re developing a new global architecture that will transform research networks into programmable systems capable of virtualization, reservation, customization, and inspection. Many of these new capabilities simply won’t be available from commercial carriers in a worldwide, multi-domain context.
We’re finally making progress at tackling the end-to-end problem, as more campus networks implement a ScienceDMZ for data mobility.

Taken together, these changes are revolutionary. They motivate a new way of thinking about research networks, as instruments for discovery, not just infrastructures for service delivery.

This idea that research networks serve as vital extensions of a discovery instrument isn’t new. In fact, it’s implicit in the architecture of the Large Hadron Collider (LHC) at CERN. LHC may have been the first large instrument designed around the premise of a highly-capable research networks, but it won’t be the last. In fact, dozens of experiments currently on the drawing board – accelerators, supercomputers, photon and neutron sources, and the extraordinary Square Kilometer Array – share exactly the same premise.

Here’s another respect in which ESnet departs from the model of simple infrastructure. The amount of traffic we carry is growing roughly twice as fast as the commercial Internet, and increasingly that traffic is very different from traffic in the Internet core: it’s dominated by a relatively small number of truly massive flows. You’ve heard this before, but modern science has entered an age of extreme-scale data. In more and more fields, scientific discovery depends on data mobility, as huge data sets from experiments or simulations move to computational and analysis facilities around the globe.

Large-scale scientific collaborations simply require advanced networks, pure and simple.

One last thing about the historical upgrade we’re finishing up. It would not have been possible without heroic efforts of four people who have moved on from the DOE family: Steve Cotter (now CEO of REANNZ in New Zealand), and Dan Hitchcock, Joe Burrescia, and Jim Gagliardi (all happily retired). Steve, Dan, Joe and Jim: we’ll drink a toast in your honor at Supercomputing this week!

– Greg

Hurricane Sandy took a terrible toll this week, and our thoughts remain with the millions of people whose lives were impacted.

From the nightly news, we learned about critical systems that temporarily failed during the extreme weather, and others that continued to function. I’d like to share an example in the second category.

Here’s a screenshot from our public web portal, http://my.es.net, showing traffic to and from Brookhaven National Laboratory on Long Island, during the worst of the storm:

ESnet's Connection to Brookhaven Lab During Hurricane Sandy — Network traffic to and from Long Island’s Brookhaven National Laboratory was not impacted by the devastating storm that struck the New York area this week.

Why does it matter that Brookhaven’s connection to ESnet, and to the broader Internet, remained functional during the disaster? It matters because modern science has entered an age of extreme-scale data. In more and more fields, scientific discovery depends on data mobility, as huge data sets from experiments or simulations move to computational and analysis facilities around the globe. Large-scale science instruments are now designed around the premise that high-speed research networks exist. In this kind of architecture, loss of network connectivity impairs scientific productivity. In the worst case, unique and vital data can be lost.

Much of the network traffic in these graphs can be attributed to ATLAS, one of the extraordinary experiments at CERN’s Large Hadron Collider, outside of Geneva. It’s been a great year at CERN, with announcement of a Higgs-like particle this summer. At ESnet, we were very happy that productivity of the ATLAS experiment was not affected by the massive storm.

Although the research networking community may have benefited from good fortune during storm, it’s important to recognize that a lot of careful planning and sound engineering – conducted over the course of many years – contributed to this outcome. Research networks like ESnet and Internet2, regional networks like NYSERNet, campus networking groups at Brookhaven and elsewhere, and exchange points like MAN LAN all work very hard to harden themselves against individual points of failure. As with all engineering, the devil is in the details: fiber diversity, elimination of shared fate in single conduits and data centers, location and specification of generators, availability of fuel, optical protection schemes, failover for dynamic circuits… the list goes on and on.

We won’t always be so fortunate, but the screenshot above is something our research networking community can take pride in. My thanks to the hundreds of people whose sound decisions made this good outcome possible.