ESnet turns 25 this year. This anniversary marks a major inflection point in our evolution as a network in terms of bandwidth, capability, and control. We are running data through our network a billion—that is 10⁶—times faster than when ESnet was established.   Yet, we are still facing even greater demands for bandwidth as the amount of scientific data explodes and global collaborations expand.

Created in 1986 to meet the Department of Energy’s needs for moving research data and enabling scientists to access remote scientific facilities, ESnet combined two networks serving researchers in fusion research; (MFEnet) and high-energy physics (HEPnet). But ESnet’s roots actually stretch back to the mid-1970s, when staff at the CTR Computer Center at Lawrence Livermore National Laboratory installed four acoustic modems on a borrowed CDC 6600 computer. Our technology morphed over the years from fractional T1s to T1s to DS3, then to ATM, and in the last ten years we have moved to packet over SONET—all driven by the needs of our thousands of scientific users.

Over the last 25 years, ESnet has built an organization of excellence driven by the DOE science mission. We have consistently provided reliable service even as the science areas we support—high energy physics, climate studies, and cosmology, to name a few—have become exponentially more data-intensive. These fields especially rely on ever more powerful supercomputers to do data crunching and modeling. Other fields, such as genomics, are growing at a rapid pace as sequencing technologies become cheaper and more distributed.

Based on the dizzying trajectory of change in scientific computing technology, we urgently need to act now to expand the capacity of scientific networks in order to stay ahead of the demand in years to come.  At this point the ESnet high-speed network carries between 7 and 10 petabytes of data monthly (a single petabyte is equivalent to 13.3 years of HD video). The level of ESnet data traffic is increasing an average of 10 times every 4 years, steeply propelled by the rise in data produced. More powerful supercomputers can create more accurate models, for instance, of drought and rainfall patterns—but greater resolution requires vastly bigger datasets. Scientific collaborations can include thousands of researchers exchanging time-sensitive data around the world. Specialized facilities like the Large Hadron Collider and digital sky surveys produce torrents of data for global distribution. All these factors are poised to overload networks and slow scientific progress.

Tonight we face our biggest milestone yet: We are launching the first phase of our brand new 100 gigabit per second (Gbps) network, currently the fastest scientific network in the world today. Called the Advanced Networking Initiative (ANI), this prototype network forms the foundation of a soon-to-be permanent national network that will vastly expand our future data handling capacity.

The ANI prototype network will initially link researchers to DOE’s supercomputing centers: the National Energy Research Scientific Computing Center (NERSC) at Berkeley Lab, Oak Ridge Leadership Computing Facility (OLCF) , and Argonne Leadership Computing Facility (ALCF), as well as MANLAN, the Manhattan Landing Exchange Point.  ESnet will also deploy 100 Gbps-capable systems throughout the San Francisco Bay Area and Chicago. The ANI prototype network will then transition into a permanent, national 100 Gbps network, paving the way to an eventual terabit-scale DOE network.

To prepare the way, ESnet acquired almost 13,000 miles of dark fiber. This gives the DOE unprecedented control over the network and its assets, enabling upgrades and expansion to capacity and capabilities according to the needs of our scientists. By owning the fiber, we are able to lock in the cost of future upgrades for decades to come.

The third facet of ANI is a nationwide testbed which is being made available to researchers both in the public and private sector as a first of its kind platform to test experimental approaches to new network protocols and architectures in a greater than 10 Gbps network environment.

Researchers are already working on multiple experiments investigating emerging technologies like Remote Direct Memory Access (RDMA)-enabled data transfers at 40 Gigabits per second (Gbps) or new TCP congestion control algorithms that scale to 100Gbps and beyond. By creating a research testbed, ESnet will enable researchers to safely experiment with disruptive technologies that will build the next generation Internet—something impossible to do on networks that also carry daily production traffic.

Bringing You the Universe at 100 Gbps speed

Just this past week our engineers completed the installation of the first phase of network – nearly 6 weeks ahead of schedule. Tonight at the SC11 conference we are showcasing the inauguration of our brand new 100 Gbps network by demonstrating how high-capacity networks can open up the universe – or at least a highly sophisticated computer simulation of the early days of the universe.  The demo will include the transfer of over 5 terabytes of data over the new network from NERSC in Oakland, CA to the Washington State Convention Center in Seattle.

This demonstration is important as astrophysicists are interested in studying high-resolution simulations to better understand the complex structural patterns in the universe. ESnet’s new 100 Gbps network will enable scientists to interactively examine large datasets at full spatial and temporal fidelity without compromising data integrity. This novel capability will also enable remotely located scientists to gain insights from large data volumes located at DOE supercomputing facilities such as NERSC. For comparison purposes, a simulation utilizing a 10 Gpbs network connection will also be displayed on a complementary screen to showcase the vast difference in quality that a magnitude difference of bandwidth can bring to scientific discovery.

To view the 100 Gbps and 10 Gbps simulations, visit: http://www.es.net/RandD/advanced-networking-initiative/

In addition to this great demo, seven other community collaborations will be leveraging the new ESnet 100 Gbps network to support innovative HPC demonstrations at SC11.

As we toast to this great new accomplishment this evening, we recognize that we are building on an amazing 25-year legacy in network innovation. We owe tremendous gratitude to the ESnet staff both past and present for their over two decades of hard work and dedication that has been keenly focused on helping the DOE community solve some of society’s greatest challenges.  Given our team’s accomplishments to date, I cannot wait to see what this next chapter in our history brings.

–Steve Cotter