Vote for ESnet5 – a good steward of Petabytes

It has almost been a year since we turned 25, and transferred a “whole universe of data” at Supercomputing 2011 – and that was over a single 100G link between NERSC and Seattle. Now we are close to the end of building out the fifth generation of our network, ESnet5.

In order to minimize the downtime for the sites, we are building ESnet5 parallel to ESnet4, with just a configuration-driven switch of traffic from one network to the other. Since the scientific community we serve depends on the network to be up, it’s important to have assurance that the transition is not disruptive in anyway. The question we have heard over and over again from some of our users – when you switch the ESnet4 production traffic to ESnet5, how confident are you that the whole network will work, and not collapse?

In this blog post, I’d like to introduce an innovative testing concept the ESnet network engineering team (with special kudos to Chris Tracy) developed and implemented to address this very problem.

The goal of our testing was to ensure that the entire set of backbone network ports would perform solidly at full 100 Gbps saturation with no packet loss, over a 24 hour period. However we had some limitations. With only one Ixia test-set with 100 GE cards at hand to generate and receive packets and not enough time to ship that equipment to every PoP and test each link, we had to create a test scenario that would generate confidence that all the deployed routers and optical hardware, optics, the fiber connections, and the underlying fiber would performing flawlessly in production.

This implied creating a scenario where the 100 Gbps traffic stream being generated by the Ixia would be carried bi-directionally over every router interface deployed in ESnet5, traverse it only once and cover the entire ESnet5 topology before being directed back to the test hardware. A creative traffic loop was created that traversed the entire footprint, and we called it the ‘Snake Test’. Even though the first possible solution was used to create the ‘snake’, I am wondering if this could be framed as a NP-hard theoretical computer science and optimization approach known as the traveling salesman problem for more complex topologies?

The diagram below illustrates the test topology:

So after sending around 1.2 petabytes of data in 24 hours, and accounting for surprise fiber maintenance events that caused the link to flap, the engineering team was happy to see a zero loss situation.

Here’s a sample portion of the data collected:

Automation is key – utility scripts had been built to do things like load/unload the config from the routers, poll the firewall counters (to check for loss ingress/egress at every interface), clear stats, parse the resulting log files and turn them into CSV (a snapshot you see in the picture) for analysis.

Phew! – the transition from ESnet4 to ESnet5 continues without a hitch. Watch out for the completion news, it may come quicker than you think…..

On behalf of the ESnet Engineering team

We’ve got Yoo

Professor Ben Yoo

ESnet is pleased to announce that UC Davis Professor S.J. Ben Yoo has been granted a joint faculty appointment with Berkeley Lab, formalizing a long-term relationship.  Yoo will be collaborating on research projects with ESnet to develop Terabit optical networks of the future to meet the upcoming data challenges triggered by Exascale thinking within the DOE.  It is an interesting research challenge, including architecture studies, software developments and networking experiments on ESnet’s ANI testbed. Yoo will also be collaborating with LBNL researchers at NERSC for applications of optical networking within high-end data centers.

“Ben is the type of highly credentialed network research scientist that we hope will take full advantage of the testbed infrastructure we are making available to the community.” said Steve Cotter, head of ESnet.

In a talk this week at Joint Techs http://bit.ly/cAtNt4, Yoo discussed the potential of next generation all-optical Label Switching (OLS) networking, a technology he invented. OLS can seamlessly integrate packet, flow, and circuit traffic. OLS has the potential to fit well within the  industry standard MPLS and GMPLS architectures, and recent experimental results show very good characteristics like extremely low latency (<100 ns) and scalability beyond 40 petabit/sec capacity. It has experimentally demonstrated a per-channel line rate of 100 Gb/s ~ 1.2 Tb/s. A centralized management station can leverage OLS to rapidly assess data flows based on real time collections of labels that contain statistical information about the data traffic.

Yoo has done extensive research with the ATD-Monet testbed in the Washington DC area, telecommunications standardization services at Bellcore, and testbed work at the Sprint Advanced Technology Laboratory. You can get a better sense of his work and research here.

We look forward to working with him on our ANI testbed as well. Yoo’s intention is to push the testbed to its limits. Should be a wild ride.

New 100GE Ethernet Standard IEEE 802.3ba (and 40GE as well)

From Charles Spurgeon's Ethernet Website


History is being written: from a simple diagram published in 1976 by Dr. Robert Metcalfe, with a data rate of 3 Mpbs, Ethernet surely has come a long way in the last 30 years. Coincidentally, the parent of ESnet, MFEnet, was also launched around the same time as a result of the new Fusion Energy supercomputer center at Lawrence Livermore National Labs (LLNL) http://www.es.net/hypertext/esnet-history.html. It is remarkable to note that right now, as the 100GE standard got ratified, ESnet engineers are very much on the ball, busy putting 100GE enabled routers through the paces within our labs.

For ESnet and the Department of Energy – it is all about the science. To enable large-scale scientific discovery, very large scientific instruments are being built. You have read on the blog about DUSEL, and are familiar with LHC. These instruments – particle accelerators, synchrotron light sources, large supercomputers, and radio telescope farms are generating massive amounts of data and involve large collaborations of scientists to extract useful research results from it. The Office of Science is looking to ESnet to build and operate a network infrastructure that can scale up to meet the highly demanding performance needs of scientific applications. The Advanced Networking Initiative (ANI) to build the nationwide 100G prototype network and a research testbed is a great start. If you are interested in being part of this exciting initiative, do bid on the 100G Transport RFP.

As a community, we need to keep advancing the state of networking to meet the oncoming age of the digital data deluge ().

To wit, the recent IEEE 802.3ba press release: – http://standards.ieee.org/announcements/2010/ratification8023ba.html Note the quote from our own Steve Cotter:

Steve Cotter, Department Head, ESnet at Lawrence Berkeley National Laboratory
“As the science community looks at collaboratively solving hard research problems to positively impact the lives of billions of people, for example research on global climate change, alternative energy and energy efficiency, as well as projects including the Large Hadron Collider that probe the fundamental nature of our universe – leveraging petascale data and information exchange is essential. To accomplish this, high-bandwidth networking is necessary for distributed exascale computing. Lawrence Berkeley National Laboratory is excited to leverage this standard to build a 100G nationwide prototype network as part of ESnet’s participation in the DOE Office of Science Advanced Networking Initiative.”

100GE around the bend?

Ever feel the exhilaration of sitting in a race car and going around the track at super high speeds? We came close to that experience when we recently received early editions of a vendor’s 100GE cards for their routers. The experience so far has been phenomenal – no issues in getting the card up and running, the optics work great and packets are getting forwarded at line-rate. We are putting those cards through our rigorous testing process, though our lips are sealed for now.

For the industry, this is significant progress – just last year we started the Advanced Networking Initiative (ANI) project and the prospects of actually seeing a 100GE interface in a router this soon seemed far off. So kudos to the vendor (you know who you are) and to the IEEE 802.3ba 40/100Gbps task force – if you are listening, this stuff is ready to go!

Listening to the drumbeats of 100G

Since we received the news of our ARRA funds for our Advanced Network Initiative (ANI), we have been working steadily towards the ambitious goal of deploying 100G technology to stimulate networking advancement in high-speed interfaces. In the course of pushing the ANI agenda over the last year we have met with many carriers and vendors. Although I cannot share my personal conversations with these vendors–the thought of flocks of lawyers descending upon me ensures reticence–I have avidly been tracking their public announcements.

Just today Cisco announced the acquisition of Core Optics, a coherent optics board vendor. I had the good fortune to see their 40G system working at OFC this year and I am sure they are working hard on getting their 100G system up and running. Google typically has been quiet about the innovations in their network to keep up with data center innovations. But they have been uncharacteristically beating the 100G drum in public – which meshes very well with our needs. If you look at ESnet, the traffic transiting our network is growing at an alarming rate of 80% year over year.



At the Packet-Optical Transport Evolution conference (http://www.lightreading.com/live/event_information.asp?event_id=29209) Bikash Koley, Senior Network Architect, Google points at machine to machine traffic (like video sensors) as the motivators for needing such bandwidth and cites hybrid networking or packet-optical integration as solving the problems of the day.

If I can quote their article in Lightreading (http://www.lightreading.com/document.asp?doc_id=192230&amp;):

“Regardless of how the network looks, Google is dead set on one thing: it wants label-switched routing and DWDM capabilities to be combined into one box. It doesn’t matter if that’s a label-switched router (LSR) with DWDM added, or a DWDM box with Layer 3 knowledge added,”Koley said. (He also stressed that the LSR doesn’t have to be a full-blown router.)

Now that is one statement we are in agreement with Bikash Koley and Google. Our own experience developing OSCARS (On-demand Secure Circuits and Advanced Reservation System – www.es.net/oscars) and the hybrid networking architecture to deal with large science flows since 2004 has led us down the path of on-demand traffic-engineered paths. With MPLS being the only choice at that time (discussing the merits of new IEEE mac-in-mac protocols will require a separate blog), we created the OSCARS open-source software to dynamically configure LSPs through our hybrid packet-optical network. That worked very well for us, though it was clear that we did not really need the service/edge capabilities built into the router. So if there is a way to make the core simpler, cheaper and more energy-efficient – sign me up for that and we will run OSCARS over it to steer those circuits to where our customers want it.

We at ESnet continue to march ahead towards the 100G-prototype network. I look forward to your comments on 40G, 100G, the new Ethernet standards and the way to higher rates (400GE, 1TBE.)

Inder Monga, Network Architect

Email me at: Imonga@es.net