DOE

Autumn means SC10

~ Jon Dugan
Jon Dugan blogs about ESnet at SC10

By autumn most folks are thinking about the holidays, but for me, fall is filled with thoughts of something different.  Since 1998, I’ve had the privilege of working with the SCinet committee of Supercomputing, a.k.a. the International Conference for High Performance Computing, Networking, Storage, and Analysis, if you are not into the whole brevity thing. SCinet is the team of people that builds the local area and wide area network for the Supercomputing conference.

This year’s conference, SC10 is in New Orleans, LA, from November 13 until November 19.  Planning for each year’s show starts a few years ahead of time. Not long after one year’s show ends, the serious planning for the next year’s SCinet begins. It’s a cycle that I’ve been through many times now, and it’s a bit like an old friend at this point.  Most of the time we enjoy each other’s company immensely but when things get stressful, we can really irritate each other.

SCinet is a pretty amazing network.  After a year of planning, there are three weeks of concentrated effort to set it up. It’s operational for about one week and it takes about two days to tear down.  This year we will have 270 Gbps of wide area network connectivity with dedicated circuits to ESnet, Internet2, NLR, and NOAA.  We will deliver over 200 network connections to the various booths on the show floor.

As amazing as the network is, the people who build it are even more amazing. They are drawn from universities, national laboratories, network equipment vendors and nationwide research and education networks.  It’s not just Americans; there are people from several different countries with strong showings from the Netherlands and Germany most years.  Many of these folks are leaders in their areas of expertise and all of them are bright, capable people.  Each of them has given up a fair bit of their own time to participate (while most have some sponsorship from their employers, it’s not unheard of for people to take vacation time to participate).

Why would people give up many evenings and weekends every fall to be a part of SCinet?  Because it’s an amazing opportunity to learn about the state of the art in computer networking and to expand your professional network as welll. I consider myself extremely fortunate to work with each of the people that make up SCinet.

So what is ESnet doing for with SCinet this year?  Glad you asked.  First off, we are bringing three 10G circuits to the show floor.  As of Friday, October 29th all three were up and operational.  One of these circuits will be used for general IP traffic, but the other two will be used to carry dynamic circuits managed by the OSCARS IDC software.

These circuits will provide substantial support for various demonstrations by exhibitors, connecting scientific and computational resources at labs and universities to the booths on the show floor.

Finally, ESnet has four people who are volunteering in various capacities within SCinet. Evangelos Chaniotakis and myself will be working with the routing team. The routing team provides IP, IPv6, IP multicast service, manages the wide area peerings, manages wide area layer 2 circuits, configures the interfaces that face the booths on the show floor and works closely with several other teams to provide a highly scalable and reliable network. John Christman is working with the fiber team, building the optical fiber infrastructure to support SCinet (all booth connections are delivered over optical fiber, which allows booths to be connected to the network using the highest-speed interfaces available.) Brian Tierney will be working with the SCinet measurement team collecting network telemetry, and using it to provide useful and meaningful visualizations of what’s happening inside SCinet as well as providing tools and hosts to allow making active network measurement such as Iperf, nuttcp, and OWAMP. The measurement data is also made accessible using the perfSONAR suite of tools. They’re also using the SNMP polling software I wrote for ESnet called ESxSNMP.


Important spots to visit:

If you are coming to SC10 this year, be sure to come by the SCinet NOC in booth 3351.  I’d be happy to meet anyone who’s read this; feel free to ask for me at the SCinet help desk at the same booth. LBNL (ESnet’s parent organization) is located in booth 2448.  Finally, I am hosting a Bird’s of a Feather (BOF) session on network measurement during the show, the details are here.

And check out the other ESnet demos: You can download a map of ESnet at SC10: SC 2010_floormapFL

LBNL Booth 2448, ESnet roundtable discussions

Inder Monga, Advanced Network Technologies Group, ESnet, will lead a roundtable discussion on: On-demand Secure Circuits and Advance Reservation System (OSCARS), 1-2 p.m., Wednesday, Nov. 17

Many of the demos at SC10 are being carried by OSCARS virtual circuits developed by ESnet with DOE support. OSCARS enables networks to reserve and schedule virtual circuits that provide bandwidth and service guarantees to support large-scale science collaborations. In the first quarter of 2011, ESnet expects to unveil OSCARS 0.6, which will offer vastly expanded capabilities, such as a modular architecture allowing for easy plug and play of the various functional modules and a flexible path computation engine (PCE) workflow architecture. Adoption of OSCARS has been accelerating as 2010 has seen deployments at Internet2 and other domestic and international research and education networks.  Since last year, ESnet saw a 30% increase in the use of virtual circuits. OSCARS virtual circuits now carry over 50% of ESnet’s monthly production traffic.  Increased use of virtual circuits was a major factor enabling ESnet to easily handle a nearly 300% rise in traffic from June 2009 to May 2010.

Brian Tierney, Advanced Network Technologies Group, ESnet, will lead a roundtable discussion on: ARRA-funded Advanced Networking Initiative (ANI) Testbed, 2- 3 p.m. Wednesday, Nov. 17

The research and education community’s needs for managing and transferring data are exploding in scope and complexity. In 2009 the DOE Office of Science awarded ESnet $62 million in Recovery Act funds to create the Advanced Networking Initiative (ANI). This next-generation, 100 Gbps network will connect DOE’s largest unclassified supercomputers. ANI is also establishing a high performance, reconfigurable network testbed for researchers to experiment with advanced networking concepts and protocols. ESnet has now opened the testbed to researchers. A variety of experiments pushing the boundaries of current network technology are underway. Another round of proposals are in the offing. The testbed will be moving from Lawrence Berkeley National Laboratory to ESnet’s dark fiber ring at Long Island (LIMAN: Long Island Metropolitan Area Network) in January 2011 and eventually the 100 Gbps national prototype network ESnet is building to accelerate deployment of 100 Gbps technologies and provide a platform for the DOE experimental facilities at Oak Ridge National Laboratory and the Magellan resources at at the National Energy Research Scientific Computing Center (NERSC)  and Argonne National Laboratory.

–Jon Dugan

Get your latest version of perfSONAR PS toolkit

~ ESnetComms

The perfSONAR collaboration has just announced the release of version 3.2 of the pS Performance Toolkit.  The latest version of perfSONAR-PS is available at http://psps.perfsonar.net/toolkit/

“perfSONAR is critical to helping our constituents achieve acceptable network performance for scientific applications” commented Eli Dart, network engineer at ESnet, who uses perfSONAR routinely. “The continued progress by the perfSONAR collaboration in developing and deploying practical test and measurement infrastructure is helping our scientists conduct critical research collaborations in many areas including climate, energy, biology and genomics.”

Release 3.2 offers features including:

– – Adoption of the CentOS Linux platform
– – Improved admin GUIs to guide the user through setup and maintenance of the host
– – Updated versions of perfSONAR-PS monitoring software
– – Updated versions of performance software including OWAMP, BWCTL, NDT, NPAD, Cacti, and Iperf
– – Stability and bugfix based improvements
– – Ability to install directly to the hard disk

For more information, check the release notes.

“Large-scale science pushes the limits of computing, storage and networking” said ESnet’s Chris Tracy, who is actively engaged in ongoing testing of trans-Atlantic circuits with US LHCNet for use by the Large Hadron Collider experiments. “perfSONAR is used daily in monitoring the network infrastructure that supports the LHC experiments, and perfSONAR provides the go-to toolset for finding, characterizing, and locating performance problems so that they can be fixed and the science can proceed unhindered.”

Collaborators on the pS Performance Toolkit urge users to subscribe to the mailing list for support questions and general discussion.

We’ve got a new standard: IEEE P802.3az Energy-Efficient Ethernet ratified

~ ESnetComms

GUEST BLOG: We’ve got EEE. Now what?

ESnet fully supports the drive for energy efficiency to reduce the amount of emissions caused by information and communication technologies (ICT). IEEE just announced that Energy-Efficient Ethernet (EEE) or IEEE P803.3az is the new standard enabling copper interfaces to reduce energy use when the network link is idle . Energy saving mechanisms of EEE can be applied in systems beyond the Ethernet physical interface, e.g. the PCI Express bus.  New hardware is required to benefit from EEE, however, so its full impact won’t be realized for a few years. ESnet is in the middle of the Advanced Network Initiative to deploy a cross-country 100G network and we would like to explore end-to-end power saving possibilities including 40G and 100G Ethernet interfaces…Here’s why:

In 2006 articles began to appear discussing the ever-increasing consumption of energy by ICT as well as how data center giants such as Google and Microsoft were locating new data centers based on the availability and cost of energy. Meanwhile, the IEEE was attempting to create a specification to reduce network energy usage, and four years later, ratified the P802.3az or Energy-Efficient Ethernet (EEE).

Earlier this year, the ITU World Summit for an Information Society reported that electricity demand by the ICT sector in industrialized countries is between 5 percent and 10 percent of total demand. But about half the electricity used is wasted by powered on equipment that is idle. So while completion of this project seems timely, the question remains how “triple-e” will impact energy use for Ethernet consumers. EEE defines a protocol to reduce energy usage during periods of low utilization for copper and backplane interfaces up to 10Gb/s.  It also reuses a couple of other IEEE protocols to allow uninterrupted communication between link partners.  While this combination of protocols can save energy, it is uncertain how much time the typical Ethernet link operates at low utilization, especially when the P802.3ba, or 40G and 100G Ethernet standard was just ratified in June, suggesting relief for pent up demand for bandwidth.

So why isn’t there an energy-efficient version of the higher-speed version of Ethernet?

The answer depends on the type of Ethernet interface and its purpose in the network, as an interface in a home desktop computer will likely be idle much longer than an uplink interface in a data center switch. A key feature of this new standard is called Low Power Idle. As the name suggests, during idle time the non-critical components of the interface go to sleep.  The link partner is activated by a wake up signal allowing the receiver time to prepare for an incoming frame.

Consider the utilization plot shown below:

File Server Bandwidth Utilization Profile

Not all links are the same

This window on a file server in an enterprise network shows plenty of idle periods. While there are several peaks over 500 Mb/s, the server is mostly idle, with average utilization under one percent. On the other hand, there are many examples of highly utilized links as well (just look at some of ESnet’s utilization plots). In those cases, less energy is saved, but the energy is being used to do something useful, like transfer information.

But when considering the number of triple-speed copper Ethernet interfaces deployed, energy savings start to add up. The P802.3az Task Force members estimated power savings in US alone can reach 5 Terawatt-hours per year, or enough energy to power 6 million 100W light bulbs. This translates into a reduction of the ICT carbon footprint by roughly 5 million tons per year.

Since EEE is built into the physical interface, new hardware will be required to take advantage of this feature and it will take a few years to reach 100% market saturation.

Getting back to the question about energy efficiency for 40G and 100G Ethernet, there are a few reasons why LPI was not specified for P802.3ba. This project overlapped with P802.3az so it is difficult to specify an energy-efficient method for the new speeds, given the record size of the project and the lack of P802.3az resources for work on optical interfaces.  This leads to another question:  Should there be an energy-efficient version of 40G and 100G Ethernet?  Or should there be an energy-efficient version of optical and P802.3ba interfaces?

To decide the scope of the project P802.3az we examined the magnitude of power consumed and number of interfaces in the market.  The power consumed for a 1000BASE-T interface is less than that used by a10GBASE-T interface, but there are orders of magnitudes more of the former. On the other hand, early in the project not many 10GBASE-T interfaces existed in the market, but the interfaces consumed power on the order of 10W-15W per interface.  These numbers are reduced by each new improvement in process technology, but they are still significant.

Considering first generation 100G transceivers can consume more than 20W each and the millions of optical Ethernet interfaces in the market, further standards development is worth pursuing.

Mike Bennett is a senior network engineer for LBLnet and chair of P802.3az. He can be reached at MJBennett@lbl.gov

ESnet recognized for outstanding performance

~ ESnetComms

ESnet’s Evangelos Chaniotakis and Chin Guok received Berkeley Lab’s Outstanding Performance Award for their work in promoting technical standards for international scientific networking. Their work is notable because the implementation of open-source  software development and new technical standards for network interoperability sets the stage for scientists around the world to better share research and collaborate.

Guok and Chaniotakis worked extensively within the DICE community on development of the Inter-domain Controller Protocol (IDCP). They are taking the principles and lessons gained from years of development efforts and applying them to the efforts in international standards bodies such as the Open Grid Forum (OGF), as well as consortia such as the Global Lambda Infrastructure Facility (GLIF).

So far, the IDCP has been adopted by more than a dozen Research and Education (R&E) networks around the world, including Internet2 (the leading US higher education network), GEANT (the trans-European R&E network), NORDUnet (Scandinavian R&E network) and USLHCNet (high speed trans-Atlantic network for the LHC community).

Guok and Chaniotakis have also advanced the widescale deployment of ESnet’s virtual circuits OSCARS (On Demand Secure Circuits and Reservation System). OSCARS, developed with DOE support, enables networks
to schedule and move the increasingly vast amounts of data generated by large-scale scientific collaborations. Since last year, ESnet has seen a 30% increase in the use of virtual circuits. OSCARS virtual circuits now carry over 50% of ESnet’s monthly production traffic.  The increased use of virtual circuits was a major factor enabling ESnet to easily handle a nearly 300% rise in traffic from June 2009 to May 2010.

A few reasons why ESnet matters to scientists.

~ ESnetComms

Keith Jackson, ESnet Guest Blogger

Recently we’ve been testing the ability to move huge amounts of scientific data in and out of commercial cloud providers like Amazon and Google. We were doing this because if you want to do scientific computation in the cloud, you need to be able to move data in and out efficiently or it will never be useful for science.

Recently we’ve been working with engineers at Google to test the performance of their cloud storage solution. We were in the midst of transferring data between Berkeley Lab servers and the Google cloud when we noticed the data wasn’t moving as fast as it should.

We tried to figure out the root of the problem. The Google folks talked to their networking people and we talked to our engineers at ESnet.

We found there was a bottleneck in the path between Berkeley and Google on the ESnet side. One path was still only 1 gigabit and was scheduled to be upgraded to 10 gigabit in the next week or so. But it limited us to no more than a gigabit per second data transfers.

Using OSCARS, not only did we find the bottleneck, but as Vangelis talked about in a prior blogpost, we were able to find a way to reroute traffic to avoid the slow link, completely bypassing the problem. ESnet was not only able to help me diagnose the problem right away, but were able to suggest and quickly deploy a solution.

In thinking about that problem, a few things occurred to me. For a scientist just concerned with getting data through the network, it is probably easier to work with ESnet than a commercial provider for several reasons.

As a research network, ESnet is completely accessible. A commercial provider would have been completely opaque because of proprietary issues and have no incentive to grant access into its network for troubleshooting by outsiders. Since serving scientists is not its main mission, its sense of urgency would be different. Moreover, a commercial network’s interfaces are not designed for the particular needs of scientists.

But ESnet exists solely to support science, and scientists. Sometimes we need to be reminded that to scientists, quite literally, the “network matters.”