The circuits behind all those SC10 demos

It is midafternoon Wednesday at SC10 and the demos are going strong. Jon Dugan supplied an automatically updating graph in psychedelic colors  http://bit.ly/9HUrqL of the traffic ESnet is able to carry with all the circuits we set up. Getting this far required many hours of work from a lot of ESnet folk to accommodate the virtual circuit needs of both ESnet sites and SCinet customers using the OSCARS IDC software.  As always, the SCinet team has put in long hours in a volatile environment to deliver a high performance network that meets the needs of the exhibitors.

Catch ESnet roundtable discussions today at SC10, 1 and 2 p.m.

Wednesday Nov. 17 at SC10:

At 1 p.m. at Berkeley Lab booth 2448, catch ESnet’s Inder Monga’s round-table discussion on OSCARS virtual circuits. OSCARS, the acronym for On- demand Secure Circuits and Advance Reservation System, allows users to reserve guaranteed bandwidth. Many of the demos at SC10 are being carried by OSCARS virtual circuits which were developed by ESnet with DOE support. Good things to come: ESnet anticipates the rollout of OSCARS 0.6 in early 2011. Version 0.6 will offer greatly expanded capabilities and versatility, such as a modular architecture enabling easy plug and play of the various functional modules and a flexible path computation engine (PCE) workflow architecture.

Then, stick around, because next at 2 p.m.  Brian Tierney from ESnet will lead a roundtable on the research being produced from the ARRA-funded Advanced Networking Initiative (ANI) testbed.

In 2009, the DOE Office of Science awarded ESnet $62 million in recovery funds to establish ANI, a next generation 100Gbps network connecting DOE’s largest unclassified supercomputers, as well as a reconfigurable network testbed for researchers to test new networking concepts and protocols.

Brian will discuss progress on the 100Gbps network, update you on the several research projects already underway on the testbed, discuss testbed capabilities and how to get access to the testbed. He will also answer your questions on how to submit proposals for the next round of testbed network research.

In the meantime, some celeb-spotting at the LBNL booth at SC10.

Inder Monga
Brian Tierney

We’ve got a new standard: IEEE P802.3az Energy-Efficient Ethernet ratified

GUEST BLOG: We’ve got EEE. Now what?

ESnet fully supports the drive for energy efficiency to reduce the amount of emissions caused by information and communication technologies (ICT). IEEE just announced that Energy-Efficient Ethernet (EEE) or IEEE P803.3az is the new standard enabling copper interfaces to reduce energy use when the network link is idle . Energy saving mechanisms of EEE can be applied in systems beyond the Ethernet physical interface, e.g. the PCI Express bus.  New hardware is required to benefit from EEE, however, so its full impact won’t be realized for a few years. ESnet is in the middle of the Advanced Network Initiative to deploy a cross-country 100G network and we would like to explore end-to-end power saving possibilities including 40G and 100G Ethernet interfaces…Here’s why:

In 2006 articles began to appear discussing the ever-increasing consumption of energy by ICT as well as how data center giants such as Google and Microsoft were locating new data centers based on the availability and cost of energy. Meanwhile, the IEEE was attempting to create a specification to reduce network energy usage, and four years later, ratified the P802.3az or Energy-Efficient Ethernet (EEE).

Earlier this year, the ITU World Summit for an Information Society reported that electricity demand by the ICT sector in industrialized countries is between 5 percent and 10 percent of total demand. But about half the electricity used is wasted by powered on equipment that is idle. So while completion of this project seems timely, the question remains how “triple-e” will impact energy use for Ethernet consumers. EEE defines a protocol to reduce energy usage during periods of low utilization for copper and backplane interfaces up to 10Gb/s.  It also reuses a couple of other IEEE protocols to allow uninterrupted communication between link partners.  While this combination of protocols can save energy, it is uncertain how much time the typical Ethernet link operates at low utilization, especially when the P802.3ba, or 40G and 100G Ethernet standard was just ratified in June, suggesting relief for pent up demand for bandwidth.

So why isn’t there an energy-efficient version of the higher-speed version of Ethernet?

The answer depends on the type of Ethernet interface and its purpose in the network, as an interface in a home desktop computer will likely be idle much longer than an uplink interface in a data center switch. A key feature of this new standard is called Low Power Idle. As the name suggests, during idle time the non-critical components of the interface go to sleep.  The link partner is activated by a wake up signal allowing the receiver time to prepare for an incoming frame.

Consider the utilization plot shown below:

File Server Bandwidth Utilization Profile

Not all links are the same

This window on a file server in an enterprise network shows plenty of idle periods. While there are several peaks over 500 Mb/s, the server is mostly idle, with average utilization under one percent. On the other hand, there are many examples of highly utilized links as well (just look at some of ESnet’s utilization plots). In those cases, less energy is saved, but the energy is being used to do something useful, like transfer information.

But when considering the number of triple-speed copper Ethernet interfaces deployed, energy savings start to add up. The P802.3az Task Force members estimated power savings in US alone can reach 5 Terawatt-hours per year, or enough energy to power 6 million 100W light bulbs. This translates into a reduction of the ICT carbon footprint by roughly 5 million tons per year.

Since EEE is built into the physical interface, new hardware will be required to take advantage of this feature and it will take a few years to reach 100% market saturation.

Getting back to the question about energy efficiency for 40G and 100G Ethernet, there are a few reasons why LPI was not specified for P802.3ba. This project overlapped with P802.3az so it is difficult to specify an energy-efficient method for the new speeds, given the record size of the project and the lack of P802.3az resources for work on optical interfaces.  This leads to another question:  Should there be an energy-efficient version of 40G and 100G Ethernet?  Or should there be an energy-efficient version of optical and P802.3ba interfaces?

To decide the scope of the project P802.3az we examined the magnitude of power consumed and number of interfaces in the market.  The power consumed for a 1000BASE-T interface is less than that used by a10GBASE-T interface, but there are orders of magnitudes more of the former. On the other hand, early in the project not many 10GBASE-T interfaces existed in the market, but the interfaces consumed power on the order of 10W-15W per interface.  These numbers are reduced by each new improvement in process technology, but they are still significant.

Considering first generation 100G transceivers can consume more than 20W each and the millions of optical Ethernet interfaces in the market, further standards development is worth pursuing.

Mike Bennett is a senior network engineer for LBLnet and chair of P802.3az. He can be reached at MJBennett@lbl.gov

ESnet recognized for outstanding performance

ESnet’s Evangelos Chaniotakis and Chin Guok received Berkeley Lab’s Outstanding Performance Award for their work in promoting technical standards for international scientific networking. Their work is notable because the implementation of open-source  software development and new technical standards for network interoperability sets the stage for scientists around the world to better share research and collaborate.

Guok and Chaniotakis worked extensively within the DICE community on development of the Inter-domain Controller Protocol (IDCP). They are taking the principles and lessons gained from years of development efforts and applying them to the efforts in international standards bodies such as the Open Grid Forum (OGF), as well as consortia such as the Global Lambda Infrastructure Facility (GLIF).

So far, the IDCP has been adopted by more than a dozen Research and Education (R&E) networks around the world, including Internet2 (the leading US higher education network), GEANT (the trans-European R&E network), NORDUnet (Scandinavian R&E network) and USLHCNet (high speed trans-Atlantic network for the LHC community).

Guok and Chaniotakis have also advanced the widescale deployment of ESnet’s virtual circuits OSCARS (On Demand Secure Circuits and Reservation System). OSCARS, developed with DOE support, enables networks
to schedule and move the increasingly vast amounts of data generated by large-scale scientific collaborations. Since last year, ESnet has seen a 30% increase in the use of virtual circuits. OSCARS virtual circuits now carry over 50% of ESnet’s monthly production traffic.  The increased use of virtual circuits was a major factor enabling ESnet to easily handle a nearly 300% rise in traffic from June 2009 to May 2010.

Why are we reincarnating OSCARS?

OSCARS ESnet traffic patterns

Some recent articles on new developments in virtual circuits such as Fenius and cloud computing with Google, Internet2’s announcements of its ION service, and the recently funded DYNES proposal are all powered by OSCARS or On Demand Secure Circuits and Reservation System, a software engine developed with DOE funding. This open-source software engine provides us with the capability of building a network with highly dynamic, traffic-engineered flows that meet the research data transport needs of scientists. The current deployed release, 0.5.2, has been deployed as a production service within ESnet for the past 3 years. We are currently enhancing 0.5.3 and plan to release the software in the Q4, 2010 time frame.

In the course of running this software as a production service and interacting with scientists, network researchers, and standards community at OGF, we realized we had to redesign the software architecture to be a much more robust and extensible platform. We wanted to be able to easily add new features to the OSCARS platform that would cater to a variety of network engineers and researchers.  With this in mind, the re-architectured OSCARS is planned as release version 0.6. Like any successful product, transitioning from a deployed release to a new one involves thorny issues like backward compatibility and feature parity. Hence, the current balancing act of taking something that is quite good and proven (0.5.2), but making it even better a.k.a. 0.6.

Here are four good reasons why OSCARS 0.6 is the way to go:

1. It can meet production requirements: The modular architecture enables features to be added through the use of distinct modules. This allows specific deployment requirements to be easily integrated into the service. For example, if it is necessary to support a federated AA implementation, the AA modules can be replaced with ones that are compliant with that AA framework (e.g. Shibboleth).  Another example would be High Availability (HA). The 0.6 architecture helps provide HA on a component basis, ensuring that the critical components do not fail.

2. It provides new complex features: As end-sites and their operators become comfortable with point to point provisioning of virtual circuits, we are getting increased requests for complex feature enhancements. The OSCARS 0.5 software architecture is not especially suitable for new features like multi-point circuits and/or multi-layer provisioning. But these new feature requests increase the urgency of moving to the 0.6 release that has been designed with such enhancements in mind. Moreover, the multi-layer ARCHSTONE research project funded by DOE will use 0.6 as the base research platform.

3. Research/GENI and other testbeds: The research community is a major constituent for OSCARS and its continuing development. This community is now conducting experiments on real infrastructure testbeds like the ANI and GENI. To really leverage the power of those testbeds, the research community wants to leverage the OSCARS software base/framework, while researching/innovating on certain algorithms and testing them. OSCARS 0.6 platform’s modular architecture enables the researcher to replace any component with new algorithmic research module. For example, with the new PCE engine re-design, one can write a flexible workflow of custom PCE’s. This flexibility does not exist with the purpose-built, but monolithic architecture of the OSCARS 0.5 codebase.

4. NSI Protocol/Standards: As the European and Asian research and education communities move towards interoperability with the US, it is important to leverage a common understanding brought through via standards. The NSI protocol standardization being discussed in the OGF NSI working group (http://ogf.org/gf/group_info/view.php?group=nsi-wg) needs to be implemented by the network middleware open source community like OSCARS. We feel that the 0.6 is the right platform to upgrade to the standard NSI protocol whenever it is ready.

At ESnet, we invest considerable time in new technology development, but balance this with our operational responsibilities. We invite the community to join in developing OSCARS 0.6, which has greatly improved capabilities over OSCARS 0.5.2. With your participation in the development process, we can accelerate the 0.6 architected software to production-quality as soon as possible.  If this excites you, we welcome you to contribute to the next stage of the OSCARS open source project.

–Chin Guok

A few reasons why ESnet matters to scientists.

Keith Jackson, ESnet Guest Blogger

Recently we’ve been testing the ability to move huge amounts of scientific data in and out of commercial cloud providers like Amazon and Google. We were doing this because if you want to do scientific computation in the cloud, you need to be able to move data in and out efficiently or it will never be useful for science.

Recently we’ve been working with engineers at Google to test the performance of their cloud storage solution. We were in the midst of transferring data between Berkeley Lab servers and the Google cloud when we noticed the data wasn’t moving as fast as it should.

We tried to figure out the root of the problem. The Google folks talked to their networking people and we talked to our engineers at ESnet.

We found there was a bottleneck in the path between Berkeley and Google on the ESnet side. One path was still only 1 gigabit and was scheduled to be upgraded to 10 gigabit in the next week or so. But it limited us to no more than a gigabit per second data transfers.

Using OSCARS, not only did we find the bottleneck, but as Vangelis talked about in a prior blogpost, we were able to find a way to reroute traffic to avoid the slow link, completely bypassing the problem. ESnet was not only able to help me diagnose the problem right away, but were able to suggest and quickly deploy a solution.

In thinking about that problem, a few things occurred to me. For a scientist just concerned with getting data through the network, it is probably easier to work with ESnet than a commercial provider for several reasons.

As a research network, ESnet is completely accessible. A commercial provider would have been completely opaque because of proprietary issues and have no incentive to grant access into its network for troubleshooting by outsiders. Since serving scientists is not its main mission, its sense of urgency would be different. Moreover, a commercial network’s interfaces are not designed for the particular needs of scientists.

But ESnet exists solely to support science, and scientists. Sometimes we need to be reminded that to scientists, quite literally, the “network matters.”

Reorganizing the way we work, the way we think, for growth forward

(from left to right) Vince Dattoria, Bill Johnston and Steve Cotter, of ESnet, Excellence.gov award luncheon, Washington D.C.

A year and some months after ESnet was honored with the Excellence.gov award, it is time to reflect on how our organization is tackling the challenges of running a production network while supporting major projects like the Advanced Networking Initiative.

Last year we added nine new people to the ESnet team: Hing Chow, Andy Lake, Chris Tracy, Josef Grosch, Inder Monga, Greg Bell, Sowmya Balasubramanian, Wendy Tsabba, and Steven Chan – some in part time roles. The challenge has been to maintain organizational excellence while scaling up.

It has been a month now after our recent reorganization, time to re-examine motivations and evaluate early results.  Prior to this, ESnet was organized into separate, well-defined teams, each responsible for their area of expertise.  The Infrastructure group managed the systems supporting ESnet’s internal business processes.  The Network Engineering group handled the design and day-to-day operations of the network. The recently created Advanced Network Technologies group had a clear mission to conduct network research to develop new capabilities and services tailored to meet the needs of the research community.  This structure worked efficiently as long as teams worked within their own domain of expertise. As the projects became more complex, a gap appeared in ownership of getting these components to work together. The old model of network engineers communicating their users’ needs to programmers who developed the tools and the system administrators who supported them resulted in slow and cumbersome integration. The ‘systems approach’ was often discussed but rarely followed.

But emerging technologies, virtualization and converged infrastructures are beginning to blur the lines between the traditional roles of R&E networking and computing.  It recently became clear to me that if ESnet was to deliver high-performance, end-to-end solutions rather than point technologies, we needed to adapt the organization to a new paradigm. The siloed approach had ceased to be fully responsive to the call for seamlessly integrated storage, network and computing. Upon closer examination we realized our network engineers were already effectively writing code, system admins were becoming a lot more familiar with networking, and operational teams were tackling research topics. It was time to formalize a more effective way of working.

The July 1st reorganization turned ESnet into a flatter organization with a greater emphasis on teamwork.  Greg Bell is the new Area Lead for Infrastructure and Support.  His primary responsibility is to ensure a consistent approach across teams toward building end-to-end solutions.  He’ll be working closely with Inder Monga, Area Lead for Research and Services.  Together, they’ll ensure project teams are formed with resources drawn from the different skill sets across the organization.  A team’s work is no longer complete when it is handed off to the next group, now at the successful conclusion of the project. While the reorganization did not dramatically change our organizational structure or roles of the existing teams, I believe it resulted in a change in mindset.

We are continuing to grow – so if you are looking for a challenge, are a network/software engineer and are interested in enabling big science through networking, then send your resume to me at steve@es.net We are looking to add new stars to this excellent team that I have the pleasure of working with every day.

See us at Joint Techs / ESCC

ESnet will be presenting at the Summer Joint Techs / ESCC meeting next week July 11-15 in Columbus, Ohio.  July 11, Joe Metzger and Brian Tierney will be giving a tutorial on “Improving End to End Bulk Data Transfer Rates” at 3 pm that focuses on the problems of moving TeraByte-scale data sets, and Jon Dugan will talk about Iperf in the Network Tools Tutorial.  July 12, at 2:40 pm, Brian Tierney will also be giving a Status Update on the DOE ANI Network Testbed.  July 13, at 10 am, ESnet’s Inder Monga will be replacing Chris Tracy on the panel “Dynamic Provisioning in Multi-Layer, Multi-Vendor Networks“, and Jon Dugan will give another presentation on ESxSNMP.  And finally, at 8:20 am on the 14th, Steve Cotter will give an ESnet Update.

Immediately following Joint Techs, the ESnet Site Coordinating Committee Meeting begins at 1 pm. The agenda is posted here:  http://indico.fnal.gov/conferenceOtherViews.py?view=standard&confId=3428 The ESnet team’s talks will outline the nuts and bolts behind 4 key areas integral to ESnet’s overall strategy:

1. Being an essential scientific resource for DOE. ESnet is making great strides in providing optimal connectivity between DOE labs as well as further developing dedicated network resources, such as our securing of dark fiber at Brookhaven. We are laying the groundwork to manage rapidly accelerating increases in DOE scientific networking traffic.  The first afternoon, Steve Cotter will give a more detailed update on ESnet’s activities at 2:10 pm and Greg Bell will lead the discussion about the ESnet implications of site reliance on cloud or externally-hosted services at 3:55 pm.

2. Knowing our users better than anyone. Steve Cotter will talk about new ways we will be reaching out to and listening to our users needs during his talk.

3. Setting a global standard for user experience.  We may not have invented the seamless user experience, but end to end data transmission is all our users care about. To that end we will be talking about our work on Graphite, URL and Weathermap.  Also, Thursday starting at 9:40 am Joe Metzger will report on the PerfSONAR Joint Interagency Demonstration Project followed by Evangelos Chaniotakis’s presentation on ESnet’s virtual circuit services status.

4. Efficiency. Helping our users optimize their networking resources in collaborations, accessing instrumentation and exascale computing needs in the most energy efficient ways possible.  Be sure not to miss Wednesday evening’s Focus Session on improving WAN network performance with Eli Dart and Joe Metzger beginning at 6:30 pm.

See you in Columbus!

The Fenius project: enabling virtual circuits around the globe

ESnet has been one of the leading research and education networks in the adoption of virtual circuit technology, which has allowed ESnet customers to sidestep traditional limitations of wide area networking and transfer data at high speed between geographically distant sites at a minimal cost. Each day, tens of terabytes of scientific data flow over ESnet’s Science Data Network between supercomputers, clusters, data storage sites, and experimental data sources like the LHC at CERN.

Essentially, virtual circuits provide an Ethernet pipeline with guaranteed bandwidth between two locations. This traffic is isolated from the rest, allowing our users to run “impolite” protocols like UDP, which would otherwise clog up their regular Internet connection. Our homegrown software code-named OSCARS, enables ESnet to easily monitor this traffic for trends and engineer its route to plan for growth and rearrange capacity according to the needs of our customers.

This is a win-win situation for both us and our customers, and we’re not alone in recognizing this. An increasing number of global research and education backbones and exchange points are deploying such services and writing their own software to manage them: Internet2 is providing the ION service (previously called DCN) based on the OSCARS platform. Across the Atlantic GEANT is developing AutoBAHN, and SURFnet is using Nortel’s DRAC. An international consortium developed Harmony under the Phosphorus project and is now starting up GEYSERS. In Japan, AIST has been developing the G-lambda suite, while Korean KISTI is in the process of coding their DynamicKL project – and there are certainly other projects out there.

Can’t we all just talk?

Now for the bad news: since there isn’t a globally accepted standard for this kind of service, the different software suites don’t quite communicate with one another. OSCARS communicates using the OSCARS application interface, DRAC uses the DRAC interface, and so forth. This, unfortunately, stymies our ambitions to automatically “stitch” virtual circuits across multiple networks. With everyone speaking a different language, this is impossible to accomplish.

A solution is to have a standard software interface; then different implementations would be able to interoperate as long as they were compliant. There is a standards effort in progress by the Open Grid Forum Network Services Interface working group, but an actual standard is probably at least several months away.

A bit of history

Several software developers made an effort to solve the interoperability issue at the GLIF meeting co-located at Joint Techs back in early 2008. After a few presentations, it became evident that all of these APIs,  stripped of their cosmetic differences and special features, looked remarkably alike in terms of the raw pieces of information they handled.  The consensus of the meeting was that there is no real reason not to have basic interoperability, even if many of the bells and whistles would be stripped. The developers then formed the GNI API task force under the umbrella of the GLIF Control Plane technical group, with the objective of duct-taping an interoperability solution together until actual standards emerged.

A mythical reference

They conceived the Fenius project, dubbed for the legendary king of Scythia, Fenius Farsaid. According to Irish folklore, after the collapse of the Tower of Babel, Fenius collected the best parts of the confused tongues of the world and invented a new language.
The Fenius Project is a fairly simple idea: it defines a bare-bones API for virtual circuit services as an interim pseudo-standard. Then developers can easily write code to automatically translate between the “standard” API and a specific software suite such as OSCARS; several translators already exist. The rest of the project is software “glue” which allows Fenius to run standalone, publishing its API as a web service, and routing incoming requests to the specific translator.

We demonstrated Fenius with good results during last year’s GLIF conference in Daejeon, Korea, as well as during Supercomputing 2009 in Portland, OR, using Fenius to provision virtual circuit services on demand across three networks – via completely different technologies, and two different software suites – from a lab in Japan to the NICT booth on the conference showfloor.

The next step for the project is to update its “standard” API according to some important lessons learned during last year’s demos, and to become the de facto external interface of production virtual circuit facilities. We plan to make an appearance at this year’s GLIF conference in Geneva, as well as in Supercomputing 2010 in New Orleans, LA. Fenius is also slated to become a component of OpenDRAC soon. http://www.opendrac.org/.

We hope that Fenius will be able to provide ESnet customers and the international research and education community wider access to the network infrastructure, and that it will enable virtual circuits to become a truly global infrastructure capability in the service of science, research, and education worldwide.

Purchase of dark fiber launches ESnet into new era

What sets us apart? ESnet has, and always will focus on anticipating the needs of the extended DOE science community.  This shapes our network strategy, from services and architecture to topology and reach. It also distinguishes ESnet from university research & education networks which are driven by the broader needs of the general university population.  Vis-à-vis commercial networks, ESnet has specialized in handling the relatively small number of very large flows of large-scale science data rather than the enormous number of relatively small data flows traversing commercial carrier networks today. Our desire to always stay a step ahead of the constantly evolving network needs of the scientific community has driven ESnet to take the bold step of purchasing and lighting our first segment of dark fiber.

Owning the road

By owning a tiny but powerful pair of optical fibers, ESnet will no longer have to rely on the vagaries of the commercial market – we will be able to deliver services when we choose and where they are needed.  For example, the DOE envisions using ESnet to link its supercomputing centers with a terabit of capacity by 2015. Our network will be key to enabling the scientific community to accomplish exascale computing by 2020.

Ramping up is no slam-dunk

But providing terabit capacity by using 10 100G waves through commercial services is no slam-dunk and could be very cost-prohibitive.  Without owning the fiber and transport infrastructure, the same is likely to be true when near-terabit waves become available around 2020. Not only does one lose spectral efficiency because a terabit wave won’t fit within ITU standard 50 Ghz spacing – it is necessary to plan for non-standard spacing, with current research pointing towards 200 Ghz to accommodate the signal.

But just solving this problem is not enough, as ESnet’s massive bandwidth requirements don’t end with the supercomputers.  ESnet must deliver steadily increasing amounts of data generated by the Large Hadron Collider as well as similar data sets shared within the climate, fusion, and genomics communities to scientists around the world.

Lighting the way forward

It is clear to us that the only way to scale the network to meet the rapidly propagating needs of large-scale science is by lighting our own dark fiber. Although this relatively small 200-mile loop linking New York City to Brookhaven National Lab barely registers with most in the networking community, it represents an exciting sea change in ESnet’s approach in serving our customers.

–Steve Cotter