ESnet

Poorly attended IPv6 conference belies urgency of Internet address depletion

~ ESnetComms

The other week the Department of Veterans Affairs sponsored the 2010 InterAgency IPv6 Information Exchange. As a pioneer in IPv6, the most fundamental protocol of the Internet, ESnet was invited to present on how it uses and implements IPv6. Over 120 agencies were invited to attend but only a handful showed, almost all from various parts of the Department of Defense, the National Institute for Standards and Technology and the Department of Veterans Affairs.

This lacklustre attendance is curious, given that IPv6 is critical to everyone. It is slated to replace IPv4, the current protocol, lock, stock, and barrel. The question is when. What we do know is that address space for existing IP will be exhausted next year. According to Geoff Huston, Adjunct Research Fellow at the Centre for Advanced Internet Architectures, we are literally running out of IPv4 Internet addresses.

Supply of IPv4 Internet Addresses Drying Up, http://www.potaroo.com

The commercial world was in denial of the need for IPv6 until a year ago. Now they are scrambling. But how is the government doing? The level of interest seems to vary by agency.

At this particular conference, presentations ranged from technical discussion of IPv6 implementation from governmental representatives, commercial IPv6 networking providers, and companies selling IPv6 management tools. The VA is implementing IPV6 to facilitate communications between nurses and patients. While ESnet has been using IPv6 for years to link DOE scientists together, some of the other applied uses of this technology, such as improving medical care, are exciting.

It was very encouraging to see the progress the Department of Defense in transitioning to IPv6 while maintaining strict controls for security and reliability. It appears that the DOD is on target for completion of the transition by 2013.

The other area of discussion was in the area of procurement requirements and the approval of new requirements for more complete IPv6 capabilities in new gear.

On the whole, the agencies present seem to be moving on a well organized plan to get to IPv6. The low response from agencies does leave one hoping it was a result of their confidence in their ability to transition in a timely manner that led to so many not participating.

–Kevin Oberman

Reorganizing the way we work, the way we think, for growth forward

~

(from left to right) Vince Dattoria, Bill Johnston and Steve Cotter, of ESnet, Excellence.gov award luncheon, Washington D.C.

A year and some months after ESnet was honored with the Excellence.gov award, it is time to reflect on how our organization is tackling the challenges of running a production network while supporting major projects like the Advanced Networking Initiative.

Last year we added nine new people to the ESnet team: Hing Chow, Andy Lake, Chris Tracy, Josef Grosch, Inder Monga, Greg Bell, Sowmya Balasubramanian, Wendy Tsabba, and Steven Chan – some in part time roles. The challenge has been to maintain organizational excellence while scaling up.

It has been a month now after our recent reorganization, time to re-examine motivations and evaluate early results.  Prior to this, ESnet was organized into separate, well-defined teams, each responsible for their area of expertise.  The Infrastructure group managed the systems supporting ESnet’s internal business processes.  The Network Engineering group handled the design and day-to-day operations of the network. The recently created Advanced Network Technologies group had a clear mission to conduct network research to develop new capabilities and services tailored to meet the needs of the research community.  This structure worked efficiently as long as teams worked within their own domain of expertise. As the projects became more complex, a gap appeared in ownership of getting these components to work together. The old model of network engineers communicating their users’ needs to programmers who developed the tools and the system administrators who supported them resulted in slow and cumbersome integration. The ‘systems approach’ was often discussed but rarely followed.

But emerging technologies, virtualization and converged infrastructures are beginning to blur the lines between the traditional roles of R&E networking and computing.  It recently became clear to me that if ESnet was to deliver high-performance, end-to-end solutions rather than point technologies, we needed to adapt the organization to a new paradigm. The siloed approach had ceased to be fully responsive to the call for seamlessly integrated storage, network and computing. Upon closer examination we realized our network engineers were already effectively writing code, system admins were becoming a lot more familiar with networking, and operational teams were tackling research topics. It was time to formalize a more effective way of working.

The July 1st reorganization turned ESnet into a flatter organization with a greater emphasis on teamwork.  Greg Bell is the new Area Lead for Infrastructure and Support.  His primary responsibility is to ensure a consistent approach across teams toward building end-to-end solutions.  He’ll be working closely with Inder Monga, Area Lead for Research and Services.  Together, they’ll ensure project teams are formed with resources drawn from the different skill sets across the organization.  A team’s work is no longer complete when it is handed off to the next group, now at the successful conclusion of the project. While the reorganization did not dramatically change our organizational structure or roles of the existing teams, I believe it resulted in a change in mindset.

We are continuing to grow – so if you are looking for a challenge, are a network/software engineer and are interested in enabling big science through networking, then send your resume to me at steve@es.net We are looking to add new stars to this excellent team that I have the pleasure of working with every day.

We’ve got Yoo

~ ESnetComms
Professor Ben Yoo

ESnet is pleased to announce that UC Davis Professor S.J. Ben Yoo has been granted a joint faculty appointment with Berkeley Lab, formalizing a long-term relationship.  Yoo will be collaborating on research projects with ESnet to develop Terabit optical networks of the future to meet the upcoming data challenges triggered by Exascale thinking within the DOE.  It is an interesting research challenge, including architecture studies, software developments and networking experiments on ESnet’s ANI testbed. Yoo will also be collaborating with LBNL researchers at NERSC for applications of optical networking within high-end data centers.

“Ben is the type of highly credentialed network research scientist that we hope will take full advantage of the testbed infrastructure we are making available to the community.” said Steve Cotter, head of ESnet.

In a talk this week at Joint Techs http://bit.ly/cAtNt4, Yoo discussed the potential of next generation all-optical Label Switching (OLS) networking, a technology he invented. OLS can seamlessly integrate packet, flow, and circuit traffic. OLS has the potential to fit well within the  industry standard MPLS and GMPLS architectures, and recent experimental results show very good characteristics like extremely low latency (<100 ns) and scalability beyond 40 petabit/sec capacity. It has experimentally demonstrated a per-channel line rate of 100 Gb/s ~ 1.2 Tb/s. A centralized management station can leverage OLS to rapidly assess data flows based on real time collections of labels that contain statistical information about the data traffic.

Yoo has done extensive research with the ATD-Monet testbed in the Washington DC area, telecommunications standardization services at Bellcore, and testbed work at the Sprint Advanced Technology Laboratory. You can get a better sense of his work and research here.

We look forward to working with him on our ANI testbed as well. Yoo’s intention is to push the testbed to its limits. Should be a wild ride.

See us at Joint Techs / ESCC

~

ESnet will be presenting at the Summer Joint Techs / ESCC meeting next week July 11-15 in Columbus, Ohio.  July 11, Joe Metzger and Brian Tierney will be giving a tutorial on “Improving End to End Bulk Data Transfer Rates” at 3 pm that focuses on the problems of moving TeraByte-scale data sets, and Jon Dugan will talk about Iperf in the Network Tools Tutorial.  July 12, at 2:40 pm, Brian Tierney will also be giving a Status Update on the DOE ANI Network Testbed.  July 13, at 10 am, ESnet’s Inder Monga will be replacing Chris Tracy on the panel “Dynamic Provisioning in Multi-Layer, Multi-Vendor Networks“, and Jon Dugan will give another presentation on ESxSNMP.  And finally, at 8:20 am on the 14th, Steve Cotter will give an ESnet Update.

Immediately following Joint Techs, the ESnet Site Coordinating Committee Meeting begins at 1 pm. The agenda is posted here:  http://indico.fnal.gov/conferenceOtherViews.py?view=standard&confId=3428 The ESnet team’s talks will outline the nuts and bolts behind 4 key areas integral to ESnet’s overall strategy:

1. Being an essential scientific resource for DOE. ESnet is making great strides in providing optimal connectivity between DOE labs as well as further developing dedicated network resources, such as our securing of dark fiber at Brookhaven. We are laying the groundwork to manage rapidly accelerating increases in DOE scientific networking traffic.  The first afternoon, Steve Cotter will give a more detailed update on ESnet’s activities at 2:10 pm and Greg Bell will lead the discussion about the ESnet implications of site reliance on cloud or externally-hosted services at 3:55 pm.

2. Knowing our users better than anyone. Steve Cotter will talk about new ways we will be reaching out to and listening to our users needs during his talk.

3. Setting a global standard for user experience.  We may not have invented the seamless user experience, but end to end data transmission is all our users care about. To that end we will be talking about our work on Graphite, URL and Weathermap.  Also, Thursday starting at 9:40 am Joe Metzger will report on the PerfSONAR Joint Interagency Demonstration Project followed by Evangelos Chaniotakis’s presentation on ESnet’s virtual circuit services status.

4. Efficiency. Helping our users optimize their networking resources in collaborations, accessing instrumentation and exascale computing needs in the most energy efficient ways possible.  Be sure not to miss Wednesday evening’s Focus Session on improving WAN network performance with Eli Dart and Joe Metzger beginning at 6:30 pm.

See you in Columbus!

The Fenius project: enabling virtual circuits around the globe

~ haniotak

ESnet has been one of the leading research and education networks in the adoption of virtual circuit technology, which has allowed ESnet customers to sidestep traditional limitations of wide area networking and transfer data at high speed between geographically distant sites at a minimal cost. Each day, tens of terabytes of scientific data flow over ESnet’s Science Data Network between supercomputers, clusters, data storage sites, and experimental data sources like the LHC at CERN.

Essentially, virtual circuits provide an Ethernet pipeline with guaranteed bandwidth between two locations. This traffic is isolated from the rest, allowing our users to run “impolite” protocols like UDP, which would otherwise clog up their regular Internet connection. Our homegrown software code-named OSCARS, enables ESnet to easily monitor this traffic for trends and engineer its route to plan for growth and rearrange capacity according to the needs of our customers.

This is a win-win situation for both us and our customers, and we’re not alone in recognizing this. An increasing number of global research and education backbones and exchange points are deploying such services and writing their own software to manage them: Internet2 is providing the ION service (previously called DCN) based on the OSCARS platform. Across the Atlantic GEANT is developing AutoBAHN, and SURFnet is using Nortel’s DRAC. An international consortium developed Harmony under the Phosphorus project and is now starting up GEYSERS. In Japan, AIST has been developing the G-lambda suite, while Korean KISTI is in the process of coding their DynamicKL project – and there are certainly other projects out there.

Can’t we all just talk?

Now for the bad news: since there isn’t a globally accepted standard for this kind of service, the different software suites don’t quite communicate with one another. OSCARS communicates using the OSCARS application interface, DRAC uses the DRAC interface, and so forth. This, unfortunately, stymies our ambitions to automatically “stitch” virtual circuits across multiple networks. With everyone speaking a different language, this is impossible to accomplish.

A solution is to have a standard software interface; then different implementations would be able to interoperate as long as they were compliant. There is a standards effort in progress by the Open Grid Forum Network Services Interface working group, but an actual standard is probably at least several months away.

A bit of history

Several software developers made an effort to solve the interoperability issue at the GLIF meeting co-located at Joint Techs back in early 2008. After a few presentations, it became evident that all of these APIs,  stripped of their cosmetic differences and special features, looked remarkably alike in terms of the raw pieces of information they handled.  The consensus of the meeting was that there is no real reason not to have basic interoperability, even if many of the bells and whistles would be stripped. The developers then formed the GNI API task force under the umbrella of the GLIF Control Plane technical group, with the objective of duct-taping an interoperability solution together until actual standards emerged.

A mythical reference

They conceived the Fenius project, dubbed for the legendary king of Scythia, Fenius Farsaid. According to Irish folklore, after the collapse of the Tower of Babel, Fenius collected the best parts of the confused tongues of the world and invented a new language.
The Fenius Project is a fairly simple idea: it defines a bare-bones API for virtual circuit services as an interim pseudo-standard. Then developers can easily write code to automatically translate between the “standard” API and a specific software suite such as OSCARS; several translators already exist. The rest of the project is software “glue” which allows Fenius to run standalone, publishing its API as a web service, and routing incoming requests to the specific translator.

We demonstrated Fenius with good results during last year’s GLIF conference in Daejeon, Korea, as well as during Supercomputing 2009 in Portland, OR, using Fenius to provision virtual circuit services on demand across three networks – via completely different technologies, and two different software suites – from a lab in Japan to the NICT booth on the conference showfloor.

The next step for the project is to update its “standard” API according to some important lessons learned during last year’s demos, and to become the de facto external interface of production virtual circuit facilities. We plan to make an appearance at this year’s GLIF conference in Geneva, as well as in Supercomputing 2010 in New Orleans, LA. Fenius is also slated to become a component of OpenDRAC soon. http://www.opendrac.org/.

We hope that Fenius will be able to provide ESnet customers and the international research and education community wider access to the network infrastructure, and that it will enable virtual circuits to become a truly global infrastructure capability in the service of science, research, and education worldwide.

Purchase of dark fiber launches ESnet into new era

~

What sets us apart? ESnet has, and always will focus on anticipating the needs of the extended DOE science community.  This shapes our network strategy, from services and architecture to topology and reach. It also distinguishes ESnet from university research & education networks which are driven by the broader needs of the general university population.  Vis-à-vis commercial networks, ESnet has specialized in handling the relatively small number of very large flows of large-scale science data rather than the enormous number of relatively small data flows traversing commercial carrier networks today. Our desire to always stay a step ahead of the constantly evolving network needs of the scientific community has driven ESnet to take the bold step of purchasing and lighting our first segment of dark fiber.

Owning the road

By owning a tiny but powerful pair of optical fibers, ESnet will no longer have to rely on the vagaries of the commercial market – we will be able to deliver services when we choose and where they are needed.  For example, the DOE envisions using ESnet to link its supercomputing centers with a terabit of capacity by 2015. Our network will be key to enabling the scientific community to accomplish exascale computing by 2020.

Ramping up is no slam-dunk

But providing terabit capacity by using 10 100G waves through commercial services is no slam-dunk and could be very cost-prohibitive.  Without owning the fiber and transport infrastructure, the same is likely to be true when near-terabit waves become available around 2020. Not only does one lose spectral efficiency because a terabit wave won’t fit within ITU standard 50 Ghz spacing – it is necessary to plan for non-standard spacing, with current research pointing towards 200 Ghz to accommodate the signal.

But just solving this problem is not enough, as ESnet’s massive bandwidth requirements don’t end with the supercomputers.  ESnet must deliver steadily increasing amounts of data generated by the Large Hadron Collider as well as similar data sets shared within the climate, fusion, and genomics communities to scientists around the world.

Lighting the way forward

It is clear to us that the only way to scale the network to meet the rapidly propagating needs of large-scale science is by lighting our own dark fiber. Although this relatively small 200-mile loop linking New York City to Brookhaven National Lab barely registers with most in the networking community, it represents an exciting sea change in ESnet’s approach in serving our customers.

–Steve Cotter

Got a networking idea you want to test? ANI testbed opening for business

~ ESnetComms

Want try out some new ideas in network research? ESnet invites you to submit a proposal to run experiments on its reconfigurable testbed.  ESnet’s ARRA-funded Advanced Networking Initiative testbed is a high-performance environment where researchers will have the opportunity to prototype, test, and validate cutting edge networking concepts.

Instructions for submitting proposals can be found here https://sites.google.com/a/lbl.gov/ani-testbed/. Proposals are due October 1, 2010. Decisions will be made January 10, 2011 when the Phase 1 version of the testbed is up and running. The phase I version is a set of 10 Gbps connected layer 1, 2, and 3 equipment that will be deployed on a dark fiber ring we  acquired in Long Island (LIMAN: Long Island Metropolitan Area Network). This will mainly be of interest to researchers doing experiments at layers 1-3, or middleware/application research at 10 Gbps.

The testbed will support research including multi-layer multi-domain hybrid networks, network protocols, component testing for future capabilities, protection and recovery,  automatic classification of large bulk data flows, high-throughput middleware and applications, and any other innovative ideas you may want to try out in a realistic network environment, but with no risk of breaking anything.

Try us. We’re open to suggestions.

The collaboration portal across continents

~ Inder Monga

When organizations are driven by the similar goal of excellence, it is amazing what can be accomplished in two days of non-stop discussions. As I try to consolidate my thoughts from meetings the past two days, the following quotation rings particularly true.

Coming together is a beginning, staying together is progress, and working together is success. ~ Henry Ford

There is still a lot of work to be done, milestones to be hit and unforeseen roadblocks to overcome before we give ourselves a pat on the back, but we are off to a good start.

For the perpetually curious, SURFnet, ESnet and NORDUnet stated their intentions to collaborate on furthering innovation in network research by working on open-source middleware to reserve and seamlessly allocate bandwidth across multiple domains  (http://www.lbl.gov/cs/Archive/news030910.html).  While our first planned meeting was delayed by volcano ash from Iceland, skies and schedules cleared enough for the Dutch to visit us at our facilities at Berkeley lab.  This week’s two day meeting of minds is a baby step taken towards an ambitious goal of open sharing of tools, knowledge, open-source software and other network-related research with the community.

Taking a Break

We did let our visitors out of the conference room for just few minutes to enjoy the beautiful, albeit faint, backdrop of the Golden Gate Bridge.

Down another pit, looking for the secrets of the universe

~ ESnetComms

The Large Hadron Collider is the world’s largest particle collider. At the bottom of a huge shaft dug into a mountain, two beams of subatomic particles, dubbed “hadrons”, shoot around an enclosed racetrack accelerating with every lap until they collide. The idea is to recreate on a small scale the conditions in the universe immediately after the Big Bang.

I actually saw the LHC under construction. I was visiting another accelerator used to generate antimatter, escorted by a physicist with spiky hair who looked like he played in a band. The question of why the universe is composed of matter versus antimatter could give humans a glimpse of “God’s big toe” according to this recent NYTimes interview http://www.nytimes.com/2010/05/18/science/space/18cosmos.html , and points to a fundamental asymmetry in the universe. The accelerator was wrapped in tinfoil and duct tape as a sort of low-tech insulation.

It is not such a bad way to spend one’s career figuring out how the universe got started. Here at ESnet, we are helping. ESnet is part of the network that carries the data from the LHC in Switzerland to groups of physicists in the U.S.

This view is at the inception of traffic on April 1st, 2010.

The Large Hadron Collider is projected to generate 15 petabytes of data yearly from six different detector experiments. The data, too massive to handle internally, is sent to 12 tier 1 sites around the world. CERN data from the ATLAS and CMS detectors travels the Atlantic on USLHCNet http://lhcnet.caltech.edu/ ESnet then carries data to Brookhaven National Laboratory and Fermilab, US tier 1 sites, for processing and archiving. The data is then distributed to tier 2 facilities, mainly of universities and research institutions around the U.S., including the Berkeley Lab’s National Energy Research Scientific Computing Division (NERSC).–Wendy Tsabba

What we see here is an upward spike in traffic. This view is taken within the last 96 hours.