Why are we reincarnating OSCARS?

OSCARS ESnet traffic patterns

Some recent articles on new developments in virtual circuits such as Fenius and cloud computing with Google, Internet2’s announcements of its ION service, and the recently funded DYNES proposal are all powered by OSCARS or On Demand Secure Circuits and Reservation System, a software engine developed with DOE funding. This open-source software engine provides us with the capability of building a network with highly dynamic, traffic-engineered flows that meet the research data transport needs of scientists. The current deployed release, 0.5.2, has been deployed as a production service within ESnet for the past 3 years. We are currently enhancing 0.5.3 and plan to release the software in the Q4, 2010 time frame.

In the course of running this software as a production service and interacting with scientists, network researchers, and standards community at OGF, we realized we had to redesign the software architecture to be a much more robust and extensible platform. We wanted to be able to easily add new features to the OSCARS platform that would cater to a variety of network engineers and researchers.  With this in mind, the re-architectured OSCARS is planned as release version 0.6. Like any successful product, transitioning from a deployed release to a new one involves thorny issues like backward compatibility and feature parity. Hence, the current balancing act of taking something that is quite good and proven (0.5.2), but making it even better a.k.a. 0.6.

Here are four good reasons why OSCARS 0.6 is the way to go:

1. It can meet production requirements: The modular architecture enables features to be added through the use of distinct modules. This allows specific deployment requirements to be easily integrated into the service. For example, if it is necessary to support a federated AA implementation, the AA modules can be replaced with ones that are compliant with that AA framework (e.g. Shibboleth).  Another example would be High Availability (HA). The 0.6 architecture helps provide HA on a component basis, ensuring that the critical components do not fail.

2. It provides new complex features: As end-sites and their operators become comfortable with point to point provisioning of virtual circuits, we are getting increased requests for complex feature enhancements. The OSCARS 0.5 software architecture is not especially suitable for new features like multi-point circuits and/or multi-layer provisioning. But these new feature requests increase the urgency of moving to the 0.6 release that has been designed with such enhancements in mind. Moreover, the multi-layer ARCHSTONE research project funded by DOE will use 0.6 as the base research platform.

3. Research/GENI and other testbeds: The research community is a major constituent for OSCARS and its continuing development. This community is now conducting experiments on real infrastructure testbeds like the ANI and GENI. To really leverage the power of those testbeds, the research community wants to leverage the OSCARS software base/framework, while researching/innovating on certain algorithms and testing them. OSCARS 0.6 platform’s modular architecture enables the researcher to replace any component with new algorithmic research module. For example, with the new PCE engine re-design, one can write a flexible workflow of custom PCE’s. This flexibility does not exist with the purpose-built, but monolithic architecture of the OSCARS 0.5 codebase.

4. NSI Protocol/Standards: As the European and Asian research and education communities move towards interoperability with the US, it is important to leverage a common understanding brought through via standards. The NSI protocol standardization being discussed in the OGF NSI working group (http://ogf.org/gf/group_info/view.php?group=nsi-wg) needs to be implemented by the network middleware open source community like OSCARS. We feel that the 0.6 is the right platform to upgrade to the standard NSI protocol whenever it is ready.

At ESnet, we invest considerable time in new technology development, but balance this with our operational responsibilities. We invite the community to join in developing OSCARS 0.6, which has greatly improved capabilities over OSCARS 0.5.2. With your participation in the development process, we can accelerate the 0.6 architected software to production-quality as soon as possible.  If this excites you, we welcome you to contribute to the next stage of the OSCARS open source project.

–Chin Guok

A few reasons why ESnet matters to scientists.

Keith Jackson, ESnet Guest Blogger

Recently we’ve been testing the ability to move huge amounts of scientific data in and out of commercial cloud providers like Amazon and Google. We were doing this because if you want to do scientific computation in the cloud, you need to be able to move data in and out efficiently or it will never be useful for science.

Recently we’ve been working with engineers at Google to test the performance of their cloud storage solution. We were in the midst of transferring data between Berkeley Lab servers and the Google cloud when we noticed the data wasn’t moving as fast as it should.

We tried to figure out the root of the problem. The Google folks talked to their networking people and we talked to our engineers at ESnet.

We found there was a bottleneck in the path between Berkeley and Google on the ESnet side. One path was still only 1 gigabit and was scheduled to be upgraded to 10 gigabit in the next week or so. But it limited us to no more than a gigabit per second data transfers.

Using OSCARS, not only did we find the bottleneck, but as Vangelis talked about in a prior blogpost, we were able to find a way to reroute traffic to avoid the slow link, completely bypassing the problem. ESnet was not only able to help me diagnose the problem right away, but were able to suggest and quickly deploy a solution.

In thinking about that problem, a few things occurred to me. For a scientist just concerned with getting data through the network, it is probably easier to work with ESnet than a commercial provider for several reasons.

As a research network, ESnet is completely accessible. A commercial provider would have been completely opaque because of proprietary issues and have no incentive to grant access into its network for troubleshooting by outsiders. Since serving scientists is not its main mission, its sense of urgency would be different. Moreover, a commercial network’s interfaces are not designed for the particular needs of scientists.

But ESnet exists solely to support science, and scientists. Sometimes we need to be reminded that to scientists, quite literally, the “network matters.”

Poorly attended IPv6 conference belies urgency of Internet address depletion

The other week the Department of Veterans Affairs sponsored the 2010 InterAgency IPv6 Information Exchange. As a pioneer in IPv6, the most fundamental protocol of the Internet, ESnet was invited to present on how it uses and implements IPv6. Over 120 agencies were invited to attend but only a handful showed, almost all from various parts of the Department of Defense, the National Institute for Standards and Technology and the Department of Veterans Affairs.

This lacklustre attendance is curious, given that IPv6 is critical to everyone. It is slated to replace IPv4, the current protocol, lock, stock, and barrel. The question is when. What we do know is that address space for existing IP will be exhausted next year. According to Geoff Huston, Adjunct Research Fellow at the Centre for Advanced Internet Architectures, we are literally running out of IPv4 Internet addresses.

Supply of IPv4 Internet Addresses Drying Up, http://www.potaroo.com

The commercial world was in denial of the need for IPv6 until a year ago. Now they are scrambling. But how is the government doing? The level of interest seems to vary by agency.

At this particular conference, presentations ranged from technical discussion of IPv6 implementation from governmental representatives, commercial IPv6 networking providers, and companies selling IPv6 management tools. The VA is implementing IPV6 to facilitate communications between nurses and patients. While ESnet has been using IPv6 for years to link DOE scientists together, some of the other applied uses of this technology, such as improving medical care, are exciting.

It was very encouraging to see the progress the Department of Defense in transitioning to IPv6 while maintaining strict controls for security and reliability. It appears that the DOD is on target for completion of the transition by 2013.

The other area of discussion was in the area of procurement requirements and the approval of new requirements for more complete IPv6 capabilities in new gear.

On the whole, the agencies present seem to be moving on a well organized plan to get to IPv6. The low response from agencies does leave one hoping it was a result of their confidence in their ability to transition in a timely manner that led to so many not participating.

–Kevin Oberman

Reorganizing the way we work, the way we think, for growth forward

(from left to right) Vince Dattoria, Bill Johnston and Steve Cotter, of ESnet, Excellence.gov award luncheon, Washington D.C.

A year and some months after ESnet was honored with the Excellence.gov award, it is time to reflect on how our organization is tackling the challenges of running a production network while supporting major projects like the Advanced Networking Initiative.

Last year we added nine new people to the ESnet team: Hing Chow, Andy Lake, Chris Tracy, Josef Grosch, Inder Monga, Greg Bell, Sowmya Balasubramanian, Wendy Tsabba, and Steven Chan – some in part time roles. The challenge has been to maintain organizational excellence while scaling up.

It has been a month now after our recent reorganization, time to re-examine motivations and evaluate early results.  Prior to this, ESnet was organized into separate, well-defined teams, each responsible for their area of expertise.  The Infrastructure group managed the systems supporting ESnet’s internal business processes.  The Network Engineering group handled the design and day-to-day operations of the network. The recently created Advanced Network Technologies group had a clear mission to conduct network research to develop new capabilities and services tailored to meet the needs of the research community.  This structure worked efficiently as long as teams worked within their own domain of expertise. As the projects became more complex, a gap appeared in ownership of getting these components to work together. The old model of network engineers communicating their users’ needs to programmers who developed the tools and the system administrators who supported them resulted in slow and cumbersome integration. The ‘systems approach’ was often discussed but rarely followed.

But emerging technologies, virtualization and converged infrastructures are beginning to blur the lines between the traditional roles of R&E networking and computing.  It recently became clear to me that if ESnet was to deliver high-performance, end-to-end solutions rather than point technologies, we needed to adapt the organization to a new paradigm. The siloed approach had ceased to be fully responsive to the call for seamlessly integrated storage, network and computing. Upon closer examination we realized our network engineers were already effectively writing code, system admins were becoming a lot more familiar with networking, and operational teams were tackling research topics. It was time to formalize a more effective way of working.

The July 1st reorganization turned ESnet into a flatter organization with a greater emphasis on teamwork.  Greg Bell is the new Area Lead for Infrastructure and Support.  His primary responsibility is to ensure a consistent approach across teams toward building end-to-end solutions.  He’ll be working closely with Inder Monga, Area Lead for Research and Services.  Together, they’ll ensure project teams are formed with resources drawn from the different skill sets across the organization.  A team’s work is no longer complete when it is handed off to the next group, now at the successful conclusion of the project. While the reorganization did not dramatically change our organizational structure or roles of the existing teams, I believe it resulted in a change in mindset.

We are continuing to grow – so if you are looking for a challenge, are a network/software engineer and are interested in enabling big science through networking, then send your resume to me at steve@es.net We are looking to add new stars to this excellent team that I have the pleasure of working with every day.