Bandwidth on Demand Will Only Get You So Far

This spring ESnet achieved something akin to global presence, figuratively with our network, and in-person at conferences as we traded ideas with the technical community, such as the limitations of bandwidth on demand, and how to compose services that are easy for end users to understand and use. In May Steve Cotter, Bill Johnston, and Inder Monga were invited speakers at the TERENA Networking Conference in Prague.

Bill described ESnet’s experience with bandwidth on demandand reviewed new OSCARS collaborative research projects.

Inder Monga followed with a presentation on “Network Service Interface: Concepts and Architecture,”  that discussed the motivation, concepts and architecture in the upcoming Open Grid Forum standard that has the promise to enable researchers simple abstract constructs to dynamically create and manage their communication infrastructure to serve their science. During the talk he explained some of the differentiating attributes of the protocol: Recursive and flexible request and response framework, abstraction of physical topology into a service layer representation—and declared that composable services are the next logical step in network design. The key to dealing with complex infrastructure is to abstract it into objects the users can understand, but that is just the beginning. A composable services model contains essential elements like abstracted technical requirements in a language that all users can understand, failsafe backups, service changes that are transparent, transport efficiency and monitoring for “soft” failures. He pointed out that a Topology Service would be the next target for standardization once the Connection Service was fully specified.

Steve Cotter talked about meeting user expectations in “Fighting a Culture of ‘Bad is Good Enough,” asserting that bandwidth on demand on its own is inadequate to meet the growing needs of science. In ESnet’s surveys scientists report that while the technology is often there, they don’t know how to access it or how to make it work. The result is that poor network performance is often the norm at various sites and scientists are left to fend for themselves without technical assistance. Frustrated, many simply give up attempting to send data via the network and instead use ‘sneakernet’. But it doesn’t have to be this way. Cotter cited the LHC as one example of investment of time and commitment to do networking right. “For us as a community to succeed, we need to provide intuitive services to researchers, and documentation and assistance to make it easy for them.” said Cotter, before he launched into a run-down of new ESnet tools and ventures.

Now that OSCARS version 0.6 is code-complete, ESnet is taking offers to help test the code. ESnet is also working with its sites to build secure, dedicated enclaves on the perimeters of networks, dubbed Science DMZs, which are fully instrumented with perfSONAR.  Separating the campus science traffic from converged network services like VOIP makes it easier to debug and improves performance across the WAN. To make it easier to test and troubleshoot infrastructure, ESnet has created a community knowledge base, that regularly receives more than 2500 hits a week.  ESnet is also developing a multi-function web portal called MyESnet that it will launch at ESCC/Joint Techs in a few weeks. MyESnet will have lots of tools and new features for the scientist and networker, including:  traffic flow visualizations, high-level information about ‘site health’, the ESnet maintenance calendar, a discussion forum and idea repository, as well as one-stop shop where users will be able to log in with Shibboleth or OpenID, initiate perfSONAR tests, and open trouble tickets.

Going beyond just bandwidth on demand
Three weeks later at the NORDUnet conference in Reykjavik, Iceland, Inder Monga discussed the ins and outs of developing composable network services on demand. Given new developments in network virtualization, co-scheduling, cloud services and 100G bandwidth, the network is playing an ever larger role in providing scientists new services.

Incidentally, Inder used high-speed networking to accomplish the enviable feat of being two places at once without violating any laws of physics. Upon landing in Iceland, Inder promptly presented a talk on green networking from Iceland for the conference on Green and Sustainable ICT in Delhi, India.

Designing “greener” networks is one of ESnet’s key priorities, and something you will be hearing more about from ESnet in the future.

Get Ready for World IPv6 Day

On June 8, 2011, content providers, universities, ISPs, and other network organizations will be engaging in activities related to World IPv6 Day. This massive exercise is akin to a “test drive” where content is shared, and networks are configured to support Internet Protocol version 6 (IPv6) which is already supplanting the current Internet Protocol version 4 (IPv4). The point of this dry run is to raise awareness of IPv6, provide feedback on what works and what gets broken, to allow for an overall smoother transition to the new IPv6 protocol as IPv4 addresses are running out. We don’t anticipate that everything–or even a majority of things–will work perfectly, but we do hope that we can learn from any problems that arise.

As a network pioneer, ESnet has made most of its public content available over IPv6 for several years. However, ESnet will still be taking a number of actions to further engage the community on World IPv6 Day.

Additional IPv6 content: ESnet will be enabling IPv6 on some of its secondary websites, such as  The goal, of course, is for all of ESnet’s public-facing websites to have full dual-stack (IPv4/IPv6) capability, and we will take additional steps toward that goal on June 8.

IPv6 on perfSONAR: ESnet’s perfSONAR infrastructure has been configured to support IPv6 at ESnet’s major points-of-presence across the network.  ESnet will begin doing regular performance tests over IPv6 on World IPv6 Day.  We hope to be able to investigate and fix any performance issues (or work with vendors to address these issues) as part of the lessons learned from this exercise.

ESnet also makes certain perfSONAR configuration files are made available as a service to the perfSONAR community.  This includes configuration files that provide information about research and education networks to the bwctl bandwith-testing service.  We will begin distributing IPv6-aware versions of these configuration files in time for World IPv6 Day.

Additional IPv6 monitoring and data collection: ESnet has operated dual-stack open-source software mirrors (e.g. and for some time.  We have created a mechanism for collecting additional flow data to measure IPv6 traffic to and from the mirrors before, during, and after World IPv6 Day.  ESnet plans to continue these instrumentation efforts and–as appropriate–make data on IPv6 usage available to the community.

ECS Videoconferencing: To the extent it can be supported by vendor-supplied equipment, ECS will be adopting IPv6 transport in time for World IPv6 Day.  ESnet’s ECS gatekeepers are able to be configured for IPv6 and will have IPv6 capability by World IPv6 Day.  Unfortunately, the multi-point control units (MCUs) that are part of ECS cannot currently be configured to support IPv6.  ESnet will work with vendors to achieve a fully dual-stack conferencing system in the future.

ESnet staff will be closely monitoring the ESnet network and its peering points to ensure that both IPv4 and IPv6 traffic is flowing optimally during World IPv6 Day.  We will make every attempt to address issues as they come up. We also look forward, although with the rest of the Internet community, to what we hope will be a valuable learning experience.

–Michael Sinatra

101 Excellent Reasons to Send in your SCiNet Research Sandbox Proposal by June 5

As co-chair of the SCinet11 Research Sandbox (SRS), I’d like to remind you that research proposals are due this Sunday, June 5.  The SRS Sandbox is a great way for researchers to play with and test ideas for innovative network architectures, applications, tools and protocols in the high-powered live environment of the SCinet network. This year is notable, as the SRS will provide researchers with access to over 100 Gigabits per second of capacity and the flexibility of a software-programmable testbed network on the SCinet infrastructure.

Digging into 100Gbps and OpenFlow too

At SC11, SRS plans a 10 Gigabit per second (Gbps), multi-vendor OpenFlow network testbed connecting the Washington State Convention Center in Seattle to several national research networks to provide wide area OpenFlow capabilities. OpenFlow allows the implementation of software-defined network policy. While originally conceived to allow researchers to implement new protocols without entirely new hardware, OpenFlow has begun to allow innovation and optimization with current protocols.  SRS is highlighting OpenFlow because it can potentially revolutionize the networking environment and in turn its ability to support HPC applications. OpenFlow allows an HPC data center to easily reconfigure the network on the fly, separating bulk data flows from other flows, for example. Openflow will also provide virtualization of the data center network to support cloud environments, allowing resource allocation to individual virtual machines, or providing multiple clouds from the same infrastructure.

This year, for the first time, submissions to the Disruptive Technologies (DT) program can also demonstrate their research as part of the SRS.  Disruptive Technologies, which has taken place as part of the SC technical program since 2006, examines new computing and networking architectures and interfaces that will significantly impact the high-performance computing field throughout the next five to 15 years, but have not yet emerged in current systems. Submissions to Disruptive Technologies should indicate their interest in demonstrating their research as part of SRS where indicated in the DT online submissions form.

How to submit

SRS submissions should describe the nature of the experiments, desired outcomes, the relevance to the HPC community, as well as a description of the network requirements and vendor collaborations (if appropriate). Submissions do not have to utilize OpenFlow to be considered for SRS. Submissions may be up to 3 pages long, and must address the approach and what will be learned or demonstrated by the effort.

Those whose submissions are accepted will be able to present their experiment in a technical panel session, and submissions will be included in the SC11 proceedings. In addition, accepted projects are expected to write up the results obtained during SC11, and all SRS write-ups will be assembled for journal submission. SCinet may provide additional fiber to a booth to support an SRS experiment as well. Submissions are due by June 5, 2011.  To submit to the SRS, visit:

— Brian Tierney, group lead for the Advanced Network Technologies Group, ESnet.