100G

ESnet & its regional, national and international partners have built high speed, high performance 100 Gbps networks. See how the research community is using these networks to accelerate science & research.

ESnet 2010 Round-up: Part 2

~ Inder Monga

Our take on ANI, OSCARS, perfSONAR, and the state of things to come.

ANI Testbed

In 2010 ESnet led the technology curve in the testbed by putting together a great multi-layer design, deploying specially tuned 10G IO Testers, became early investors in the Openflow protocol by deploying the NEC switches, and built a research breadboard of end-hosts leveraging open-source virtualization and cloud technologies.

The first phase of the ANI testbed is concluding. After 6+ months of operational life, with exciting research projects like ARCHSTONE, Flowbench, HNTES, climate studies, and more leveraging the facilities, we are preparing to move the testbed to its second phase on the dark fiber ring in Long Island. Our call for proposals that closed October 1st garnered excellent ideas from researchers and was reviewed by the academic and industry stalwarts in the panel. We are tying up loose ends as we light the next phase of testbed research.

OSCARS

This year the OSCARS team has been extremely productive. We added enhancements to create the next version (0.5.3) of currently production OSCARS software, progressed on architecting and developing a highly modular and flexible platform for the next-generation OSCARS (0.6), a PCE-SDK targeted towards network researchers focused on creating complex algorithms for path computation, and developing FENIUS to support the GLIF Automated GOLE demonstrator.

Not only did the ESnet team multitask on various ANI, operational network and OSCARS deliverables, it also spent significant time supporting our R&E partners like Internet2, SURFnet, NORDUnet, RNP and others interested in investigating the capabilities of this open-source software. We also appreciate Internet2’s participation by dedicating testing resources for OSCARS 0.6 starting next year to ensure a thoroughly vetted and stable platform during the April timeframe. This is just one example of the accomplishments possible for the R&E community by commiting to partnership and collaboration.

perfSONAR collaboration

perfSONAR kept up its rapid pace of feature additions and new releases in joint collaboration with Internet2 and others. In addition to rapid progress in software capabilities, ESnet is aggressively rolling out perfSONAR nodes in its 10G and 1G POPs, creating an infrastructure where the network can be tuned to hum. With multiple thorny network problems now solved, perfSONAR has proven to be great tool delivering value. This year we focused on making perfSONAR easily deployable and adding the operational features to transform it into a production service. An excellent workshop in August succinctly captured the challenges and opportunities to leverage perfSONAR for operational troubleshooting and also by researchers in understanding further how to improve networks. Joint research projects continue to stimulate further development with a focus on solving end-to-end performance issues.

The next networking challenge?

2011

Life in technology tends to be interesting, even though people keep warning about the commoditization of networking gear. The focus area for innovation just shifts, but never goes away.  Some areas of interest as we evaluate our longer term objectives next year:

  • Enabling the end-to-end world: What new enhancements or innovations are needed to deploy performance measurement, and control techniques to enable a seamless end-to-end application performance?
  • Life in a Terabit digital world: What network innovations are needed to fully exploit the requirement for Terabit connectivity between supercomputer centers in the 2015-2018 timeframe?
  • Life in a carbon economy: What are the low-hanging fruit for networks to become more energy-efficient and/or enable energy-efficiency in the IT ecosystem they play? Cloud-y or Clear?

We welcome your comments and contributions,

Happy New Year

Inder Monga and the folks at ESnet

ESnet 2010 Round-up: Part 1

~ Inder Monga

Part 1: Considering the state of 100G and the state we’re in

The past year slipped by at a dizzying pace for us at ESnet, as we made new forays into cutting-edge technologies. In this two-part blogpost, we will recap accomplishments of the year, but also consider the challenges facing us in the one to come as we progress towards delivering the Advanced Networking Initiative.

Stimulating 100G

One of our prime directives with ANI funding was to stimulate the 100G market towards increasing spectral efficiency. In the last year, we have had wonderful engagement with the vendors that are moving products in this direction. Coherent receivers and DP-QPSK modulation are now standard fare for the 40G/100G solutions. At the latest conference, IEEE ANTS, in Mumbai last week, the 100G question was considered solved. Researchers are now exploring innovative solutions to deliver a next generation of 100G with higher power efficiency, or jump to the next level in delivering 400G. One researcher at the Indian Institute of Technology, Mumbai, is looking at revolutionizing the power consumption curve of the digital processing paradigm of coherent solutions by investigating analog processing techniques (super secret, so we will just have to wait and see).

IEEE-ANTS Conference

A representative from OFS, the optical fiber company, described research on new fibers which cater to the coherent world that will enable better performance. He quoted hero experiments, papers, and research presented at this years’ OFC, touting the advantages of new fiber captured through the joint work of Alcatel-Lucent Bell Labs and OFS (ex-Lucent) research collaborators. There is a lot of fiber still being laid out in the developing countries and they are well positioned to take advantage of this new research to bring cheaper broadband connectivity in so far underserved communities.

Some selected points raised at the panel regarding 400G and beyond:

  • Raman amplification is coming back in vogue
  • 50GHz ITU-Grid needs to evolve to flexi-grid technology. With flexi-grid, some of the basic modem concepts of negotiation (remember the auto-sensing modems of late 90’s) is back – where based on distance, and loss, the appropriate grid spacing can be negotiated for each wavelength.
  • If the industry sticks with electrical compensation, optical equipment will see increased electricity consumption by the power-hungry Analog-Digital Conversion (ADC) and Digital Signal Processing (DSP) ASICS. With advances in CMOS, the status quo might not suffice in a few years, especially since the whole industry is out there sticking the router vendors with a big “power-hungry” sticker. The equations in power-consumption tradeoffs still need to be studied and appropriate comparisons made. I hope the vendors also develop a perspective in that direction.
  • Comcast, the only other vendor on the panel, mentioned current capacities of 30x40G (coherent) on some links of their backbone and their eagerness to deploy 100G solutions. They are WAY ahead in deploying 100G, though the industry seems to not broadcast such news widely.
  • Comcast felt that coherent optics in the Metro Area is overkill and entreated the vendors not to build one-size-fits-all solutions even if simpler (and, they hope making 100G more affordable, as well).
  • Speeding into next year

    There was little discussion on the 100GE standards, although there was a clear message that LR-10 is here to stay, mainly supported by Data Center customers, though almost all traditional carriers intend deploy LR-4, in the case it starts costing less than a Ferrari.

At Supercomputing 2010, the SCinet community orchestrated and deployed 100G-capable equipment from Brocade, Cisco, Ciena and Juniper, to name a few vendors, and included 100G host demonstrations of data transfers by NASA. It was encouraging to see the Academic and R&E community lead deployment and testing of 100G [See a sample poster below].

100G Testing by German Academia

The SCinet community lives on the “bleeding edge” and supported a demonstration by Internet2, ESnet, and other partners carrying live 100Gbps application data over a 100G wave from Chicago to New Orleans show floor.

We are looking forward to Seattle (SC11) and can already predict multiple 100G’s of bandwidth coming towards the show floor – if you have any cool demonstrations that you would like to collaborate with us, please drop us a note.

— Inder Monga

A few grace notes to SC10

~ ESnetComms

As SC10 wound down, ESnet started disassembling the network of connections that brought experimental data from the rest of the country to New Orleans, (and at least a bit of the universe as well). We detected harbingers of 100Gbps in all sorts of places. We will be sharing our observations on promising and significant networking technologies with you in blogs to come.

We were impressed by the brilliant young people we saw at the SC Student Cluster Competition organized collaboratively part of SC Communities, which brings together programs designed to support emerging leaders and groups that have traditionally been under-represented in computing.  Teams came from U.S. universities, including Purdue, Florida A&M, SUNY Stonybrook, and University of Texas at Austin, as well as universities from China and Russia.

Florida A&M team
Nizhni Novgorod State University team

At ESnet, we are always looking for bright, committed students interested in networking internships (paid!). We are also still hiring.

 

As SC10 concluded, the computer scientists and network engineers on the streets of the city dissipated, replaced by a conference of anthropologists. SC11 is scheduled for Seattle. But before we go, a note of appreciation to New Orleans.

Katrina memorial

Across from the convention center is a memorial to the people lost to Katrina; a sculpture of a wrecked house pinioned in a tree. But if you walk down the street to the corner of Bourbon and Canal, each night you will hear the trumpets of the ToBeContinued Brass Band. The band is a group of friends who met in their high school marching bands and played together for years until scattered by Katrina. Like the city, they are regrouping, and are profiled in a new documentary.

Our mission at ESnet is to help scientists to collaborate and share research. But a number of ESnet people are also musicians and music lovers, and we draw personal inspiration from the energy, technical virtuosity and creativity of artists as well as other engineers and scientists. We are not alone in this.

New Orleans is a great American city, and we wish it well.

100G: it may be voodoo, but it certainly works

~ ESnetComms

SC10, Thursday morning.

During the SC10 conference, NASA, NOAA, ESnet, the Dutch Research Consortium, US LHCNet and CANARIE announced that they would transmit 100Gbps of scientific data between Chicago and New Orleans.  Through the use of 14 10GigE interconnects, researchers attempted to  completely utilize the full 100 Gbps worth of bandwidth by producing up to twelve 8.5-to-10Gbps individual data flows.

Brian Tierney reports: “We are very excited that a team from NASA Goddard completely filled the 100G connection from the show floor to Chicago.  It is certainly the first time for the supercomputing conference that a single wavelength over the WAN achieved 100Gbps. The other thing that is so exciting about it that they used a single sending host to do it.”

“Was this just voodoo?” asked NERSC’s Brent Draney.

Tierney assures us that indeed it must have been… but whatever they did, it certainly works.

Catch ESnet roundtable discussions today at SC10, 1 and 2 p.m.

~ ESnetComms

Wednesday Nov. 17 at SC10:

At 1 p.m. at Berkeley Lab booth 2448, catch ESnet’s Inder Monga’s round-table discussion on OSCARS virtual circuits. OSCARS, the acronym for On- demand Secure Circuits and Advance Reservation System, allows users to reserve guaranteed bandwidth. Many of the demos at SC10 are being carried by OSCARS virtual circuits which were developed by ESnet with DOE support. Good things to come: ESnet anticipates the rollout of OSCARS 0.6 in early 2011. Version 0.6 will offer greatly expanded capabilities and versatility, such as a modular architecture enabling easy plug and play of the various functional modules and a flexible path computation engine (PCE) workflow architecture.

Then, stick around, because next at 2 p.m.  Brian Tierney from ESnet will lead a roundtable on the research being produced from the ARRA-funded Advanced Networking Initiative (ANI) testbed.

In 2009, the DOE Office of Science awarded ESnet $62 million in recovery funds to establish ANI, a next generation 100Gbps network connecting DOE’s largest unclassified supercomputers, as well as a reconfigurable network testbed for researchers to test new networking concepts and protocols.

Brian will discuss progress on the 100Gbps network, update you on the several research projects already underway on the testbed, discuss testbed capabilities and how to get access to the testbed. He will also answer your questions on how to submit proposals for the next round of testbed network research.

In the meantime, some celeb-spotting at the LBNL booth at SC10.

Inder Monga
Brian Tierney

Autumn means SC10

~ Jon Dugan
Jon Dugan blogs about ESnet at SC10

By autumn most folks are thinking about the holidays, but for me, fall is filled with thoughts of something different.  Since 1998, I’ve had the privilege of working with the SCinet committee of Supercomputing, a.k.a. the International Conference for High Performance Computing, Networking, Storage, and Analysis, if you are not into the whole brevity thing. SCinet is the team of people that builds the local area and wide area network for the Supercomputing conference.

This year’s conference, SC10 is in New Orleans, LA, from November 13 until November 19.  Planning for each year’s show starts a few years ahead of time. Not long after one year’s show ends, the serious planning for the next year’s SCinet begins. It’s a cycle that I’ve been through many times now, and it’s a bit like an old friend at this point.  Most of the time we enjoy each other’s company immensely but when things get stressful, we can really irritate each other.

SCinet is a pretty amazing network.  After a year of planning, there are three weeks of concentrated effort to set it up. It’s operational for about one week and it takes about two days to tear down.  This year we will have 270 Gbps of wide area network connectivity with dedicated circuits to ESnet, Internet2, NLR, and NOAA.  We will deliver over 200 network connections to the various booths on the show floor.

As amazing as the network is, the people who build it are even more amazing. They are drawn from universities, national laboratories, network equipment vendors and nationwide research and education networks.  It’s not just Americans; there are people from several different countries with strong showings from the Netherlands and Germany most years.  Many of these folks are leaders in their areas of expertise and all of them are bright, capable people.  Each of them has given up a fair bit of their own time to participate (while most have some sponsorship from their employers, it’s not unheard of for people to take vacation time to participate).

Why would people give up many evenings and weekends every fall to be a part of SCinet?  Because it’s an amazing opportunity to learn about the state of the art in computer networking and to expand your professional network as welll. I consider myself extremely fortunate to work with each of the people that make up SCinet.

So what is ESnet doing for with SCinet this year?  Glad you asked.  First off, we are bringing three 10G circuits to the show floor.  As of Friday, October 29th all three were up and operational.  One of these circuits will be used for general IP traffic, but the other two will be used to carry dynamic circuits managed by the OSCARS IDC software.

These circuits will provide substantial support for various demonstrations by exhibitors, connecting scientific and computational resources at labs and universities to the booths on the show floor.

Finally, ESnet has four people who are volunteering in various capacities within SCinet. Evangelos Chaniotakis and myself will be working with the routing team. The routing team provides IP, IPv6, IP multicast service, manages the wide area peerings, manages wide area layer 2 circuits, configures the interfaces that face the booths on the show floor and works closely with several other teams to provide a highly scalable and reliable network. John Christman is working with the fiber team, building the optical fiber infrastructure to support SCinet (all booth connections are delivered over optical fiber, which allows booths to be connected to the network using the highest-speed interfaces available.) Brian Tierney will be working with the SCinet measurement team collecting network telemetry, and using it to provide useful and meaningful visualizations of what’s happening inside SCinet as well as providing tools and hosts to allow making active network measurement such as Iperf, nuttcp, and OWAMP. The measurement data is also made accessible using the perfSONAR suite of tools. They’re also using the SNMP polling software I wrote for ESnet called ESxSNMP.


Important spots to visit:

If you are coming to SC10 this year, be sure to come by the SCinet NOC in booth 3351.  I’d be happy to meet anyone who’s read this; feel free to ask for me at the SCinet help desk at the same booth. LBNL (ESnet’s parent organization) is located in booth 2448.  Finally, I am hosting a Bird’s of a Feather (BOF) session on network measurement during the show, the details are here.

And check out the other ESnet demos: You can download a map of ESnet at SC10: SC 2010_floormapFL

LBNL Booth 2448, ESnet roundtable discussions

Inder Monga, Advanced Network Technologies Group, ESnet, will lead a roundtable discussion on: On-demand Secure Circuits and Advance Reservation System (OSCARS), 1-2 p.m., Wednesday, Nov. 17

Many of the demos at SC10 are being carried by OSCARS virtual circuits developed by ESnet with DOE support. OSCARS enables networks to reserve and schedule virtual circuits that provide bandwidth and service guarantees to support large-scale science collaborations. In the first quarter of 2011, ESnet expects to unveil OSCARS 0.6, which will offer vastly expanded capabilities, such as a modular architecture allowing for easy plug and play of the various functional modules and a flexible path computation engine (PCE) workflow architecture. Adoption of OSCARS has been accelerating as 2010 has seen deployments at Internet2 and other domestic and international research and education networks.  Since last year, ESnet saw a 30% increase in the use of virtual circuits. OSCARS virtual circuits now carry over 50% of ESnet’s monthly production traffic.  Increased use of virtual circuits was a major factor enabling ESnet to easily handle a nearly 300% rise in traffic from June 2009 to May 2010.

Brian Tierney, Advanced Network Technologies Group, ESnet, will lead a roundtable discussion on: ARRA-funded Advanced Networking Initiative (ANI) Testbed, 2- 3 p.m. Wednesday, Nov. 17

The research and education community’s needs for managing and transferring data are exploding in scope and complexity. In 2009 the DOE Office of Science awarded ESnet $62 million in Recovery Act funds to create the Advanced Networking Initiative (ANI). This next-generation, 100 Gbps network will connect DOE’s largest unclassified supercomputers. ANI is also establishing a high performance, reconfigurable network testbed for researchers to experiment with advanced networking concepts and protocols. ESnet has now opened the testbed to researchers. A variety of experiments pushing the boundaries of current network technology are underway. Another round of proposals are in the offing. The testbed will be moving from Lawrence Berkeley National Laboratory to ESnet’s dark fiber ring at Long Island (LIMAN: Long Island Metropolitan Area Network) in January 2011 and eventually the 100 Gbps national prototype network ESnet is building to accelerate deployment of 100 Gbps technologies and provide a platform for the DOE experimental facilities at Oak Ridge National Laboratory and the Magellan resources at at the National Energy Research Scientific Computing Center (NERSC)  and Argonne National Laboratory.

–Jon Dugan

We’ve got Yoo

~ ESnetComms
Professor Ben Yoo

ESnet is pleased to announce that UC Davis Professor S.J. Ben Yoo has been granted a joint faculty appointment with Berkeley Lab, formalizing a long-term relationship.  Yoo will be collaborating on research projects with ESnet to develop Terabit optical networks of the future to meet the upcoming data challenges triggered by Exascale thinking within the DOE.  It is an interesting research challenge, including architecture studies, software developments and networking experiments on ESnet’s ANI testbed. Yoo will also be collaborating with LBNL researchers at NERSC for applications of optical networking within high-end data centers.

“Ben is the type of highly credentialed network research scientist that we hope will take full advantage of the testbed infrastructure we are making available to the community.” said Steve Cotter, head of ESnet.

In a talk this week at Joint Techs http://bit.ly/cAtNt4, Yoo discussed the potential of next generation all-optical Label Switching (OLS) networking, a technology he invented. OLS can seamlessly integrate packet, flow, and circuit traffic. OLS has the potential to fit well within the  industry standard MPLS and GMPLS architectures, and recent experimental results show very good characteristics like extremely low latency (<100 ns) and scalability beyond 40 petabit/sec capacity. It has experimentally demonstrated a per-channel line rate of 100 Gb/s ~ 1.2 Tb/s. A centralized management station can leverage OLS to rapidly assess data flows based on real time collections of labels that contain statistical information about the data traffic.

Yoo has done extensive research with the ATD-Monet testbed in the Washington DC area, telecommunications standardization services at Bellcore, and testbed work at the Sprint Advanced Technology Laboratory. You can get a better sense of his work and research here.

We look forward to working with him on our ANI testbed as well. Yoo’s intention is to push the testbed to its limits. Should be a wild ride.

New 100GE Ethernet Standard IEEE 802.3ba (and 40GE as well)

~ Inder Monga
From Charles Spurgeon's Ethernet Website


History is being written: from a simple diagram published in 1976 by Dr. Robert Metcalfe, with a data rate of 3 Mpbs, Ethernet surely has come a long way in the last 30 years. Coincidentally, the parent of ESnet, MFEnet, was also launched around the same time as a result of the new Fusion Energy supercomputer center at Lawrence Livermore National Labs (LLNL) http://www.es.net/hypertext/esnet-history.html. It is remarkable to note that right now, as the 100GE standard got ratified, ESnet engineers are very much on the ball, busy putting 100GE enabled routers through the paces within our labs.

For ESnet and the Department of Energy – it is all about the science. To enable large-scale scientific discovery, very large scientific instruments are being built. You have read on the blog about DUSEL, and are familiar with LHC. These instruments – particle accelerators, synchrotron light sources, large supercomputers, and radio telescope farms are generating massive amounts of data and involve large collaborations of scientists to extract useful research results from it. The Office of Science is looking to ESnet to build and operate a network infrastructure that can scale up to meet the highly demanding performance needs of scientific applications. The Advanced Networking Initiative (ANI) to build the nationwide 100G prototype network and a research testbed is a great start. If you are interested in being part of this exciting initiative, do bid on the 100G Transport RFP.

As a community, we need to keep advancing the state of networking to meet the oncoming age of the digital data deluge (D³).

To wit, the recent IEEE 802.3ba press release: – http://standards.ieee.org/announcements/2010/ratification8023ba.html Note the quote from our own Steve Cotter:

Steve Cotter, Department Head, ESnet at Lawrence Berkeley National Laboratory
“As the science community looks at collaboratively solving hard research problems to positively impact the lives of billions of people, for example research on global climate change, alternative energy and energy efficiency, as well as projects including the Large Hadron Collider that probe the fundamental nature of our universe – leveraging petascale data and information exchange is essential. To accomplish this, high-bandwidth networking is necessary for distributed exascale computing. Lawrence Berkeley National Laboratory is excited to leverage this standard to build a 100G nationwide prototype network as part of ESnet’s participation in the DOE Office of Science Advanced Networking Initiative.”

Got a networking idea you want to test? ANI testbed opening for business

~ ESnetComms

Want try out some new ideas in network research? ESnet invites you to submit a proposal to run experiments on its reconfigurable testbed.  ESnet’s ARRA-funded Advanced Networking Initiative testbed is a high-performance environment where researchers will have the opportunity to prototype, test, and validate cutting edge networking concepts.

Instructions for submitting proposals can be found here https://sites.google.com/a/lbl.gov/ani-testbed/. Proposals are due October 1, 2010. Decisions will be made January 10, 2011 when the Phase 1 version of the testbed is up and running. The phase I version is a set of 10 Gbps connected layer 1, 2, and 3 equipment that will be deployed on a dark fiber ring we  acquired in Long Island (LIMAN: Long Island Metropolitan Area Network). This will mainly be of interest to researchers doing experiments at layers 1-3, or middleware/application research at 10 Gbps.

The testbed will support research including multi-layer multi-domain hybrid networks, network protocols, component testing for future capabilities, protection and recovery,  automatic classification of large bulk data flows, high-throughput middleware and applications, and any other innovative ideas you may want to try out in a realistic network environment, but with no risk of breaking anything.

Try us. We’re open to suggestions.