During the SC10 conference, NASA, NOAA, ESnet, the Dutch Research Consortium, US LHCNet and CANARIE announced that they would transmit 100Gbps of scientific data between Chicago and New Orleans. Through the use of 14 10GigE interconnects, researchers attempted to completely utilize the full 100 Gbps worth of bandwidth by producing up to twelve 8.5-to-10Gbps individual data flows.
Brian Tierney reports: “We are very excited that a team from NASA Goddard completely filled the 100G connection from the show floor to Chicago. It is certainly the first time for the supercomputing conference that a single wavelength over the WAN achieved 100Gbps. The other thing that is so exciting about it that they used a single sending host to do it.”
“Was this just voodoo?” asked NERSC’s Brent Draney.
Tierney assures us that indeed it must have been… but whatever they did, it certainly works.
At 1 p.m. at Berkeley Lab booth 2448, catch ESnet’s Inder Monga’s round-table discussion on OSCARS virtual circuits. OSCARS, the acronym for On- demand Secure Circuits and Advance Reservation System, allows users to reserve guaranteed bandwidth. Many of the demos at SC10 are being carried by OSCARS virtual circuits which were developed by ESnet with DOE support. Good things to come: ESnet anticipates the rollout of OSCARS 0.6 in early 2011. Version 0.6 will offer greatly expanded capabilities and versatility, such as a modular architecture enabling easy plug and play of the various functional modules and a flexible path computation engine (PCE) workflow architecture.
Then, stick around, because next at 2 p.m. Brian Tierney from ESnet will lead a roundtable on the research being produced from the ARRA-funded Advanced Networking Initiative (ANI) testbed.
In 2009, the DOE Office of Science awarded ESnet $62 million in recovery funds to establish ANI, a next generation 100Gbps network connecting DOE’s largest unclassified supercomputers, as well as a reconfigurable network testbed for researchers to test new networking concepts and protocols.
Brian will discuss progress on the 100Gbps network, update you on the several research projects already underway on the testbed, discuss testbed capabilities and how to get access to the testbed. He will also answer your questions on how to submit proposals for the next round of testbed network research.
In the meantime, some celeb-spotting at the LBNL booth at SC10.
ESnet fully supports the drive for energy efficiency to reduce the amount of emissions caused by information and communication technologies (ICT). IEEE just announced that Energy-Efficient Ethernet (EEE) or IEEE P803.3az is the new standard enabling copper interfaces to reduce energy use when the network link is idle . Energy saving mechanisms of EEE can be applied in systems beyond the Ethernet physical interface, e.g. the PCI Express bus. New hardware is required to benefit from EEE, however, so its full impact won’t be realized for a few years. ESnet is in the middle of the Advanced Network Initiative to deploy a cross-country 100G network and we would like to explore end-to-end power saving possibilities including 40G and 100G Ethernet interfaces…Here’s why:
In 2006 articles began to appear discussing the ever-increasing consumption of energy by ICT as well as how data center giants such as Google and Microsoft were locating new data centers based on the availability and cost of energy. Meanwhile, the IEEE was attempting to create a specification to reduce network energy usage, and four years later, ratified the P802.3az or Energy-Efficient Ethernet (EEE).
Earlier this year, the ITU World Summit for an Information Society reported that electricity demand by the ICT sector in industrialized countries is between 5 percent and 10 percent of total demand. But about half the electricity used is wasted by powered on equipment that is idle. So while completion of this project seems timely, the question remains how “triple-e” will impact energy use for Ethernet consumers. EEE defines a protocol to reduce energy usage during periods of low utilization for copper and backplane interfaces up to 10Gb/s. It also reuses a couple of other IEEE protocols to allow uninterrupted communication between link partners. While this combination of protocols can save energy, it is uncertain how much time the typical Ethernet link operates at low utilization, especially when the P802.3ba, or 40G and 100G Ethernet standard was just ratified in June, suggesting relief for pent up demand for bandwidth.
So why isn’t there an energy-efficient version of the higher-speed version of Ethernet?
The answer depends on the type of Ethernet interface and its purpose in the network, as an interface in a home desktop computer will likely be idle much longer than an uplink interface in a data center switch. A key feature of this new standard is called Low Power Idle. As the name suggests, during idle time the non-critical components of the interface go to sleep. The link partner is activated by a wake up signal allowing the receiver time to prepare for an incoming frame.
Consider the utilization plot shown below:
Not all links are the same
This window on a file server in an enterprise network shows plenty of idle periods. While there are several peaks over 500 Mb/s, the server is mostly idle, with average utilization under one percent. On the other hand, there are many examples of highly utilized links as well (just look at some of ESnet’s utilization plots). In those cases, less energy is saved, but the energy is being used to do something useful, like transfer information.
But when considering the number of triple-speed copper Ethernet interfaces deployed, energy savings start to add up. The P802.3az Task Force members estimated power savings in US alone can reach 5 Terawatt-hours per year, or enough energy to power 6 million 100W light bulbs. This translates into a reduction of the ICT carbon footprint by roughly 5 million tons per year.
Since EEE is built into the physical interface, new hardware will be required to take advantage of this feature and it will take a few years to reach 100% market saturation.
Getting back to the question about energy efficiency for 40G and 100G Ethernet, there are a few reasons why LPI was not specified for P802.3ba. This project overlapped with P802.3az so it is difficult to specify an energy-efficient method for the new speeds, given the record size of the project and the lack of P802.3az resources for work on optical interfaces. This leads to another question: Should there be an energy-efficient version of 40G and 100G Ethernet? Or should there be an energy-efficient version of optical and P802.3ba interfaces?
To decide the scope of the project P802.3az we examined the magnitude of power consumed and number of interfaces in the market. The power consumed for a 1000BASE-T interface is less than that used by a10GBASE-T interface, but there are orders of magnitudes more of the former. On the other hand, early in the project not many 10GBASE-T interfaces existed in the market, but the interfaces consumed power on the order of 10W-15W per interface. These numbers are reduced by each new improvement in process technology, but they are still significant.
Considering first generation 100G transceivers can consume more than 20W each and the millions of optical Ethernet interfaces in the market, further standards development is worth pursuing.
Mike Bennett is a senior network engineer for LBLnet and chair of P802.3az. He can be reached at MJBennett@lbl.gov
ESnet is now soliciting research proposals for its ARRA-funded testbed. It currently provides network researchers with a rapidly reconfigurable high-performance network research environment where reproducible tests can be run. This will eventually evolve into a nationwide 100Gbps testbed, available for use to any researcher whose proposal is accepted.
Researchers can use the testbed to prototype, test, and validate cutting edge networking concepts, for example, projects including:
Path computation algorithms that incorporate information about hybrid layer 1, 2 and 3 paths, and support ‘cut-through’ routing.
New transport protocols for high speed networks
Protection and recovery algorithms
Automatic classification of large bulk data flows
New routing protocols
New network management techniques
Novel packet processing algorithms
High-throughput middleware and applications research
Please look at the descriptionto get a more detailed idea of the current testbed capabilities.
The proposal review panel will discuss and review proposals twice yearly. The first round of proposals is due October 1, 2010, and decisions will be made by Dec 10, 2010. After that the committee will meet approximately every six months to accept additional proposals and review progress of current projects.
Proposals should be sent to: firstname.lastname@example.org
More details on the testbed and the brief proposal process can be found right here.
While we have been busy working towards a 100G ANI prototype wide area network (WAN), researchers at Intel are making sure that we have plenty to do in the future. Yesterday’s Wall Street Journal article (http://on.wsj.com/dcf5ko) on Intel demonstrating 50Gbps communication between chips with silicon-based lasers, is just the tip of the iceberg of competitive research looming in the arena of photon-electron integration.
This demonstration from Intel (Kudos to them!) is a great reminder of how such innovations can revolutionize the computing model by making it easier to move large amounts of data between the chips on a motherboard or between thousands of multi-core processors, leading the way towards exascale computing. Just imagine the multi-terabit fire hose of capacity ESnet would have to turn on to keep those chips satisfied! This seamless transition from electronics to photonics without dependence on expensive sets of photonic components has the potential to transform the entire computing industry and give an additional boost to the “Cloud” industry. Thomas J. Watson has been credited with saying “The world needs only five computers”. We look to be collecting the innovations to just prove him right one day.
While we do get excited about the fantastic future of silicon integration, I would like to point out the PIC (Photonic Integrated Chip) has been a great innovation by a company, Infinera, just down the Silicon Valley – they are actually mass-producing integrated lasers on a chip for a different application – long distance communication, by using a substrate material different than silicon. This technology is for real. You can get to play with the Infinera’s in our ANI testbed – you just need to come up with a cool research problem and write a proposal by October 1st, 2010.
ESnet is pleased to announce that UC Davis Professor S.J. Ben Yoo has been granted a joint faculty appointment with Berkeley Lab, formalizing a long-term relationship. Yoo will be collaborating on research projects with ESnet to develop Terabit optical networks of the future to meet the upcoming data challenges triggered by Exascale thinking within the DOE. It is an interesting research challenge, including architecture studies, software developments and networking experiments on ESnet’s ANI testbed. Yoo will also be collaborating with LBNL researchers at NERSC for applications of optical networking within high-end data centers.
“Ben is the type of highly credentialed network research scientist that we hope will take full advantage of the testbed infrastructure we are making available to the community.” said Steve Cotter, head of ESnet.
In a talk this week at Joint Techs http://bit.ly/cAtNt4, Yoo discussed the potential of next generation all-optical Label Switching (OLS) networking, a technology he invented. OLS can seamlessly integrate packet, flow, and circuit traffic. OLS has the potential to fit well within the industry standard MPLS and GMPLS architectures, and recent experimental results show very good characteristics like extremely low latency (<100 ns) and scalability beyond 40 petabit/sec capacity. It has experimentally demonstrated a per-channel line rate of 100 Gb/s ~ 1.2 Tb/s. A centralized management station can leverage OLS to rapidly assess data flows based on real time collections of labels that contain statistical information about the data traffic.
Yoo has done extensive research with the ATD-Monet testbed in the Washington DC area, telecommunications standardization services at Bellcore, and testbed work at the Sprint Advanced Technology Laboratory. You can get a better sense of his work and research here.
We look forward to working with him on our ANI testbed as well. Yoo’s intention is to push the testbed to its limits. Should be a wild ride.
What sets us apart? ESnet has, and always will focus on anticipating the needs of the extended DOE science community. This shapes our network strategy, from services and architecture to topology and reach. It also distinguishes ESnet from university research & education networks which are driven by the broader needs of the general university population. Vis-à-vis commercial networks, ESnet has specialized in handling the relatively small number of very large flows of large-scale science data rather than the enormous number of relatively small data flows traversing commercial carrier networks today. Our desire to always stay a step ahead of the constantly evolving network needs of the scientific community has driven ESnet to take the bold step of purchasing and lighting our first segment of dark fiber.
Owning the road
By owning a tiny but powerful pair of optical fibers, ESnet will no longer have to rely on the vagaries of the commercial market – we will be able to deliver services when we choose and where they are needed. For example, the DOE envisions using ESnet to link its supercomputing centers with a terabit of capacity by 2015. Our network will be key to enabling the scientific community to accomplish exascale computing by 2020.
Ramping up is no slam-dunk
But providing terabit capacity by using 10 100G waves through commercial services is no slam-dunk and could be very cost-prohibitive. Without owning the fiber and transport infrastructure, the same is likely to be true when near-terabit waves become available around 2020. Not only does one lose spectral efficiency because a terabit wave won’t fit within ITU standard 50 Ghz spacing – it is necessary to plan for non-standard spacing, with current research pointing towards 200 Ghz to accommodate the signal.
But just solving this problem is not enough, as ESnet’s massive bandwidth requirements don’t end with the supercomputers. ESnet must deliver steadily increasing amounts of data generated by the Large Hadron Collider as well as similar data sets shared within the climate, fusion, and genomics communities to scientists around the world.
Lighting the way forward
It is clear to us that the only way to scale the network to meet the rapidly propagating needs of large-scale science is by lighting our own dark fiber. Although this relatively small 200-mile loop linking New York City to Brookhaven National Lab barely registers with most in the networking community, it represents an exciting sea change in ESnet’s approach in serving our customers.
History is being written: from a simple diagram published in 1976 by Dr. Robert Metcalfe, with a data rate of 3 Mpbs, Ethernet surely has come a long way in the last 30 years. Coincidentally, the parent of ESnet, MFEnet, was also launched around the same time as a result of the new Fusion Energy supercomputer center at Lawrence Livermore National Labs (LLNL) http://www.es.net/hypertext/esnet-history.html. It is remarkable to note that right now, as the 100GE standard got ratified, ESnet engineers are very much on the ball, busy putting 100GE enabled routers through the paces within our labs.
For ESnet and the Department of Energy – it is all about the science. To enable large-scale scientific discovery, very large scientific instruments are being built. You have read on the blog about DUSEL, and are familiar with LHC. These instruments – particle accelerators, synchrotron light sources, large supercomputers, and radio telescope farms are generating massive amounts of data and involve large collaborations of scientists to extract useful research results from it. The Office of Science is looking to ESnet to build and operate a network infrastructure that can scale up to meet the highly demanding performance needs of scientific applications. The Advanced Networking Initiative (ANI) to build the nationwide 100G prototype network and a research testbed is a great start. If you are interested in being part of this exciting initiative, do bid on the 100G Transport RFP.
As a community, we need to keep advancing the state of networking to meet the oncoming age of the digital data deluge (D³).
Steve Cotter, Department Head, ESnet at Lawrence Berkeley National Laboratory “As the science community looks at collaboratively solving hard research problems to positively impact the lives of billions of people, for example research on global climate change, alternative energy and energy efficiency, as well as projects including the Large Hadron Collider that probe the fundamental nature of our universe – leveraging petascale data and information exchange is essential. To accomplish this, high-bandwidth networking is necessary for distributed exascale computing. Lawrence Berkeley National Laboratory is excited to leverage this standard to build a 100G nationwide prototype network as part of ESnet’s participation in the DOE Office of Science Advanced Networking Initiative.”
Want try out some new ideas in network research? ESnet invites you to submit a proposal to run experiments on its reconfigurable testbed. ESnet’s ARRA-funded Advanced Networking Initiative testbed is a high-performance environment where researchers will have the opportunity to prototype, test, and validate cutting edge networking concepts.
Instructions for submitting proposals can be found here https://sites.google.com/a/lbl.gov/ani-testbed/. Proposals are due October 1, 2010. Decisions will be made January 10, 2011 when the Phase 1 version of the testbed is up and running. The phase I version is a set of 10 Gbps connected layer 1, 2, and 3 equipment that will be deployed on a dark fiber ring we acquired in Long Island (LIMAN: Long Island Metropolitan Area Network). This will mainly be of interest to researchers doing experiments at layers 1-3, or middleware/application research at 10 Gbps.
The testbed will support research including multi-layer multi-domain hybrid networks, network protocols, component testing for future capabilities, protection and recovery, automatic classification of large bulk data flows, high-throughput middleware and applications, and any other innovative ideas you may want to try out in a realistic network environment, but with no risk of breaking anything.
All those cheers and whoops from the top of the Berkeley Hill would be us. ESnet just nailed another ANI milestone. We got out our RFP to vendors for the next stage in the nationwide 100Gbps prototype network.
The network will deliver data at scorching speeds and link three of the Department of Energy’s major supercomputing centers— NERSC at Berkeley Lab, the Argonne Leadership Computing Facility at Argonne National Laboratory in Illinois and the Oak Ridge Leadership Computing Facility at Oak Ridge National Laboratory in Tennessee— and MANLAN, the international exchange point in New York.
Your ARRA stimulus money is busy, this time building the infrastructure to help scientists communicate and deal with all that data proliferation from places like the Large Hadron Collider. We handle petabytes of data a month with our usual aplomb; but terabit networking is not far in our future. Good to get prepared.