At 1 p.m. at Berkeley Lab booth 2448, catch ESnet’s Inder Monga’s round-table discussion on OSCARS virtual circuits. OSCARS, the acronym for On- demand Secure Circuits and Advance Reservation System, allows users to reserve guaranteed bandwidth. Many of the demos at SC10 are being carried by OSCARS virtual circuits which were developed by ESnet with DOE support. Good things to come: ESnet anticipates the rollout of OSCARS 0.6 in early 2011. Version 0.6 will offer greatly expanded capabilities and versatility, such as a modular architecture enabling easy plug and play of the various functional modules and a flexible path computation engine (PCE) workflow architecture.
Then, stick around, because next at 2 p.m. Brian Tierney from ESnet will lead a roundtable on the research being produced from the ARRA-funded Advanced Networking Initiative (ANI) testbed.
In 2009, the DOE Office of Science awarded ESnet $62 million in recovery funds to establish ANI, a next generation 100Gbps network connecting DOE’s largest unclassified supercomputers, as well as a reconfigurable network testbed for researchers to test new networking concepts and protocols.
Brian will discuss progress on the 100Gbps network, update you on the several research projects already underway on the testbed, discuss testbed capabilities and how to get access to the testbed. He will also answer your questions on how to submit proposals for the next round of testbed network research.
In the meantime, some celeb-spotting at the LBNL booth at SC10.
ESnet fully supports the drive for energy efficiency to reduce the amount of emissions caused by information and communication technologies (ICT). IEEE just announced that Energy-Efficient Ethernet (EEE) or IEEE P803.3az is the new standard enabling copper interfaces to reduce energy use when the network link is idle . Energy saving mechanisms of EEE can be applied in systems beyond the Ethernet physical interface, e.g. the PCI Express bus. New hardware is required to benefit from EEE, however, so its full impact won’t be realized for a few years. ESnet is in the middle of the Advanced Network Initiative to deploy a cross-country 100G network and we would like to explore end-to-end power saving possibilities including 40G and 100G Ethernet interfaces…Here’s why:
In 2006 articles began to appear discussing the ever-increasing consumption of energy by ICT as well as how data center giants such as Google and Microsoft were locating new data centers based on the availability and cost of energy. Meanwhile, the IEEE was attempting to create a specification to reduce network energy usage, and four years later, ratified the P802.3az or Energy-Efficient Ethernet (EEE).
Earlier this year, the ITU World Summit for an Information Society reported that electricity demand by the ICT sector in industrialized countries is between 5 percent and 10 percent of total demand. But about half the electricity used is wasted by powered on equipment that is idle. So while completion of this project seems timely, the question remains how “triple-e” will impact energy use for Ethernet consumers. EEE defines a protocol to reduce energy usage during periods of low utilization for copper and backplane interfaces up to 10Gb/s. It also reuses a couple of other IEEE protocols to allow uninterrupted communication between link partners. While this combination of protocols can save energy, it is uncertain how much time the typical Ethernet link operates at low utilization, especially when the P802.3ba, or 40G and 100G Ethernet standard was just ratified in June, suggesting relief for pent up demand for bandwidth.
So why isn’t there an energy-efficient version of the higher-speed version of Ethernet?
The answer depends on the type of Ethernet interface and its purpose in the network, as an interface in a home desktop computer will likely be idle much longer than an uplink interface in a data center switch. A key feature of this new standard is called Low Power Idle. As the name suggests, during idle time the non-critical components of the interface go to sleep. The link partner is activated by a wake up signal allowing the receiver time to prepare for an incoming frame.
Consider the utilization plot shown below:
File Server Bandwidth Utilization Profile
Not all links are the same
This window on a file server in an enterprise network shows plenty of idle periods. While there are several peaks over 500 Mb/s, the server is mostly idle, with average utilization under one percent. On the other hand, there are many examples of highly utilized links as well (just look at some of ESnet’s utilization plots). In those cases, less energy is saved, but the energy is being used to do something useful, like transfer information.
But when considering the number of triple-speed copper Ethernet interfaces deployed, energy savings start to add up. The P802.3az Task Force members estimated power savings in US alone can reach 5 Terawatt-hours per year, or enough energy to power 6 million 100W light bulbs. This translates into a reduction of the ICT carbon footprint by roughly 5 million tons per year.
Since EEE is built into the physical interface, new hardware will be required to take advantage of this feature and it will take a few years to reach 100% market saturation.
Getting back to the question about energy efficiency for 40G and 100G Ethernet, there are a few reasons why LPI was not specified for P802.3ba. This project overlapped with P802.3az so it is difficult to specify an energy-efficient method for the new speeds, given the record size of the project and the lack of P802.3az resources for work on optical interfaces. This leads to another question: Should there be an energy-efficient version of 40G and 100G Ethernet? Or should there be an energy-efficient version of optical and P802.3ba interfaces?
To decide the scope of the project P802.3az we examined the magnitude of power consumed and number of interfaces in the market. The power consumed for a 1000BASE-T interface is less than that used by a10GBASE-T interface, but there are orders of magnitudes more of the former. On the other hand, early in the project not many 10GBASE-T interfaces existed in the market, but the interfaces consumed power on the order of 10W-15W per interface. These numbers are reduced by each new improvement in process technology, but they are still significant.
Considering first generation 100G transceivers can consume more than 20W each and the millions of optical Ethernet interfaces in the market, further standards development is worth pursuing.
Mike Bennett is a senior network engineer for LBLnet and chair of P802.3az. He can be reached at MJBennett@lbl.gov
While we have been busy working towards a 100G ANI prototype wide area network (WAN), researchers at Intel are making sure that we have plenty to do in the future. Yesterday’s Wall Street Journal article (http://on.wsj.com/dcf5ko) on Intel demonstrating 50Gbps communication between chips with silicon-based lasers, is just the tip of the iceberg of competitive research looming in the arena of photon-electron integration.
50G Silicon Photonics Link (image from Intel white paper)
This demonstration from Intel (Kudos to them!) is a great reminder of how such innovations can revolutionize the computing model by making it easier to move large amounts of data between the chips on a motherboard or between thousands of multi-core processors, leading the way towards exascale computing. Just imagine the multi-terabit fire hose of capacity ESnet would have to turn on to keep those chips satisfied! This seamless transition from electronics to photonics without dependence on expensive sets of photonic components has the potential to transform the entire computing industry and give an additional boost to the “Cloud” industry. Thomas J. Watson has been credited with saying “The world needs only five computers”. We look to be collecting the innovations to just prove him right one day.
While we do get excited about the fantastic future of silicon integration, I would like to point out the PIC (Photonic Integrated Chip) has been a great innovation by a company, Infinera, just down the Silicon Valley – they are actually mass-producing integrated lasers on a chip for a different application – long distance communication, by using a substrate material different than silicon. This technology is for real. You can get to play with the Infinera’s in our ANI testbed – you just need to come up with a cool research problem and write a proposal by October 1st, 2010.
ESnet is pleased to announce that UC Davis Professor S.J. Ben Yoo has been granted a joint faculty appointment with Berkeley Lab, formalizing a long-term relationship. Yoo will be collaborating on research projects with ESnet to develop Terabit optical networks of the future to meet the upcoming data challenges triggered by Exascale thinking within the DOE. It is an interesting research challenge, including architecture studies, software developments and networking experiments on ESnet’s ANI testbed. Yoo will also be collaborating with LBNL researchers at NERSC for applications of optical networking within high-end data centers.
“Ben is the type of highly credentialed network research scientist that we hope will take full advantage of the testbed infrastructure we are making available to the community.” said Steve Cotter, head of ESnet.
In a talk this week at Joint Techs http://bit.ly/cAtNt4, Yoo discussed the potential of next generation all-optical Label Switching (OLS) networking, a technology he invented. OLS can seamlessly integrate packet, flow, and circuit traffic. OLS has the potential to fit well within the industry standard MPLS and GMPLS architectures, and recent experimental results show very good characteristics like extremely low latency (<100 ns) and scalability beyond 40 petabit/sec capacity. It has experimentally demonstrated a per-channel line rate of 100 Gb/s ~ 1.2 Tb/s. A centralized management station can leverage OLS to rapidly assess data flows based on real time collections of labels that contain statistical information about the data traffic.
Yoo has done extensive research with the ATD-Monet testbed in the Washington DC area, telecommunications standardization services at Bellcore, and testbed work at the Sprint Advanced Technology Laboratory. You can get a better sense of his work and research here.
We look forward to working with him on our ANI testbed as well. Yoo’s intention is to push the testbed to its limits. Should be a wild ride.
You must be logged in to post a comment.