Author: Inder Monga

ESnet 2010 Round-up: Part 2

~ Inder Monga

Our take on ANI, OSCARS, perfSONAR, and the state of things to come.

ANI Testbed

In 2010 ESnet led the technology curve in the testbed by putting together a great multi-layer design, deploying specially tuned 10G IO Testers, became early investors in the Openflow protocol by deploying the NEC switches, and built a research breadboard of end-hosts leveraging open-source virtualization and cloud technologies.

The first phase of the ANI testbed is concluding. After 6+ months of operational life, with exciting research projects like ARCHSTONE, Flowbench, HNTES, climate studies, and more leveraging the facilities, we are preparing to move the testbed to its second phase on the dark fiber ring in Long Island. Our call for proposals that closed October 1st garnered excellent ideas from researchers and was reviewed by the academic and industry stalwarts in the panel. We are tying up loose ends as we light the next phase of testbed research.

OSCARS

This year the OSCARS team has been extremely productive. We added enhancements to create the next version (0.5.3) of currently production OSCARS software, progressed on architecting and developing a highly modular and flexible platform for the next-generation OSCARS (0.6), a PCE-SDK targeted towards network researchers focused on creating complex algorithms for path computation, and developing FENIUS to support the GLIF Automated GOLE demonstrator.

Not only did the ESnet team multitask on various ANI, operational network and OSCARS deliverables, it also spent significant time supporting our R&E partners like Internet2, SURFnet, NORDUnet, RNP and others interested in investigating the capabilities of this open-source software. We also appreciate Internet2’s participation by dedicating testing resources for OSCARS 0.6 starting next year to ensure a thoroughly vetted and stable platform during the April timeframe. This is just one example of the accomplishments possible for the R&E community by commiting to partnership and collaboration.

perfSONAR collaboration

perfSONAR kept up its rapid pace of feature additions and new releases in joint collaboration with Internet2 and others. In addition to rapid progress in software capabilities, ESnet is aggressively rolling out perfSONAR nodes in its 10G and 1G POPs, creating an infrastructure where the network can be tuned to hum. With multiple thorny network problems now solved, perfSONAR has proven to be great tool delivering value. This year we focused on making perfSONAR easily deployable and adding the operational features to transform it into a production service. An excellent workshop in August succinctly captured the challenges and opportunities to leverage perfSONAR for operational troubleshooting and also by researchers in understanding further how to improve networks. Joint research projects continue to stimulate further development with a focus on solving end-to-end performance issues.

The next networking challenge?

2011

Life in technology tends to be interesting, even though people keep warning about the commoditization of networking gear. The focus area for innovation just shifts, but never goes away.  Some areas of interest as we evaluate our longer term objectives next year:

  • Enabling the end-to-end world: What new enhancements or innovations are needed to deploy performance measurement, and control techniques to enable a seamless end-to-end application performance?
  • Life in a Terabit digital world: What network innovations are needed to fully exploit the requirement for Terabit connectivity between supercomputer centers in the 2015-2018 timeframe?
  • Life in a carbon economy: What are the low-hanging fruit for networks to become more energy-efficient and/or enable energy-efficiency in the IT ecosystem they play? Cloud-y or Clear?

We welcome your comments and contributions,

Happy New Year

Inder Monga and the folks at ESnet

ESnet 2010 Round-up: Part 1

~ Inder Monga

Part 1: Considering the state of 100G and the state we’re in

The past year slipped by at a dizzying pace for us at ESnet, as we made new forays into cutting-edge technologies. In this two-part blogpost, we will recap accomplishments of the year, but also consider the challenges facing us in the one to come as we progress towards delivering the Advanced Networking Initiative.

Stimulating 100G

One of our prime directives with ANI funding was to stimulate the 100G market towards increasing spectral efficiency. In the last year, we have had wonderful engagement with the vendors that are moving products in this direction. Coherent receivers and DP-QPSK modulation are now standard fare for the 40G/100G solutions. At the latest conference, IEEE ANTS, in Mumbai last week, the 100G question was considered solved. Researchers are now exploring innovative solutions to deliver a next generation of 100G with higher power efficiency, or jump to the next level in delivering 400G. One researcher at the Indian Institute of Technology, Mumbai, is looking at revolutionizing the power consumption curve of the digital processing paradigm of coherent solutions by investigating analog processing techniques (super secret, so we will just have to wait and see).

IEEE-ANTS Conference

A representative from OFS, the optical fiber company, described research on new fibers which cater to the coherent world that will enable better performance. He quoted hero experiments, papers, and research presented at this years’ OFC, touting the advantages of new fiber captured through the joint work of Alcatel-Lucent Bell Labs and OFS (ex-Lucent) research collaborators. There is a lot of fiber still being laid out in the developing countries and they are well positioned to take advantage of this new research to bring cheaper broadband connectivity in so far underserved communities.

Some selected points raised at the panel regarding 400G and beyond:

  • Raman amplification is coming back in vogue
  • 50GHz ITU-Grid needs to evolve to flexi-grid technology. With flexi-grid, some of the basic modem concepts of negotiation (remember the auto-sensing modems of late 90’s) is back – where based on distance, and loss, the appropriate grid spacing can be negotiated for each wavelength.
  • If the industry sticks with electrical compensation, optical equipment will see increased electricity consumption by the power-hungry Analog-Digital Conversion (ADC) and Digital Signal Processing (DSP) ASICS. With advances in CMOS, the status quo might not suffice in a few years, especially since the whole industry is out there sticking the router vendors with a big “power-hungry” sticker. The equations in power-consumption tradeoffs still need to be studied and appropriate comparisons made. I hope the vendors also develop a perspective in that direction.
  • Comcast, the only other vendor on the panel, mentioned current capacities of 30x40G (coherent) on some links of their backbone and their eagerness to deploy 100G solutions. They are WAY ahead in deploying 100G, though the industry seems to not broadcast such news widely.
  • Comcast felt that coherent optics in the Metro Area is overkill and entreated the vendors not to build one-size-fits-all solutions even if simpler (and, they hope making 100G more affordable, as well).
  • Speeding into next year

    There was little discussion on the 100GE standards, although there was a clear message that LR-10 is here to stay, mainly supported by Data Center customers, though almost all traditional carriers intend deploy LR-4, in the case it starts costing less than a Ferrari.

At Supercomputing 2010, the SCinet community orchestrated and deployed 100G-capable equipment from Brocade, Cisco, Ciena and Juniper, to name a few vendors, and included 100G host demonstrations of data transfers by NASA. It was encouraging to see the Academic and R&E community lead deployment and testing of 100G [See a sample poster below].

100G Testing by German Academia

The SCinet community lives on the “bleeding edge” and supported a demonstration by Internet2, ESnet, and other partners carrying live 100Gbps application data over a 100G wave from Chicago to New Orleans show floor.

We are looking forward to Seattle (SC11) and can already predict multiple 100G’s of bandwidth coming towards the show floor – if you have any cool demonstrations that you would like to collaborate with us, please drop us a note.

— Inder Monga

ESnet gives CISCO Nerd Lunch talk, learns televangelism is harder than it seems

~ Inder Monga

As science transitions from lab-oriented to a distributed computational and data-intensive activity, the research and education (R&E) networking community is tracking the growing data needs of scientists. Huge instruments like the Large Hadron Collider are being planned and built. These projects require global-scale collaborations and contributions from thousands of scientists, and as the data deluge from the instruments grows, even more scientists are interested in analyzing it for the next breakthrough discovery. Suffice it to say that even though worldwide video consumption on the Internet is driving a similar increase in commercial bandwidth, the scale, characteristics, and requirements of scientific data traffic is quite different.

And this is why ESnet got invited to Cisco Systems’ headquarters last week to talk about how we how we handle data as part of their regular Nerd Lunch talk series. What I found interesting although not surprising, was that with Cisco being a big evangelist of telepresence, more employees attended the talk from their desks than in person.  This was a first for me and I came away with a new appreciation for the challenges of collaborating across distances.

From a speaker’s perspective, the lesson learnt by me was to brush up my acting skills. My usual preparations are to rehearse the difficult transitions and  focus on remembering the few important points to make on every slide. When presenting, that slide presentation portion of my brain goes on auto-pilot, while my focus turns towards evaluating the impact on the audience. When speaking at a podium one can observe when someone in the audience opens a notebook to jot down a thought, when their attention drifts to email on the laptop in front of them, or when a puzzled look appears on the face of someone as they try to figure out the impact of the point I’m trying to make. But these visual cues go missing with a largely webcast audience, making it harder to know when to stop driving home a point or when to explain the point further to the audience.  In the future, I’ll have to be better at keeping the talk interesting without the usual clues from my audience.

Maybe the next innovation in virtual-reality telepresence is just waiting to happen?

Notwithstanding the challenges of presenting to a remote audience, enabling remote collaboration is extremely important to ESnet. Audio, video and web collaboration is a key service offered by us to the DOE labs. ESnet employees use video extensively in our day-to-day operations. The “ESnet watercooler”, a 24×7 open video bridge, is used internally by our distributed workforce to discuss technical issues, as well as, to have ad-hoc meetings on topics of interest. As science goes increasingly global, scientists are also using this important ESnet service for their collaborations.

With my brief stint in front of a stage now over, it is back to ESnet and then on to the 100G invited panel/talk at IEEE ANTS conference in Mumbai. Wishing all of you a very Happy New Year!

Inder Monga

Scaling up – when computing meets optical transport

~ Inder Monga

While we have been busy working towards a 100G ANI prototype wide area network (WAN), researchers at Intel are making sure that we have plenty to do in the future. Yesterday’s Wall Street Journal article (http://on.wsj.com/dcf5ko) on Intel demonstrating 50Gbps communication between chips with silicon-based lasers, is just the tip of the iceberg of competitive research looming in the arena of photon-electron integration.

50G Silicon Photonics Link (image from Intel white paper)

This demonstration from Intel (Kudos to them!) is a great reminder of how such innovations can revolutionize the computing model by making it easier to move large amounts of data between the chips on a motherboard or between thousands of multi-core processors, leading the way towards exascale computing.  Just imagine the multi-terabit fire hose of capacity ESnet would have to turn on to keep those chips satisfied! This seamless transition from electronics to photonics without dependence on expensive sets of photonic components has the potential to transform the entire computing industry and give an additional boost to the “Cloud” industry. Thomas J. Watson has been credited with saying “The world needs only five computers”. We look to be collecting the innovations to just prove him right one day.

While we do get excited about the fantastic future of silicon integration, I would like to point out the PIC (Photonic Integrated Chip) has been a great innovation by a company, Infinera, just down the Silicon Valley – they are actually mass-producing integrated lasers on a chip for a different application – long distance communication, by using a substrate material different than silicon. This technology is for real. You can get to play with the Infinera’s in our ANI testbed – you just need to come up with a cool research problem and write a proposal by October 1st, 2010.

Fire away!

—-

August 4th, 2010

Computing at the Speed of Light – Read MIT Technology Review’s take on the same topic.

New 100GE Ethernet Standard IEEE 802.3ba (and 40GE as well)

~ Inder Monga
From Charles Spurgeon's Ethernet Website


History is being written: from a simple diagram published in 1976 by Dr. Robert Metcalfe, with a data rate of 3 Mpbs, Ethernet surely has come a long way in the last 30 years. Coincidentally, the parent of ESnet, MFEnet, was also launched around the same time as a result of the new Fusion Energy supercomputer center at Lawrence Livermore National Labs (LLNL) http://www.es.net/hypertext/esnet-history.html. It is remarkable to note that right now, as the 100GE standard got ratified, ESnet engineers are very much on the ball, busy putting 100GE enabled routers through the paces within our labs.

For ESnet and the Department of Energy – it is all about the science. To enable large-scale scientific discovery, very large scientific instruments are being built. You have read on the blog about DUSEL, and are familiar with LHC. These instruments – particle accelerators, synchrotron light sources, large supercomputers, and radio telescope farms are generating massive amounts of data and involve large collaborations of scientists to extract useful research results from it. The Office of Science is looking to ESnet to build and operate a network infrastructure that can scale up to meet the highly demanding performance needs of scientific applications. The Advanced Networking Initiative (ANI) to build the nationwide 100G prototype network and a research testbed is a great start. If you are interested in being part of this exciting initiative, do bid on the 100G Transport RFP.

As a community, we need to keep advancing the state of networking to meet the oncoming age of the digital data deluge (D³).

To wit, the recent IEEE 802.3ba press release: – http://standards.ieee.org/announcements/2010/ratification8023ba.html Note the quote from our own Steve Cotter:

Steve Cotter, Department Head, ESnet at Lawrence Berkeley National Laboratory
“As the science community looks at collaboratively solving hard research problems to positively impact the lives of billions of people, for example research on global climate change, alternative energy and energy efficiency, as well as projects including the Large Hadron Collider that probe the fundamental nature of our universe – leveraging petascale data and information exchange is essential. To accomplish this, high-bandwidth networking is necessary for distributed exascale computing. Lawrence Berkeley National Laboratory is excited to leverage this standard to build a 100G nationwide prototype network as part of ESnet’s participation in the DOE Office of Science Advanced Networking Initiative.”

The collaboration portal across continents

~ Inder Monga

When organizations are driven by the similar goal of excellence, it is amazing what can be accomplished in two days of non-stop discussions. As I try to consolidate my thoughts from meetings the past two days, the following quotation rings particularly true.

Coming together is a beginning, staying together is progress, and working together is success. ~ Henry Ford

There is still a lot of work to be done, milestones to be hit and unforeseen roadblocks to overcome before we give ourselves a pat on the back, but we are off to a good start.

For the perpetually curious, SURFnet, ESnet and NORDUnet stated their intentions to collaborate on furthering innovation in network research by working on open-source middleware to reserve and seamlessly allocate bandwidth across multiple domains  (http://www.lbl.gov/cs/Archive/news030910.html).  While our first planned meeting was delayed by volcano ash from Iceland, skies and schedules cleared enough for the Dutch to visit us at our facilities at Berkeley lab.  This week’s two day meeting of minds is a baby step taken towards an ambitious goal of open sharing of tools, knowledge, open-source software and other network-related research with the community.

Taking a Break

We did let our visitors out of the conference room for just few minutes to enjoy the beautiful, albeit faint, backdrop of the Golden Gate Bridge.

100GE around the bend?

~ Inder Monga

Ever feel the exhilaration of sitting in a race car and going around the track at super high speeds? We came close to that experience when we recently received early editions of a vendor’s 100GE cards for their routers. The experience so far has been phenomenal – no issues in getting the card up and running, the optics work great and packets are getting forwarded at line-rate. We are putting those cards through our rigorous testing process, though our lips are sealed for now.

For the industry, this is significant progress – just last year we started the Advanced Networking Initiative (ANI) project and the prospects of actually seeing a 100GE interface in a router this soon seemed far off. So kudos to the vendor (you know who you are) and to the IEEE 802.3ba 40/100Gbps task force – if you are listening, this stuff is ready to go!

Listening to the drumbeats of 100G

~ Inder Monga

Since we received the news of our ARRA funds for our Advanced Network Initiative (ANI), we have been working steadily towards the ambitious goal of deploying 100G technology to stimulate networking advancement in high-speed interfaces. In the course of pushing the ANI agenda over the last year we have met with many carriers and vendors. Although I cannot share my personal conversations with these vendors–the thought of flocks of lawyers descending upon me ensures reticence–I have avidly been tracking their public announcements.

Just today Cisco announced the acquisition of Core Optics, a coherent optics board vendor. I had the good fortune to see their 40G system working at OFC this year and I am sure they are working hard on getting their 100G system up and running. Google typically has been quiet about the innovations in their network to keep up with data center innovations. But they have been uncharacteristically beating the 100G drum in public – which meshes very well with our needs. If you look at ESnet, the traffic transiting our network is growing at an alarming rate of 80% year over year.



At the Packet-Optical Transport Evolution conference (http://www.lightreading.com/live/event_information.asp?event_id=29209) Bikash Koley, Senior Network Architect, Google points at machine to machine traffic (like video sensors) as the motivators for needing such bandwidth and cites hybrid networking or packet-optical integration as solving the problems of the day.

If I can quote their article in Lightreading (http://www.lightreading.com/document.asp?doc_id=192230&):

“Regardless of how the network looks, Google is dead set on one thing: it wants label-switched routing and DWDM capabilities to be combined into one box. It doesn’t matter if that’s a label-switched router (LSR) with DWDM added, or a DWDM box with Layer 3 knowledge added,”Koley said. (He also stressed that the LSR doesn’t have to be a full-blown router.)

Now that is one statement we are in agreement with Bikash Koley and Google. Our own experience developing OSCARS (On-demand Secure Circuits and Advanced Reservation System – www.es.net/oscars) and the hybrid networking architecture to deal with large science flows since 2004 has led us down the path of on-demand traffic-engineered paths. With MPLS being the only choice at that time (discussing the merits of new IEEE mac-in-mac protocols will require a separate blog), we created the OSCARS open-source software to dynamically configure LSPs through our hybrid packet-optical network. That worked very well for us, though it was clear that we did not really need the service/edge capabilities built into the router. So if there is a way to make the core simpler, cheaper and more energy-efficient – sign me up for that and we will run OSCARS over it to steer those circuits to where our customers want it.

We at ESnet continue to march ahead towards the 100G-prototype network. I look forward to your comments on 40G, 100G, the new Ethernet standards and the way to higher rates (400GE, 1TBE.)

Inder Monga, Network Architect

Email me at: Imonga@es.net