Research and Development

Learn how ESnet and its partners are investigating and testing emerging services, protocols, routing techniques, and tools necessary to meet the expanding needs of science.

5G For Science: How Research Will Benefit from Advanced Wireless?

~ ESnetComms

5G Andrew

5G is the next-generation wireless network that will give you much faster Internet connections. That means massive files, like high-definition movies, that take you about six minutes to download over a 4G LTE network, could be downloaded in a matter of seconds over the 5G network. And because of its innovative design, 5G is about to change the way things like cars, TVs, and even buildings connect to the Internet.

The Department of Energy’s national labs, sponsored by the Office of Science, are currently working to identify opportunities on how science can leverage 5G and other advanced wireless technologies. The Office of Science recently published a report on its findings.     

ESnet Computer Systems Engineer Andrew Wiedlea helped facilitate discussions and report findings. We recently caught up with him to talk about the benefits of 5G and other advanced wireless technologies for science, and what it will take to make it available for research. 

What is 5G and Advanced Wireless? And, how could science benefit from it?

Floating Robots
Berkeley Lab and UC Berkeley researchers unleashed 100 floating sensors to understand how water flows through the Sacramento-San Joaquin Delta on its way to pumping stations and San Francisco Bay.  The sensors transmitted data to the National Energy Research Scientific Computing Center for assimilation and analysis. (Photo by Roy Kaltschmidt).

Scientific data movement is on the cusp of a new era for flexible, low-cost deployment of scientific sensors and data mobility. Advanced wireless capabilities offer the promise of solving the “last mile problem” for science, which is creating new ways for scientists to connect data from sensors, vehicles, and isolated locations, with U.S. Department of Energy’s world-class supercomputers. It’s important to note that advanced wireless will not replace high speed scientific optical networks for large-scale wired “backbone” connectivity, rather we will solve the last mile problem through the integration of advanced wireless- and wired- backhaul. 

5G technology is one part of this emerging wireless data connectivity era. In addition to emerging low-orbit satellite constellation non-terrestrial networks, terrestrial millimeter wireless (mmWave), 5G “New Radio” capabilities will be deployed both by commercial vendors and non-commercial entities (using open parts of the radio frequency spectrum-space) to support myriad uses. Because 5G operates over a very wide range of radio frequencies (600 MHz to 27 GHz) and also leverages advances made since the deployment of earlier cellular radio communication standards, such as software defined networking, beam steering, and improved signal processing, 5G will allow users (including the scientific community) to engineer wireless data transmission supporting novel sensing applications for the world around us.

What makes 5G different from previous wireless standards for science? 

accessibility-browsing-5g-business5G is built around three standards, each of which leverages network resources in different ways.  Each of these application models will be leveraged by scientists depending on their needs:

Enhanced Mobile Broadband: The main benefit of 5G comes from a great increase in the ability to spatially reuse the radio spectrum. In comparison to previous cellular network standards, 5G networks will support higher data rates, and an ability to support many more subscribing devices wherever this is needed.  For scientists, this will mean much improved options for sensor networks, Internet of Things (IoT) applications, lower wireless data costs, and (hopefully) less reliance on “sneakernet” or other improvised methods for data collection and movement.

Ultra Reliable and Low Latency Communications: 5G supports deployment modes based around defined service levels, which means users will be able to reserve “slices” of capacity in a way similar to reserving circuits on a wired network. This, combined with other capabilities, will allow 5G to support scientific uses where communications reliability is essential, such as when measurements depend on near-real-time interaction with instrument control systems or as part of operating mobile systems such as unmanned aerial vehicles.

Massive Machine Type Communications: 5G is also built to support deployment modes in support for low power, automated systems.  These capabilities will be of benefit for all kinds of urban applications, but particularly so for scientists leveraging 5G for urban or building applications.  Leveraging this standard, scientists will be able to deploy hundreds or even thousands of small, very power-efficient, sensors throughout buildings or other areas to measure energy or environmental factors.

Taken as a whole, the capabilities provided by advanced wireless (5G, non-terrestrial networks, and mmWave) will allow new kinds of science, both within the confines of the laboratory and outside in a world via commercial and national laboratory dense sensor networks. Both the types and amounts of data generated will greatly increase – as will the scientific opportunities to learn new things.

What role is ESnet playing in creating a 5G network for scientists?

Cori Supercomputer
Advanced wireless capabilities are creating new ways for scientists to connect data from sensors, vehicles, and isolated locations, with world-class supercomputers like the National Energy Research Scientific Computing Center’s (NERSC’s) Cori system. (Picture by Roy Kaltschmidt)

ESnet’s mission is to ensure that science collaborations—at every scale and in every scientific domain—have the information and tools they need to achieve maximum benefit from global networks. This mission is not defined by a particular technology. ESnet works to integrate the compute, storage, and analytic resources operated by sites within the Department of Energy complex, and our scientific customers. 

Unlike previous generations of sensor or data infrastructure development, such as the Internet, Advanced Wireless and 5G advances are largely occurring without the US National Laboratory system playing lead roles. The challenge for scientific users is primarily one of connecting wireless technology (when needed) into the toolset made available by the Department of Energy to support US and global science objectives.  

ESnet inherently must support these customer efforts because we operate the high-speed scientific data network upon which the community depends now, and in the future as next-generation capabilities (ESnet 6) come to life.  We are also at the forefront of thinking about next-generation data movement and analytics through leadership roles with the National Science Foundation’s FABRIC program, software defined networking, and other projects supporting the Department of Energy’s future vision for the science laboratory system. 

At the Lawrence Berkeley National Laboratory (Berkeley Lab), where ESnet is headquartered, we are working to develop a community of interest on 5G and advanced wireless applications, and have been using this as a forum to develop ideas, and bring in external speakers to provide technical talks on 5G state of the art.  

ESnet’s Science Engagement Team is also starting to work with the Applied Physics Program and others to test aspects of advanced wireless technology, as well as how we can connect this to ESnet work in edge computing, our ScienceDMZ architecture, and other Berkeley Lab resources.  We have also started to develop research relationships with the UC Berkeley advanced wireless community, especially the Wireless Research Center to explore mmWave capabilities.  Outside of Berkeley Lab, we have been very active in the Department of Energy’s Enabled Energy Innovation Workshop (5GEEIW)  and related discussions for science uses of 5G and future requirements, as well as discussions with other labs and commercial entities about collaboration on testbeds and prototyping use cases.  These efforts will grow over the next year and hopefully, the report just released from the 5GEEIW will contribute to this progress. [link here]

Are there any experiments looking to use 5G? 

16538042638_cf555c1932_k
Argonne National Laboratory’s  Waggle platform is a novel wireless sensor with advanced edge computing capabilities to enable a new breed of sensor-driven environmental science and smart city research. (Photo by Mark Lopez, Argonne National Laboratory)

Around the Department of Energy complex, many teams are starting to look at the use of 5G to support experiments.  There are also developing applications for inside building and laboratory use as well using unlicensed 5G spectrum—some of this application space is now served by either Wifi or wired connectivity. There is a need for some general networking research to explore how ESnet wired capabilities, such as caching and data transfer nodes, should be deployed as part of wired-wireless interfaces, and to develop patterns for scientific support for projects making use of advanced wireless technologies as part of ESnet support for science.

We, along with Argonne National Laboratory, Pacific Northwest National Laboratory, and other Labs, are developing ideas for 5G/Advanced Wireless testbed and prototype application testing environments.  At present, the availability of equipment and service is limited, but this is expected to change rapidly as the first generation of 5G handsets and other devices begin to flood the market.

What is the state of 5G now? How long will it be until scientists can access it?

5G is being commercially rolled out by carriers now, and the build-out of this service is expected to take several years.  Other resources, such as IoT 5G toolsets and hardware are also beginning to reach the market from Ericsson and other vendors.  Similarly, non-terrestrial network constellations such as StarLink are beginning to support limited communities of beta-testers, and mmWave resources are also becoming commercially available.  

Thanks to Berkeley Lab IT’s stellar work with Verizon, however, we hope that there will be options over this next year for Berkeley-community access to 5G testing resources, and similar opportunities to explore mmWave or non-terrestrial networks tools as we build relationships and capabilities.  We also believe that opportunities and resources will start to become available over this next year from the Department of Energy, and other funding sources to support science user testing and the uptake of advanced wireless.

How did you get into this work and what do you enjoy most about it?

I got into this area at the start of my career working on satellite mobile telephony, and later with the Department of Defense working on sensors and analysis systems. When I was supporting military forces in the field with analytics, the problem was always how to handle really data thin-pipes, and as part of this, we had a lot of trouble with existing radio, cellular and satellite options. 

As part of the research-support community, I’m most interested in how we can use 5G and advanced wireless technologies to allow scientists to do new things. It is really fascinating to be at a point of inflection, for RF wireless technology and the ability for almost anyone to be able to affordably collect data from the world, backhaul that data globally, and make sense of it.  

I think that we are in a great position to lead the way with open science “out in the world” which will leverage these new technologies and ESnet is a wonderful place to serve that cause.

Interviewed by Linda Vu, Berkeley Lab Computing Sciences

Into the Medical Science DMZ

~ ESnetComms
iStock-629606180
Speeding research. The Medical Science DMZ expedites data transfers for scientists working on large-scale research such as biomedicine and genomics while maintaining federally-required patient privacy.

In a new paper, Lawrence Berkeley National Laboratory (Berkeley Lab) computer scientist Sean Peisert and Energy Sciences Network (ESnet) researcher Eli Dart and their collaborators outline a “design pattern” for deploying specialized research networks and ancillary computing equipment for HIPAA-protected biomedical data that provides high-throughput network data transfers and high-security protections.

“The original Science DMZ model provided a way of securing high-throughput data transfer applications without the use of enterprise firewalls,” says Dart. “You can protect data transfers using technical controls that don’t impose performance limitations.”

Read More at Science Node: https://sciencenode.org/feature/into-the-science-dmz.php 

Sean-and-Eli
Left: Eli Dart, ESnet Engineer | Right:  Sean Peisert, Berkeley Lab Computer Scientist

Berkeley Lab and ESnet Document Flow, Performance of 56 Terabytes Climate Data Transfer

~ ESnetComms
Visualization by Prabhat (Berkeley Lab).
The simulated storms seen in this visualization are generated from the finite volume version of NCAR’s Community Atmosphere Model. Visualization by Prabhat (Berkeley Lab).

In a recent paper entitled “An Assessment of Data Transfer Performance for Large‐Scale Climate Data Analysis and Recommendations for the Data Infrastructure for CMIP6,” experts from Lawrence Berkeley National Laboratory (Berkeley Lab) and ESnet (the Energy Sciences Network, document the data transfer workflow, data performance, and other aspects of transferring approximately 56 terabytes of climate model output data for further analysis.

The data, required for tracking and characterizing extratropical storms, needed to be moved from the distributed Coupled Model Intercomparison Project (CMIP5) archive to the National Energy Research Supercomputing Center (NERSC) at Berkeley Lab.

The authors found that there is significant room for improvement in the data transfer capabilities currently in place for CMIP5, both in terms of workflow mechanics and in data transfer performance. In particular, the paper notes that performance improvements of at least an order of magnitude are within technical reach using current best practices.

To illustrate this, the authors used Globus to transfer the same raw data set between NERSC and Argonne Leadership Computing Facility (ALCF) at Argonne National Lab.

Read the Globus story: https://www.globus.org/user-story-lbl-and-esnet
Read the paper: https://arxiv.org/abs/1709.09575

How the World’s Fastest Science Network Was Built

~ ESnetComms

Created in 1986, the U.S. Department of Energy’s (DOE’s) Energy Sciences Network (ESnet) is a high-performance network built to support unclassified science research. ESnet connects more than 40 DOE research sites—including the entire National Laboratory system, supercomputing facilities and major scientific instruments—as well as hundreds of other science networks around the world and the Internet.

Funded by DOE’s Office of Science and managed by the Lawrence Berkeley National Laboratory (Berkeley Lab), ESnet moves about 51  petabytes of scientific data every month. This is a 13-step guide about how ESnet has evolved over 30 years.

Step 1: When fusion energy scientists inherit a cast-off supercomputer, add 4 dialup modems so the people at the Princeton lab can log in. (1975)

Online3

Step 2: When landlines prove too unreliable, upgrade to satellites! Data screams through space. (1981)

18ogxd

Step 3: Whose network is best? High Energy Physics (HEPnet)? Fusion Physics (MFEnet)?  Why argue? Merge them into one-Energy Sciences Network (ESnet)-run by the Department of Energy!  Go ESnet! (1986)

ESnetListicle

Step 4: Make it even faster with DUAL Satellite links! We’re talking 56 kilobits per second! Except for the Princeton fusion scientists – they get 112 Kbps! (1987)

Satellite

Step 5:  Whoa, when an upgrade to 1.5 MEGAbits per second isn’t enough, add ATM (not the money machine, but Asynchronous Transfer Mode) to get more bang for your buck. (1995)

18qlbh

Step 6: Duty now for the future—roll out the very first IPv6 address to ensure there will be enough Internet addresses for decades to come. (2000)

18s8om

Step 7: Crank up the fastest links in the network to 10 GIGAbits per second—16 times faster than the old gear—a two-generation leap in network upgrades at one time. (2003)

18qlnc

Step 8: Work with other networks to develop really cool tools, like the perfSONAR toolkit for measuring and improving end-to-end network performance and OSCARS (On-Demand Secure Circuit and Advance Reservation), so you can reserve a high-speed, end-to-end connection to make sure your data is delivered on time. (2006)

18qn9e

Step 9: Why just rent fiber? Pick up your own dark fiber network at a bargain price for future expansion. In the meantime, boost your bandwidth to 100G for everyone. (2012)

18on55

Step 10: Here’s a cool idea, come up with a new network design so that scientists moving REALLY BIG DATASETS can safely avoid institutional firewalls, call it the Science DMZ, and get research moving faster at universities around the country. (2012)

18onw4

18oo6c

Step 11: We’re all in this science thing together, so let’s build faster ties to Europe. ESnet adds three 100G lines (and a backup 40G link) to connect researchers in the U.S. and Europe. (2014)

18qnu6

Step 12: 100G is fast, but it’s time to get ready for 400G. To pave the way, ESnet installs a production 400G network between facilities in Berkeley and Oakland, Calif., and even provides a 400G testbed so network engineers can get up to speed on the technology. (2015)

18oogv

Step 13: Celebrate 30 years as a research and education network leader, but keep looking forward to the next level. (2016)

ESnetFireworks

ESnet Connections Peak at 270 Gbps Flow In, Out of SC14 Conference

~ ESnetComms

The booths have been dismantled, the routers and switchers shipped back home and the SC14 conference in New Orleans officially ended Nov. 21, but many attendees are still reflecting on important connections made during the annual gathering of the high performance computing and networking community.

Among those helping make the right connections were ESnet staff, who used ESnet’s infrastructure to bring a combined network capacity of 400 gigabits-per-second (Gbps) in the Ernest Morial Convention Center. Those links accounted for one third of SC14’s total 1.22 Tbps connectivity, provided by SCinet, the conference’s network infrastructure designed and built by volunteers. The network links were used for a number of demonstrations between booths on the exhibition floor and sites around the world.

A quick review of the ESnet traffic patterns at https://my.es.net/demos/sc14#/summary shows that traffic apparently peaked at 12:15 p.m. Thursday, Nov. 20, with 79.2 Gbps of inbound data and 190 Gbps flowing out.

Among the largest single users of ESnet’s bandwidth was a demo by the Naval Research Laboratory, which used ESnet’s 100 Gbps testbed to conduct a 100 Gbps remote I/O demonstration at SC14. Read the details at: http://www.nrl.navy.mil/media/news-releases/2014/nrl-and-collaborators-conduct-100-gigabit-second-remote-io-demonstration#sthash.RttfV8kw.dpuf

NRL and Collaborators Conduct 100 Gigabit/Second Remote I/O Demonstration

~ ESnetComms
The Naval Research Laboratory (NRL), in collaboration with the DOE’s Energy Sciences Network (ESnet), the International Center for Advanced Internet Research (iCAIR) at Northwestern University, the Center for Data Intensive Science (CDIS) at the University of Chicago, the Open Cloud Consortium (OCC) and significant industry support, have conducted a 100 gigabits per second (100G) remote I/O demonstration at the SC14 supercomputing conference in New Orleans, LA.

The remote I/O demonstration illustrates a pipelined distributed processing framework and software defined networking (SDN) between distant operating locations. The demonstration shows the capability to dynamically deploy a production quality 4K Ultra-High Definition Television (UHDTV) video workflow across a nationally distributed set of storage and computing resources that is relevant to emerging Department of Defense data processing challenges.

Visit the My Esnet Portal at https://my.es.net/demos/sc14#/nrl to view real-time network traffic on ESnet.
Visit the My Esnet Portal at https://my.es.net/demos/sc14#/nrl to view real-time network traffic on ESnet.

Read more: http://www.nrl.navy.mil/media/news-releases/2014/nrl-and-collaborators-conduct-100-gigabit-second-remote-io-demonstration#sthash.35f9S8Wy.dpu

ESnet partners with Corsa, REANNZ and Google in first end-to-end trans-Pacific SDN BGP multi-AS network

~ ESnetComms

Corsa Technology, ESnet, and REANNZ have successfully demonstrated the first international Software Defined Networking (SDN)-only IP transit network of three Autonomous Systems (AS) managed as SDN domains.  The partners took the approach of building and testing an Internet-scale SDN solution that not only embodies the SDN vision of separation of control and data, but enables seamless integration of SDN networks with the Internet.

This first implementation passed through 3 AS domains, namely Energy Sciences Network (ESnet) at Berkeley, REANNZ at Wellington, and Google research deployment at Victoria University, Wellington (NZ).  ESnet’s node used the Corsa DP6420 640Gbps data plane as the OpenFlow hardware packet forwarder, controlled by the open-source VANDERVECKEN SDN controller stack (based on RouteFlow and Quagga).

Read more.

ESnet’s Inder Monga Co-authors Article on Growing Role of Optical Networks

~ ESnetComms

ESnet Chief Technologist Inder Monga is co-author of “Optical Networks Come of Age,” was has just been published in the September 2014 issue of Optics and Photonics News.

Although fiber optic transmission capacity has grown by seven orders of magnitude in just 20 years, these systems serve mainly as the “fat pipes,” the large-scale plumbing of the Internet, according to the article. But that is changing.

“Greater use of optical networks—particularly in network edge applications that carry less aggregated, more “bursty,” service traffic—and continued traffic growth will soon revise this picture,” the authors write. “A changing landscape in fiber optic communication technologies is stimulating a resurgence of interest in optical switching. These trends are coming together in ways that hold promise for the long-anticipated widespread deployment of optically switched fiber networks that respond in real time to changing traffic and operator requirements.

“The ultimate mission is to enable the next-generation Internet—one that can support terabit-per-second speeds, but that remains economical and energy efficient,” the authors write.

In addition to Monga, the authors are Daniel Kilper, University of Arizona; Keren Bergman, Columbia University; Vincent W.S. Chan, MIT; George Porter, University of California, San Diego; and Kristin Rauschenbach, Notchway Solutions LLC.

A pdf of the article can be found at: lightwave.ee.columbia.edu/files/kilper2014a.pdf

ESnet Chief Technologist Inder Monga
ESnet Chief Technologist Inder Monga

ESnet Student Assistant Henrique Rodrigues Wins Best Student Paper Award at Hot Interconnects

~ ESnetComms

Henrique Rodrigues, a Ph.D. student in computer science at the University of California, San Diego, who is working with ESnet, won the best student paper award at the Hot Interconnects conference held Aug. 26-28 in Mountain View, Calif. Known formally as the 2014 IEEE 22nd Annual Symposium on High-Performance Interconnects, Hot Interconnects is the premier international forum for researchers and developers of state-of-the-art hardware and software architectures and implementations for interconnection networks of all scales.

Rodrigues’ paper, “Traffic Optimization in Multi-Layered WANs using SDN,” was co-authored by Inder Monga, Chin Guok and Eric Poulyou of ESnet, Abhinava Sadasivarao and Sharfuddin Syed of Infinera Corp. and Tajana Rosing of UC San Diego.

“Special thanks to ESnet that gave me the opportunity to work on such an important and interesting topic,” Rodrigues wrote to his ESnet colleagues. “Also to the reviewers of my endless drafts, making themselves available to provide feedback at all times. I hope to continue with the good collaboration moving forward!”

OSCARS 0.6 hits the limelight

~ ESnetComms

At the recent Supercomputing11 conference, the bubbly was flowing. ESnet launched its ANI 100 gigabit-per-second network, and marked a quarter century of networking for DOE science. That big news may have overshadowed another milestone—SC11 was the first time OSCARS 0.6 was publicly demonstrated in a production environment. Now we’d like to give OSCARS its due.

OSCARS, or On-Demand Secure Circuits and Advance Reservation System, allows users to set up virtual circuits on demand to reserve bandwidth, streamlining the transfer of massive data sets across multiple network domains. OSCARS originated at ESnet, but we open-sourced it to the community long ago. Last spring the more modular OSCARS version 0.6 was released for testers and early adopters.

Other famous OSCARS

The performance of OSCARS 0.6 at SC11 showed us that we met our design goal of creating a flexible and modular framework. This was reflected in the demos, which were easy for folks to customize according to their needs. In the demo, “Enabling Large Scale Science using Inter-domain Circuits over OpenFlow” Tom Lehman of ISI used OSCARS to provide the functionality to control Openflow switches. Thanks to the flexibility to customize software built into OSCARS 0.6, ESnet’s Eric Pouyol was able to produce a variation of that application, customizing OSCARS 0.6 for resource brokering. OSCARS also played a part in the successful demonstration of Internet2’s Dynamic Network System (DYNES).The goal of DYNES is to work with regional networks and campuses, using OSCARS to schedule and support scientific data flows from the LHC, and other data intensive science programs such as LIGO, Virtual Observatory, and other large-scale sky surveys.

Most of the 100 Gbps demos at SC were supported by both the ANI 100 Gbps network and the 100 Gbps SCinet showfloor network. OSCARS 0.6 was used to schedule all eight of the demos using the 100 Gbps ANI network, which included complex visualizations of climate models, the Large Hadron Collider and the VERY early history—13.5 billion years ago, or 100 billion in dog years— of the Universe. OSCARS also controlled the approximately 100 different connections at SCInet, as well as connecting to three other OSCARS instances on the show floor.

OSCAR the Grouch

We used OSCARS 0.6 to provision the network, scheduling user time-slices of the 100 gigabit-per-second ANI and SCinet network, 24 hours a day, over the period of a week so they could test the demos in advance without having to get up at 3:00 a.m. to do it.

OSCARS 0.6 ended up making certain network engineers’ lives much easier. According to my colleague Evangelos Chaniatakis a.k.a. Vangelis, who was involved in the gritty details of setting up OSCARS 0.6 at the show, his team was required to make last-minute changes to the pre-existing network framework to work with the new hardware but didn’t receive the equipment until the week before the conference. The modularity ESnet built into OSCARS 0.6 helped the team get the network working at short notice.

 Less of a Software, More of a Service

Every year the number of reservations and circuits at SC continues to grow. The SC11 network required roughly twice the number of VLANs over the previous year. While the bandwidth wasn’t much bigger, and there were approximately the same number of customers, this year’s users definitely had more requirements. “On the whole OSCARS 0.6 was really stable.” Vangelis reports. “It worked fine.”  But the lessons learned at SC11 made us rethink the OSCARS 0.6 service module and requirements. In the near future, we intend to tweak OSCARS 0.6 to provide users more flexibility, making it less of a software and more of a service.