Research and Development

Learn how ESnet and its partners are investigating and testing emerging services, protocols, routing techniques, and tools necessary to meet the expanding needs of science.

ESnet’s Wireless Edge: Extending Our Network to Support Field Science

~ Andrew Wiedlea

Throughout the world, earth and environmental scientists are deploying new kinds of sensors to measure and understand how the climate is changing and how we can best manage key infrastructure and resources in response. 

Operation and data analysis of these sensors can often be challenging, as they are deployed in areas with limited power, sometimes with no data connectivity beyond the periodic physical collection of memory cards. Sensors may be in areas where weather and other factors make access laborious and challenging, such as at the top of a mountain, down a borehole, or under dense forest canopy.

Solar-powered meteorological and hydrological sensors deployed at the Snodgrass Field Site, Crested Butte, July 2022 at approximately 9,000 ft. elevation. (Photo: Andrew Wiedlea)

As the number, types, and capabilities of these sensors increases, the U.S. Department of Energy’s (DOE) Energy Sciences Network (ESnet) is working on ways to extend its high-speed network to support the needs of scientists working in remote, resource-challenged environments where our fiber backbone cannot be extended. Using advanced wireless technologies such as low-Earth orbit constellations, 5G, and private citizen band radio system cellular, mmWave, and Internet-of-Things tools like long-range (LoRa) mesh networks, we are developing ways to remove the limits of geographical constraints from field scientists, just as we have traditionally sought to do for laboratory scientists around the DOE complex.

In early July this year, ESnet took a step forward in these efforts by installing a private cellular network near Crested Butte, Colorado, supporting sensor fields being used by Earth and environmental scientists on Lawrence Berkeley National Laboratory’s (Berkeley Lab’s) Surface Atmosphere Integrated Laboratory program.  

The purpose of this effort is to assess requirements for operation of a private 4G/5G wireless network in a remote and changing environment, which can pull ESnet capabilities and services supporting scientific research out beyond our performant 13,000 km optical backbone. We are also using this research to identify specific operational, workflow, and data movement needs for the Earth and environmental science community as part of building ESnet’s logistics, operational, and human capital resources available to support the Earth and environmental science mission.

Our system, which is currently being configured, is built around a Nokia Digital Automation Cloud private cellular capability, with antennas being placed across a valley from sensor fields at the Snodgrass Field Site in Crested Butte. The intent is to use this cellular service to automate and improve the efficiency of data collection from sensors, using cellular routers and radios, depending on the specific capabilities of each sensor system. For those sensor systems that cannot be directly connected to a cellular network, we are establishing solar-powered sensor stations that will provide local area bridge (several hundred meter) connectivity to local sensors via wifi, LoRa, or direct ethernet cable. 

Once data is backhauled from a sensor field through our private cellular network, it will be transmitted back to ESnet via SpaceX’s Starlink low earth orbit satellite system, connecting to ESnet at a peering location in Seattle, Washington, and then through our optical backbone to the National Energy Research Scientific Computing Center at Berkeley Lab for processing and storage.

With fantastic assistance and collaboration from the Atmospheric Radiation Monitoring program, the Rocky Mountain Biological Laboratory, and Dan Feldman and Charulekha Varadarajan in the Watershed Function Science Focus Area at Berkeley Lab, our first field campaign was both great fun and extremely productive. 

We will return later in the Fall to complete network configuration and connection of sensors to the network. Once this is done, we can begin the next phase of this research: studying the operational performance and service requirements necessary to support field science through the demanding conditions provided by winter in the Colorado High Rockies. We will also begin to develop standard deployment equipment specifications and practices that we can use to support ESnet wireless edge deployments supporting science in other regions and for other purposes.  

This effort is being made possible by teamwork across ESnet and Berkeley Lab, including outstanding support at Berkeley Lab from Chris Tracy, Jackson Gor with ESnet network engineering, and Steve Nobles and many others with IT Telephone Services. The Colorado deployment success depended on the hard (often physical) work of Stijn Wielandt-EESA, Kate Robinson (ESnet Network Engineering), Jeff D’Ambrogia (IT-Science IT), and Jeff Chavez with Nokia.

ESnet Highlights from ZeekWeek’21

Fatema Bannat Wala presenting at ZeekWeek21
~ ESnetComms

Slides and videos from ZeekWeek have just been made available — here are links to ESnet highlights.

ZeekWeek, an annual Fall conference organized by the Zeek Project, took place online from October 13-15 this year. The conference had over 2000 registered participants from the open source user community this year, who got together to share the latest and greatest about this cyber-security and network monitoring software tool.

Berkeley Lab staff member Vern Paxson developed the precursor to the Zeek intrusion detection software, then called Bro, in 1994. As an early adopter, ESnet’s cybersecurity team has strong relationships with the Zeek community, and this ZeekWeek was an opportunity to showcase advances and uses made by the software by ESnet and the entire Research and Educational Networking Community.

The talk “DNS and Spoofed traffic investigation with Zeek,” presented by Fatema Bannat Wala, discussed how Zeek is being used to do network traffic analysis/investigations at ESnet by triaging abnormal activities when these occur on our network.

The talks “A Better Way to Capture Packets with DPDK” and “Details for DPDK plugin development and performance measurement” presented by Vlad Grigorescu and Scott Campbell, detailed the development process of the plugin and the performance enhancements it brings to the network packet capture technology.

Fatema Bannat Wala also did a training session on “Introduction to Zeek,” which provided hands-on experience with Zeek tools and information about how to get involved with the collaboration.

ESnet’s cybersecurity team looks forward to continued collaboration with the Zeek community, attending next year’s ZeekWeek, and to contributing future code enhancements to this great software ecosystem.

ESnet Machine Learning Researchers Win Best Paper at MLN ‘2021!

MLN '2021 Best Paper Award Notification
~ ESnetComms

Sheng Shen, Mariam Kiran, and Bashir Mohammed have just been awarded the Best Paper award at the International Conference on Machine Learning for Networking (MLN). Sponsored by the Conservatoire National des Arts et Métiers (CNAM), the École Supérieure d’Ingénieurs en Électrotechnique et Électronique (ESIEE), and Laboratoire d’Informatique Gaspard-Monge (LIGM), MLN is being held virtually 1-3 December 2021.

The paper, “DynamicDeepFlow: An Approach for Identifying Changes in Network Traffic Flow Using Unsupervised Clustering,” uses a hybrid of deep learning variational autoencoder model and a shallow learning k-means to help identify unique traffic patterns across ESnet. These unique patterns can help identify if a new experiment has started or whether current network bandwidth is changing.

DynamicDeepFlow (DDF) model structure

“We’re very excited to receive this recognition and the conference was a wonderful opportunity to exchange thoughts and ideas with peers in France. MLN is a conference dedicated to discussing machine learning applications in networks. Our next task is to integrate DynamicDeepflow with Netpredict to show real-time information in ESnet data” — Mariam Kiran

Papers from MLN will be published as post-proceedings in Springer’s Lecture Notes in Computer Science (LNCS).

ESnet Highlights from the National Science Foundation’s Cybersecurity Summit ’21

~ Fatema Bannat Wala

The National Science Foundation (NSF) Cybersecurity Center of Excellence, Trusted CI Project hosts a yearly cybersecurity summit, inviting people from various NSF-funded research organizations to share innovations and ideas. Here are some videos of ESnet presentations.

Scott Campbell presented “ESnet Security Group Impact on Network Architecture” where he discussed some of the social, technical, and architectural outcomes of the ESnet6 network upgrade that were beneficial to the organization. By being involved early, security design elements were incorporated into workflows at early stages and were both tightly integrated and vetted during the core design process. This early involvement also heightened the security group’s visibility, which led to a better understanding of how the various groups interact and their different methods of problem-solving and time management.

Eli Dart and Fatema Bannat Wala presented “Best practices for securing Science DMZ,” focusing on disentangling security policies and enforcement for science flows from traditional security approaches for business systems, and use of the Science DMZ model to protect high-performance science flows. They discussed thinking of the Science DMZ as a security architecture that provides useful and implementable security controls without impacting performance. 

ESnet Scientists awarded best paper at SC21 INDIS!

~ ESnetComms

A combined team from ESnet and Lehigh University was awarded the best paper for Exploring the BBRv2 Congestion Control Algorithm for use on Data Transfer Nodes at the 8th IEEE/ACM International Workshop on Innovating the Network for Data-Intensive Science (INDIS 2021), which was held in conjunction with the 2021 IEEE/ACM International Conference for High Performance Computing, Networking, Storage and Analysis (SC21) on Monday, November 15, 2021.

The team was comprised of:

  • Brian Tierney, Energy Sciences Network (ESnet)
  • Eli Dart, Energy Sciences Network (ESnet)
  • Ezra Kissel, Energy Sciences Network (ESnet)
  • Eashan Adhikarla, Lehigh University

The paper can be found here. Slides from the presentation are here. In this Q+A, ESnet spoke with the award-winning team about their research — answers are from the team as a whole.

INDIS 21 Best Paper Certificate

The paper is based on extensive testing and controlled experiments with the BBR (Bottleneck Bandwidth and Round-trip propagation time), BBRv2 and the Cubic Function Binary Increase Congestion Control (CUBIC) Transmission Control Protocol (TCP) Internet congestion algorithms. What was the biggest lesson from this testing?

BBRv2 represents a fundamentally different approach to TCP congestion control. CUBIC (as well as Hamilton, Reno, and many others) are loss-based, meaning that they interpret packet loss as congestion and therefore require significant network engineering effort to achieve high performance. BBRv2 is different in that it measures the network path and builds a model of the path – it then paces itself to avoid loss and queueing. In practical terms, this means that BBRv2 is resilient to packet loss in a way that CUBIC is not. This comes through loud and clear in our data.

What part of the testing was the most difficult and/or interesting?

We ran a large number of tests in a wide range of scenarios. It can be difficult to keep track of all the test configurations, so we wrote a “test harness” in python that allowed us to keep track of all the testing parameters and resulting data sets.

The harness also allowed us to better compare results collected over real-world paths to those in our testbed environments. Managing the deployment of the testing environment though containers also allowed for rapid setup and improved reproducibility. 

You provide readers with links to great resources so they can do their own testing and learn more about BBRv2. What do you hope readers will learn?

We hope others will test BBRv2 in high-performance research and education environments. There are still some things that we don’t fully understand, for example there are some cases where CUBIC outperforms BBRv2 on paths with very large buffers. It would be great for this to be better characterized, especially in R&E network environments.

What’s the next step for ESnet research into BBRv2? How will you top things next year?

We want to further explore how well BBRv2 performs at 100G and 400G. We would also like to spend additional time performing a deeper analysis of the current (and newly generated) results to gain insights into how BBRv2 performs compared to other algorithms across varied networking infrastructure. Ideally we would like to provide strongly substantiated recommendations on where it makes sense to deploy BBRv2 in the context of research and educational network applications.

Science begins as a Conversation! See how ESnet creates a world where conversations become discovery. Watch our new video now!

~ ESnetComms

Ever want to know how big research data moves around the globe? ESnet plays a significant role in supporting the great scientific conversations, collaborations, and experiments underway, wherever and whenever they occur. We move Exabytes of data around the world creating a global laboratory that accelerates scientific discovery.

In order to meet these needs of scientists, we are constantly looking for opportunities to expand our capabilities with our next generation network ESnet6, intelligent edge analytics, advanced network testbeds, 5G wireless, quantum networking and more.

https://www.es.net/scienceconversation/

Programmable Per-Packet Network Telemetry: From Wire to Kafka at Scale

~ Richard Cziva

High-speed intelligent Research and Educational Networks (RENs),  such as the one we’re building as part of the ESnet 6 program, will require a greater ability to understand and manage traffic flows. One research program underway to provide this capability is the High Touch effort,  a programmable, scalable, and expressive hardware and software solution that produces and analyzes per-packet telemetry information with nanosecond-accurate timing. Along with Zhang Liu, Bruce Mah, Yatish Kumar, and Chin Guok, I have just released a presentation for the Proceedings of the 2021 Virtual Meeting on Systems and Network Telemetry and Analytics, describing work underway to create a programmable, very high speed, packet monitoring, and telemetry capability as part of bringing High-Touch to life.

Richard Cziva presenting at the SNTA ’21: Proceedings of the 2021 on Systems and Network Telemetry and Analytics

For more information on this talk, please see this link.

Graduate students publish on network telemetry with ESnet

~ Richard Cziva

Two graduate students working with ESnet have published their papers recently in IEEE and ACM workshops.

  • Bibek Shrestha
  • Zhang Liu

Bibek Shrestha, a graduate student at the University of Nevada, Reno, and his advisor Engin Arslan worked with Richard Cziva from ESnet to publish a work on “INT Based Network-Aware Task Scheduling for Edge Computing”. In the paper, Bibek investigated the use of in-band network telemetry (INT) for real-time in-network task scheduling. Bibek’s experimental analysis using various workload types and network congestion scenarios revealed that enhancing task scheduling of edge computing with high-precision network telemetry can lead up to a 40% reduction in data transfer times and up to 30% reduction in total task execution times by favoring edge servers in uncongested (or mildly congested) sections of the network when scheduling tasks. The paper will appear in the 3rd Workshop on Parallel AI and Systems for the Edge (PAISE) co-conducted with IEEE IPDPS 2021 conference to be held on May 21st, 2021, in Portland, Oregon. 

Zhang Liu, a former ESnet intern and a current graduate student at the University of Colorado at Boulder, worked with the ESnet High Touch Team – Chin Guok, Bruce Mah, Yatish Kumar, and Richard Cziva – on fastcapa-ng, ESnet’s telemetry processing software. In the paper “Programmable Per-Packet Network Telemetry: From Wire to Kafka at Scale,” Zhang showed the scaling and performance characteristics of fastcapa-ng, and highlighted the most critical performance considerations that allow the pushing of 10.4 million telemetry packets per second to Kafka with only 5 CPU cores, which is more than enough to handle 170 Gbit/s of original traffic with 1512B MTU. This paper will appear in the 4th International Workshop on Systems and Network Telemetry and Analytics (SNTA 2021) held at the ACM HPCD 2021 conference in Stockholm, Sweden between 21-25 June 2021.

Congratulations Bibek and Zhang!

If you are a networked systems research student looking to collaborate with us on network measurements, please reach out to Richard Cziva. If you are interested in a summer internship with ESnet, please visit this page.

IPv6 past, present, future with Michael Sinatra and Nick Buraglio

~ Michael Sinatra

In March 2020, the U.S. Government Office of Management and Budget (OMB) released a draft memo outlining a required migration to IPv6 only. Memorandum M-21-07 was made official on November 19, 2020. Among other things, this memo mandates that 80% of IP-enabled assets on Federal networks are operating in IPv6-only environments by the end of FY 2025.

ESnet is in the process of planning this transition now, to ensure that we provide our users with the support and resources they need to continue their work uninterrupted and unimpeded by the transition. Practically speaking, this means for ESnet that by 2025, all of our nodes will be transitioned to IPv6 address space, and we will not support dual-stacking with IPv4 and IPv6 addresses. 

Transitioning to an IPv6-only network has been over a quarter-century in the making for ESnet.  Here’s a look back at our history with IPv6

IPv6: Past and Present

ESnet’s history of helping to develop, support, and operationalize new protocols begins well before the advent of IPv6.  

In the early 1990s, Cathy Aronson, an employee of Lawrence Livermore National Laboratory working on ESnet, helped establish a production implementation and support plan for the Open Systems Interconnect (OSI) Connectionless-mode Network Service (CLNS) suite of network protocols. Crucially, Aronson developed a scalable network addressing plan that provided a model for the utilization of the kinds of massive address spaces that OSI CLNS and, later, IPv6 would come to use. CLNS itself was a logical progression from DECnet which had been embraced and supported by ESnet’s precursors (MFEnet and HEPnet).  

As the IPv6 draft standard (RFC2460) developed in the 1990s, ESnet staff created an operational support model for the new protocol. The stakes were high; if IPv6 were to succeed in supplanting IPv4, and prevent the ill effects of IPv4 address exhaustion, it would need a smooth roll-out. Bob Fink, Tony Hain, and Becca Nitzan spearheaded early IPv6 adoption processes, and their efforts reached far beyond ESnet and the Department of Energy (DOE).  The trio were instrumental in establishing a set of operational practices and testbeds under the auspices of the Internet Engineering Task Force–the body where IPv6 was standardized–and this led to the development of a worldwide collaboration known as the 6bone.  6bone was a set of tunnels that allowed IPv6 “islands” to be connected, forming a global overlay network.  More importantly, it was a collaboration that brought together commercial and research networks, vendors, and scientists, all with the goal of creating a robust internet protocol for the future.

Not only were Fink, Hain, and Nitzan critical in this development of what would become a production IPv6 network (their names appear on a number of IETF RFCs), they would also spearhead the adoption of the protocol within ESnet and DOE. In the summer of 1996, ESnet was officially connected to the 6bone; by 1999, the Regional Internet Registries had received their production allocations of IPv6 address space. Just one month later, the first US allocation of that space was made–to ESnet.  ESnet has the distinction of being the first IPv6 allocation from ARIN – assigned on August 3, 1999, with the prefix 2001:0400::/32. 

Nitzan continued her pioneering work, establishing native IPv6 support on ESnet, and placing what we believe was the first workstation on a production IPv6 network. This was part of becoming the first production network in North America to adopt IPv6 in tandem with IPv4 via the use of an IPv6 “dual-stack.” As US Government requirements and mandates developed in 2005, 2012, and 2014, the ESnet team met these requirements for increased IPv6 adoption, while also providing support and consultation for the DOE community. 

Although Aronson, Fink, Hain, and Nitzan have all moved on from ESnet, a new generation of staff continued the spirit of innovation and early adoption. In the early 2010s, ESnet’s internal routing protocols were consolidated around the use of multi-topology Intermediate System to Intermediate System or IS-IS. This allowed for the deployment of flexible and disparate IPv4 and IPv6 topologies, paving the way for the creation of IPv6-only portions of ESnet, allowing the use of optimized routing protocols for the entire network.  ESnet’s acquisition strategy has long emphasized IPv6 support and feature parity between IPv4 and IPv6.  

All IPv6: Switching over, and the future

As ESnet moves into ESnet6, it is well-positioned to build and expand an IPv6-only network, while retaining legacy support for IPv4 where needed. ESnet will soon finish a two-year project to switch our management plane entirely over to IPv6. 

For our customers and those connected to us, here’s what this means:

  • ESnet will be ready, willing, and able to support connectors, constituents, and partners in their journey to deploying IPv6-only across our international network. 
  • ESnet planning and architecture team members have been included in the Department of Energy Integration and Product Team (DOE IPT) for migration to IPv6-only, and are supporting planning and documentation efforts for the DOE Complex.
  • We look forward to supporting our customers and users, as we all make this change to IPv6 together.