ESnet’s Wireless Edge: Extending Our Network to Support Field Science

Throughout the world, earth and environmental scientists are deploying new kinds of sensors to measure and understand how the climate is changing and how we can best manage key infrastructure and resources in response. 

Operation and data analysis of these sensors can often be challenging, as they are deployed in areas with limited power, sometimes with no data connectivity beyond the periodic physical collection of memory cards. Sensors may be in areas where weather and other factors make access laborious and challenging, such as at the top of a mountain, down a borehole, or under dense forest canopy.

Solar-powered meteorological and hydrological sensors deployed at the Snodgrass Field Site, Crested Butte, July 2022 at approximately 9,000 ft. elevation. (Photo: Andrew Wiedlea)

As the number, types, and capabilities of these sensors increases, the U.S. Department of Energy’s (DOE) Energy Sciences Network (ESnet) is working on ways to extend its high-speed network to support the needs of scientists working in remote, resource-challenged environments where our fiber backbone cannot be extended. Using advanced wireless technologies such as low-Earth orbit constellations, 5G, and private citizen band radio system cellular, mmWave, and Internet-of-Things tools like long-range (LoRa) mesh networks, we are developing ways to remove the limits of geographical constraints from field scientists, just as we have traditionally sought to do for laboratory scientists around the DOE complex.

In early July this year, ESnet took a step forward in these efforts by installing a private cellular network near Crested Butte, Colorado, supporting sensor fields being used by Earth and environmental scientists on Lawrence Berkeley National Laboratory’s (Berkeley Lab’s) Surface Atmosphere Integrated Laboratory program.  

The purpose of this effort is to assess requirements for operation of a private 4G/5G wireless network in a remote and changing environment, which can pull ESnet capabilities and services supporting scientific research out beyond our performant 13,000 km optical backbone. We are also using this research to identify specific operational, workflow, and data movement needs for the Earth and environmental science community as part of building ESnet’s logistics, operational, and human capital resources available to support the Earth and environmental science mission.

Our system, which is currently being configured, is built around a Nokia Digital Automation Cloud private cellular capability, with antennas being placed across a valley from sensor fields at the Snodgrass Field Site in Crested Butte. The intent is to use this cellular service to automate and improve the efficiency of data collection from sensors, using cellular routers and radios, depending on the specific capabilities of each sensor system. For those sensor systems that cannot be directly connected to a cellular network, we are establishing solar-powered sensor stations that will provide local area bridge (several hundred meter) connectivity to local sensors via wifi, LoRa, or direct ethernet cable. 

Once data is backhauled from a sensor field through our private cellular network, it will be transmitted back to ESnet via SpaceX’s Starlink low earth orbit satellite system, connecting to ESnet at a peering location in Seattle, Washington, and then through our optical backbone to the National Energy Research Scientific Computing Center at Berkeley Lab for processing and storage.

With fantastic assistance and collaboration from the Atmospheric Radiation Monitoring program, the Rocky Mountain Biological Laboratory, and Dan Feldman and Charulekha Varadarajan in the Watershed Function Science Focus Area at Berkeley Lab, our first field campaign was both great fun and extremely productive. 

We will return later in the Fall to complete network configuration and connection of sensors to the network. Once this is done, we can begin the next phase of this research: studying the operational performance and service requirements necessary to support field science through the demanding conditions provided by winter in the Colorado High Rockies. We will also begin to develop standard deployment equipment specifications and practices that we can use to support ESnet wireless edge deployments supporting science in other regions and for other purposes.  

This effort is being made possible by teamwork across ESnet and Berkeley Lab, including outstanding support at Berkeley Lab from Chris Tracy, Jackson Gor with ESnet network engineering, and Steve Nobles and many others with IT Telephone Services. The Colorado deployment success depended on the hard (often physical) work of Stijn Wielandt-EESA, Kate Robinson (ESnet Network Engineering), Jeff D’Ambrogia (IT-Science IT), and Jeff Chavez with Nokia.

Chris Cummings Speaks About Service Orchestration at Networking Field Day

ESnet’s Chris Cummings from our Orchestration and Core Data team was asked to present at the industry event “Networking Field Day: Service Provider”. 

Networking Field Day (NFD) is a unique event in which industry professionals are invited to join a panel of delegates who are presented with new products and offerings from networking vendors. These presentations then follow an open format where delegates can ask questions of the vendors and understand more about the products being discussed. 

For this particular instance of NFD, Chris was invited as a community member to present on how ESnet has built service orchestration and intent-based networking tooling, which allows us to abstract our service offerings from the technical implementation details that compose them. This is a topic that has a lot of mystique and buzz-words surrounding it in the networking industry, but this presentation contains concrete examples and demonstrations of the software that ESnet uses daily. 

The talk Chris gave explored ESnet’s approach to building a service orchestration software suite. He also gave a few demonstrations of the software in action. This presentation is not an exhaustive explanation of how to build your own intent-based networking environment, but rather an example and overview of a real-world stack that is being used in a production network today and the principles behind it. 

Watch Chris’s presentation here: https://techfieldday.com/appearance/a-real-world-approach-to-intent-based-networking-and-service-orchestration/

3Q with Juan Antunez, ESnet’s new Network Operations Engineer!

Before joining ESnet as a network operations engineer, Juan Antunez was an infrastructure engineer at Lowe’s. He brings seven years of experience in IT help desk support (customer support) and holds a Bachelor of Business Administration in Management Information Systems from the University of Houston-Downtown. Outside of work, he enjoys playing soccer with his seven-year-old daughter and traveling. 

Juan Antunez smiling into the camera
Juan Antunez

Question 1: What brought you to ESnet? 

After working for many years in the aerospace and retail industries, I’ve decided to join ESnet, which has a significant footprint supporting scientific research and development. The opportunity to contribute to the Department of Energy’s large-scale scientific research is exciting and fulfilling. I’m thrilled about joining, and I’m looking forward to continuing my career development!

Question 2: What is the most exciting thing going on in your field right now? 

I would say that Automation is something that many organizations are integrating into their network to avoid daily repetitive tasks that take time away from the workforce. It also helps prevent zero-day attacks with automation and response.

Question 3: What book, movie, or podcast would you recommend? 

I highly recommend The Art of Networking Engineering podcast to anyone interested in listening to other network engineers speak of their real-life experiences in the industry.  Also, Darknet Diaries is another fascinating podcast; you get to listen to and learn about what’s happening in the cybersecurity world. 

3Q with David Oh – ESnet’s new Network Operations Engineer!

Before joining ESnet, David Oh was a Senior Data Center Engineer at XR Trading. He lives in the Los Angeles area with his wife and his dog, a Havanese, poodle, and super mutt mix. Before moving to LA, he liked to spend his time working on his BMW E46 M3, not necessarily because he wanted to, but more often because he had to (15-year-old BMWs WILL break down on a near-weekly basis).

A photo of David Oh
David Oh

What brought you to ESnet?

I was working on cutting edge infrastructure in the trading world. I helped make things go faster so the company could flourish. After learning how ESnet supports research and development labs all over the world, I was excited at the prospect of helping science and humanity flourish.

What is the most exciting thing going on in your field right now?

I feel like automation is being used in amazing and creative ways to make tedious and mundane tasks in the networking world easier to accomplish.

What book, movie, or podcast would you recommend?

I enjoy listening to the Art of Network Engineering and, of course, the occasional crime mystery podcasts!

Watch our latest video: Serving Conversations That Matter

Title card from the Serving Conversations That Matter video

ESnet exists to support research into some of the most important questions of our time. The traffic that travels over our network on a daily basis contains data from tens of thousands of researchers – data that could lead to the next major discovery or scientific breakthrough.

In our latest video, learn about just a few of the revolutionary research collaborations we support, and the questions they’re working to answer.

The my.es.net portal has been updated!

Last week, we updated the my.es.net portal to better communicate the quantities of data that traverse the network on a daily basis!

The portal now shows larger dynamic traffic ranges between 0 and 800+ Gbps on the map – the previous portal’s visualizations maxed out at 80 Gbps.

The new color scheme is also color blind-friendly and works in grayscale!

Check it out for yourself here

CERN’s Large Hadron Collider Begins Run 3 

Today, the world’s most powerful particle accelerator, the Large Hadron Collider (LHC), begins Run 3, a new period of data taking! This comes after more than three years of upgrade and maintenance work.

Image: CERN

ESnet is looking forward to continuing to support the US LHC community for Run 3 through its connectivity to CERN. ESnet carries all traffic for LHC from Europe to the US, to the two Tier 1 centers at Fermilab and Brookhaven National Lab, and for all the physicists in Tier 2 centers in Universities. You can see the list of LHCONE collaborators at my.es.net/lhcone/list.

View the traffic on our trans-Atlantic links live at my.es.net.

Check out CERN’s livestream of the event here.

We want YOU  for #ESnet6Week!

Registration is now open for the unveiling of the ESnet6 network, Confab22 and other events planned for #ESnet6Week! Come join us either in person in Berkeley, CA or virtually.  

#ESnet6Week Schedule (October 11-14, 2022):

  • October 11: ESnet6 Unveiling – the Next-Generation Network (Free to attend virtually. In-person attendance is invite only)
  • October 12-13 : Confab22 – the Annual User Meeting ($300 to attend in-person, $150 to attend virtually – save $50 if you register before 30 June!)
  • October 14 : ESnet Site Coordinator Committee Meeting (open to ESCC invitees only)

To register or for more information, please see: https://go.lbl.gov/confab22!

ESnet is heading to Italy to share networking expertise with international colleagues at TNC22

ESnet will be in Trieste, Italy from June 13-17, 2022 sharing their knowledge about software-defined networking at TNC22, the research and education networking conference.

Here’s a look at where to find ESnet at the conference: 

Wednesday, 6/15

CEO Roundtable
Wednesday 6/15 – 8:30-16:30 CEST | Room B
Invitation only

ESnet executive director Inder Monga will join 70 CEOs from research networks around the world for a conversation about best practices, advancements in technology, and the future of scientific networking. 

Data Mobility Exhibition
Wednesday 6/15 – 10:30-11:00 CEST | Demonstration Area
Presenter: Eli Dart

Data mobility is a critical component of the process of science. Being able to predictably and efficiently move scientific data between experimental source, processing facilities, long term storage, and collaborators is a common use case that transcends the boundaries of research disciplines. The demo transfers reference data sets from well-tuned Data Transfer Nodes (DTNs) to existing campus CI components. Participants will upload, download, measure, and potentially improve their scientific data movement capabilities as well as review previous tests.

Thursday 6/16

Session: Real Life Network Orchestration

Building a Realistic Orchestration Validation Environment for netwoRks (ROVER)
Thursday 6/16 – 09:00 – 10:30 CEST | Auditorium
Speakers: Chris Cummings, Nick Buraglio 

Building software that controls network equipment has many similarities to traditional software engineering, however, testing this software introduces many complexities unique to the network orchestration world. Join this talk to learn how we approached these challenges by building a Realistic Orchestration Validation Environment for netwoRks (ROVER) at ESnet.

Session: Data Driven Networking

Superfacility – A Blueprint for Supporting Large-Scale Scientific Workflows
Thursday 6/16 – 11:00 – 12:30 CEST | Auditorium
Speaker: Chin Guok

Data-intensive workflows have been a mainstay of large scale collaborative scientific research for several decades. However, with the growing exponential rate of data that instruments can produce today, there needs to be a paradigm shift in how distributed resources are requested and utilized across the various facilities that contribute to these data-intensive workflows. A “Superfacility” model, which seamlessly integrates instrument, computing, storage, and networking facilities, is required to more effectively support these increasingly demanding workflows. This talk will present on the desired characteristics for a Superfacility, discuss specific use cases, and highlight some activities and initiatives in this area.

Exploring the BBRv2 Congestion Control Algorithm for use on Data Transfer Nodes
Thursday 6/16 – 11:00 – 12:30 CEST | Auditorium
Speaker: Eli Dart 

It is well known that loss-based TCP congestion control algorithms are problematic for high-speed, high-latency flows that are common in Big Science. In 2016 Google released a new congestion control algorithm called ‘BBR’ (Bottleneck Bandwidth and Round-trip time) that uses a model-based approach, and the design has since been refined in an alpha release of BBRv2. In this paper, we describe and perform a set of experiments that assess the suitability of BBRv2 for use on Data Transfer Nodes (DTNs).


Registration deadlines for TNC are coming up! Registration for regular in-person passes is open until June 6 (unless you want to pay via bank transfer – registration for that closed on May 30th). Free online passes are also available!

[[Editors note: This post has been updated to include information about the Data Mobility Exhibition on 6/15 that was previously excluded]].

ESnet staff attend strategic on-site meetings for the first time in years!

Last week, over 50 ESnet employees gathered at Berkeley Lab for a week of strategizing and socializing. Here are some pictures from their adventures!

Jealous of all the fun we had? Want to hang out with us, too? Good news – Registration will open soon for Confab22, ESnet’s first user meeting! Keep an eye on the blog or pre-register for updates!