ESnet

Simplification and Advancement in Segment Routing with Nicholas Buraglio

~ ESnetComms

Segment Routing is a way to increase network efficiency by prepending a set of route instructions to a packet, allowing it to traverse directly to a specific destination. Much has been said about advantages and disadvantages of segment routing in the networking industry. There are the more obvious advantages like the ability to simplify the network and reduce resource utilization and reducing the number of nodes that need to be touched for path provisioning and changes but there are also many limitations. 

In this blog piece, Nicholas Buraglio, computer systems engineer on the Planning and Architecture team, discusses segment routing in scientific networks and how it can be highly beneficial. 

Segment Routing: Simplification and Advancement for Science Networks

Over the last few years, much of the networking industry has been abuzz about segment routing (SR) – a technology that seemingly straddles the line between the promised benefits of software defined networking (SDN) and the operational needs of large, complex, geographically diverse networks. Meeting that confluence of “granular control” and extreme scalability is no easy task. Add to that the prospect of simplification of a well known complex and uncommon set of controls and protocol stacks, and one starts to understand why SR is so highly desirable. 

So what is SR and what does this solution bring that makes it so desirable? In a nutshell, SR is a networking technology that combines the features of Multi Protocol Label Switching (MPLS), with the flexibility of SDN. It allows for controller augmented and source-based routing without the need for maintaining state across a network core and for seamless fallback to traditional network protocols in the case of failures. Alone, each of these attributes are very compelling, but together they make for an extremely robust solution. SR “provides more with less,” in that it requires fewer protocols to enable more and increasingly complex features. 

Setting aside the fact that as operators of boundary pushing high performance networks we are able to take advantage of more simple configuration (and therefore easier to provision and operate), SR opens up the world of SDN by offloading computationally complex tasks, such as path calculation and re-routing, but leaves behind the overhead often associated with controller-based networking technologies such as OpenFlow, which place the controller in the critical path for most control plane functions. SR controllers allow for a far more seamless transition from traditional, discreet router based networking decisions and the ability to offload tasks such as pre-calculating data paths and re-optimizing the network.   

In addition to the already lengthy list of advantages, SR also boasts a version that is derivative of the way that most large networks have been built by leveraging MPLS. This derivation makes for a significantly easier shift in operations as the day to day concepts are very similar and often well known to existing support and engineering staff. On a more technical level, SR contains many of the powerful and widely deployed features of MPLS in addition to many functional improvement and extensions, such as Traffic Engineering, used for guaranteeing bandwidth for experiments and other related functions, path engineering, robust failure protection, and compatibility with legacy protocols such as RSVP-TE. These features are especially compelling for ESnet since this allows for our OSCARS service to flourish and expand. 

Practically speaking, SR allows for complex operations on a large network, especially in the realm of traffic engineering. As an example, an intricate path from point A to point B can be calculated, provisioned, re-routed, and adjusted from an external interface that only needs to speak to a single device at the start of the requested path. For example, using the following five router topology, paths can be easily provisioned that connect resources using guaranteed bandwidth via non-default paths.   

Diagram 1: Five router topology using segment routing. Credit: Nick Buraglio.

Diagram1 shows that the red dotted line is a far longer path from System C to System D. While this may seem like a simple process, it is counter to traditional routing which would, by default, choose the direct path between router 4 and router 5. In addition to this capability, SR allows for additional criteria that is not available for consideration in the legacy protocol suites to be taken into account when building a path. Again referencing Diagram 1, we consider the blue path. Asserting the path between System A and system B is lower latency, SR allows for latency to be considered in path selection. Practically speaking this again allows for non-traditional network traffic engineering to be leveraged in order to meet a far greater variety of requirements that researchers and scientists may require.  

Want to know more about the protocols used within SR and to incorporate a Path Computation Element (PCE)? Find information on that subject and more here.

Chin Guok Takes on Role as ESnet’s Chief Technology Officer

~ ESnetComms

Chin Guok has been named as Energy Sciences Network (ESnet)’s Chief Technology Officer (CTO). Guok will lead the Planning and Innovation Department while taking on the additional role of CTO.

Guok has been with ESnet for more than 25 years and has led many innovative projects during that time. In 2006, Guok conceived and led the On-Demand Secure Circuits and Advance Reservation System (OSCARS) project which won the R&D 100 award as well as the Department of Energy Secretary’s Honor award. 

“Chin has long demonstrated leadership and innovation across the global research and experimentation ecosystem,” said Inder Monga, executive director of ESnet. “It is gratifying to see him take on this more prominent role as well as realize his technical vision and implement strategy at ESnet.” 

More recently, Guok led the design for ESnet’s next-generation network, ESnet6, which launched in early October 2022. He was also the lead in delivering the market-leading ESnet High Touch project and the in-network cache deployment, along with being deeply engaged with the SENSE/Rucio collaboration and the ExaFEL project. Guok is also a sought-after speaker internationally and most recently gave a keynote address at the Korea Institute of Science and Technology Information (KISTI) anniversary event. 

“I am honored to be asked to take on this new position at ESnet,” said Chin Guok. “ESnet has long been at the forefront of scientific networking and I am excited to have a larger role in guiding its future success.” 

Guok’s research interests include high-performance networking and network protocols, dynamic network resource provisioning, network tuning issues, and hybrid network traffic engineering. Guok received an M.S. in Computer Science from the University of Arizona in 1997 and a B.S. in Computer Science from the University of Pacific in 1991.

Quantum Networking Basics With ESnet’s Wenji Wu

~ ESnetComms

Quantum networks may provide new capabilities for information processing and transport, potentially transformative for science, economy and natural science uses. These capabilities, provably impossible for existing “classical” physics based networking technologies, are of key interest to many U.S. Department of Energy (DOE) mission areas, such as climate and Earth system science, astronomy, materials discovery, and life sciences, etc.

In August of 2021, the Advanced Scientific Computing Research (ASCR) division of the US Department of Energy’s Office of Science announced a funding award for several quantum information system projects in support of the U.S. National Quantum Initiative. One of these projects is QUANT-NET (Quantum Application Network Testbed for Novel Entanglement Technology), a collaboration between Berkeley Lab, UC Berkeley, University of Innsbruck, and Caltech.

QUANT-NET research is focused on building a software-controlled quantum computing network, linking Berkeley Lab and UC Berkeley. ESnet executive director Inder Monga is the project principal investigator. The idea for QUANT-NET was born out of the 2020 DOE Quantum Internet Blueprint workshop, where representatives from DOE national laboratories, universities, industry, and other U.S. agencies came together to define a roadmap for building the first nationwide quantum Internet.

In this post, Dr. Wenji Wu, an ESnet networking researcher who is part of the QUANT-NET team, describes what future capabilities quantum networking may provide and why researchers believe quantum networks will transform scientific activities. 

Why Quantum Networks?

In the past thirty years, significant progress has been made in the fields of quantum technologies. The combination of quantum mechanics and information science forms a new area – quantum information science (QIS). In the broad context of QIS, quantum networks have an important role for the physical implementation of quantum computing, communication, and metrology. Quantum networks are envisioned to achieve novel capabilities that are provably impossible using classical networks and could be transformative to science, the economy, and national security. These novel capabilities range from cryptography, sensing and metrology, distributed systems, to secure quantum cloud computing. 

A few examples of this include: 

  • Secure Quantum Communication: Quantum networks take advantage of the laws of quantum physics (i.e., superposition and entanglement) to transmit information, potentially achieving a level of privacy and security that is impossible to achieve with today’s Internet. See Figure 1a.
  • A Quantum Network of Clocks: Recent research shows that a quantum network of atomic clocks can result in a substantial boost of the overall precision if multiple clocks are properly connected by quantum mechanical means. Compared to a single clock, the ultimate precision will improve as much as 1/K, where K is the number of clocks. If the same clocks are connected via a classical network, the precision scales as much as 1/SQRT(K). Ultimately, a quantum network of atomic clocks can surpass the Standard Quantum Limit (SQL) to reach the ultimate precision allowed by quantum theory — the Heisenberg limit. See Figure 1b.
  • Upscaling Quantum Computing: An individual quantum computer is typically limited in size. Connected by quantum networks, multiple quantum computers can work together as one big quantum computer to address larger problems. See Figure 1c.
Diagram Description automatically generated
Figure 1a: Secure quantum communication (credit: Chen et al. https://journals.aps.org/prl/abstract/10.1103/PhysRevLett.124.070501).Figure 1b: A quantum network of clocks (credit: Komar, Peter et al. “A quantum network of clocks.” Nature Physics 10.8 (2014:582-587).Figure 1c: Upscale quantum computing (credit: Thor Swift, Berkeley Lab).

Quantum Network Basics

Quantum networks are distributed systems of quantum systems, which are able to exchange quantum bits (qubits) and generate and distribute entangled quantum states. As illustrated by Figure 2, a quantum network conceptually consists of three essential quantum components: 

  1. Quantum nodes, which are physical quantum systems (e.g., trapped ions, quantum dots, Nitrogen-vacancy centers) connected to the quantum network. Well-characterized matter qubits are typically defined and created from these physical quantum systems. Quantum information is generated, processed, and stored locally by matter qubits in quantum nodes.  Matter qubits, often referred to as stationary qubits, are typically isolated from the surrounding environment to minimize decoherence and facilitate various quantum operations. 
  2. Quantum channels, which connect physically separated quantum components in the quantum network and transfer quantum states faithfully from place to place using the flying qubits. Optical fibers and free-space communications are typically implemented as quantum channels because they have a reduced chance of decoherence and loss. Photons with polarization or time-bin encoding are the flying qubit of choice. The implementation of quantum channels also requires that information encoded in a stationary qubit is reliably transferred to a flying qubit, and vice versa. 
  3. Quantum repeaters, which allow the end-to-end generation of quantum entanglement, and thus, the end-to-end transmission of qubits by using quantum teleportation. Quantum repeaters typically implement entanglement-related operations such as entanglement swapping and entanglement purification.

Figure 2: A quantum network consists of three essential quantum systems

In quantum networks, qubits cannot be copied due to the no-cloning theorem, which forbids the creation of identical copies of an arbitrary unknown quantum state. Therefore, qubits can not be physically transmitted over long distances without being hindered by the effects of signal loss and decoherence inherent to most transport mediums such as optical fiber. However, qubits can share a special relation known as entanglement. Entangled qubits have interesting non-local properties, even if they are located at distant nodes. Consuming an entangled qubit pair, a data qubit can be sent deterministically to a remote node. Entanglement is the fundamental building block of quantum networks. 

As illustrated in Figure 3, key entanglement-related operations include: 

  • Entanglement Purification: Multiple low-quality entanglements can be purified into a high-quality entanglement. 
  • Entanglement Swapping: Long-distance entanglement can be built from shorter segments, with flying qubits transmitted locally.
  • Teleportation: to enable the end-to-end transmission of qubits.

Figure 3: Key entanglement-related operations

Classic networks typically concern the performance metrics such as bandwidth, throughput, and latency. Likewise, quantum networks care for performance metrics related to quantum operations. Critical quantum quality metrics include entanglement generation rate, decoherence rate, and fidelity. In quantum networks, fidelity is a key indicator to characterize the quality of quantum states or operations. In general, a minimum fidelity (Fmin) is required to support quantum operations.

It is envisioned that quantum networks will operate in parallel with classic networks. Quantum networks are not meant to replace classic networks but rather to supplement them with quantum capabilities.

Current Status

Today, quantum networks are in their infancy. Like the Internet, quantum networks are expected to undergo different stages of research and development until they reach their full functionality. There are many promising R&D efforts underway looking to develop quantum network technologies. The DOE unveiled a quantum Internet blueprint in 2020 to accelerate research in quantum science and technology, with the emphasis on the creation of a quantum Internet.

Q&A with Jessy Schmit, ESnet’s Network Engineering Group Lead!

~ ESnetComms

Jessy Schmit came to ESnet from Pilot Fiber in New York, NY, where for the last six years, she was the Senior Manager of Network Operations and Support. Before Pilot Fiber, Jessy worked at a creative advertising agency and spent several years in the arts as a performer and director. Her background includes strategic leadership in marketing, customer experience, design, and technology. 

Schmit recently earned her Master’s Degree in Technology Management from New York University’s Tandon School of Engineering. 

Originally from Seattle, she currently resides in Brooklyn and spent seven years in the San Francisco Bay Area getting her undergraduate degree. She looks forward to reconnecting with her West Coast roots at ESnet.  

Question 1: What brought you to ESnet?

I had the opportunity to work with Jay Stewart at my last company, his recommendation and an instant connection to the people I met during the interview process made the decision a no-brainer.  ESnet’s mission and values are something I can really get behind!

Question 2: What is the most exciting thing in your field right now?

I nerd out on customer experience and process improvements, so I am excited about the modernization of IT back office, technical support, and self-service for engineering organizations. Increasing automation strategically without sacrificing the beneficial human elements of customer and end-user support can speed execution and ease the burden on engineering and support teams. Network automations can also reduce error and improve availability and resilience. In other sectors, specifically healthcare, we’re seeing how self-service, increased resiliency and the improved application of technology can make people feel more connected to their provider or service.  

Question 3: What excites you most about your role?

The people! The candid and thoughtful approach to questions and discussion was really refreshing during my interview process. And now I have the opportunity to work beside those totally awesome ESnet folks everyday and they’ve surpassed my expectations. I am excited to continue collaborating with such a talented and dedicated team of performers across the organization and learning all I can in my new environment. Working to further such a worthy mission makes it pretty easy to feel passionate about my new job.

Question 4: What challenges/opportunities are you looking forward to tackling?

I’m excited to figure out what motivates my team. I’ve found that what drives an engineer is wildly different from what motivates an accountant or a professor or chef (or any other role). Creating an environment where everyone feels supported and enabled to perform exemplary work that betters the larger organizational goals but also, ideally, their own development goals, is a focus for me.  

Question 5: How do you feel your past experience will transfer to your role at ESnet?

Looking at the typical pedigree of a team lead in technology or science, the benefits of a background in the arts might not be immediately obvious. While traditional technical skills may get a candidate in the door, it’s really the interpersonal and communication skills that allow them to thrive in their role. Entering the realm of technology and science from another discipline provides me with a unique perspective that can add diversity to the viewpoints of the team. My previous role was at a startup ISP in Manhattan and the pace of progress on our network operations and engineering meant I had to be agile, speedy, creative, and responsive – around the clock – to emergencies and customer needs. I’m hopeful the transfer of my work ethic, adaptability, and empathy will allow me to provide individualized support for my team(s) and future customers. 

Question 6: What book, movie, or podcast would you recommend?

I could talk about movies for days, but I’d say “KIMI” for a little tech industry suspense. I would also recommend “The Woman King” for some stellar performances and an inspiring story, and “Severance” (TV show) for a fascinating, and sometimes super funny, dystopian drama.

Join ESnet at SC22!

~ ESnetComms

The International Conference for High Performance Computing, Networking, Storage, and Analysis (SC22) is just around the corner and ESnet staff will be there to connect, learn, and share their knowledge with the HPC community. SC22 will take place November 13 – 18 in Dallas, Texas, and is primarily in person for the first time since 2019. 

Here are some staff highlights:

Sunday, November 13

  • 8:30 AM – 5:00 PM  INDIS 2022: Annual International Workshop on Innovating the Network for Data-Intensive Science, Mariam Kiran, Anu Mercian, Room C156 
  • 8:55 AM     INDIS 2022: Panel Discussion: Network Research Exhibition: the Future of Networking and Computing with Big Data Streams, Tom Lehman, C
  • 3:30 PM     INDIS 2022 Featured Technical Talk: Quantum Communication: A Physics Experiment of a Network Paradigm Shift, Inder Monga, Room C156
  • 4:10 PM     Paper: EJ-FAT Joint ESnet JLab FPGA Accelerated Transport Load Balancer, Stacey Sheldon, Yatish Kumar, Michael Goodrich, Graham Heyes, Room C156  

Tuesday, November 15

  • 10:30 AM – 12:00 PM    Paper: HPC Network Architecture, Mariam Kiran, Room C141-143-149
  • 12:00 PM – 1:00 PM    Demo: Global Petascale to Exascale Workflows for Data Intensive Science, Mariam Kiran, DOE Booth #1600
  • 3:15 PM    Featured DOE Booth Talk: ESnet6: How ESnet’s Next-Generation Infrastructure Will Enable Integrated Research Initiative Workflows, Inder Monga, DOE Booth #1600

Wednesday, November 16

  • 11:00 AM    SC22 Network Research Exhibition, SC22-NRE-15, SENSE and Rucio/FTS/XRootD Interoperation, Tom Lehman, Xi Yang, Caltech Booth #2820
  • 2:00 PM    SC22 Network Research Exhibition, SC22-NRE-13, AutoGOLE/SENSE: End-to-End Network Services and Workflow Integration, Tom Lehman, Xi Yang, Caltech Booth #2820

Thursday, November 17

  • 10:00 AM     Demo: Janus Container Management and the EScp Data Mover, Ezra Kissel, Charles Shiflett, Md Arrifuzzaman, DOE Booth #1600

Under Budget and Ahead of Schedule, ESnet6 Project Receives Final CD-4 Approval

~ ESnetComms

The Department of Energy’s Office of Project Assessment recently issued its final CD-4 Review Report on the Energy Sciences Network (ESnet)’s ESnet6 upgrade project. The review, held on July 12 – 13, 2022, and conducted at the request of Barbara Helland, Associate Director of Science for Advanced Scientific Computing Research (ASCR), assessed the project’s readiness to proceed to the approval of project completion. The project completed all threshold key performance parameters (KPPs) six months ahead of the early finish date, two years ahead of the CD-4 Level 1 milestone date, and well under budget. The committee assessed that the project was ready to proceed to CD-4 approval, which was achieved on July 29, 2022. The Final Closeout report and Lessons Learned are being submitted next week within the specified 90-day window.

“I want to congratulate the entire ESnet organization especially the ESnet6 project director, Kate Mace, and the project team,” said Inder Monga, executive director of ESnet. “When the team set out to deliver on a project scope as vast as the ESnet6 launch, we did not imagine a global pandemic would interrupt the process. Despite that, the team delivered the entire project ahead of the deadline and, even with supply-chain issues, managed to complete the scope below the projected budget. Most of the team attended the ESnet6 Unveiling Event on October 11 and heard their accomplishments praised by the lab directorate as well as congresspeople and DOE staff.”

The committee commended the project team for their “unique and innovative approach” in completing the project objectives and complimented ESnet for their agility in following through with the project scope while dealing with the difficult environment generated by the COVID-19 pandemic. The report stated that COVID-19 restrictions, limitations, and supply-chain issues presented “no significant impact” on the project’s critical path. The report also identified the distributed nature of operations and ESnet’s support for a remote workforce as an “invaluable” approach and a best practice to be shared beyond the DOE complex.

The final reports required for the official Project Closeout will be submitted to DOE this week. The ESnet team has continued to keep up the pace as they work toward additional enhancements to the ESnet6 Facility. “The #ESnet6Week festivities the week of October 9 energized the team. Not only were the project accomplishments celebrated at the ESnet6 Unveiling event, but the team also heard firsthand about the impact the project has already had on scientific discovery,” said Kathryn Mace, the ESnet6 project director, and Network Engineering group lead. “Hearing about the expansion of scientific collaborations made easier with the ESnet6 network and automated operations provided the team with newfound motivation to keep moving full speed ahead. ESnet6 sets the foundation for global scientific innovation over the next 10 years.”

The ESnet6 project team, DOE staff, IPR Committee, and members of the Berkeley Lab directorate during the final CD-4 IPR closeout session.

ESnet6 Unveiled Tomorrow!

~ Andrew Wiedlea

We’re getting things set up for the ESnet6 Unveiling tomorrow – our tent has gone up, we’re holding final rehearsals for the presentations, printing badges, and doing a thousand other small things.  

Photo: Lauren Rotman, LBL
Photo: Lauren Rotman, LBL

The only thing missing from our pictures is you! 

See you tomorrow for the big day, if you are visiting in person, travel safe, and if you are joining us virtually, the show starts at 9:00 AM on https://streaming.lbl.gov.

Why We Designed and Deployed ESnet6: It is All About the Science!

~ ESnetComms

We’re just a few days away from the ESnet6 unveiling and Confab22!

Here’s a great video interview with Ann Almgren, Senior Scientist in CCSE and the Department Head of the Applied Mathematics Department in the Applied Mathematics and Computational Research Division at Berkeley Lab. In it she discusses her research into wind power generation/distribution, and how she will use ESnet6.

Ann Almgren, Berkeley Lab

To watch the unveiling of ESnet6 and learn more about Ann’s research, join us 11 October from 900AM – 12 PM PT at streaming.lbl.gov!

ESnet team to give multiple talks about networking, automation, and QUANT-NET at NORDUnet Conference and GEANT’s SIG-NGN meeting

~ ESnetComms

The 31st NORDUnet Conference will take place in Reykjavik, Iceland from September 13-15, 2022. 

ESnet staff will also be in attendance at the Special Interest Group on Next Generation Networking (SIG-NGN) on September 12, 2022, the day before the NORDUnet Conference.

Here’s where you can find ESnet team’s talks during these events: 

Monday, September 12, 2022: SIG-NGN

The next generation NREN lightning talks 
09:05 – 10:30am GMT

The lightning talks will feature two presentations from ESnet: 

  • ESnet Effort to Build Upon the NML and MRML – John MacAuley
  • LHC Next Generation Requirements Gathering – Eli Dart

–

Future network architectures. Technological change to support data moving / data planes 
11:00am – 12:30pm GMT

This session will start with 10-minute presentations, including two by ESnet staff:

  • ESnet7 – Chin Guok
  • Underlay Packet Inspection, Making Traffic Engineering Decisions at L2 – Yatish Kumar

These talks will be followed by a panel discussion. 

–

How do we stitch and share our L1-L3+ networks to introduce better and new services
12:00pm – 3:30pm GMT

This session includes a series of short talks, including:

  • Real Time Data Processing Requirements – Yatish Kumar

Yatish Kumar will also host a discussion on future networking technologies from =2:00pm – 3:20pm GMT

Tuesday, September 13, 2022: NORDUnet Conference

The ESnet6 Approach to Network Orchestration and Automation
11:00am – 12:30pm GMT | Track 1 / Room: Silfurberg B

Speaker: Scott Richmond

Abstract: Network Orchestration is a defining factor in next generation networks, enabling operators to deliver more consistent and reliable services. ESnet has leveraged a combination of internally developed tools, open source software, and commercial software to orchestrate and automate network configuration deployment. This approach has enabled rapid deployment of new network services, as well as ensuring that configuration standards are well enforced when deploying network services.

During this talk, we will provide a brief history of automation at ESnet, dive into what our goals were for orchestration and automation in the ESnet6 project, and describe the technology and process that we used to meet those goals. Finally, we will discuss the hurdles encountered and lessons we learned along the way while developing this tooling.

–

Eli Dart was part of the technical program committee and is the chair for the HPC session, taking place in Track 2 / Room: Rima from 2:00 – 3:30 pm GMT. 

Wednesday, September 14, 2022: NORDUnet Conference

Experimenting with Teleportation Based Physical Layer for the Network: QUANT-NET
1:30pm – 3:00pm GMT | Track 1 / Room: Silfurberg B

Speaker: Inder Monga

Abstract: QUANT-NET takes an application-centric and systems-based approach to building a Quantum Internet testbed. The main thrust of this effort is to build a three-node distributed quantum computing testbed between two sites, Lawrence Berkeley National Lab and the University of California Berkeley (UCB) connected with an entanglement swapping substrate over optical fiber and managed by a quantum network protocol stack. We will implement the most basic building block of distributed quantum computing by teleporting a controlled-NOT gate between two nodes. This approach will enable research, prototyping, measurement and testing of the entire quantum network stack from physical layer to the application. The talk will describe our proposed testbed and progress.

ESnet’s Wireless Edge: Extending Our Network to Support Field Science

~ Andrew Wiedlea

Throughout the world, earth and environmental scientists are deploying new kinds of sensors to measure and understand how the climate is changing and how we can best manage key infrastructure and resources in response. 

Operation and data analysis of these sensors can often be challenging, as they are deployed in areas with limited power, sometimes with no data connectivity beyond the periodic physical collection of memory cards. Sensors may be in areas where weather and other factors make access laborious and challenging, such as at the top of a mountain, down a borehole, or under dense forest canopy.

Solar-powered meteorological and hydrological sensors deployed at the Snodgrass Field Site, Crested Butte, July 2022 at approximately 9,000 ft. elevation. (Photo: Andrew Wiedlea)

As the number, types, and capabilities of these sensors increases, the U.S. Department of Energy’s (DOE) Energy Sciences Network (ESnet) is working on ways to extend its high-speed network to support the needs of scientists working in remote, resource-challenged environments where our fiber backbone cannot be extended. Using advanced wireless technologies such as low-Earth orbit constellations, 5G, and private citizen band radio system cellular, mmWave, and Internet-of-Things tools like long-range (LoRa) mesh networks, we are developing ways to remove the limits of geographical constraints from field scientists, just as we have traditionally sought to do for laboratory scientists around the DOE complex.

In early July this year, ESnet took a step forward in these efforts by installing a private cellular network near Crested Butte, Colorado, supporting sensor fields being used by Earth and environmental scientists on Lawrence Berkeley National Laboratory’s (Berkeley Lab’s) Surface Atmosphere Integrated Laboratory program.  

The purpose of this effort is to assess requirements for operation of a private 4G/5G wireless network in a remote and changing environment, which can pull ESnet capabilities and services supporting scientific research out beyond our performant 13,000 km optical backbone. We are also using this research to identify specific operational, workflow, and data movement needs for the Earth and environmental science community as part of building ESnet’s logistics, operational, and human capital resources available to support the Earth and environmental science mission.

Our system, which is currently being configured, is built around a Nokia Digital Automation Cloud private cellular capability, with antennas being placed across a valley from sensor fields at the Snodgrass Field Site in Crested Butte. The intent is to use this cellular service to automate and improve the efficiency of data collection from sensors, using cellular routers and radios, depending on the specific capabilities of each sensor system. For those sensor systems that cannot be directly connected to a cellular network, we are establishing solar-powered sensor stations that will provide local area bridge (several hundred meter) connectivity to local sensors via wifi, LoRa, or direct ethernet cable. 

Once data is backhauled from a sensor field through our private cellular network, it will be transmitted back to ESnet via SpaceX’s Starlink low earth orbit satellite system, connecting to ESnet at a peering location in Seattle, Washington, and then through our optical backbone to the National Energy Research Scientific Computing Center at Berkeley Lab for processing and storage.

With fantastic assistance and collaboration from the Atmospheric Radiation Monitoring program, the Rocky Mountain Biological Laboratory, and Dan Feldman and Charulekha Varadarajan in the Watershed Function Science Focus Area at Berkeley Lab, our first field campaign was both great fun and extremely productive. 

We will return later in the Fall to complete network configuration and connection of sensors to the network. Once this is done, we can begin the next phase of this research: studying the operational performance and service requirements necessary to support field science through the demanding conditions provided by winter in the Colorado High Rockies. We will also begin to develop standard deployment equipment specifications and practices that we can use to support ESnet wireless edge deployments supporting science in other regions and for other purposes.  

This effort is being made possible by teamwork across ESnet and Berkeley Lab, including outstanding support at Berkeley Lab from Chris Tracy, Jackson Gor with ESnet network engineering, and Steve Nobles and many others with IT Telephone Services. The Colorado deployment success depended on the hard (often physical) work of Stijn Wielandt-EESA, Kate Robinson (ESnet Network Engineering), Jeff D’Ambrogia (IT-Science IT), and Jeff Chavez with Nokia.