At ESnet, Innovation and Collaboration Build Solutions for Today and Tomorrow

Inder Monga reflects on 2022’s highlights and looks ahead to the future.

ESnet Executive Director Inder Monga at the launch event celebrating the unveiling of ESnet6, the sixth generation of the Department of Energy’s (DOE’s) dedicated high-speed scientific network.

Dear Friends, Well-wishers, Colleagues, and all of ESnet,

It’s been less than a year since ESnet formally introduced ESnet6, the latest iteration of the U.S. Department of Energy’s Energy Sciences Network. And we’ve already made much progress in enhancing research capabilities and data sharing across a broad spectrum of scientific applications.

For more than 35 years, ESnet – headquartered at Lawrence Berkeley National Laboratory – has served as the data circulatory system for the DOE, connecting all of its national laboratories, tens of thousands of DOE-funded researchers, and DOE’s premier scientific instruments and supercomputing centers. This interconnected system enables data to move quickly between sites and collaborators, accelerating time-to-discovery.

ESnet6, unveiled in October 2022 in conjunction with Confab, our first user meeting, takes the network’s capabilities to the next level. ESnet6 represents a transformational change in the way networks are built for research, with improved capacity, resiliency, and flexibility. With more than 46 Terabits per second of aggregate bandwidth deployed, it features a significant increase over prior generations of the network. This boost in capacity enables scientists to more quickly process, analyze, visualize, share, and store the mountains of research data produced by experiments, modeling, and simulations.

But the new network – which was completed under budget and ahead of schedule – does more than just increase capacity. With ESnet6, our engineers have developed smart, programmable, and automated services uniquely built to support the multi-petabyte dataflows typical of science research today. In addition, they are future-proofed to manage the emerging exabyte data era, streaming data from instruments and high-impact digital twins that require predictability and low latency. 

For example, ESnet is a critical component of Berkeley Lab’s Superfacility Project, which offers researchers seamless analysis of their experimental data in real time and regardless of their location. Additionally, with the recent ‘Superlab’ demonstration of the ARIES project by the National Renewable Energy Laboratory (NREL), we demonstrated how these new capabilities can be used to “address large-scale emergent challenges to meet the nation’s clean energy goals and to reinforce the energy security needs of every community,” as Rob Hovsapian, ARIES research lead in hybrid energy systems at NREL, noted in a collaborative news release announcing this project. With this in mind, we’re already looking to what users and stakeholders would like to see next.

Four strategic thrusts will define our efforts: 

  1. Transform Operations: While priority one is to operate a highly performant and robust network, we are also exploring new architectures, infrastructure enhancements, improvements to business processes, additional orchestration and automation capabilities, and ways to integrate new technologies like AI/ML – all to improve the resiliency, efficiency, and effectiveness of the user facility.
  2. Expand Services Portfolio: Our current services are foundational to the national labs and science communities. As we enter into an exascale era, with data-intensive instruments and widely distributed experiments, the network will play a key role in providing critical data services and supporting distributed data workflows, both for our scientists and the sites. The staff continue to innovate, experiment, prototype, and transition to production new data and network services. In addition, we actively look to expand the modalities through which scientists acquire data, from private 5G to low-Earth-orbiting satellites in remote locations, and potentially through quantum networks.
  3. Increase Stakeholder Value: As high-speed and big-data networking experts, we can co-design solutions based on upcoming requirements with our scientific and site user community to ensure that ESnet provides the most value to all of DOE as well as the worldwide research and education community stakeholders.
  4. Build Accountability and Transparency: We will foster the culture of accountability and transparency that provides the right environment for our users and our employees to perform at their personal best. 

ESnet exemplifies the team science value of Berkeley Lab. Our partnerships with all of the DOE national labs, vendors, global research and education networks, and academia have been essential to the design and build of ESnet6 and our future endeavors. Our integration of experimental, networking, and computational facilities gives scientists the ability to take a giant leap forward in gaining insight from massive datasets produced by experiments that use large-scale instruments such as genome sequencers, telescope observatories, X-ray light sources, and particle accelerators, among many others. We know we cannot do this alone. Participating in community-based collaborative initiatives better positions us to address future needs for all users and stakeholders. Some examples include:

  1. Co-design with science collaborations: SENSE/Rucio integration (collaboration with U.S. CMS [Compact Muon Solenoid experiment]) and GRETA networking (collaboration with Nuclear Physics) in co-designing data/science workflows with scientists.
  2. Open source contributions: Collaborating with and contributing to the SURFnet Workflow Orchestrator for network automation. (Please see the “From Zero to Orchestrated—A Workflow Orchestrator Beginners’ Workshop” at TNC, June 2023, co-organized with SURFNet.) Contributions to perfSONAR, iperf3, Grafana, and many others are part of ESnet’s work with the larger networking community.
  3. Strategic collaborations with worldwide R&E partners: Transatlantic MOU with ANA (Advanced North Atlantic) collaboration partners to make “gap on oceans” irrelevant when it comes to scientists.
  4. Enabling impactful networking research through multi-organization collaboration: Research collaborations on the FABRIC Testbed to supercharge network and distributed systems research within the U.S. and internationally. The Berkeley Lab–led Quantum Testbed (QUANT-NET) will accomplish the same for quantum communications and computing.

We are applying the same thoughtfulness to our staffing efforts. People want to work in organizations that have meaningful impact and contribute to humanity, and we are building the foundation to support this. Between 2018 and 2022, ESnet grew by 200%, hiring and adding a diverse array of skillsets to realize a dedicated staff of more than 100. As we look to the future, we strive to build a balanced workplace that represents a diversity of backgrounds, skillsets, regions, and states. 

Ultimately, ESnet’s success depends on the sum of its people – those who work in or with our organization have ample opportunity to have a meaningful impact on humanity and science.

Ultimately, ESnet’s success depends on the sum of its people – those who work in or with our organization have ample opportunity to have a meaningful impact on humanity and science. In addition to our commitment to next-generation enabling technologies, this is a key focus for ESnet over the next 10 years and beyond. ESnet6 is designed to support the DOE’s multi-billion dollars of investments in scientific research that touches our everyday lives, and we will continue to invest in these and related technologies, services, and people to support the needs of the DOE, HPC, and global science communities. 

The ESnet Portal: Visualizing the Network

As a facility that provides reliable, high bandwidth interconnectivity to scientists at national laboratories, universities, and research institutions, it is important for ESnet to share timely and accessible information about the network and its current status. The my.es.net portal has been an innovative place for sharing this information since its inception. 

Following a six-month upgrade project, ESnet’s public-facing portal now provides improved visualization capabilities. It allows users to easily view the core network and its connections to national lab sites. With this updated version, users can now see the network in a more detailed view that also provides the ability to zoom in and pan. By utilizing an in-house developed network visualization library, this enhanced version synthesizes a longer time range of data to provide faster, more accurate, and more detailed network topology updates.

You can experience the new version and look at all its new capabilities at https://my.es.net

Building on a Treasure Trove of Measurement Data

In 2021, ESnet rolled out its Stardust system, which collects precise network measurement data and allows users to retrieve information about specific equipment over a given time range. This updated release builds on Stardust’s capabilities, giving users a window into measurement data over the entire network topology.

A New View of the Network

Historically, ESnet’s portal has only offered a logical view of the network, visualizing the connections between sites but only approximating the network footprint. The updated version offers two visualization options: an updated and expanded logical “subway-style” view and a geographically referenced view. While building both views of the network, care was taken to strike a balance between providing a visualization that is as rich and accurate as possible while minimizing visual clutter. The approach uses interactive layers to help target the most important network information.

ESnet pictured in “logical” view

ESnet pictured in “geographic” view

Understanding Network Utilization

One of the most important things to understand when building and maintaining a network is measurements of bandwidth utilization, particularly at peak and near-peak conditions. To give users a clearer understanding of these key peak bandwidth utilization measurements, the map sends Stardust queries for traffic aggregations that display the “95th percentile,” or near-peak traffic, and a “maximum,” or peak measurement for a given period of time. These measurements give a sense of the “high water marks” for the network, letting ESnet know when a “flood” might occur and helping visualize the available headroom and plan for times of highest utilization.

Map (with options) picturing the “high water marks” for the last week

Tools for the Future

In conjunction with ESnet’s Stardust system, this update to the portal allows for much more responsiveness to changes in the network topology in the future. When a new router or site is added, ESnet can bring a visualization of it online in minutes rather than days. 

With this added capability and flexibility, ESnet may enhance site-centric capabilities, providing tailored views or new network layers (e.g., layers of university sites or peering points with commercial and R&E networks) to better inform users.
The portal’s new visualization of ESnet is a significant upgrade that provides researchers, network engineers, and other stakeholders with a more comprehensive, detailed, and accurate view of our network. With new statistical methods, more extensive time-based analysis, and a greater range of visualization capabilities, our portal update provides valuable insights into network behavior and performance. If you’re an ESnet user, check out the new tool and see how it can help you understand the network in greater detail.