ESnet’s second annual Confab gathering is designed for scientists across all disciplines who want to vastly improve their workflows and collaborations to accelerate time to discovery; for network engineers from national labs and universities who support science IT services for researchers on their campuses; and for the research networking professionals who partner with ESnet to move data across the world.
Last year’s inaugural Confab in Berkeley, held concurrently with the unveiling of ESnet6, was a fun and resounding success — and we believe Confab23 will continue and broaden the conversation we started.
Confab23 will showcase scientists who use ESnet today to perform real-time data analysis, leverage multiple supercomputers in parallel for large-scale simulations, and collaborate with colleagues on experiments as if side by side while thousands of miles apart – among many other applications.
Together we can chart the future of scientific data management and integrated scientific infrastructure.
In addition to lively conversation and informal technical discussions between our ESnet, DOE, and scientific community attendees, the program includes:
Updates and discussions with the Department of Energy on major initiatives, such as the Integrated Research Infrastructure initiative and the High Performance Data Facility, that support our shared vision — that scientific progress will be completely unconstrained by the physical location of instruments, people, computational resources, or data.
Scientific keynotes by Nicholas Schwarz on Light Source and BES developments, and Amadeo Perazzo, on LCLS and DOE HPC collaboration
An overview of quantum networking activities by Inder Monga, part of a lively evening program
We will also hold a meeting of the ESnet Site Coordinator Committee at the same venue on October 19 and 20.
If you have any questions or suggestions – or would like to offer a presentation or session topic – please email our workshop team directly at confab@es.net.
Quantum networks may provide new capabilities for information processing and transport, potentially transformative for science, economy and natural science uses. These capabilities, provably impossible for existing “classical” physics based networking technologies, are of key interest to many U.S. Department of Energy (DOE) mission areas, such as climate and Earth system science, astronomy, materials discovery, and life sciences, etc.
In August of 2021, the Advanced Scientific Computing Research (ASCR) division of the US Department of Energy’s Office of Science announced a funding award for several quantum information system projects in support of the U.S. National Quantum Initiative. One of these projects is QUANT-NET (Quantum Application Network Testbed for Novel Entanglement Technology), a collaboration between Berkeley Lab, UC Berkeley, University of Innsbruck, and Caltech.
QUANT-NET research is focused on building a software-controlled quantum computing network, linking Berkeley Lab and UC Berkeley. ESnet executive director Inder Monga is the project principal investigator. The idea for QUANT-NET was born out of the 2020 DOE Quantum Internet Blueprint workshop, where representatives from DOE national laboratories, universities, industry, and other U.S. agencies came together to define a roadmap for building the first nationwide quantum Internet.
In this post, Dr. Wenji Wu, an ESnet networking researcher who is part of the QUANT-NET team, describes what future capabilities quantum networking may provide and why researchers believe quantum networks will transform scientific activities.
Why Quantum Networks?
In the past thirty years, significant progress has been made in the fields of quantum technologies. The combination of quantum mechanics and information science forms a new area – quantum information science (QIS). In the broad context of QIS, quantum networks have an important role for the physical implementation of quantum computing, communication, and metrology. Quantum networks are envisioned to achieve novel capabilities that are provably impossible using classical networks and could be transformative to science, the economy, and national security. These novel capabilities range from cryptography, sensing and metrology, distributed systems, to secure quantum cloud computing.
A few examples of this include:
Secure Quantum Communication: Quantum networks take advantage of the laws of quantum physics (i.e., superposition and entanglement) to transmit information, potentially achieving a level of privacy and security that is impossible to achieve with today’s Internet. See Figure 1a.
A Quantum Network of Clocks: Recent research shows that a quantum network of atomic clocks can result in a substantial boost of the overall precision if multiple clocks are properly connected by quantum mechanical means. Compared to a single clock, the ultimate precision will improve as much as 1/K, where K is the number of clocks. If the same clocks are connected via a classical network, the precision scales as much as 1/SQRT(K). Ultimately, a quantum network of atomic clocks can surpass the Standard Quantum Limit (SQL) to reach the ultimate precision allowed by quantum theory — the Heisenberg limit. See Figure 1b.
Upscaling Quantum Computing: An individual quantum computer is typically limited in size. Connected by quantum networks, multiple quantum computers can work together as one big quantum computer to address larger problems. See Figure 1c.
Quantum networks are distributed systems of quantum systems, which are able to exchange quantum bits (qubits) and generate and distribute entangled quantum states. As illustrated by Figure 2, a quantum network conceptually consists of three essential quantum components:
Quantum nodes, which are physical quantum systems (e.g., trapped ions, quantum dots, Nitrogen-vacancy centers) connected to the quantum network. Well-characterized matter qubits are typically defined and created from these physical quantum systems. Quantum information is generated, processed, and stored locally by matter qubits in quantum nodes. Matter qubits, often referred to as stationary qubits, are typically isolated from the surrounding environment to minimize decoherence and facilitate various quantum operations.
Quantum channels, which connect physically separated quantum components in the quantum network and transfer quantum states faithfully from place to place using the flying qubits. Optical fibers and free-space communications are typically implemented as quantum channels because they have a reduced chance of decoherence and loss. Photons with polarization or time-bin encoding are the flying qubit of choice. The implementation of quantum channels also requires that information encoded in a stationary qubit is reliably transferred to a flying qubit, and vice versa.
Quantum repeaters, which allow the end-to-end generation of quantum entanglement, and thus, the end-to-end transmission of qubits by using quantum teleportation. Quantum repeaters typically implement entanglement-related operations such as entanglement swapping and entanglement purification.
Figure 2: A quantum network consists of three essential quantum systems
In quantum networks, qubits cannot be copied due to the no-cloning theorem, which forbids the creation of identical copies of an arbitrary unknown quantum state. Therefore, qubits can not be physically transmitted over long distances without being hindered by the effects of signal loss and decoherence inherent to most transport mediums such as optical fiber. However, qubits can share a special relation known as entanglement. Entangled qubits have interesting non-local properties, even if they are located at distant nodes. Consuming an entangled qubit pair, a data qubit can be sent deterministically to a remote node. Entanglement is the fundamental building block of quantum networks.
As illustrated in Figure 3, key entanglement-related operations include:
Entanglement Purification: Multiple low-quality entanglements can be purified into a high-quality entanglement.
Entanglement Swapping: Long-distance entanglement can be built from shorter segments, with flying qubits transmitted locally.
Teleportation: to enable the end-to-end transmission of qubits.
Figure 3: Key entanglement-related operations
Classic networks typically concern the performance metrics such as bandwidth, throughput, and latency. Likewise, quantum networks care for performance metrics related to quantum operations. Critical quantum quality metrics include entanglement generation rate, decoherence rate, and fidelity. In quantum networks, fidelity is a key indicator to characterize the quality of quantum states or operations. In general, a minimum fidelity (Fmin) is required to support quantum operations.
It is envisioned that quantum networks will operate in parallel with classic networks. Quantum networks are not meant to replace classic networks but rather to supplement them with quantum capabilities.
Current Status
Today, quantum networks are in their infancy. Like the Internet, quantum networks are expected to undergo different stages of research and development until they reach their full functionality. There are many promising R&D efforts underway looking to develop quantum network technologies. The DOE unveiled a quantum Internet blueprint in 2020 to accelerate research in quantum science and technology, with the emphasis on the creation of a quantum Internet.
Scientists at Oak Ridge, Argonne, and Lawrence Livermore National Laboratories are collaborating on the next generation of integrated Earth climate models using Exascale Computing Project computers and simulation models. The Earth System Grid Federation program is building vast simulation models using data collected about our planet at all levels, from space to far below the surface. Predictions from these models are vital to our understanding of climate, ocean, and other complex systems that make life possible. Read more about this and ESnet’s role in this important international science conversation in a new phys.org article from Oak Ridge National Laboratory.
Visualization from the Earth System Model, one component of the Earth System Grid Federation program. ESnet provides the data connectivity necessary to stitch teams and computers at different labs together. Credit: LLNL, U.S. Dept. of Energy
The ESnet6 Unveiling Ceremony is 4 days away! Come celebrate our new network and the great science we support, like the Earth System Grid Federation. Join us from 9 a.m. – 12 a.m., 11 October on https://streaming.lbl.gov.
ESnet6 marks a new era of our high-performance network supporting the needs of scientists. We’re able to handle massive flows of data in a reliable, nimble way, and we can specifically configure our setup to match the needs of individual experiments. The upgrade ensures that ESnet is ready to support the future of science today, including the significant increase in the amount of data produced by scientific experiments and the increasingly complex needs of scientists and the way they interact with our network.
Come watch the ESnet6 unveiling ceremony 9AM -12 PM PT, October 11, at streaming.lbl.gov!
The 31st NORDUnet Conference will take place in Reykjavik, Iceland from September 13-15, 2022.
ESnet staff will also be in attendance at the Special Interest Group on Next Generation Networking (SIG-NGN)on September 12, 2022, the day before the NORDUnet Conference.
Here’s where you can find ESnet team’s talks during these events:
Monday, September 12, 2022: SIG-NGN
The next generation NREN lightning talks 09:05 – 10:30am GMT
The lightning talks will feature two presentations from ESnet:
ESnet Effort to Build Upon the NML and MRML – John MacAuley
LHC Next Generation Requirements Gathering – Eli Dart
–
Future network architectures. Technological change to support data moving / data planes 11:00am – 12:30pm GMT
This session will start with 10-minute presentations, including two by ESnet staff:
ESnet7 – Chin Guok
Underlay Packet Inspection, Making Traffic Engineering Decisions at L2 – Yatish Kumar
These talks will be followed by a panel discussion.
–
How do we stitch and share our L1-L3+ networks to introduce better and new services 12:00pm – 3:30pm GMT
This session includes a series of short talks, including:
Real Time Data Processing Requirements – Yatish Kumar
Yatish Kumar will also host a discussion on future networking technologies from =2:00pm – 3:20pm GMT
Tuesday, September 13, 2022: NORDUnet Conference
The ESnet6 Approach to Network Orchestration and Automation 11:00am – 12:30pm GMT | Track 1 / Room: Silfurberg B
Speaker: Scott Richmond
Abstract: Network Orchestration is a defining factor in next generation networks, enabling operators to deliver more consistent and reliable services. ESnet has leveraged a combination of internally developed tools, open source software, and commercial software to orchestrate and automate network configuration deployment. This approach has enabled rapid deployment of new network services, as well as ensuring that configuration standards are well enforced when deploying network services.
During this talk, we will provide a brief history of automation at ESnet, dive into what our goals were for orchestration and automation in the ESnet6 project, and describe the technology and process that we used to meet those goals. Finally, we will discuss the hurdles encountered and lessons we learned along the way while developing this tooling.
–
Eli Dart was part of the technical program committee and is the chair for the HPC session, taking place in Track 2 / Room: Rima from 2:00 – 3:30 pm GMT.
Wednesday, September 14, 2022: NORDUnet Conference
Experimenting with Teleportation Based Physical Layer for the Network: QUANT-NET 1:30pm – 3:00pm GMT | Track 1 / Room: Silfurberg B
Speaker: Inder Monga
Abstract: QUANT-NET takes an application-centric and systems-based approach to building a Quantum Internet testbed. The main thrust of this effort is to build a three-node distributed quantum computing testbed between two sites, Lawrence Berkeley National Lab and the University of California Berkeley (UCB) connected with an entanglement swapping substrate over optical fiber and managed by a quantum network protocol stack. We will implement the most basic building block of distributed quantum computing by teleporting a controlled-NOT gate between two nodes. This approach will enable research, prototyping, measurement and testing of the entire quantum network stack from physical layer to the application. The talk will describe our proposed testbed and progress.
Throughout the world, earth and environmental scientists are deploying new kinds of sensors to measure and understand how the climate is changing and how we can best manage key infrastructure and resources in response.
Operation and data analysis of these sensors can often be challenging, as they are deployed in areas with limited power, sometimes with no data connectivity beyond the periodic physical collection of memory cards. Sensors may be in areas where weather and other factors make access laborious and challenging, such as at the top of a mountain, down a borehole, or under dense forest canopy.
Solar-powered meteorological and hydrological sensors deployed at the Snodgrass Field Site, Crested Butte, July 2022 at approximately 9,000 ft. elevation. (Photo: Andrew Wiedlea)
As the number, types, and capabilities of these sensors increases, the U.S. Department of Energy’s (DOE) Energy Sciences Network (ESnet) is working on ways to extend its high-speed network to support the needs of scientists working in remote, resource-challenged environments where our fiber backbone cannot be extended. Using advanced wireless technologies such as low-Earth orbit constellations, 5G, and private citizen band radio system cellular, mmWave, and Internet-of-Things tools like long-range (LoRa) mesh networks, we are developing ways to remove the limits of geographical constraints from field scientists, just as we have traditionally sought to do for laboratory scientists around the DOE complex.
The purpose of this effort is to assess requirements for operation of a private 4G/5G wireless network in a remote and changing environment, which can pull ESnet capabilities and services supporting scientific research out beyond our performant 13,000 km optical backbone. We are also using this research to identify specific operational, workflow, and data movement needs for the Earth and environmental science community as part of building ESnet’s logistics, operational, and human capital resources available to support the Earth and environmental science mission.
Our system, which is currently being configured, is built around a Nokia Digital Automation Cloud private cellular capability, with antennas being placed across a valley from sensor fields at the Snodgrass Field Site in Crested Butte. The intent is to use this cellular service to automate and improve the efficiency of data collection from sensors, using cellular routers and radios, depending on the specific capabilities of each sensor system. For those sensor systems that cannot be directly connected to a cellular network, we are establishing solar-powered sensor stations that will provide local area bridge (several hundred meter) connectivity to local sensors via wifi, LoRa, or direct ethernet cable.
Once data is backhauled from a sensor field through our private cellular network, it will be transmitted back to ESnet via SpaceX’s Starlink low earth orbit satellite system, connecting to ESnet at a peering location in Seattle, Washington, and then through our optical backbone to the National Energy Research Scientific Computing Center at Berkeley Lab for processing and storage.
With fantastic assistance and collaboration from the Atmospheric Radiation Monitoring program, the Rocky Mountain Biological Laboratory, and Dan Feldman and Charulekha Varadarajan in the Watershed Function Science Focus Area at Berkeley Lab, our first field campaign was both great fun and extremely productive.
We will return later in the Fall to complete network configuration and connection of sensors to the network. Once this is done, we can begin the next phase of this research: studying the operational performance and service requirements necessary to support field science through the demanding conditions provided by winter in the Colorado High Rockies. We will also begin to develop standard deployment equipment specifications and practices that we can use to support ESnet wireless edge deployments supporting science in other regions and for other purposes.
This effort is being made possible by teamwork across ESnet and Berkeley Lab, including outstanding support at Berkeley Lab from Chris Tracy, Jackson Gor with ESnet network engineering, and Steve Nobles and many others with IT Telephone Services. The Colorado deployment success depended on the hard (often physical) work of Stijn Wielandt-EESA, Kate Robinson (ESnet Network Engineering), Jeff D’Ambrogia (IT-Science IT), and Jeff Chavez with Nokia.
An ESnet Private Cellular Antenna on the side of the ARM Crested Butte Meteorological Radar. (Photo: Andrew Wiedlea)
The view back toward Snodgrass from the ARM Radar Site (Photo: Andrew Wiedlea).
ARM shipping container containing the ESnet cellular system. Note the Starlink Antenna being used for data backhaul to ESnet’s optical backbone. (Photo: Andrew Wiedlea)
Learning about the rigors of field science, part 1: A sudden storm at altitude traps Jeff D’Ambroglia (Berkeley Lab-Science IT) and Stijn Wielandt (Berkeley Lab-EESA) in the ARM trailer as we deploy the CBRS system. (Photo: Andrew Wiedlea)
Learning about the rigors of field science, part 2 Kate Robinson (ESnet) and Andrew Wiedlea (ESnet) drip dry in the ARM trailer as the storm passes. (Photo: Andrew Wiedlea)
A completed solar sensor station, providing Cellular, Wifi, and (soon) LoRa backhaul to the sensors located in the field via our ESnet Private Cellular/CBRS. (Photo: Andrew Wiedlea)
Learning about the rigors of field science, part 3 Stijn Wielandt (LBNL-EESA) loads solar cellular sensor station equipment onto an ATV for deployment at Snodgrass (Photo: Andrew Wiedlea)
Learning about the rigors of field science, part 4 Jeff D’Ambrogia, Stijn Wieland, and a member of RMBL staff setting up a solar sensor station at Snodgrass. Everything is heavy and there’s always a need to improvise building atop a mountain at altitude. (Photo: Andrew Wiedlea)
Last week, over 50 ESnet employees gathered at Berkeley Lab for a week of strategizing and socializing. Here are some pictures from their adventures!
Photo by: Rae Pendergrass Featuring: Kapil Agrawal, Fatema Bannat WalaPhoto by: Unknown Featuring: (counter-clockwise around table from front): Ed Balas, Samir Faci, Biju Jacob, Bruce Mah, Katrina Turner, Jeremy Randolph, Sarah Larsen, Sowmya BalasubramanianPhoto by: Andrew Wiedlea Featuring: Eli Dart, Jason Zurawski, and Ken MillerPhoto by: Dale W. Carder Featuring (clockwise around table from left): Andy Lake, Sarah Larsen, Samir Faci, Sartaj Baveja, Sowmya Balasubramanian, Biju Jacob, James Kafader, Katrina Turner, Bruce Mah, Sasank Maganti, Jeremy Randolph, Ed BalasPhoto by: Ed Balas Featuring: Biju Jacob, Bruce Mah, wSasank Maganti, Sarah Larsen, Andy Lake, Asma Aldaghar, Dale Carder, Chris Cummings, Evangelos (Vangelis) Chaniotakis , Ryan VredenburgPhoto by: Ed Balas Featuring: Biju Jacob, Sowmya Balasubramanian, Sasaki Maganti, Katrina Turner, Joyce Johnson, Yoshi TurnerPhoto by: bmwt Featuring: Anne WhitePhoto by: bmwt Featuring: Charles Shiflett, Sam Moats, Jason Kang, Dhiva M, Goran Pejovic, Deb HellerPhoto by: bmwt Featuring: George B. Robb IIIPhoto by: bmwt Featuring: Charles Shiflett, Jason Kang, Goran Pejovic, Dhiva M.Photo by: bmwt Featuring: George B. Robb III, Giselle Jiles, Shawn KwangPhoto by: bmwt Featuring: Deb Heller, Dhiva M., Paul PorterPhoto by: bmwt Featuring: Deb Heller, Dhiva M., Paul Porter, Luke Baker, Goran PejovicPhoto by: bmwt Featuring: Anne White, Remy DoucetPhoto by: Dale Carder Featuring: Derek Howard
Join us 12–13 October 2022 for ESnet’s first science user meeting!
Science is a conversation! Come join the conversation and help shape the future of scientific networking at the inaugural ESnet yearly science user meeting – Confab22.
Dear Friends, Well-wishers, Colleagues, and all of ESnet,
In October of this year we will launch ESnet6, a next-generation network featuring an entirely new, software-driven network design that enhances the ability to rapidly invent, test, and deploy new innovations to meet the data needs of the Office of Science/DOE.
We put forth the vision for ESnet6 in 2016. Since then, this $151M project (total project cost – DOE 413.3 parlance including contingency) has overcome pandemic-induced issues like site lockdowns, differing vaccination and inter-state travel policies, and variable supply chain delays, and is now in its final stages of implementation. As I prepare this historic unveiling, I can’t help but look back at what the team accomplished last year.
This is the first post in a series of blog posts about the people, partnerships, and innovations that have paved the road to ESnet6.
2021 was a year for growth within ESnet. We have 100+ people in the organization now—a 30% increase from last year—and it has been great to have new employees on-boarded, integrated, and productive in this challenging environment.
The dimensions of growth for ESnet
Looking towards the future, we think of ESnet growing around four dimensions. The three spatial axes are:
Foundations: Next Generation Network and Services
Innovation: Testbeds and Advanced Research and Development
and Co-design: Partnerships with Science for new data and network solutions.
The fourth axis, Culture, is pervasive across all three dimensions.
The main reason for choosing this very technical representation is to illustrate that these are not independent thrusts—success in each of these dimensions depends on the capabilities of the other.
In this post, I’d like to focus on that first axis: Foundations. In the next few posts, I will focus on the Innovation and Co-Design dimensions and share more thoughts about our focus for 2022 and beyond.
Major capacity improvements
In 2021, we installed a brand new routing infrastructure on our network backbone, while decommissioning a portion of the previous generation packet processors in parallel. We seamlessly transitioned all ESnet customers and peers onto the forty new backbone routers before the holidays, and the remaining router upgrades at our customer sites are in progress and scheduled through 2022.
The greenfield optical infrastructure (installed at 300 locations in 2020— another noteworthy accomplishment) is getting a wonderful upgrade: 400G wavelengths are being standardized across our national backbone as we complete the second phase of optical upgrades.
In addition to our team’s intricate efforts to decommission the existing network, we added another 100G on the ring in Europe (thanks to our collaboration with GEANT). This ensured that the first Large Hadron Collider Data Challenge had enough bandwidth to accommodate both ESnet scientific data and LHC data challenge (test) streams. We also established a new point of presence in Dallas to support new peerings and the FABRIC project.
ESnet network map showing LHC data challenge traffic sending nearly 100Gbps from Amsterdam to Boston.
Creating a smarter network
The vision laid out in 2016 focused not only on capacity, but also on improving the essential framework of how we operate with the network.
We made a significant investment in building out a high-availability site within 10ms of our main data center, in addition to our disaster-recovery site on the east coast. So any planned or unplanned power outages will be handled without a scramble. While the supply chain issues prevented the site from being ready for operations, we are making steady progress and look forward to completing it this year.
The software orchestration team made tremendous progress in laying down the vision and framework for automation. They were supported by strong internal collaboration with the engineering team. Many repetitious deployments were automated, and I know it took diligent effort to make these tools available in the right time frame, aligned with evolving constraints of the deployments. A few examples of where automation was used include:
Deployment of optical wavelengths on our backbone
Deployment of routers and base configurations, and service provisioning
Customer migration configurations from old network to the new equipment automatically generated from ESnet Database (ESDB)
Virtualized test environment was developed to test out new tools and services before actual in-field deployment.
This year, we prepare to bring the official DOE 413.3 ESnet6 project to a close, but as you know the network never sleeps, data never stops growing, and we have to constantly evolve the network. I can proudly say that we have the core foundations of the enduring ESnet user facility ready to handle the next big challenges of Data, AI, and Integrated multi-facility research that the scientists and National Labs are actively pursuing.
Wishing you all a very Happy New Year from ESnet.
Inder
This post is part of a series of posts reflecting on the road to ESnet6. Check back soon to see upcoming posts from Inder focusing on innovation, co-design, and his vision for ESnet6 and beyond.
ZeekWeek, an annual Fall conference organized by the Zeek Project, took place online from October 13-15 this year. The conference had over 2000 registered participants from the open source user community this year, who got together to share the latest and greatest about this cyber-security and network monitoring software tool.
Berkeley Lab staff member Vern Paxson developed the precursor to the Zeek intrusion detection software, then called Bro, in 1994. As an early adopter, ESnet’s cybersecurity team has strong relationships with the Zeek community, and this ZeekWeek was an opportunity to showcase advances and uses made by the software by ESnet and the entire Research and Educational Networking Community.
Fatema Bannat Wala also did a training session on “Introduction to Zeek,” which provided hands-on experience with Zeek tools and information about how to get involved with the collaboration.
ESnet’s cybersecurity team looks forward to continued collaboration with the Zeek community, attending next year’s ZeekWeek, and to contributing future code enhancements to this great software ecosystem.
You must be logged in to post a comment.