Across the physical sciences, new instruments and capabilities are continuing a relentless growth in data production and need for high speed networking and analysis resources.
ESnet stays on-top of these trends via the Network Requirements Review process, which for the past 15 years has been a remarkable and useful collaboration between the DOE Office of Advanced Supercomputing Research (ASCR), ESnet and science programs across the DOE Office of Science.
The latest Network Requirements Review for the Office of Science High Energy Physics program office (HEP) is now available — among many other findings, this review confirms that the exponential growth of scientific data generation will continue unabated as we proceed into what may well be a new golden age for high energy physics research. Some samples include:
⇾ The upcoming High Luminosity era for the Large Hadron Collider (beyond 2027, or Run-4) will require multi-Tbps network speeds to support globally dispersed “Tier 1” HPC resources. Scientists will use the LHC to uncover how the Higgs-Boson interacts and gives mass to other particles, and explore emerging evidence for particle behaviors not explained by current physics models. Each data-taking year, the experiments, ATLAS and CMS combined, are expected to accumulate roughly 1 EB of new data and it is estimated that complete data set sizes may routinely exceed 100 PB.
⇾ Scientists at the Deep Underground Neutrino Experiment (DUNE) in South Dakota and at Fermilab in Illinois, will use high speed data transfer to identify supernova events, as part of ongoing measurement of neutrino interactions. Supernovae measured by DUNE will generate over 200TB of compressed data per event, and Research and Educational Networks (REN) must be able to supply highly reliable, predictable data transfer capabilities to provide telescope targeting data to global arrays.
⇾ The Cosmic Microwave Background, Stage 4 (CMB-S4) experiment will require data management and transfer capabilities in some of the most demanding locations on earth. Operating two observational locations, and multiple telescopes with a combined total of 500,000 cryogenically-cooled superconducting detectors at the South Pole and in the Chilean Atacama Desert, CMB-S4 will provide an unprecedented picture back into the start of the Universe. Operating for seven years in these conditions, 22 TB (~8 TB at the South Pole and ~14 TB in Chile) of data will be generated daily, leading to an accrual of 3 PB annually, and as much as 100 TB over the full program lifecycle.
Network Requirements Reviews analyze the current, near, and long-term needs of the HEP community, providing a network and data-centric understanding of the scientific process used by the researchers and scientists. These requirements reviews drive ESnet’s investments in new services and capabilities, and enable ESnet to build strong partnerships with Office of Science (SC) programs, PIs, and user facilities. More information on this ESnet requirements review process can be found here.
We would like to thank the 13 HEP projects, and all of the HEP & DOE Office of Science collaborators who generously gave of their time, expertise, and most importantly, their enthusiasm for the future of high energy physics, as part of creating this report.
We want to especially thank the entire Science Engagement team plus Kate Robinson, and Dale Carder with our Network Engineering group who all provided outstanding support and technical expertise.
Three years ago, ESnet unveiled its plan to build ESnet6, its next-generation network dedicated to serving the Department of Energy (DOE) national lab complex and overseas collaborators. With a projected early finish in 2023, ESnet6 will feature an entirely new software-driven network design that enhances the ability to rapidly invent, test, and deploy new innovations. The design includes:
State-of-the-art optical, core and service edge equipment deployed on ESnet’s dedicated fiber optic cable backbone
A scalable switching core architecture coupled with a programmable services edge to facilitate high-speed data movement
100–400Gbps optical channels, with up to eight times the potential capacity compared to ESnet5
Services that monitor and measure the network 24/7/365 to ensure it is operating at peak performance, and
Advanced cybersecurity capabilities to protect the network, assist its connected sites, and defend its devices in the event of a cyberattack
Later this month, ESnet staff will present an online update on ESnet6 to the ESnet Site Coordinators Committee (ESCC). Despite the challenges of deploying new equipment at over 300 distinct sites across the country and lighting up approximately 15,000 of miles of dark fiber during a pandemic, the team is making great progress, according to ESnet6 Project Director Kate Mace.
“We’ve had some delays, but our first priority is making sure the work is being done safely,” Mace said. “We have a lot of subcontractors and we are working closely with them to make sure they’re safe, they’re following local pandemic rules and they’re getting the access they need for installs.
“The bottom line is that we have a lot of pretty amazing people putting in a lot of hours and hard work to keep the project moving forward,” Mace said.
When completed in 2023, ESnet6 will provide the DOE science community with a dedicated backbone capable of carrying at least 400 Gigabits per second (Gbps), with some spans capable of carrying more than 1 Terabit per second.
The current network, known as ESnet5, comprises a series of interconnected backbone rings, each with 100Gbps or higher bandwidth. ESnet5 operates on a fiber footprint owned by and shared with Internet2. Once the switch is complete, Internet2 will take over ESnet’s share of the fiber spectrum to provide more bandwidth to the U.S. education community.
“We’re almost done with the optical layer, which is a big deal,” Mace said. “It’s been a major procurement of new optical line equipment from Infinera to light up the new optical footprint.”
Mapping the road to ESnet6
Back in 2011, using Recovery Act funds for its Advanced Networking Initiative, ESnet secured the long-term rights to a pair of fibers on a national fiber network that had been built, but not yet used. Because there was a surplus of installed fiber cable at the time, ESnet was able to negotiate advantageous terms for the network.
As part of the ESnet6 project, ESnet and its subcontractors began installing optical equipment along the ESnet fiber footprint starting in November 2019. The optical network consists of seven large fiber rings east to west across the U.S., and smaller “metro” rings in the Chicago and San Francisco Bay areas.
At this point, Infinera has completed the installation of the equipment at all locations. The four large eastern-most rings have passed ESnet’s rigorous testing and verification process ensuring that they are configured and working as designed, and most ESnet services in these areas have been transitioned over to the new optical system.
Infinera has turned over the other three large rings and is working closely with ESnet staff to address a number of minor issues identified during testing.
ESnet and Infinera are collaborating on turning up, testing, and rolling services to the new network in the Chicago and Bay Area rings. The installation in these areas is more complex because it is re-using the ESnet5 fiber going into the DOE Laboratories.
“The ESnet and Infinera teams have worked really well together to overcome all of the typical challenges we expected on a network build of this scale, as well as some unexpected obstacles,” said Joe Metzger, the ESnet6 Implementation Lead.
The typical expected challenges ranged from installing thousands of perfectly clean (microscopically verified) fiber connections, to the unexpected, such as engineers driving for hours to get to a remote isolated location to install the equipment only to find the access road is drifted in with snow, or somebody changed the lock.
Most of the unexpected challenges were related to COVID-19.
“It was amazing to see how the facility providers, including the DOE Laboratories, ESnet and Infinera teams worked together to find safe, workable solutions to the COVID-19-related access constraints that we encountered during the installation,” said Metzger.
The team expects the optical system build to be fully accepted and all services transitioned over to it by Oct. 1, completing what they are calling ESnet5.5, the first major step in the transition from ESnet5 to ESnet6.
To get to this point, ESnet’s network engineers needed extensive, hands-on training on the new Infinera equipment and built a specialized test lab at Berkeley Lab. To do this, a test lab was built at Berkeley Lab to provide hands-on training. Engineers take a weeklong session learning how to configure, operate, and troubleshoot the equipment deployed in the field.
The next major step will be the installation of new routers for the packet layer, which is expected to begin in early 2021, Mace said.
And of course, this is all being carried out while ESnet keeps its production network and services in regular operation and with the undercurrent of stress from the COVID-19 pandemic.
While ESnet staff are known for building an ever-evolving network that’s super fast and super reliable, along with specialized tools to help researchers make effective use of the bandwidth, there is also a side of the organization where things are pushed, tested, broken and rebuilt: ESnet’s testbed.
For example, in conjunction with the rollout of its nationwide 100Gbps backbone network, the staff opened up a 100Gbps testbed in 2009 with Advanced Networking Initiative funding through the American Reinvestment and Recovery Act. This allowed scientists to test their ideas on a separate but equally fast network so if something crashed, ESnet traffic would continue to flow unimpeded across the network. Six years later, ESnet upped the ante and launched the 400Gbps network — the first science network to hit this speed — to help NERSC move its massive data archive from Oakland to Berkeley Lab.
Eric Pouyoul is the principal investigator for the testbed and the things he’s learned on past projects can be applied to others. His most recent project also pushed the boundaries of what the organization does in supporting DOE science. With funding from the lab’s Nuclear Physics Division, Pouyoul developed a pair of uniquely specialized data processing systems for the GRETA experiment, short for Gamma Ray Energy Tracking Array. The gamma ray detector will be installed at DOE’s Facility for Rare Isotope Beams (FRIB) located at Michigan State University in East Lansing.
When an early version of GRETA goes online at the end of 2023 it will house an array of 120 detectors that will produce up to 480,000 messages per second—totaling 4 gigabytes of data per second—and send them through a computing cluster for analysis. Not only did Pouyoul write the software for the first stage that will reduce the amount of data by an order of magnitude—in real-time—he also designed the physics simulation software to generate realistic data generation to test the system.
For the second data handling phase of GRETA, called the Global Event Builder, he wrote the software that will take all of the data from the first phase and, using the timestamps, aggregate them in order, as well as sort them by event. This data will then be stored for future analysis.
Even though he designed and built the systems to simulate the behavior of the nuclear physics that will occur inside the detector, “don’t expect me to understand it,” Pouyoul said. “I never did anything like this before.”
GRETA is the first of its kind in that it will track the positions of the scattering paths of the gamma rays using an algorithm specifically developed for the project. This capability will help scientists understand the structure of nuclei, which is not only important for understanding the synthesis of heavy elements in stellar environments, but also for applied-science topics in nuclear energy, nuclear forensics, and stockpile stewardship.
“This has been my most exciting project and it only could have happened here,” he said. “I think it takes me back to the origins of the Lab when scientists and engineers worked together to create new physics. We know it will work, but we don’t even know how the results will turn out, we don’t know what will be discovered.”
Before joining ESnet at Berkeley Lab 11 years ago, he had worked in the private sector. At one point in his career, he wrote code for control systems for nuclear power plants. Looking back, he estimates that maybe three lines of his code made it into the final library. He’s quick to point out that he doesn’t consider himself a software engineer, nor does he think of himself as a network engineer. At ESnet, those engineers are responsible for designing and deploying robust systems that keep the data moving in support of DOE’s research missions.
“I really like to work with prototypes, one-time projects like in the testbed,” he said. “I know how to build stuff.”
He developed that skill as a high school student in Paris, where he preferred to roam the sidewalks, looking for discarded electronics he could take home, repair, and sell. He did manage to attend classes often enough to pass his exams and graduate with a degree. That was the only diploma he’s ever received.
Since then, he’s learned by working on things, not sitting in lecture halls. Some of it he picked up working for a supercomputing startup company. He learned how to tune networks for maximum performance by tweaking data transfer nodes, the equipment that takes in data from experiments, observations, or computations and speeds them on their way to end-users.
He sees the GRETA project as a pilot and it’s already drawing interest from other researchers. The idea is that if ESnet can work with scientists from the start, it will be more efficient and effective than trying to tack on the networking components afterward. Pouyoul looking forward to the next one.
“I’m really not specialized, but I do understand different aspects of projects,” he said. “I only have fun when I’m not in my comfort zone — and I had a lot of fun working on GRETA.”
As modern computers begin to reach the limit of their processing power, quantum computing has the potential to solve more specialized problems that require immensely robust computing. With that potential capturing the imagination of many, a consensus is building that a communication system using quantum mechanics represents one of the most important technological frontiers of the 21st century. To harness the full promise of quantum computing and sensing, we need to build scalable quantum communication networks that can support applications across science, industry, and national security. Scientists now believe that the construction of a prototype “quantum Internet” will be within reach over the next decade.
Toward this end, on July 23 the U.S. Department of Energy (DOE) unveiled a report that lays out a blueprint strategy to accelerate research in quantum science and technology, with an emphasis on the creation of a quantum Internet. “The Department of Energy is proud to play an instrumental role in the development of the national quantum Internet,” U.S. Secretary of Energy Dan Brouillette said in a news release. “By constructing this new and emerging technology, the United States continues with its commitment to maintaining and expanding our quantum capabilities.”
In support of this and related efforts, the Energy Sciences Network (ESnet) – a DOE Office of Science user facility managed by Lawrence Berkeley National Laboratory – is actively tracking multiple quantum networking projects and collaborating with the research community to help lay the groundwork for scalable quantum communication networks and a quantum Internet. In this interview with ESnet Director Inder Monga, he talks about the future of quantum networking and its role in facilitating quantum information science across the DOE and beyond.
How will the DOE’s quantum networking blueprint impact the development of quantum communications and applications?
Researchers believe that a quantum Internet could have a profound impact on a number of application areas critical to science, national security, and industry. Application areas include upscaling quantum computing by connecting distributed quantum computers, quantum sensing through a network of quantum telescopes, quantum metrology, and secure communications.
DOE’s 17 National Laboratories will serve as the backbone of a proposed nationwide quantum Internet, which will rely on the laws of quantum mechanics to control and transmit information more securely than ever before. Earlier this year I co-chaired the first Quantum Internet Blueprint Workshop, designed to begin laying the groundwork for this complex undertaking. (Details can be found in the workshop report released July 23.) This meeting was a great first step in articulating what challenges need to be addressed in order to create a quantum Internet with a coordinated research roadmap. The diversity of attendance at the meeting itself was an example of how such grand challenges can be tackled with collaboration across DOE national labs, universities, industry, and various government entities, including NASA, NIST, NSF, and others.
The workshop explored the specific research and engineering advances needed to build a quantum Internet in the near term, along with the path to move from today’s limited tabletop experiments to a viable, metro-to-wide-area, secure quantum Internet. Participants identified four priority research opportunities and five key milestones that must be achieved to build the foundation for a quantum network:
Provide the foundational building blocks for a quantum Internet
Integrate multiple quantum networking devices
Create repeating, switching, and routing for quantum entanglement
Enable error correction of quantum networking functions
Verification of secure quantum protocols over fiber networks
Inter-campus and intra-city entanglement distribution
Intercity quantum communication using entanglement swapping
Interstate quantum entanglement distribution using quantum repeaters
Build a multi-institutional ecosystem to transition from demonstration to operational infrastructure
What role will a high-speed, high-performance, dedicated science research network like ESnet play in facilitating the adoption of quantum technologies?
ESnet is a DOE user facility that connects all of the DOE national labs with a high-speed “classical” network built over leased fiber pairs that span more than 15,000 miles nationwide. In order to build an operational and manageable quantum network, we need to research and build new quantum devices and build new protocols and control systems to integrate, control, manage, and monitor those devices and systems to achieve end-to-end communications. While the classical networks will not have access to the secure data being carried by the quantum channel, using the classical network for control and management is critical to a quantum network’s operational success.
ESnet not only brings the existing nationwide infrastructure and connections to National Labs, it also brings its operational expertise and protocol knowledge to work hand-in-hand with the quantum physicists, scientists, and device and system manufacturers to ensure the right mechanisms are in place to realize DOE’s vision of a quantum Internet.
What research is ESnet helping to enable, and what more still needs to happen to make long-distance quantum communications a practical reality?
Just like the first modems developed in the 1960s leveraged the well-established copper-based telephone network to connect teletype terminals and send data at 110 bits per second (compared to 800 gigabits-per-second modems over fiber today), quantum networking technologies are at an early stage of development. Most current quantum networking research uses photons over either fiber or free-space as the preferred medium of transmission.
At this point, there are many significant investments in small but highly collaborative quantum networking research and prototype deployments in different areas of the country, including Brookhaven/Stony Brook, MIT/Harvard/Lincoln Labs, ANL/University of Chicago, FermiLab/Caltech/JPL/Northwestern, Oak Ridge National Laboratory, and more.
As researchers across the nation build and test fundamental technologies needed to build the quantum internet, ESnet is helping build the infrastructure and provide support to those projects. For example, ESnet engineers worked with researchers at Brookhaven and Stony Brook to test quantum entanglement across the Stony Brook campus leveraging existing ESnet fiber pairs, achieving long-distance entanglement of 18 km using unique quantum entanglement sources and an existing ESnet communications fiber network. Since that initial work, Stony Brook and Brookhaven have established an 80-mile quantum network testbed. ESnet is also working closely with Stony Brook, FERMI, ANL, and Caltech researchers to see how we can support the infrastructure build to expand the reach of their quantum networking research, and we are in conversation with other exploratory testbed projects.
One of the key building blocks to scale these regional demonstrations is the quantum repeater. The repeater is an essential piece of the quantum network that will enable transmission of quantum information across large distances. Many of the testbeds mentioned above are aggressively building breadboard versions of the quantum repeater, competing with other nations to create a first viable repeater system that can be deployed widely. Each of them have a different scientific approach to the problem, for example, the use of quantum memories, and that diversity of research ideas at this stage is extremely important for us to find the right solution that will scale.
What do you see as the next steps to realizing the goal of building a nationwide quantum Internet?
The blueprint report describes in detail the five key milestones that will demonstrate progress toward the ultimate goal of building a nationwide quantum Internet. Here I provide my own perspective on the next steps that will help us realize this goal.
First, we need capable quantum network devices that blend the quantum protocols with classical control. In addition to the grand challenge of building a deployable quantum repeater, an ecosystem of quantum devices from efficient quantum memory; transducers for quantum sources; high-speed, low-loss quantum switches; and much more are needed. Many research labs across the U.S. are working on these technologies, and the first big milestone will be to take these devices, and reliable quantum entanglement/distribution, from laboratory-level readiness to acceptable field-level readiness. Following the “team science” philosophy of Berkeley Lab, this step will not just involve the researchers and physicists but will require collaboration with engineers who have experience deploying and managing components in the field.
Second, once we have deployable and supportable components, we need to gain experience running and operating these devices. It is wonderful that Brookhaven, Fermi, Argonne, UChicago, Northwestern, Oakridge, Caltech, MIT, and others have built or are planning to build capable free-space and/or fiber-based regional testbeds. We can also think about expanding these testbeds using existing dark fiber from Berkeley to SLAC to Caltech, or Brookhaven/Stonybrook to MIT/Harvard, or the Argonne/Fermi/Chicago regional testbed to Oakridge as the capabilities of the devices expand.
In addition to physical devices, quantum entanglement, and teleportation techniques, the classical networking protocols and techniques to control, manage, and operate the quantum network are extremely important. At this stage of development, it is critical to let a thousand flowers bloom. ESnet, with its practical experience, can help design testbeds, connect these testbeds to quantum applications, and support end-to-end tests to help the researchers focus on the techniques that are most viable and easily scalable across the nation. This principle of co-design has been impactful across the DOE Office of Science projects and I hope will be applied to the quantum Internet efforts as well.
Finally, I remember working on one of the leading multi-protocol routers in the 90s with protocols like X.25, Appletalk, IPX, OSI, and many others that have now faded away. With the many approaches to quantum routing and potential protocols to control these devices, we will need a testbed that allows neutral testing of these various research approaches and maybe even determines the interoperability of the various free-space, satellite, and fiber-based techniques. We will also need to build strong collaborations not only between the DOE labs, quantum centers, and the science applications, but also with other agencies like NASA, NIST, NSF, and others that have investments in this space. This is the only way we can leverage the knowledge and expertise of the broader scientific community to reach the vision outlined in the Quantum Internet Blueprint report.
Interview by Kathy Kincade, Berkeley Lab Computing Sciences
5G is the next-generation wireless network that will give you much faster Internet connections. That means massive files, like high-definition movies, that take you about six minutes to download over a 4G LTE network, could be downloaded in a matter of seconds over the 5G network. And because of its innovative design, 5G is about to change the way things like cars, TVs, and even buildings connect to the Internet.
The Department of Energy’s national labs, sponsored by the Office of Science, are currently working to identify opportunities on how science can leverage 5G and other advanced wireless technologies. The Office of Science recently published a report on its findings.
ESnet Computer Systems Engineer Andrew Wiedlea helped facilitate discussions and report findings. We recently caught up with him to talk about the benefits of 5G and other advanced wireless technologies for science, and what it will take to make it available for research.
What is 5G and Advanced Wireless? And, how could science benefit from it?
Scientific data movement is on the cusp of a new era for flexible, low-cost deployment of scientific sensors and data mobility. Advanced wireless capabilities offer the promise of solving the “last mile problem” for science, which is creating new ways for scientists to connect data from sensors, vehicles, and isolated locations, with U.S. Department of Energy’s world-class supercomputers. It’s important to note that advanced wireless will not replace high speed scientific optical networks for large-scale wired “backbone” connectivity, rather we will solve the last mile problem through the integration of advanced wireless- and wired- backhaul.
5G technology is one part of this emerging wireless data connectivity era. In addition to emerging low-orbit satellite constellation non-terrestrial networks, terrestrial millimeter wireless (mmWave), 5G “New Radio” capabilities will be deployed both by commercial vendors and non-commercial entities (using open parts of the radio frequency spectrum-space) to support myriad uses. Because 5G operates over a very wide range of radio frequencies (600 MHz to 27 GHz) and also leverages advances made since the deployment of earlier cellular radio communication standards, such as software defined networking, beam steering, and improved signal processing, 5G will allow users (including the scientific community) to engineer wireless data transmission supporting novel sensing applications for the world around us.
What makes 5G different from previous wireless standards for science?
5G is built around three standards, each of which leverages network resources in different ways. Each of these application models will be leveraged by scientists depending on their needs:
Enhanced Mobile Broadband: The main benefit of 5G comes from a great increase in the ability to spatially reuse the radio spectrum. In comparison to previous cellular network standards, 5G networks will support higher data rates, and an ability to support many more subscribing devices wherever this is needed. For scientists, this will mean much improved options for sensor networks, Internet of Things (IoT) applications, lower wireless data costs, and (hopefully) less reliance on “sneakernet” or other improvised methods for data collection and movement.
Ultra Reliable and Low Latency Communications: 5G supports deployment modes based around defined service levels, which means users will be able to reserve “slices” of capacity in a way similar to reserving circuits on a wired network. This, combined with other capabilities, will allow 5G to support scientific uses where communications reliability is essential, such as when measurements depend on near-real-time interaction with instrument control systems or as part of operating mobile systems such as unmanned aerial vehicles.
Massive Machine Type Communications: 5G is also built to support deployment modes in support for low power, automated systems. These capabilities will be of benefit for all kinds of urban applications, but particularly so for scientists leveraging 5G for urban or building applications. Leveraging this standard, scientists will be able to deploy hundreds or even thousands of small, very power-efficient, sensors throughout buildings or other areas to measure energy or environmental factors.
Taken as a whole, the capabilities provided by advanced wireless (5G, non-terrestrial networks, and mmWave) will allow new kinds of science, both within the confines of the laboratory and outside in a world via commercial and national laboratory dense sensor networks. Both the types and amounts of data generated will greatly increase – as will the scientific opportunities to learn new things.
What role is ESnet playing in creating a 5G network for scientists?
ESnet’s mission is to ensure that science collaborations—at every scale and in every scientific domain—have the information and tools they need to achieve maximum benefit from global networks. This mission is not defined by a particular technology. ESnet works to integrate the compute, storage, and analytic resources operated by sites within the Department of Energy complex, and our scientific customers.
Unlike previous generations of sensor or data infrastructure development, such as the Internet, Advanced Wireless and 5G advances are largely occurring without the US National Laboratory system playing lead roles. The challenge for scientific users is primarily one of connecting wireless technology (when needed) into the toolset made available by the Department of Energy to support US and global science objectives.
ESnet inherently must support these customer efforts because we operate the high-speed scientific data network upon which the community depends now, and in the future as next-generation capabilities (ESnet 6) come to life. We are also at the forefront of thinking about next-generation data movement and analytics through leadership roles with the National Science Foundation’s FABRIC program, software defined networking, and other projects supporting the Department of Energy’s future vision for the science laboratory system.
At the Lawrence Berkeley National Laboratory (Berkeley Lab), where ESnet is headquartered, we are working to develop a community of interest on 5G and advanced wireless applications, and have been using this as a forum to develop ideas, and bring in external speakers to provide technical talks on 5G state of the art.
ESnet’s Science Engagement Team is also starting to work with the Applied Physics Program and others to test aspects of advanced wireless technology, as well as how we can connect this to ESnet work in edge computing, our ScienceDMZ architecture, and other Berkeley Lab resources. We have also started to develop research relationships with the UC Berkeley advanced wireless community, especially the Wireless Research Center to explore mmWave capabilities. Outside of Berkeley Lab, we have been very active in the Department of Energy’s Enabled Energy Innovation Workshop (5GEEIW) and related discussions for science uses of 5G and future requirements, as well as discussions with other labs and commercial entities about collaboration on testbeds and prototyping use cases. These efforts will grow over the next year and hopefully, the report just released from the 5GEEIW will contribute to this progress. [link here]
Are there any experiments looking to use 5G?
Around the Department of Energy complex, many teams are starting to look at the use of 5G to support experiments. There are also developing applications for inside building and laboratory use as well using unlicensed 5G spectrum—some of this application space is now served by either Wifi or wired connectivity. There is a need for some general networking research to explore how ESnet wired capabilities, such as caching and data transfer nodes, should be deployed as part of wired-wireless interfaces, and to develop patterns for scientific support for projects making use of advanced wireless technologies as part of ESnet support for science.
We, along with Argonne National Laboratory, Pacific Northwest National Laboratory, and other Labs, are developing ideas for 5G/Advanced Wireless testbed and prototype application testing environments. At present, the availability of equipment and service is limited, but this is expected to change rapidly as the first generation of 5G handsets and other devices begin to flood the market.
What is the state of 5G now? How long will it be until scientists can access it?
5G is being commercially rolled out by carriers now, and the build-out of this service is expected to take several years. Other resources, such as IoT 5G toolsets and hardware are also beginning to reach the market from Ericsson and other vendors. Similarly, non-terrestrial network constellations such as StarLink are beginning to support limited communities of beta-testers, and mmWave resources are also becoming commercially available.
Thanks to Berkeley Lab IT’s stellar work with Verizon, however, we hope that there will be options over this next year for Berkeley-community access to 5G testing resources, and similar opportunities to explore mmWave or non-terrestrial networks tools as we build relationships and capabilities. We also believe that opportunities and resources will start to become available over this next year from the Department of Energy, and other funding sources to support science user testing and the uptake of advanced wireless.
How did you get into this work and what do you enjoy most about it?
I got into this area at the start of my career working on satellite mobile telephony, and later with the Department of Defense working on sensors and analysis systems. When I was supporting military forces in the field with analytics, the problem was always how to handle really data thin-pipes, and as part of this, we had a lot of trouble with existing radio, cellular and satellite options.
As part of the research-support community, I’m most interested in how we can use 5G and advanced wireless technologies to allow scientists to do new things. It is really fascinating to be at a point of inflection, for RF wireless technology and the ability for almost anyone to be able to affordably collect data from the world, backhaul that data globally, and make sense of it.
I think that we are in a great position to lead the way with open science “out in the world” which will leverage these new technologies and ESnet is a wonderful place to serve that cause.
Interviewed by Linda Vu, Berkeley Lab Computing Sciences
Todd Anderson may be new to his role as the Energy Sciences Network’s (ESnet) Director of Systems and Software Engineering, but he’s already made history.
Anderson is the first ESnet staff member to be hired and onboarded completely virtually. And because of the Bay Area’s extended shelter-in-place order, he will be spending his first months on the job managing his team remotely from home in Lafayette, California.
Before coming to ESnet, he spent 20 years working at the executive level of a technology company that provided software-as-a-service (SaaS) solutions to financial institutions, including services in account identity risk management, fraud prevention, and digital payments, specifically the Zelle payment platform.
“It was a good gig. We were preventing fraud, but at the end of the day, our mission was to help big banks optimize their bottom line,” said Anderson. “I have an engineering background and I wanted to do something a little more meaningful. So, I looked for organizations around the Bay Area working in the fields of sustainability, cleantech, and renewable energy. That’s when I saw this opportunity at ESnet.”
At ESnet, he gets to apply his experience to manage the teams that develop and deploy tools to allow scientific users to optimize their use of the network.
“It’s really exciting to hear about the science projects I will be supporting as a member of the ESnet staff,” said Anderson. “And ESnet is doing its own networking research, too. It’s cool to be on a conference call and hear people talking about quantum networking and 5G.”
As a child during the space race, Anderson remembers watching the Apollo and space shuttle missions. That experience inspired him to major in mechanical engineering at the University of Colorado, Boulder, so he could one day build spacecraft for NASA. But after graduation, life took a different turn. Following in the footsteps of a friend he admired, he joined the Peace Corps and spent two years teaching math and science to middle school children in Botswana’s Kalahari Desert.
“Service really makes you feel like you are doing something worthwhile and making the world a better place,” said Anderson.
When he returned to the United States, a friend asked him to help write a software application to detect merchant credit card fraud. This move kicked-off of his 30-year career in technology.
In his free time, Anderson enjoys doing things around the house. When the world isn’t in the midst of a pandemic, he likes to be out in nature and to sample the diverse culture and geography of the Bay Area with his family.
Nearly two months into California’s shelter-in-place order, we’ve all been in more than our fair share of video conferences. To boost morale during this difficult time, the Energy Sciences Network (ESnet) staff held a Zoom Background Competition during their all-to-all meeting on Monday, April 27.
Staff were encouraged to create their own backgrounds and display them during the meeting. There were 21 entries. ESnet employees voted. Submissions were judged on overall artistry, functionality (not too distracting as a background), whether it elevated the voter’s mood, and if it made them feel included in the ESnet community.
The top three winners got bragging rights. Here they are:
First place: Jeff Berman, NOC Engineer
This Zoom challenge inspired Berman, an avid sailor, to take to the sea. He won this competition with an hour video of the San Francisco skyline, one he filmed while sailing on the Bay. Although he typically likes to go sailing with friends and family, he says that sailing solo brings him a sense of peace, calm, and tranquility.
“What is sailing? Most books define it as hours of sheer boredom scattered with white knuckle periods of terror. On a good day, both are true. Both give you an equal sense of accomplishment. How to be with yourself with nothing to do, good training for our current situation,” said Berman.
Second Place: Sartaj Baveja, Software Engineer
This challenge inspired Baveja to create a background meme of office life. In the background, someone (Baveja) is looking over your shoulder to catch a glimpse of your screen and make sure you don’t procrastinate.
Third Place: Joe Metzger, Network Engineer
This challenge inspired Metzger to use a picture that he took in Barcelona. The focal point of the picture (the blur) is a little girl in a red coat, black dress and white tights who was just running back and forth between the pools of light and shadow created by the stone arches and rosette windows, while her family was sitting in the cafe.
“I used this as my zoom background because I think it is a really cool picture. It brings to mind a fun evening strolling around the little squares and back streets in Barcelona and sitting in cafes with a good glass of wine relaxing,” said Metzger.
Written by Linda Vu, Berkeley Lab Computing Sciences.
A network diagnostic and performance measurement tool developed by engineers at the U.S. Department of Energy’s (DOE’s) Energy Sciences Network (ESnet) and Lawrence Berkeley National Laboratory (LBNL) is being used by Comcast to fine-tune the largest residential Internet network in the United States and help ensure its services remain up and running at top speed during the COVID-19 crisis.
“We are seeing an unprecedented shift in network usage, but it’s within the capability of our network,” Comcast states on its COVID-19 Network Update webpage.
Prompted by shelter-in-place orders across the U.S. and the world in recent months, extraordinary demand is being placed on residential Internet service providers (ISPs) as people increasingly rely on home-based Internet connections for entertainment, education, shopping, and work.
ESnet’s open-source iperf3 tool is helping Comcast meet this challenge by giving them the ability to make timing and buffering changes across their network in real-time. Originally developed in 2009 as part of the perfSONAR toolkit, iperf3 is designed to measure the available network bandwidth between two hosts on an IP network. It supports tuning of features related to timing, protocols, and buffers; for each test, it reports the measured throughput, loss, and other parameters.
At present, Comcast – which is seeing a 30% uptick in network traffic since the country’s shelter-in-place orders took effect – is using iperf3 to run 700,000+ diagnostic speed tests per day. This helps them engineer the network for peak capacity and better handle spikes and shifts in usage patterns.
“Comcast is one of the largest ISPs in the world, and they are using iperf3 – part of their normal troubleshooting workflow – to make sure their network is delivering the performance that is required in this situation,” said Bruce Mah, a software engineer in ESnet’s Software Engineering Group who works with Comcast as part of their open-source relationship.
ESnet, a DOE Office of Science user facility managed by LBNL, is the fastest network in the world dedicated to science. It supports a multi-100Gbps fiber optic backbone that connects the DOE’s national laboratory system and experimental facilities with research and commercial networks around the globe.
While people around the world hunker down in their homes to try to slow the advance of the COVID-19 virus and many services have decreased or stopped, two user facilities operated by the U.S. Department of Energy’s Office of Science continue to provide critical computing and networking resources to thousands of scientists, including some who are exploring ways to fight the pandemic.
The National Energy Research Scientific Computing Center (NERSC) and the Energy Sciences Network (ESnet) are managed by Lawrence Berkeley National Laboratory, which has reduced operations and onsite staffing under state-wide shelter-in-place orders. But NERSC and ESnet, deemed to provide essential services to the nation, continue to support “science as usual” as staff remotely manage the facilities from their homes.
NERSC has been named to the COVID-19 High Performance Computing Consortium. Led by the White House Office of Science and Technology Policy, industrial partners, and DOE, the consortium will give researchers access to supercomputers at DOE’s Argonne, Lawrence Livermore, Los Alamos, Oak Ridge, and Sandia national laboratories. ESnet will provide robust, high-bandwidth connections and peerings allowing scientists to tap into these computing resources and move data from across the world to those sites for analysis.
With an eye on pandemic-related research, NERSC staff have set up dedicated priority queues to run COVID-19-related research projects on a supercomputer. In one project, scientists at the Beckman Research Institute at the City of Hope are running molecular dynamic simulations that apply to a range of COVID-19 research areas. In particular, they are looking at the difference between the Chinese and Italian strains of the virus as well as potential antiviral treatments.
“It’s very challenging for everyone, it’s unprecedented,” said NERSC Division Director Sudip Dosanjh. “Our staff are very dedicated, and I think this also shows their passion for the science mission of NERSC, ESnet, and the laboratory.”
In fact, shelter-in-place policies across the country appear to be fostering even greater demands on the supercomputers at NERSC. With travel plans and conferences delayed or canceled, many of NERSC’s more than 7,000 users are spending their time at home but still want to advance their research by running projects on the center’s systems, Dosanjh surmised.
“Thanks to the dedicated efforts of NERSC personnel to keep computing systems running and supporting users’ requests, our ‘computing lab’ (NERSC) remains open and operational at full capacity,” said Manos Mavrikakis, a NERSC user and distinguished professor at the University of Wisconsin-Madison whose work focuses on understanding catalytic process principles and the discovery of new materials that would enable more efficient energy production. “As a result, we have been able to continue pursuing our research on catalytic reaction mechanisms, pretty much at the same pace as before coronavirus dominated everybody’s lifestyle. We are enormously grateful to NERSC personnel for an excellent job under highly stressful conditions.”
“We recognize the importance of that and are seeing that the utilization of Cori, our primary computer, is at 97 percent, an all-time high,” Dosanjh said. “A lot of other people can’t do their work unless we do our job, and I couldn’t be more proud of our staff.”
Cori, a Cray XC40 supercomputer able to perform nearly 30 quadrillion calculations per second, is used to create detailed models of scientific problems and analyze massive amounts of data from experimental facilities operated by DOE.
ESnet provides the critical high-bandwidth connection between tens of thousands of researchers at national labs, universities, user facilities and supercomputer centers like NERSC. ESnet operates a dedicated multi-100-gigabits-per-second network that crisscrosses the country and has four similar links crossing the Atlantic Ocean for collaborations in Europe. Almost all network traffic passing to and from DOE laboratories traverses the network.
Although ESnet’s operations center is in Berkeley, about 40 percent of the staff live in other states across four time zones and are used to working offsite. The network operates 24 hours a day, 365 days a year, enabling scientists to seamlessly access data portals, transfer massive research data sets, and tap into remote scientific instruments — all in real-time from anywhere.
The dispersed staff are both closer to other facilities and bring different perspectives to solving network issues, said Tony Ferrelli, head of ESnet’s Network Engineering and Operations Team. With so many people across the country working from home, ESnet has seen a blip in traffic moving onto the network, Ferrelli said, but there is still bandwidth to spare. One interesting note is that with more people working from home, they are finding that their home network connections are much slower than expected, which is compounded by increased demand, Ferrelli said.
Network staff is also on hand to help researchers should they need help to manage the large datasets that are typical of DOE science, ESnet Director Inder Monga said.
“It’s all about the people – those of us whose job is to provide these resources and those who tap into them to support our nation’s scientific leadership,” Monga said. “With all these efforts, science is proceeding as usual.”
By connecting with other research and education networks, ESnet is providing a critical link for scientists and consortium members like those from COVID-19 High Performance Computing Consortium with DOE supercomputer centers, thereby supporting research efforts into the COVID-19 pandemic.
FABRIC, a project funded by the National Science Foundation, announces the formation of a Scientific Advisory Committee (SAC) tasked with facilitating collaboration and providing scientific and technological review for the project. FABRIC will create a unique national research infrastructure for testing novel architectures aimed at building an extensible, more secure Internet.
With leadership from the Renaissance Computing Institute (RENCI) at the University of North Carolina at Chapel Hill, the FABRIC project will build a large-scale platform with storage, computational and network hardware nodes across the country that are connected by dedicated high-speed optical links. FABRIC will also link major national research facilities such as universities, national labs and supercomputing centers that generate and process enormous scientific data sets.
The SAC will help guide the project by providing recommendations and critical feedback. Initially, the focus will be on reviewing the FABRIC design to ensure it can meet the diverse research needs of the future. The committee will also facilitate critical partnerships between collaborating institutions both within and outside of the US. As work progresses, the SAC will develop grand challenges that focus on solving key research problems using the FABRIC infrastructure.
“We are excited to have key research leaders across diverse career stages in fields such as networking, computing, software and security as our Scientific Advisory Committee,” said Inder Monga, co-PI of the FABRIC project and executive director of the U.S. Department of Energy’s Energy Sciences Network at Lawrence Berkeley National Laboratory. “The work is progressing well with FABRIC, and we look forward to the committee’s guidance on building an infrastructure that can facilitate testing of radical new ideas and approaches that will help lay the groundwork for the future Internet.”