New Ground-Shaking Science with Dark Fiber

As a Network Engineer at ESnet, I am no stranger to the importance of designing and maintaining a robust fiber-optic network. To operate a network that will “enable and accelerate scientific discovery by delivering unparalleled network infrastructure, capabilities, and tools,” ESnet has acquired an impressive US continental footprint of more than 21,000 kilometers of leased fiber-optic cable. We spend a great deal of effort designing and sourcing redundant fiber-optic paths to support network data connectivity between scores of DOE Office of Science facilities and research collaborators across the country.

But network data transfer is only one of the uses for fiber-optic cable. What about using buried fiber-optic cable for some truly “ground-shaking” science? The answer is “Yes, absolutely!” – and I was fortunate to play a part in exploring new uses for fiber-optic cable networks this past year.

Back in 2017, the majority of our 21,000 km fiber footprint was still considered “dark fiber,” meaning it was not yet in use. At that time, ESnet was actively working on the design to upgrade from our current production network “ESnet5” to our next-generation network “ESnet6,” but we hadn’t yet put our fiber into production.

At the same time, Dr. Jonathan Ajo-Franklin, then graduate students Nate Lindsey and Shan Dou, and the Berkeley Lab’s Earth and Environmental Science Area (EESA) were exploring the use of distributed acoustic sensing (DAS) technology to detect seismic waves by using laser pulses across buried fiber optic cable. The timing was perfect to try and expand on the short-range tests that Dr. Ajo-Franklin and his team had been performing at the University of California’s Richmond Field Station by using a section of the unused ESnet dark fiber footprint in the West Sacramento area for more extensive testing. ESnet’s own Chris Tracy worked with Dr. Ajo-Franklin and team to demonstrate how the underground fiber-optic cables running from West Sacramento northwest toward Woodland in California’s Central Valley made an excellent sensor platform for early earthquake detection, monitoring groundwater, and mapping new sources of potential geothermal energy.

The Sacramento ESnet Dark Fiber Route (left) and seismic events recorded on the array from around the world including the massive M 8.1 earthquake in Chiapas, Mexico.

Fast forward to May 2019, and Dr. Ajo-Franklin was heading up a new collaborative scientific research project for the DOE’s Geothermal Technology Office based on his prior DAS experimentation successes using ESnet fiber. The intent was to map potential geothermal energy locations in the California Imperial Valley south of the Salton Sea, near Calipatria and El Centro. The team, including scientists in EESA, Lawrence Livermore National Laboratory (LLNL), and Rice University needed a fiber path to conduct the experiment. It would make sense to assume that ESnet’s fiber footprint, which runs through that area, would be an excellent candidate for this experiment. Fortunately for ESnet’s other users, but unfortunately for the DAS team, by 2018 the ESnet6 team was already “lighting” this previously dark fiber. 

However, just because ESnet fiber in the Imperial Valley was no longer a candidate for DAS-based experiments, that didn’t mean there weren’t ways to gain access to unused dark fiber. For every piece of fiber that has been put into production to support ESnet6, there are dozens if not hundreds of other fibers running right alongside it. When fiber-optic providers install new fiber paths, they pull large cables consisting of many individual fibers to lease or sell to as many customers as possible. Because the ESnet fiber footprint was running right through the Imperial Valley, we knew that there was likely unused fiber in the ground, and only had to find a provider that would be willing to lease a small section to Berkeley Lab for Dr. Ajo-Franklin’s experiment. 

Making the search a little more complicated, the DAS equipment utilized for this experiment has an effective sensing range that is limited to less than 30 kilometers. Most fiber providers expect to lease long sections of fiber connecting metropolitan areas. For example, the fiber circuits that run through the Imperial Valley are actually intended to connect metropolitan areas of Arizona to large cities in Southern California. Finding a provider that would be willing to break up a continuous 600 km circuit connecting Phoenix to Los Angeles just to sell a 30 km piece for a year-long research project would be a difficult task.  

One of my contributions to the ESnet6 project was sourcing new dark fiber circuits and data center colocation spaces to “fill out” our existing footprint and get ready for our optical system deployments. Because of those efforts, I knew that there were often entire sections of fiber that had been damaged across the country and would likely not be repaired until there was a new customer that wanted to lease the fiber. I was asked to assist Dr. Ajo-Franklin and his team to engineer a new fiber solution for the experiment. I just had to find someone willing to lease us one of these small damaged sections.

After speaking with many providers in the area, the communications company Zayo was able to find a section of fiber starting in Calipatria, heading south through El Centro and then west to Plaster City, that was a great candidate for DAS use. This section of fiber had been accidentally cut near Plaster City and was considered unusable for networking purposes. Working with Zayo, we were able to negotiate a lease on this “broken” fiber span along with a small amount of rack space and power to house the DAS equipment that Dr. Ajo-Franklin’s team would need to move forward with their research.  

The Imperial Valley Dark Fiber Array: (A) Team Co-PI Veronica Rodriguez Tribaldos (LBNL) turning on the DAS system. (B) The ILA used to house the equipment in Calipatria. (C) The Zayo fiber section currently being used in the experiment. (D) The corresponding DAS data showing a magnitude 2.6 earthquake located near the Salton Sea, to the north. 

This cut fiber segment was successfully “turned up” for the project on November 10, 2020 by a team including Co-PI Veronica Rodriguez Tribaldos, Michelle Robertson, and Todd Wood (EESA/LBNL), and seismic data collection equipment is now up and running. The figure above (D) shows some great initial data recorded on the array, a small earthquake many miles to the north. There will be many more articles and reports from the Imperial Valley Dark Fiber Team as they continue to gather data and perform their experiments, and I’m sure we’ll begin to see fiber across the country put to use for this type of sensing and research.

I’ve had a great experience working with the different groups that were assembled for this project. By seeing how new technologies and methods are being developed to use fiber-optic cable for important research outside of computing science, I’ve developed a greater appreciation for how our labs and universities are tackling some of our biggest energy and public safety challenges.

100G DTN Experiment: Testing Technologies for Next-Generation File Transfer

ESnet has recently completed an experiment testing high-performance, file-based data transfers using Data Transfer Nodes (DTNs) on the 100G ESnet Testbed. Within ESnet, new ways to provide optimized, on-demand data movement tools to our network users are being prototyped. One such potential new data movement tool is offered by Zettar, Inc. Zettar’s “zx” product integrates with several storage technologies with an API for automation. This ESnet data movement experiment allowed us to test the use of tools like zx on our network. 

Two 100Gbps capable DTNs were deployed on the ESnet Testbed for this work, each with 8 x NVMe SSDs for fast disk-to-disk transfers, and connected using an approximately 90ms round trip time network path.  As many readers are aware, this combination of fast storage and fast networking requires careful tuning from both a file I/O and network protocol standpoint to achieve expected end-to-end transfer rates, and this evaluation was no exception. With the help of a storage throughput baseline achieved using the freely available elbencho tool, a single tuning profile for zx was found that struck an impressive performance balance when moving a sweep of hyperscale data sets (>1TB total size or >1M total files or both, see figure below) between the testbed DTNs.

A combined line chart with the measured storage throughput for each file size (blue line), together with both the Zettar zxtransfer data rates attained with a single run carried out by Zettar (orange line), and the average of five runs carried out by ESnet (green line)

To keep things interesting, the DTN software under evaluation was configured and launched within Docker containers to understand any performance and management impacts, and to establish a potential use case for more broadly deploying DTNs as-a-Service using containerization approaches. Spoiler: the testing was a great success! When configured appropriately, our evaluation has shown that modern container namespaces using performance-oriented Linux networking impart little to no impact on achievable storage and network performance at the 100Gbps scale while enabling a great deal of potential for distributed deployment of DTNs.  More critically, the problem of service orchestration and automation becomes the next great challenge when considering any large-scale deployment of dynamic data movement endpoints.

Our takeaways:

  • When properly provisioned and configured, a containerized environment has a high potential to provide an optimized, on-demand data movement service.
  • Data movers such as zx demonstrate that when modern TCP is used efficiently to move data at scale and speed, network latency becomes less of a factor – the same level of data rates are attainable over LAN, Metro, and WAN as long as packet loss rates can be effectively kept low
  • Finally, creating a holistic data movement solution demands integrated consideration of storage, computing, networking, and highly concurrent and intrinsically scale-out data mover software that incorporates a proper understanding of the variety in data movement scenarios.

For more information, a project report detailing the testing environment, performance comparisons, and best practices may be found here

40G Data Transfer Node (DTN) now Available for User Testing!

ESnet’s first 40 Gb/s public data transfer node (DTN) has been deployed and is now available for community testing. This new DTN is the first of a new generation of publicly available networking test units, provided by ESnet to the global research and engineering network community as part of promoting high-speed scientific data mobility. This 40G DTN will provide four times the speed of previous-generation DTN test units, as well as the opportunity to test a variety of network transfer tools and calibrated data sets.

The 40G DTN server, located at ESnet’s El Paso location, is based on an updated reference implementation of our Science DMZ architecture. This new DTN (and others that will soon follow in other locations) will allow our collaborators throughout the global research and engineering network community to test high speed, large, demanding data transfers as part of improving their own network performance. The deployment provides a resource enabling the global science community to reach levels of data networking performance first demonstrated in 2017 as part of the ESnet Petascale DTN project

The El Paso 40G DTN has Globus installed for gridFTP and parallel file transfer testing. Additional data transfer applications may be installed in the future. To facilitate user evaluation of their own network capabilities ESnet Data Mobility Exhibition (DME), test data sets will be loaded on this new 40G DTN shortly. 

All ESnet DTN public servers can be found at https://app.globus.org/file-manager. ESnet will continue to support existing 10G DTNs located at Sunnyvale, Starlight, New York, and CERN. 

ESnet's 40G DTN Reference Architecture Block Diagram
ESnet’s 40G DTN Reference Architecture Block Diagram

The full 40G DTN Reference architecture and more information on the design of these new DTN can be found here:

A second 40G DTN will be available in the next few weeks, and will be deployed in Boston. It will feature Google’s bottleneck bandwidth and round-trip propagation time (BBR2) software, allowing improved round-trip-time measurement and the ability for users to explore BBR2 enhancements to standard TCP congestion control algorithms.

In an upcoming blog post, I will describe the Boston/BBR2-enabled 40G DTN and perfSONAR servers. In the meantime, ESnet and the deployment team hope that the new El Paso DTN will be of great use to the global research community!  

Re-imagining perfSONAR to gain new network insights

Scientific discovery increasingly relies on the ability to perform large data transfers across networks operated by many different providers (including ESnet) around the globe. But what happens when a researcher initiates one of these large data transfers and data movement is slow? What does “slow” even mean? These can be surprisingly complex questions and it is important to have the right tools to help answer them. perfSONAR is an open source software tool designed to measure network performance and pinpoint issues that occur as data travels across many different networks on the way to a destination.

perfSONAR has been around for more than 15 years and is primarily maintained today by a collaboration of ESnet, GEANT, Indiana University, Internet2, and the University of Michigan. perfSONAR has an active community that extends well beyond the five core organizations that maintain the software with more than 2000 public deployments that span six continents and hundreds of organizations. perfSONAR deployments are capable of scheduling  and running tests that calculate metrics including (but not limited to) how fast a transfer can be performed (throughput), if a unit of information makes it to a desired destination (packet loss), if so how long did it take (latency) and what path did it take to get there (traceroute). What is novel about perfSONAR is not just these metrics, but the set of tools to feature these metrics in dashboards built by multiple collaborating organizations.  These dashboards aim to clearly identify patterns that signify potential issues and provide the means to drill-down into graphs that give more information.

Example perfSONAR dashboard grid highlighting packet loss to an ANL test node (top). Example line graphs that further illustrate aspects of the problem (bottom).

While perfSONAR has had great success in providing the current set of capabilities, there is more that can be done. For example, perfSONAR is very good at correlating metrics it collects with the other perfSONAR metrics with at least one similar endpoint. But what if we want to correlate the metrics by location, intermediate network or with non-perfSONAR collected statistics like flow statistics and interface counters? These are all key questions the perfSONAR project is looking to answer. 

Building upon a strong foundation

PerfSONAR has the ability to add analytics from other software tools using a plug-in framework. Recently, we have begun to use Elastic Search via this framework, to ingest log data and enable improved search and analytics on perfSONAR data.

For example, traditionally perfSONAR has viewed an individual measurement as something between a pair of IP addresses. But what do these IP addresses represent and where are they located? Using off-the-shelf tools Elastic Search in combination with Logstash, perfSONAR is able to answer questions like “What geographic areas are showing the most packet loss?”.

Example map showing packet loss hotspots to different locations around the globe. It also contains a menu to filter results by intermediate network.

Additionally, we can apply this same principle to traceroute (and similar tools) that yield a list of IP addresses giving an idea of the path a measurement takes between source and destination. Each IP address is a key to more information about the path including not only geographic information but also the organization at each point. This means you can ask questions such as “What is the throughput of all results that transit a given organization?”. Previously a user would not only have to know the exact address of the IPs, but it would have to be the first (source) or last (destination) address in the path. 

Integration with non-perfSONAR data is another area the project is looking to expand. By putting perfSONAR data in a well established data store like Elasticsearch, the door is open to leverage other off-the-shelf open source tools like Grafana to display results. What’s interesting about this platform is not only its ability to build new visualizations, but also the diverse set of backends it is capable of querying. If data such as host metrics, network interface counters and flow statistics are kept in any of the supported data stores, then there is a means to present this information along perfSONAR data. 

Example of perfSONAR statistics combined with host statistics from a completely different database being displayed in Grafana

These efforts are very much still in their early stages of development, but initial indicators are promising. Leveraging the perfSONAR architecture in conjunction with the wealth of off-the-shelf open source tools available on the market today create opportunities to gain new insights from the network, like those described above, not previously possible with the traditional perfSONAR tools. 

Getting involved and learning more

The perfSONAR project will continue to provide updates as this work progresses. You can also see the perfSONAR web site for updates and more information on keeping in touch through our mailing lists. The perfSONAR project looks forward to working with the community to provide exciting new network measurement capabilities.

Zeek and stream asymmetry research at ESnet

In my previous post, we discussed use of the open-source Zeek software to support network security monitoring at ESnet.  In this post, I’ll talk a little about work underway to improve Zeek’s ability to support network traffic monitoring when faced with stream asymmetry.

This comes from recent work by two of my colleagues on the ESnet Security team.

Scott Campbell and Sam Oehlert presented ‘Running Zeek on the WAN: Experiences and solutions for large scale flow asymmetry’ during a workshop held last year at CERN Geneva that explained the phases and deployment of the Zeek-on-the-WAN (ZoW) pilot in detail.

Scott Campbell at CERN presenting ‘Running Zeek on the WAN’
The asymmetry problem on a WAN (example)

Some of the significant findings and results from this presentation are highlighted below:

  • Phase I: Initial Zeek Node Design Considerations 
    • Select locations that provide an interesting network vantage point – in the case of our ESnet network, we deployed Zeek nodes on our commodity internet peerings (eqx-sj, eqx-chi, eqx-ash) since they represent the interface to the vast majority of hostile traffic.
    • Identifying easy traffic to test with and using spanning ports to forward traffic destined to the stub network on each of the routers used for collection.
  • Phase I: Initial Lessons learned from testing and results
    • Some misconfigurations were found in the ACL prefix lists. 
    • We increased visibility into our WAN side traffic through implementation of new background methods.
    • Establishing a new process for end-to-end testing, installing and verifying Zeek system reporting. 
  • Phase II:  Prove there is more useful data to be seen
    • For phase II we moved towards collection of full peer connection records, from statistical sampling based techniques. Started running Zeek on traffic crossing the interfaces which connect ESnet network peers to the internet from the AS (Autonomous system) responsible for most notices. .
    • To get high fidelity connection information without being crushed by data volume, define a subset of packets that are interesting – zero length control packets (Syn/Syn-Ack/Fin/Rst) from peerings.
  • Phase II: Results
    • A lot of interesting activity got discovered like information leakage in syslogs, logins (and attempted logins) using poorly secure authentication protocols, and analysis of the amount of asymmetric traffic patterns gave valuable insights to understand better the asymmetric traffic problems.
  • Ongoing Phase III: Expanding the reach of traffic collection on WAN
    • We are currently in the process of deploying Zeek nodes at another three WAN locations for monitoring commodity internet peering – PNWG (peering at Seattle WA), AM-SIX (peering at Amsterdam) and LOND (peering at London)
Locations for the ZoW systems, the pink shows ongoing Phase III deployment

As our use of Zeek on the WAN side of ESnet continues to grow, the next phase to the ZoW pilot is currently being defined.  We’re working to incorporate these lessons learned on how to handle traffic asymmetry into these next phases of effort. 

Some (not all) solutions being taken into consideration include: 

  • Aggregating traffic streams at a central location to make sense out of the asymmetric packet streams and then run Zeek on the aggregated traffic, or
  • Running Zeek on the individual asymmetric streams and then aggregating these Zeek streams @ 5-tuple which will be aggregation of connection metadata rather than the connection stream itself. 

We are currently exploring these WAN solutions as part of providing better solutions to both ESnet, and connected sites.

Zeekurity at ESnet

Zeek is an open source network security monitoring software extensively used by ESnet.  Zeek (formally called Bro) was initially developed by researchers at Berkeley Lab. Zeek allows users to identify & manage cyber threats by monitoring network traffic. It acts as a passive monitoring software (NSM – Network Security Monitor), that gives a holistic view of what is transpiring in the network and gives visibility into the network traffic. 

In order to better understand network behavior and provide flexible security services, we use Zeek as an important part of our data center security architecture and are experimenting with placing Zeek clusters on various WAN high value points. This is providing technical insights as well as significant challenges. 

In this post we would present some of our efforts in approaching the WAN security using Zeek for network monitoring, with successes and challenges hit during the process and interesting things learned.

Zeek on the ESnet LAN:

Monitoring local area and data center networks is a familiar and less complex network traffic monitoring design, and ESnet is no different. The traffic flowing through the LAN networks is currently monitored using two Zeek clusters, one at Brookhaven National Lab and another for the west coast at Berkeley Lab. We have implemented BHR (black hole routing) functionality on our data center routers to block external actors which violate our established policies based on Zeek detections on both IPv4 and IPv6 protocol stacks. 

Apart from network security monitoring using “standard” Zeek detection scripts, many enhancements and custom scripts written by the ESnet Security team members serve a vital role in detecting various kinds of suspicious activity. Recently, a Zeek package – Zeek-Known-outbound contributed by Michael “Dop” Dopheide won the first prize in the Zeek Package Contest-2 held in May 2020. The package provides the ability to track and alert on outbound service usage to a list of ‘watched’ countries, and also adds the country codes for the origin and recipient hosts in one of the log files that Zeek generates called conn.log, to log all the connection attempts seen on the network. The motivation behind this work came from the discovery of few systems contacting hosts in foreign countries for package updates, and DNS services found during routine log analysis. 

Zeek on the ESnet WAN:

To augment our LAN efforts on a wider scale, we have been experimenting with monitoring the network traffic on the WAN side of the network using Zeek in order to get more visibility and to provide improved security/network services. Most of this work is experimental: iterative design changes as we use what we learn from stage 1 to stage 3 and beyond.

  • Some notable differences and challenges from typical LAN network: 
    • Data Volume: There are a large number of WAN links that run at 1-400Gb/s
    • Data Encapsulation: Data with variable length headers is problematic, so we have been employing a load balancer to address this problem. 
    • Asymmetric Data Flows: This is a hard problem to solve, especially when the network is distributed across the country. When the packets corresponding inbound and outbound flows between two network nodes follow different paths, it can be challenging to reconcile conversation activities as part of network monitoring.
    • Technical Integration: Coordinating activities between teams distributed geographically  introduces challenges, which we are developing ways to overcome.

At ESnet we thrive to push the boundaries and try innovative ways to address challenges, Zeek on the WAN is an example of that and in my next article I will discuss some ways we have been experimenting with to address above noted complex problems and specifically going into details of the research been done in addressing Asymmetric Data Flows on WAN.

New DOE Blueprint to Pave the Way for a Nationwide Quantum Internet

Quantum Internet Blueprint Workshop Steering Committee

DOE Quantum Internet Blueprint Workshop ReportAs modern computers begin to reach the limit of their processing power, quantum computing has the potential to solve more specialized problems that require immensely robust computing. With that potential capturing the imagination of many, a consensus is building that a communication system using quantum mechanics represents one of the most important technological frontiers of the 21st century. To harness the full promise of quantum computing and sensing, we need to build scalable quantum communication networks that can support applications across science, industry, and national security. Scientists now believe that the construction of a prototype “quantum Internet” will be within reach over the next decade.

Toward this end, on July 23 the U.S. Department of Energy (DOE) unveiled a report that lays out a blueprint strategy to accelerate research in quantum science and technology, with an emphasis on the creation of a quantum Internet. “The Department of Energy is proud to play an instrumental role in the development of the national quantum Internet,” U.S. Secretary of Energy Dan Brouillette said in a news release. “By constructing this new and emerging technology, the United States continues with its commitment to maintaining and expanding our quantum capabilities.”

In support of this and related efforts, the Energy Sciences Network (ESnet) – a DOE Office of Science user facility managed by Lawrence Berkeley National Laboratory – is actively tracking multiple quantum networking projects and collaborating with the research community to help lay the groundwork for scalable quantum communication networks and a quantum Internet. In this interview with ESnet Director Inder Monga, he talks about the future of quantum networking and its role in facilitating quantum information science across the DOE and beyond.

How will the DOE’s quantum networking blueprint impact the development of quantum communications and applications?

Quantum Internet Blueprint Workshop Steering Committee
The Quantum Internet Blueprint Workshop chairs. Top row left to right: Inder Monga (ESnet) and Gabriella Carini (BNL). Bottom row left to right: Nicolas Peters (ORNL), Kerstin Kleese van Dam (BNL), Joseph Lykken (Fermilab), Thomas Schenkel (Berkeley Lab).

Researchers believe that a quantum Internet could have a profound impact on a number of application areas critical to science, national security, and industry. Application areas include upscaling quantum computing by connecting distributed quantum computers, quantum sensing through a network of quantum telescopes, quantum metrology, and secure communications.

DOE’s 17 National Laboratories will serve as the backbone of a proposed nationwide quantum Internet, which will rely on the laws of quantum mechanics to control and transmit information more securely than ever before. Earlier this year I co-chaired the first Quantum Internet Blueprint Workshop, designed to begin laying the groundwork for this complex undertaking. (Details can be found in the workshop report released July 23.) This meeting was a great first step in articulating what challenges need to be addressed in order to create a quantum Internet with a coordinated research roadmap. The diversity of attendance at the meeting itself was an example of how such grand challenges can be tackled with collaboration across DOE national labs, universities, industry, and various government entities, including  NASA, NIST, NSF, and others.

The workshop explored the specific research and engineering advances needed to build a quantum Internet in the near term, along with the path to move from today’s limited tabletop experiments to a viable, metro-to-wide-area, secure quantum Internet. Participants identified four priority research opportunities and five key milestones that must be achieved to build the foundation for a quantum network:

  • Research Priorities
    • Provide the foundational building blocks for a quantum Internet
    • Integrate multiple quantum networking devices
    • Create repeating, switching, and routing for quantum entanglement
    • Enable error correction of quantum networking functions
  • Roadmap Milestones
    • Verification of secure quantum protocols over fiber networks
    • Inter-campus and intra-city entanglement distribution
    • Intercity quantum communication using entanglement swapping
    • Interstate quantum entanglement distribution using quantum repeaters
    • Build a multi-institutional ecosystem to transition from demonstration to operational infrastructure

What role will a high-speed, high-performance, dedicated science research network like ESnet play in facilitating the adoption of quantum technologies?

Inder Monga and Eden Figueroa
ESnet engineers worked with researchers at Brookhaven National Laboratory and Stony Brook University to test quantum entanglement across the Stony Brook campus leveraging existing ESnet fiber pairs, achieving long-distance entanglement of 18 km using an existing ESnet communications fiber network. Since that initial work, Stony Brook and Brookhaven have established an 80-mile quantum network testbed.ESnet’s Inder Monga (right) talks with Stony Brook’s Eden Figueroa, lead investigator of the quantum networking testbed project.

ESnet is a DOE user facility that connects all of the DOE national labs with a high-speed “classical” network built over leased fiber pairs that span more than 15,000 miles nationwide. In order to build an operational and manageable quantum network, we need to research and build new quantum devices and build new protocols and control systems to integrate, control, manage, and monitor those devices and systems to achieve end-to-end communications. While the classical networks will not have access to the secure data being carried by the quantum channel, using the classical network for control and management is critical to a quantum network’s operational success.

ESnet not only brings the existing nationwide infrastructure and connections to National Labs, it also brings its operational expertise and protocol knowledge to work hand-in-hand with the quantum physicists, scientists, and device and system manufacturers to ensure the right mechanisms are in place to realize DOE’s vision of a quantum Internet.

What research is ESnet helping to enable, and what more still needs to happen to make long-distance quantum communications a practical reality?

Just like the first modems developed in the 1960s leveraged the well-established copper-based telephone network to connect teletype terminals and send data at 110 bits per second (compared to 800 gigabits-per-second modems over fiber today), quantum networking technologies are at an early stage of development. Most current quantum networking research uses photons over either fiber or free-space as the preferred medium of transmission.

At this point, there are many significant investments in small but highly collaborative quantum networking research and prototype deployments in different areas of the country, including Brookhaven/Stony Brook, MIT/Harvard/Lincoln Labs, ANL/University of Chicago, FermiLab/Caltech/JPL/Northwestern, Oak Ridge National Laboratory, and more.

As researchers across the nation build and test fundamental technologies needed to build the quantum internet, ESnet is helping build the infrastructure and provide support to those projects. For example, ESnet engineers worked with researchers at Brookhaven and Stony Brook to test quantum entanglement across the Stony Brook campus leveraging existing ESnet fiber pairs, achieving long-distance entanglement of 18 km using unique quantum entanglement sources and an existing ESnet communications fiber network. Since that initial work, Stony Brook and Brookhaven have established an 80-mile quantum network testbed. ESnet is also working closely with Stony Brook, FERMI, ANL, and Caltech researchers to see how we can support the infrastructure build to expand the reach of their quantum networking research, and we are in conversation with other exploratory testbed projects.

One of the key building blocks to scale these regional demonstrations is the quantum repeater. The repeater is an essential piece of the quantum network that will enable transmission of quantum information across large distances. Many of the testbeds mentioned above are aggressively building breadboard versions of the quantum repeater, competing with other nations to create a first viable repeater system that can be deployed widely. Each of them have a different scientific approach to the problem, for example, the use of quantum memories, and that diversity of research ideas at this stage is extremely important for us to find the right solution that will scale.

What do you see as the next steps to realizing the goal of building a nationwide quantum Internet?

The blueprint report describes in detail the five key milestones that will demonstrate progress toward the ultimate goal of building a nationwide quantum Internet. Here I provide my own perspective on the next steps that will help us realize this goal.

First, we need capable quantum network devices that blend the quantum protocols with classical control. In addition to the grand challenge of building a deployable quantum repeater, an ecosystem of quantum devices from efficient quantum memory; transducers for quantum sources; high-speed, low-loss quantum switches; and much more are needed. Many research labs across the U.S. are working on these technologies, and the first big milestone will be to take these devices, and reliable quantum entanglement/distribution, from laboratory-level readiness to acceptable field-level readiness. Following the “team science” philosophy of Berkeley Lab, this step will not just involve the researchers and physicists but will require collaboration with engineers who have experience deploying and managing components in the field.

Second, once we have deployable and supportable components, we need to gain experience running and operating these devices. It is wonderful that Brookhaven, Fermi, Argonne, UChicago, Northwestern, Oakridge, Caltech, MIT, and others have built or are planning to build capable free-space and/or fiber-based regional testbeds. We can also think about expanding these testbeds using existing dark fiber from Berkeley to SLAC to Caltech, or Brookhaven/Stonybrook to MIT/Harvard, or the Argonne/Fermi/Chicago regional testbed to Oakridge as the capabilities of the devices expand.

In addition to physical devices, quantum entanglement, and teleportation techniques, the classical networking protocols and techniques to control, manage, and operate the quantum network are extremely important. At this stage of development, it is critical to let a thousand flowers bloom. ESnet, with its practical experience, can help design testbeds, connect these testbeds to quantum applications, and support end-to-end tests to help the researchers focus on the techniques that are most viable and easily scalable across the nation. This principle of co-design has been impactful across the DOE Office of Science projects and I hope will be applied to the quantum Internet efforts as well.

Finally, I remember working on one of the leading multi-protocol routers in the 90s with protocols like X.25, Appletalk, IPX, OSI, and many others that have now faded away. With the many approaches to quantum routing and potential protocols to control these devices, we will need a testbed that allows neutral testing of these various research approaches and maybe even determines the interoperability of the various free-space, satellite, and fiber-based techniques. We will also need to build strong collaborations not only between the DOE labs, quantum centers, and the science applications, but also with other agencies like NASA, NIST, NSF, and others that have investments in this space. This is the only way we can leverage the knowledge and expertise of the broader scientific community to reach the vision outlined in the Quantum Internet Blueprint report.

Interview by Kathy Kincade, Berkeley Lab Computing Sciences

 

5G For Science: How Research Will Benefit from Advanced Wireless?

5G Andrew

5G is the next-generation wireless network that will give you much faster Internet connections. That means massive files, like high-definition movies, that take you about six minutes to download over a 4G LTE network, could be downloaded in a matter of seconds over the 5G network. And because of its innovative design, 5G is about to change the way things like cars, TVs, and even buildings connect to the Internet.

The Department of Energy’s national labs, sponsored by the Office of Science, are currently working to identify opportunities on how science can leverage 5G and other advanced wireless technologies. The Office of Science recently published a report on its findings.     

ESnet Computer Systems Engineer Andrew Wiedlea helped facilitate discussions and report findings. We recently caught up with him to talk about the benefits of 5G and other advanced wireless technologies for science, and what it will take to make it available for research. 

What is 5G and Advanced Wireless? And, how could science benefit from it?

Floating Robots
Berkeley Lab and UC Berkeley researchers unleashed 100 floating sensors to understand how water flows through the Sacramento-San Joaquin Delta on its way to pumping stations and San Francisco Bay.  The sensors transmitted data to the National Energy Research Scientific Computing Center for assimilation and analysis. (Photo by Roy Kaltschmidt).

Scientific data movement is on the cusp of a new era for flexible, low-cost deployment of scientific sensors and data mobility. Advanced wireless capabilities offer the promise of solving the “last mile problem” for science, which is creating new ways for scientists to connect data from sensors, vehicles, and isolated locations, with U.S. Department of Energy’s world-class supercomputers. It’s important to note that advanced wireless will not replace high speed scientific optical networks for large-scale wired “backbone” connectivity, rather we will solve the last mile problem through the integration of advanced wireless- and wired- backhaul. 

5G technology is one part of this emerging wireless data connectivity era. In addition to emerging low-orbit satellite constellation non-terrestrial networks, terrestrial millimeter wireless (mmWave), 5G “New Radio” capabilities will be deployed both by commercial vendors and non-commercial entities (using open parts of the radio frequency spectrum-space) to support myriad uses. Because 5G operates over a very wide range of radio frequencies (600 MHz to 27 GHz) and also leverages advances made since the deployment of earlier cellular radio communication standards, such as software defined networking, beam steering, and improved signal processing, 5G will allow users (including the scientific community) to engineer wireless data transmission supporting novel sensing applications for the world around us.

What makes 5G different from previous wireless standards for science? 

accessibility-browsing-5g-business5G is built around three standards, each of which leverages network resources in different ways.  Each of these application models will be leveraged by scientists depending on their needs:

Enhanced Mobile Broadband: The main benefit of 5G comes from a great increase in the ability to spatially reuse the radio spectrum. In comparison to previous cellular network standards, 5G networks will support higher data rates, and an ability to support many more subscribing devices wherever this is needed.  For scientists, this will mean much improved options for sensor networks, Internet of Things (IoT) applications, lower wireless data costs, and (hopefully) less reliance on “sneakernet” or other improvised methods for data collection and movement.

Ultra Reliable and Low Latency Communications: 5G supports deployment modes based around defined service levels, which means users will be able to reserve “slices” of capacity in a way similar to reserving circuits on a wired network. This, combined with other capabilities, will allow 5G to support scientific uses where communications reliability is essential, such as when measurements depend on near-real-time interaction with instrument control systems or as part of operating mobile systems such as unmanned aerial vehicles.

Massive Machine Type Communications: 5G is also built to support deployment modes in support for low power, automated systems.  These capabilities will be of benefit for all kinds of urban applications, but particularly so for scientists leveraging 5G for urban or building applications.  Leveraging this standard, scientists will be able to deploy hundreds or even thousands of small, very power-efficient, sensors throughout buildings or other areas to measure energy or environmental factors.

Taken as a whole, the capabilities provided by advanced wireless (5G, non-terrestrial networks, and mmWave) will allow new kinds of science, both within the confines of the laboratory and outside in a world via commercial and national laboratory dense sensor networks. Both the types and amounts of data generated will greatly increase – as will the scientific opportunities to learn new things.

What role is ESnet playing in creating a 5G network for scientists?

Cori Supercomputer
Advanced wireless capabilities are creating new ways for scientists to connect data from sensors, vehicles, and isolated locations, with world-class supercomputers like the National Energy Research Scientific Computing Center’s (NERSC’s) Cori system. (Picture by Roy Kaltschmidt)

ESnet’s mission is to ensure that science collaborations—at every scale and in every scientific domain—have the information and tools they need to achieve maximum benefit from global networks. This mission is not defined by a particular technology. ESnet works to integrate the compute, storage, and analytic resources operated by sites within the Department of Energy complex, and our scientific customers. 

Unlike previous generations of sensor or data infrastructure development, such as the Internet, Advanced Wireless and 5G advances are largely occurring without the US National Laboratory system playing lead roles. The challenge for scientific users is primarily one of connecting wireless technology (when needed) into the toolset made available by the Department of Energy to support US and global science objectives.  

ESnet inherently must support these customer efforts because we operate the high-speed scientific data network upon which the community depends now, and in the future as next-generation capabilities (ESnet 6) come to life.  We are also at the forefront of thinking about next-generation data movement and analytics through leadership roles with the National Science Foundation’s FABRIC program, software defined networking, and other projects supporting the Department of Energy’s future vision for the science laboratory system. 

At the Lawrence Berkeley National Laboratory (Berkeley Lab), where ESnet is headquartered, we are working to develop a community of interest on 5G and advanced wireless applications, and have been using this as a forum to develop ideas, and bring in external speakers to provide technical talks on 5G state of the art.  

ESnet’s Science Engagement Team is also starting to work with the Applied Physics Program and others to test aspects of advanced wireless technology, as well as how we can connect this to ESnet work in edge computing, our ScienceDMZ architecture, and other Berkeley Lab resources.  We have also started to develop research relationships with the UC Berkeley advanced wireless community, especially the Wireless Research Center to explore mmWave capabilities.  Outside of Berkeley Lab, we have been very active in the Department of Energy’s Enabled Energy Innovation Workshop (5GEEIW)  and related discussions for science uses of 5G and future requirements, as well as discussions with other labs and commercial entities about collaboration on testbeds and prototyping use cases.  These efforts will grow over the next year and hopefully, the report just released from the 5GEEIW will contribute to this progress. [link here]

Are there any experiments looking to use 5G? 

16538042638_cf555c1932_k
Argonne National Laboratory’s  Waggle platform is a novel wireless sensor with advanced edge computing capabilities to enable a new breed of sensor-driven environmental science and smart city research. (Photo by Mark Lopez, Argonne National Laboratory)

Around the Department of Energy complex, many teams are starting to look at the use of 5G to support experiments.  There are also developing applications for inside building and laboratory use as well using unlicensed 5G spectrum—some of this application space is now served by either Wifi or wired connectivity. There is a need for some general networking research to explore how ESnet wired capabilities, such as caching and data transfer nodes, should be deployed as part of wired-wireless interfaces, and to develop patterns for scientific support for projects making use of advanced wireless technologies as part of ESnet support for science.

We, along with Argonne National Laboratory, Pacific Northwest National Laboratory, and other Labs, are developing ideas for 5G/Advanced Wireless testbed and prototype application testing environments.  At present, the availability of equipment and service is limited, but this is expected to change rapidly as the first generation of 5G handsets and other devices begin to flood the market.

What is the state of 5G now? How long will it be until scientists can access it?

5G is being commercially rolled out by carriers now, and the build-out of this service is expected to take several years.  Other resources, such as IoT 5G toolsets and hardware are also beginning to reach the market from Ericsson and other vendors.  Similarly, non-terrestrial network constellations such as StarLink are beginning to support limited communities of beta-testers, and mmWave resources are also becoming commercially available.  

Thanks to Berkeley Lab IT’s stellar work with Verizon, however, we hope that there will be options over this next year for Berkeley-community access to 5G testing resources, and similar opportunities to explore mmWave or non-terrestrial networks tools as we build relationships and capabilities.  We also believe that opportunities and resources will start to become available over this next year from the Department of Energy, and other funding sources to support science user testing and the uptake of advanced wireless.

How did you get into this work and what do you enjoy most about it?

I got into this area at the start of my career working on satellite mobile telephony, and later with the Department of Defense working on sensors and analysis systems. When I was supporting military forces in the field with analytics, the problem was always how to handle really data thin-pipes, and as part of this, we had a lot of trouble with existing radio, cellular and satellite options. 

As part of the research-support community, I’m most interested in how we can use 5G and advanced wireless technologies to allow scientists to do new things. It is really fascinating to be at a point of inflection, for RF wireless technology and the ability for almost anyone to be able to affordably collect data from the world, backhaul that data globally, and make sense of it.  

I think that we are in a great position to lead the way with open science “out in the world” which will leverage these new technologies and ESnet is a wonderful place to serve that cause.

Interviewed by Linda Vu, Berkeley Lab Computing Sciences

Into the Medical Science DMZ

iStock-629606180
Speeding research. The Medical Science DMZ expedites data transfers for scientists working on large-scale research such as biomedicine and genomics while maintaining federally-required patient privacy.

In a new paperLawrence Berkeley National Laboratory (Berkeley Lab) computer scientist Sean Peisert and Energy Sciences Network (ESnet) researcher Eli Dart and their collaborators outline a “design pattern” for deploying specialized research networks and ancillary computing equipment for HIPAA-protected biomedical data that provides high-throughput network data transfers and high-security protections.

“The original Science DMZ model provided a way of securing high-throughput data transfer applications without the use of enterprise firewalls,” says Dart. “You can protect data transfers using technical controls that don’t impose performance limitations.”

Read More at Science Node: https://sciencenode.org/feature/into-the-science-dmz.php 

Sean-and-Eli
Left: Eli Dart, ESnet Engineer | Right:  Sean Peisert, Berkeley Lab Computer Scientist

Berkeley Lab and ESnet Document Flow, Performance of 56 Terabytes Climate Data Transfer

Visualization by Prabhat (Berkeley Lab).
The simulated storms seen in this visualization are generated from the finite volume version of NCAR’s Community Atmosphere Model. Visualization by Prabhat (Berkeley Lab).

In a recent paper entitled “An Assessment of Data Transfer Performance for Large‐Scale Climate Data Analysis and Recommendations for the Data Infrastructure for CMIP6,” experts from Lawrence Berkeley National Laboratory (Berkeley Lab) and ESnet (the Energy Sciences Network, document the data transfer workflow, data performance, and other aspects of transferring approximately 56 terabytes of climate model output data for further analysis.

The data, required for tracking and characterizing extratropical storms, needed to be moved from the distributed Coupled Model Intercomparison Project (CMIP5) archive to the National Energy Research Supercomputing Center (NERSC) at Berkeley Lab.

The authors found that there is significant room for improvement in the data transfer capabilities currently in place for CMIP5, both in terms of workflow mechanics and in data transfer performance. In particular, the paper notes that performance improvements of at least an order of magnitude are within technical reach using current best practices.

To illustrate this, the authors used Globus to transfer the same raw data set between NERSC and Argonne Leadership Computing Facility (ALCF) at Argonne National Lab.

Read the Globus story: https://www.globus.org/user-story-lbl-and-esnet
Read the paper: https://arxiv.org/abs/1709.09575