ESnet Team Showcases New Services, Pushes Networking Boundaries at OFC2023

ESnet software engineers Sarah Larsen, Dan Doyle, and Bruce Mah at ESnet’s High Touch demonstration booth

After a pandemic-related in-person hiatus, the Optical Fiber Communication Conference and Exhibition (OFC), sponsored by Optica, IEEE Communication Society, and IEEE Photonics Society, resumed operations with a sold-out event in March 2023 at the San Diego Convention center. More than 11,500 participants and 515 exhibitors attended this global event for optical communications and networking, including almost two dozen from ESnet. Planning & Architecture Acting Group Lead Chris Tracy led ESnet’s multifaceted involvement at OFC23, which ranged from a booth demonstrating ESnet’s High Touch project and panel discussion to helping implement OFCnet, an unconventional high-speed network connecting the show floor to a research center in Chicago. 

Staffed by ESnet software engineers Bruce Mah, Sarah Larsen, and Dan Doyle, the ESnet booth presented a high-level technical overview and showed examples of data and analysis from the High-Touch system being deployed in ESnet6, the latest version of ESnet’s backbone network for supporting scientific collaborations and research around the globe.​ The High-Touch project uses a combination of software and programmable, off-the-shelf hardware to deliver new network services. Its first applications provide high-precision network telemetry, including summarization of network flows and capture of packet headers, which are computed from unsampled streams of packets from multiple 100GE and 400GE links. This demonstration relied heavily on the efforts of ESnet’s Infrastructure team to install and configure dozens of data collection servers across ESnet’s network footprint.

ESnet High-Touch Architecture and Design: This diagram shows the flow of packets and network measurements through ESnet’s High-Touch system, which uses a combination of programmable, off-the-shelf hardware and software to provide high-precision network telemetry.

ESnet Executive Director Inder Monga and Chris also realized that OFC2023 offered potential for demonstrating network capabilities that went beyond the exhibition floor. Prior to OFC2022, there was no high-speed, “external” network connectivity at the event suitable for data-intensive demonstrations. The conference consisted of technical talks about papers that were being published and vendor booths. At OFC2022, Optica, Lumen, CENIC, Ciena and Smart City successfully showed in a modest proof of concept that external fiber could be brought into the convention center so that a live demonstration could be run on the show floor. For OFC2023, Ciena’s office of the CTO – who was leading the OFCnet effort – approached ESnet about demonstrating high-performance networking applications as well as emerging technologies, and more broadly, bringing some networking focus into the conference.

Working with Ciena staff, ESnet Network Services Optical Network Group Lead Patrick Dorn and Network Engineers Michael Blodgett, Kate Robinson, and Nathan Miller helped build an un-regenerated 400 Gbps link between the OFC show floor in San Diego and the StarLight Data Center in Chicago. “Un-regenerated” means the signal remains solely in the optical domain, e.g. as wavelengths of light, not an electrical signal, for transcontinental distances (more than 4,600 kilometers). 

Another interesting feature of this demonstration was that the ESnet team connected ESnet6’s production Infinera FlexILS line system to a Cisco NCS 1010 line system (provided by Cisco to support OFCnet), effectively bridging the purpose-built OFC exhibition network to a live, nation-scale infrastructure. In addition to the Infinera and Cisco line systems, Ciena provided the ultra-long-haul transponder equipment necessary to communicate over such distances, plus the engineering expertise – along with staff from Cisco, ESnet and CENIC – to ensure it all worked. 

Enlarge for more detail

Using the high-speed channel ESnet established between San Diego and Chicago, researchers from Northwestern University’s International Center for Advanced Internet Research (iCAIR) could showcase data transfer applications being used to move massive scientific datasets. By helping to implement this somewhat unconventional infrastructure, the ESnet team sought to show what might someday be possible when networks can transport 400 Gigabit Ethernet over such long distances without relying on bonding two 200 Gbps wavelengths using inverse muxing. 

Some of the ESnet team at OFC23

In addition to the two demonstrations, ESnet staff participated in multiple panel discussions and a bird of a feather (BoF) event at OFC23. For a panel on how high performance research networks continue to drive fundamental science and innovation, Chris Tracy and others used OFCnet and its connection to an external Research & Education network to discuss data transfer for data intensive science, detailed monitoring of science flows within the network, network security considerations in the research network environment, and applications like distributed computing that take advantage of these networks. At the BoF event, Inder presented, while Chris, ESnet staff, and other OFCnet volunteers brainstormed ideas for how OFCnet might evolve as a next-generation optical photonic network for OFC2024. One recommendation: a Sunday workshop titled: “How Can OFC with a Real-Life Testbed Accelerate Innovation in the Design and Operation of Next Generation Optical Photonic Networks?” The BoF participants believe this would provide an opportunity to invite speakers and publish papers within the context of the workshop for these kinds of networking-related topics.

Planning for next year’s iteration of OFCnet (March 24-28, 2024) has already kicked off, with ESnet once again participating in a leadership role. The goals for OFCnet24 are ambitious. The volunteer team hopes to attract attendees from different communities, such as networking science (academia and research labs); make it possible to showcase high performance networking application use cases and other emerging technologies – turning the exhibits floor as a science accelerator; and bridge the exhibit and technical programs by offering the opportunity to present advanced technical papers with live demos. 

“It was great to be able to demonstrate some of the innovative services we’re delivering through the High Touch project,” said Chris. “And of course we welcome any opportunity for ESnet to participate in something such as OFCnet that advances the state of the art for networking and allows us to showcase emerging technologies on our network. Next year is going to be even more exciting.” 

Marc Lyonnais (right), OFCnet chair and director of external research at Ciena, presented Planning & Architecture Acting Group Lead Chris Tracy (left) and Executive Director Inder Monga (center) with a plaque in thanks for ESnet’s OFCnet efforts

The Science Conversation Continues at Confab23, Oct. 16-18 in D.C.

confab23 logo

Registration is now open for Confab23, to be held October 16 to 18 in Gaithersburg, Maryland! 

ESnet’s second annual Confab gathering is designed for scientists across all disciplines who want to vastly improve their workflows and collaborations to accelerate time to discovery; for network engineers from national labs and universities who support science IT services for researchers on their campuses; and for the research networking professionals who partner with ESnet to move data across the world.

Last year’s inaugural Confab in Berkeley, held concurrently with the unveiling of ESnet6, was a fun and resounding success — and we believe Confab23 will continue and broaden the conversation we started. 

Confab23 will showcase scientists who use ESnet today to perform real-time data analysis, leverage multiple supercomputers in parallel for large-scale simulations, and collaborate with colleagues on experiments as if side by side while thousands of miles apart – among many other applications.

Together we can chart the future of scientific data management and integrated scientific infrastructure.  

In addition to lively conversation and informal technical discussions between our ESnet, DOE, and scientific community attendees, the program includes:

  • Updates and discussions with the Department of Energy on major initiatives, such as the Integrated Research Infrastructure initiative and the High Performance Data Facility, that support our shared vision — that scientific progress will be completely unconstrained by the physical location of instruments, people, computational resources, or data.
  • An overview of quantum networking activities by Inder Monga, part of a lively evening program

We will also hold a meeting of the ESnet Site Coordinator Committee at the same venue on October 19 and 20.

If you have any questions or suggestions – or would like to offer a presentation or session topic – please email our workshop team directly at confab@es.net.

Register now >

ESnet6 Unveiled Tomorrow!

We’re getting things set up for the ESnet6 Unveiling tomorrow – our tent has gone up, we’re holding final rehearsals for the presentations, printing badges, and doing a thousand other small things.  

The only thing missing from our pictures is you! 

See you tomorrow for the big day, if you are visiting in person, travel safe, and if you are joining us virtually, the show starts at 9:00 AM on https://streaming.lbl.gov.

ESnet6 Unveiling in Seven Days!

On October 11, 2022, we will welcome the newest generation of our high-performance scientific network, ESnet6, at an unveiling ceremony hosted by Berkeley Lab.

ESnet6 marks a new era of our high-performance network supporting the needs of scientists. We’re able to handle massive flows of data in a reliable, nimble way, and we can specifically configure our setup to match the needs of individual experiments. The upgrade ensures that ESnet is ready to support the future of science today, including the significant increase in the amount of data produced by scientific experiments and the increasingly complex needs of scientists and the way they interact with our network. 

Come watch the ESnet6 unveiling ceremony 9AM -12 PM PT, October 11, at streaming.lbl.gov!

ESnet team to give multiple talks about networking, automation, and QUANT-NET at NORDUnet Conference and GEANT’s SIG-NGN meeting

The 31st NORDUnet Conference will take place in Reykjavik, Iceland from September 13-15, 2022. 

ESnet staff will also be in attendance at the Special Interest Group on Next Generation Networking (SIG-NGN) on September 12, 2022, the day before the NORDUnet Conference.

Here’s where you can find ESnet team’s talks during these events: 

Monday, September 12, 2022: SIG-NGN

The next generation NREN lightning talks 
09:05 – 10:30am GMT

The lightning talks will feature two presentations from ESnet: 

  • ESnet Effort to Build Upon the NML and MRML – John MacAuley
  • LHC Next Generation Requirements Gathering – Eli Dart

Future network architectures. Technological change to support data moving / data planes 
11:00am – 12:30pm GMT

This session will start with 10-minute presentations, including two by ESnet staff:

  • ESnet7 – Chin Guok
  • Underlay Packet Inspection, Making Traffic Engineering Decisions at L2 – Yatish Kumar

These talks will be followed by a panel discussion. 



How do we stitch and share our L1-L3+ networks to introduce better and new services
12:00pm – 3:30pm GMT

This session includes a series of short talks, including:

  • Real Time Data Processing Requirements – Yatish Kumar

Yatish Kumar will also host a discussion on future networking technologies from =2:00pm – 3:20pm GMT


Tuesday, September 13, 2022: NORDUnet Conference

The ESnet6 Approach to Network Orchestration and Automation
11:00am – 12:30pm GMT | Track 1 / Room: Silfurberg B

Speaker: Scott Richmond

Abstract: Network Orchestration is a defining factor in next generation networks, enabling operators to deliver more consistent and reliable services. ESnet has leveraged a combination of internally developed tools, open source software, and commercial software to orchestrate and automate network configuration deployment. This approach has enabled rapid deployment of new network services, as well as ensuring that configuration standards are well enforced when deploying network services.

During this talk, we will provide a brief history of automation at ESnet, dive into what our goals were for orchestration and automation in the ESnet6 project, and describe the technology and process that we used to meet those goals. Finally, we will discuss the hurdles encountered and lessons we learned along the way while developing this tooling.

Eli Dart was part of the technical program committee and is the chair for the HPC session, taking place in Track 2 / Room: Rima from 2:00 – 3:30 pm GMT. 


Wednesday, September 14, 2022: NORDUnet Conference

Experimenting with Teleportation Based Physical Layer for the Network: QUANT-NET
1:30pm – 3:00pm GMT | Track 1 / Room: Silfurberg B

Speaker: Inder Monga

Abstract: QUANT-NET takes an application-centric and systems-based approach to building a Quantum Internet testbed. The main thrust of this effort is to build a three-node distributed quantum computing testbed between two sites, Lawrence Berkeley National Lab and the University of California Berkeley (UCB) connected with an entanglement swapping substrate over optical fiber and managed by a quantum network protocol stack. We will implement the most basic building block of distributed quantum computing by teleporting a controlled-NOT gate between two nodes. This approach will enable research, prototyping, measurement and testing of the entire quantum network stack from physical layer to the application. The talk will describe our proposed testbed and progress.


ESnet is heading to Italy to share networking expertise with international colleagues at TNC22

ESnet will be in Trieste, Italy from June 13-17, 2022 sharing their knowledge about software-defined networking at TNC22, the research and education networking conference.

Here’s a look at where to find ESnet at the conference: 

Wednesday, 6/15

CEO Roundtable
Wednesday 6/15 – 8:30-16:30 CEST | Room B
Invitation only

ESnet executive director Inder Monga will join 70 CEOs from research networks around the world for a conversation about best practices, advancements in technology, and the future of scientific networking. 

Data Mobility Exhibition
Wednesday 6/15 – 10:30-11:00 CEST | Demonstration Area
Presenter: Eli Dart

Data mobility is a critical component of the process of science. Being able to predictably and efficiently move scientific data between experimental source, processing facilities, long term storage, and collaborators is a common use case that transcends the boundaries of research disciplines. The demo transfers reference data sets from well-tuned Data Transfer Nodes (DTNs) to existing campus CI components. Participants will upload, download, measure, and potentially improve their scientific data movement capabilities as well as review previous tests.

Thursday 6/16

Session: Real Life Network Orchestration

Building a Realistic Orchestration Validation Environment for netwoRks (ROVER)
Thursday 6/16 – 09:00 – 10:30 CEST | Auditorium
Speakers: Chris Cummings, Nick Buraglio 

Building software that controls network equipment has many similarities to traditional software engineering, however, testing this software introduces many complexities unique to the network orchestration world. Join this talk to learn how we approached these challenges by building a Realistic Orchestration Validation Environment for netwoRks (ROVER) at ESnet.

Session: Data Driven Networking

Superfacility – A Blueprint for Supporting Large-Scale Scientific Workflows
Thursday 6/16 – 11:00 – 12:30 CEST | Auditorium
Speaker: Chin Guok

Data-intensive workflows have been a mainstay of large scale collaborative scientific research for several decades. However, with the growing exponential rate of data that instruments can produce today, there needs to be a paradigm shift in how distributed resources are requested and utilized across the various facilities that contribute to these data-intensive workflows. A “Superfacility” model, which seamlessly integrates instrument, computing, storage, and networking facilities, is required to more effectively support these increasingly demanding workflows. This talk will present on the desired characteristics for a Superfacility, discuss specific use cases, and highlight some activities and initiatives in this area.

Exploring the BBRv2 Congestion Control Algorithm for use on Data Transfer Nodes
Thursday 6/16 – 11:00 – 12:30 CEST | Auditorium
Speaker: Eli Dart 

It is well known that loss-based TCP congestion control algorithms are problematic for high-speed, high-latency flows that are common in Big Science. In 2016 Google released a new congestion control algorithm called ‘BBR’ (Bottleneck Bandwidth and Round-trip time) that uses a model-based approach, and the design has since been refined in an alpha release of BBRv2. In this paper, we describe and perform a set of experiments that assess the suitability of BBRv2 for use on Data Transfer Nodes (DTNs).


Registration deadlines for TNC are coming up! Registration for regular in-person passes is open until June 6 (unless you want to pay via bank transfer – registration for that closed on May 30th). Free online passes are also available!

[[Editors note: This post has been updated to include information about the Data Mobility Exhibition on 6/15 that was previously excluded]].

ESnet staff attend strategic on-site meetings for the first time in years!

Last week, over 50 ESnet employees gathered at Berkeley Lab for a week of strategizing and socializing. Here are some pictures from their adventures!

Jealous of all the fun we had? Want to hang out with us, too? Good news – Registration will open soon for Confab22, ESnet’s first user meeting! Keep an eye on the blog or pre-register for updates!

ESnet Scientists awarded best paper at SC21 INDIS!

A combined team from ESnet and Lehigh University was awarded the best paper for Exploring the BBRv2 Congestion Control Algorithm for use on Data Transfer Nodes at the 8th IEEE/ACM International Workshop on Innovating the Network for Data-Intensive Science (INDIS 2021), which was held in conjunction with the 2021 IEEE/ACM International Conference for High Performance Computing, Networking, Storage and Analysis (SC21) on Monday, November 15, 2021.

The team was comprised of:

  • Brian Tierney, Energy Sciences Network (ESnet)
  • Eli Dart, Energy Sciences Network (ESnet)
  • Ezra Kissel, Energy Sciences Network (ESnet)
  • Eashan Adhikarla, Lehigh University

The paper can be found here. Slides from the presentation are here. In this Q+A, ESnet spoke with the award-winning team about their research — answers are from the team as a whole.

INDIS 21 Best Paper Certificate

The paper is based on extensive testing and controlled experiments with the BBR (Bottleneck Bandwidth and Round-trip propagation time), BBRv2 and the Cubic Function Binary Increase Congestion Control (CUBIC) Transmission Control Protocol (TCP) Internet congestion algorithms. What was the biggest lesson from this testing?

BBRv2 represents a fundamentally different approach to TCP congestion control. CUBIC (as well as Hamilton, Reno, and many others) are loss-based, meaning that they interpret packet loss as congestion and therefore require significant network engineering effort to achieve high performance. BBRv2 is different in that it measures the network path and builds a model of the path – it then paces itself to avoid loss and queueing. In practical terms, this means that BBRv2 is resilient to packet loss in a way that CUBIC is not. This comes through loud and clear in our data.

What part of the testing was the most difficult and/or interesting?

We ran a large number of tests in a wide range of scenarios. It can be difficult to keep track of all the test configurations, so we wrote a “test harness” in python that allowed us to keep track of all the testing parameters and resulting data sets.

The harness also allowed us to better compare results collected over real-world paths to those in our testbed environments. Managing the deployment of the testing environment though containers also allowed for rapid setup and improved reproducibility. 

You provide readers with links to great resources so they can do their own testing and learn more about BBRv2. What do you hope readers will learn?

We hope others will test BBRv2 in high-performance research and education environments. There are still some things that we don’t fully understand, for example there are some cases where CUBIC outperforms BBRv2 on paths with very large buffers. It would be great for this to be better characterized, especially in R&E network environments.

What’s the next step for ESnet research into BBRv2? How will you top things next year?

We want to further explore how well BBRv2 performs at 100G and 400G. We would also like to spend additional time performing a deeper analysis of the current (and newly generated) results to gain insights into how BBRv2 performs compared to other algorithms across varied networking infrastructure. Ideally we would like to provide strongly substantiated recommendations on where it makes sense to deploy BBRv2 in the context of research and educational network applications.

ESnet Builds Morale and Community With a Zoom Competition

Nearly two months into California’s shelter-in-place order, we’ve all been in more than our fair share of video conferences. To boost morale during this difficult time, the Energy Sciences Network (ESnet) staff held a Zoom Background Competition during their all-to-all meeting on Monday, April 27. 

Staff were encouraged to create their own backgrounds and display them during the meeting. There were 21 entries. ESnet employees voted. Submissions were judged on overall artistry, functionality (not too distracting as a background), whether it elevated the voter’s mood, and if it made them feel included in the ESnet community. 

The top three winners got bragging rights. Here they are:

First place: Jeff Berman, NOC Engineer

This Zoom challenge inspired Berman, an avid sailoJeffrey Bermanr, to take to the sea. He won this competition with an hour video of the San Francisco skyline, one he filmed while sailing on the Bay. Although he typically likes to go sailing with friends and family, he says that sailing solo brings him a sense of peace, calm, and tranquility.  

“What is sailing? Most books define it as hours of sheer boredom scattered with white knuckle periods of terror. On a good day, both are true. Both give you an equal sense of accomplishment. How to be with yourself with nothing to do, good training for our current situation,” said Berman.

Second Place: Sartaj Baveja, Software Engineer

This challenge inspired Baveja to create a background meme of office life. In the background, someone (Baveja) is looking over your shoulder to catch a glimpse of your screen and make sure you don’t procrastinate.

Sartaj Baveja

Third Place: Joe Metzger, Network Engineer

Joe_interviewThis challenge inspired Metzger to use a picture that he took in Barcelona. The focal point of the picture (the blur) is a little girl in a red coat, black dress and white tights who was just running back and forth between the pools of light and shadow created by the stone arches and rosette windows, while her family was sitting in the cafe. 

“I used this as my zoom background because I think it is a really cool picture. It brings to mind a fun evening strolling around the little squares and back streets in Barcelona and sitting in cafes with a good glass of wine relaxing,” said Metzger.

Girl in Red

Written by Linda Vu, Berkeley Lab Computing Sciences.

Diversity & Inclusion POV: Michael Sinatra

At ESnet, we believe that a diverse workforce results in creative ideas and innovations. So we aim to create an inclusive working environment where people feel valued and can share their thoughts and ideas. In this series, we’ll be sharing perspectives from our staff in hopes of sharing our lessons learned and igniting conversations.

 As part of ESnet’s ongoing inclusion effort, we held a workshop, “Empathy: A Building Block for Inclusiveness,” last month to discuss proactive approaches to understanding others. Mukundagiri Kandadai Ramanujam (‘Ram’ for short), Lead Trainer with Love To Share Foundation, facilitated the discussion. Michael Sinatra, a network engineer at ESnet, shares his thoughts on the workshop.

Michael Sinatra, ESnet Network Engineer

XBD201101-00024
Michael Sinatra, ESnet Network Engineer

At ESnet, we deal with a lot of complex issues, which generate a lot of subtle risks and benefits. We also have a diverse staff that has different communication styles. I have found Ram’s seminars really helpful in reminding us of the benefit of understanding employees’ underlying concerns when we communicate. 

In the seminar, Ram made the distinction in interpersonal communication among statements that are intellectual or state neutral facts, versus those that evaluate, judge and eventually label. By using empathy to understand the underlying needs being expressed by our coworkers, we can better convey the important things that need to be communicated in our organization without causing emotional issues to block out the underlying needs of the organization. Emotions have a place at work, but they can also cloud our ability to see important issues in our jobs.

The discussion reminded me of an example from a previous workplace. When I was 21 I had a summer job at a factory where, as one of my assignments, I had to do some work with the plant machinist, who had been told by the plant managers that he had an “attitude problem.” In the course of our work, when my colleague began grumbling about something I was doing, I, aware of this person’s alleged attitude problem, tried to be as emotionally neutral as I could to find out—on an intellectual level— why he was grumbling. It turns out that my colleague had identified a serious safety issue in what I was doing. We were able to correct the issue quickly before it caused more problems. I realized that my colleague might not have an attitude problem at all, but he might just have had some trouble communicating important issues to others, and then he would get frustrated when people didn’t take his concerns seriously.

I have tried to apply some of these concepts by being more open about my underlying motivations, especially when expressing concerns about something. At the same time, I have tried to better, and more neutrally, understand my colleagues’ concerns and motivations. Ram showed us that empathy works both ways–by better exposing our own needs and concerns, we can better communicate about the issues that are important to ESnet, and by applying the principles of empathy to our colleagues, we can better understand where they’re coming from, regardless of differences in communication style or culture.

empathy-workshop
On January 10, 2017, Berkeley Lab’s Scientific Networking Division hosted an “Empathy: A Building Block for Inclusion” discussion, facilitated by Mukundagiri Kandadai Ramanujam,  Lead Trainer with Love to Share Foundation. (Photo Credit: Sowmya Balasubramanian, ESnet)