Looking back at ESnet’s 2020

Advancing our strategy and shaping our position on the board.
Some thoughts from Inder on the year-that-was.

Miniature from Alfonso X’s Libro del axedrez dados et tablas (Book of chess, dices and tables), c. 1283. , Public domain, via Wikimedia Commons

Dear Friends, Well-wishers, Colleagues, and all of ESnet,

Chess! 2020 has been much more challenging than this game. It’s also been a year where we communicated through the squares on our zoom screens, filled with faces of our colleagues, collaborators, and loved ones.

In January, Research and Education leaders came together in Hawaii at the Pacific Telecommunications Council meeting to discuss the future of networking across the oceans. It was impossible to imagine then that we would not be able to see each other again for such a long time. Though thanks to those underwater cables, we have been able to communicate seamlessly across the globe.

Looking back at 2020, we not only established a solid midgame position on our ESnet chessboard, but succeeded in ‘winning positions’ despite the profound challenges. The ESnet team successfully moved our network operations to be fully remote (and 24/7) and accomplished several strategic priorities. 

ESnet played some really interesting gambits this year: 

  1. Tackled COVID-related network growth and teleworking issues for the DOE complex
    • We saw a 4x spike in remote traffic and worked closely across several Labs to upgrade their connectivity. We continue to address the ever-growing demand in a timely manner. 

    • As we all shifted to telework from home, ESnet engineers developed an impromptu guide that was valuable to troubleshoot our home connectivity issues. 
  2. Progressed greatly on implementing our next-generation network, ESnet6
    • We deployed and transitioned to the ESnet6 optical backbone network, with 300 new site installations, 100’s of 100G waves provisioned, with just six months of effort, and while following pandemic safety constraints. I am grateful to our partners Infinera (Carahsoft) and Lumen for working with our engineers to make this happen. Check out below how we decommissioned the ESnet5 optical network and lit up the ESnet6 network.
    • Installed a brand new management network and security infrastructure upgrades along with significant performance improvements.
    • We awarded the new ESnet6 router RFP (Congratulations Nokia and IMPRES!); the installs start soon.
    • Issued another RFP for optical transponders, and will announce the winner shortly.
  3. Took initiative on several science collaborations to address current and future networking needs
    • We brainstormed new approaches with the Rubin Observatory project team, Amlight, DOE and NSF program managers to meet the performance and security goals for traffic originating in Chile. We moved across several countries in South America before reaching the continental U.S. in Florida (Amlight), and eventually the U.S. Data Facility at SLAC via ESnet.
    • Drew insights through deep engagement of ESnet engineers with the High Energy Physics program physicists, for serving the data needs of their current and planned experiments expediently.
      Due to the pandemic, a two-day immersive in-person meeting turned into a multi-week series of Zoom meetings, breakouts, and discussions.
    • When an instrument produces tons of data, how do you build the data pipeline reliably? ESnet engineers took on this challenge, and worked closely with the GRETA team to define and develop the networking architecture and data movement design for this instrument. This contributed to a successful CD 2/3 review of the project—a challenging enough milestone during normal times, and particularly tough when done remotely. 
    • Exciting opening positions were created with EMSL, FRIB, DUNE/SURF, LCLS-II…these games are still in progress, more will be shared soon. 
  4. Innovated to build a strong technology portfolio with a series of inspired moves
    • AI/ML
      • We demonstrated Netpredict, a tool using deep learning models and real-time traffic statistics to predict when and where the network will be congested. Mariam’s web page showcases some of the other exciting investigations in progress. 
      • Richard and his collaborators published Real-time flow classification by applying AI/ML to detailed network telemetry.
    • High-touch ESnet6 project
      • Ever dream of having the ability to look at every packet, a “packetscope”, at your fingertips? An ability to create new ways to troubleshoot, performance engineer, and gain application insights? We demonstrated a working prototype of that vision at the SC20 XNET workshop
    • SENSE
      • We deployed a beta version of software that provides science applications the ability to orchestrate large data flows across administrative domains securely. What started as a small research project five years ago (Thanks ASCR!) is now part of the AutoGOLE project initiative in addition to being used for Exascale Computing Project (ECP) project, ExaFEL.
    • TCP
      • Initiated the Q-Factor project this year, a research collaboration with Amlight, funded by NSF. The project will enable ultra-high-speed data transfer optimization by TCP parameter tuning through the use of programmable dataplane telemetry: https://q-factor.io/
      • We testbed thoroughly the interactions between TCP congestion control algorithms, BBRv2 and CUBIC. A detailed conversation with Google, the authors of the BBRv2 implementation, is in progress.
  5. Initiated strategic new games, with a high potential for impact
      • Executed on the vision and design of a nationwide @scale research testbed working alongside a superstar multi-university team.
      • With the new FAB grant, FABRIC went international with plans to put nodes in Bristol, Amsterdam, Tokyo and Geneva. More locations and partners are possibilities for the future.  
    • Edge Computing
      • Created an prototype FPGA-based edge-computing platform for data-intensive science instruments in collaboration with the Computational Research Division and Xilinx. Look for exciting news on the blog as we complete the prototype deployment of this platform.
    • Quantum
    • 5G
      • What are the benefits of widespread deployment of 5G technology on science research? We contributed to the development of this important vision at a DOE workshop. New and exciting pilots are emerging that will change the game on how science is conducted. Stay tuned. 

Growth certainly has its challenges. But, as we grew, we evolved from our old game into an adept new playing style. I am thankful for the trust that all of you placed in ESnet leadership, vital for our numerous, parallel successes. Our 2020 reminds me of the scene in Queen’s Gambit where the young Beth Harmon played all the members of a high-school chess team at the same time. 

Several achievements could not make it to this blog, but are important pieces on the ESnet chess board. They required immense support from all parts of ESnet, CS Area staff, Lab procurement, Finance, HR, IT, Facilities, and Communications partners.

I am especially grateful to the DOE Office of Science, Advanced Scientific Computing Research leadership, NSF, and our program manager Ben Brown, whose unwavering support has enabled us to adapt and execute swiftly despite blockades. 

All this has only been possible due to the creativity, resolve, and resilience of ESnet staff — I am truly proud of each one of you. I am appreciative of the new hires that trusted their careers with us and joined us remotely—without shaking hands or even stepping foot at the lab.

My wish is for all to stay safe this holiday season, celebrate your successes, and enjoy that extra time with your immediate family. In 2021, I look forward to killer moves on the ESnet chessboard, while humanity checkmates the virus. 

Signing off for the year, 

Inder Monga

Three Questions with Mark Körner

Three questions with a new staff member!  

Marc Körner is the most recent addition to the ESnet Software Engineering – Orchestration and Core Data team (OCD).  He comes to us from Join Digital in San Jose where he was Lead Engineer in their Network Services team.  

Marc has a PhD in Computer Science from the Technical University of Berlin, and has spent a number of years working as a researcher in both the Berlin and Berkeley areas. He’ll be working on the automation side of OCD getting familiar with the network services orchestrator platform and helping us achieve our ESnet6 deliverables. Marc is onboarding virtually this week but resides full time in the San Jose area.

What brought you to ESnet?

I was always very passionate about computer networks. The idea of having a global technology for the data and knowledge exchange was always very fascinating for me. It started with the LAN sessions I had with my friends and ended up with the tremendous opportunity to build the first SDN research network in Europe. After my time as a research fellow at the UC Berkeley Netsys lab group and my startup experiences in the access network provider business, the open position for the network automation at ESnet was the ultimate opportunity to take it to the next level.

What is the most exciting thing going on in your field right now?

This question is not easy to answer, there are so many things going on in computer networks. I think one of the biggest innovations in the last decade is the virtualization in general and the centralization of network management and control. However, one of the more recent trends which correlates with this particular network development is edge computing, or the slightly more generalized concept of fog computing and its seamless orchestration. It’s basically a fine granular fusion of the compute and network control plan, which we also observed in cloud computing.

What book would you recommend?

It has been a while since I read a book. As an EECS guy people are probably expecting something to hear like: “The programming language C by Brian Kernighan and Dennis Ritchie”. However, if you are interested in science in general I would probably recommend: “The Universe in a Nutshell by Stephen Hawking”. The book provides some interesting insights about modern physics and has the potential to open up interesting views on the world around us.

40G Data Transfer Node (DTN) now Available for User Testing!

ESnet’s first 40 Gb/s public data transfer node (DTN) has been deployed and is now available for community testing. This new DTN is the first of a new generation of publicly available networking test units, provided by ESnet to the global research and engineering network community as part of promoting high-speed scientific data mobility. This 40G DTN will provide four times the speed of previous-generation DTN test units, as well as the opportunity to test a variety of network transfer tools and calibrated data sets.

The 40G DTN server, located at ESnet’s El Paso location, is based on an updated reference implementation of our Science DMZ architecture. This new DTN (and others that will soon follow in other locations) will allow our collaborators throughout the global research and engineering network community to test high speed, large, demanding data transfers as part of improving their own network performance. The deployment provides a resource enabling the global science community to reach levels of data networking performance first demonstrated in 2017 as part of the ESnet Petascale DTN project

The El Paso 40G DTN has Globus installed for gridFTP and parallel file transfer testing. Additional data transfer applications may be installed in the future. To facilitate user evaluation of their own network capabilities ESnet Data Mobility Exhibition (DME), test data sets will be loaded on this new 40G DTN shortly. 

All ESnet DTN public servers can be found at https://app.globus.org/file-manager. ESnet will continue to support existing 10G DTNs located at Sunnyvale, Starlight, New York, and CERN. 

ESnet's 40G DTN Reference Architecture Block Diagram
ESnet’s 40G DTN Reference Architecture Block Diagram

The full 40G DTN Reference architecture and more information on the design of these new DTN can be found here:

A second 40G DTN will be available in the next few weeks, and will be deployed in Boston. It will feature Google’s bottleneck bandwidth and round-trip propagation time (BBR2) software, allowing improved round-trip-time measurement and the ability for users to explore BBR2 enhancements to standard TCP congestion control algorithms.

In an upcoming blog post, I will describe the Boston/BBR2-enabled 40G DTN and perfSONAR servers. In the meantime, ESnet and the deployment team hope that the new El Paso DTN will be of great use to the global research community!  

Three Questions with Jay Stewart

Three questions with a new staff member!

Jay was born in Cambridge (UK) but moved, when he was four, to the United States. Jay grew up in Brookhaven National Laboratory’s backyard, Long Island, and attended school at Suffolk University in Boston where he received a degree (BS) in Marketing with a minor in Information Systems. Jay became keenly interested in computers when a parent’s colleague gifted him a CD-ROM with a slew of MS-DOS-based games. It was through the immortal wisdom of the game “Ecco the Dolphin” that his wrists became transfixed to the computer desk and they’ve been there ever since. In his spare time, he enjoys reading and learning to live without sleep as he and his wife had their first child in October

What brought you to ESnet?

I came to ESnet as a Network Engineer who had been working for a commercial ISP, Pilot Fiber, based out of Manhattan for the last 3 years. I had joined the ranks of the Service-Provider lifestyle right out of college as a call center technician and googled every term to climb the ranks to Network Engineer. Nick Buraglio had reached out about the position at ESnet. Knowing that I will be following in my father’s footsteps by helping to ensure the highest degree of scientific collaboration made the decision, to join ESnet, an easy one. I’ll be working onsite at Brookhaven National Laboratory whilst helping to ramp up their connections onto ESnet6 and transitioning into my role as site ambassador.

What is the most exciting thing going on in your field right now?

At the highest level, I think Quantum communication is an exciting thing to read about and to try and understand. Seeing that Quantum computing, in general, has the power to move us from the law of Moore’s to the law of Neven’s is thrilling. A more grounded excitement, at least to me, is the work being done with Segment Routing. Think of it like Waze for your packets. It allows granular steering capabilities alongside ensuring a bird’s eye view of your network. The instructions are encapsulated in the header of the packet and get directed/removed at each node in the path in a matryoshka-doll-esque fashion.

What book would you recommend?

I’m a sucker for a futuristic, dystopian book and The Water Knife by Paolo Bacigalupi fits that need nicely.