Summer at ESnet: the view from our students: Part 1

Summer students are a key part of growing ESnet and supporting the scientific networking community. Every year, we host research projects with talented students working on important research topics. We benefit tremendously from their enthusiasm, talent, and fresh ideas, and they work directly with our staff across a wide set of disciplines.  Here are some thoughts from two current students on what it is like to work with ESnet, and what research excites them.

Sandesh Dhawaskar Sathyanarayana:

I am thrilled with my summer internship at ESnet. During my Multipath Transmission Control Protocol (TCP) research, I used in-kernel programs to implement receiver-based network controllers and have always wanted to work more on it as it allows one to hook into the kernel and innovate the different network protocols. Software Defined Networking (SDN) along with dataplane and kernel network programming is trending as it enables the telecom world to save billions of dollars and operate the network more efficiently. 

My goal for this summer was to work and innovate in the SDN field, and ESnet was the perfect fit for it. At ESnet, I work on the Q-factor project using technologies such as eBPF (extended Berkeley Packet Filters) and XDP (eXpress Data Path) to improve data transfer speeds in science networks. I get to play with the state-of-the-art P4 dataplane programming language for switches and programmable NICs. The project is a collaboration with Florida International University (FIU), so I get to work with amazing people. Our team is small, with great mentors like Richard Cziva and Jeronimo Bezerra. 

What I love the most is the freedom to think and solve problems with great support. Having to work in different labs, I used to be stressed most of the time to complete the work. This summer has been a very different experience with excellent mentorship. I also had other offers and chose ESnet as my advisor and co-advisors insisted strongly, and I am happy I went with ESnet.

Elias Joseph:

Interning at ESnet has been a really good learning experience for me. The regular seminars from researchers in the lab about their current projects have allowed me to learn about a lot of topics I usually wouldn’t have much exposure to, as well as see how the concepts I have learned about in school are being applied in a professional environment. It is really interesting to see how machine learning is actively being used at the laboratory, and what current advancements are being made with it.

As much as I’m learning from the seminars, I’m learning even more from the project I’m working on. This internship is giving me experience using a lot of tools that are prevalent in computer science but are underutilized in my master’s program, and my mentor has been extremely helpful in getting me up to speed on these tools.

I’ve also found working on my project very fulfilling. Primarily I’ve been working on a tool that displays internet traffic, as well as predictions for future traffic, and seeing it come together over the past month and a half has been really cool.

I do miss the social aspect of working in an office, but the networking and social activities that have been organized have done a lot to alleviate that, and overall, I have grown a lot in the first half of my internship.

If you are interested in learning more about future summer opportunities with ESnet, please see this link — we typically post notices and accept applications for the next summer starting in January or February.

ESnet History on (virtual) Display

As part of LBL’s The Next 90 campaign, new features on the past, present and future of the Computing Sciences Area (CSA) and ESnet have just been released!

From the first documented demonstration of 10 gigabit Ethernet in 2002….

For a vision of where we, and all of CSA, started back in the 1990s, including a video of early ESnet telecommuting experiments well before Zoom, please see this link.

For an overview of CSA with Jonathan Carter, please see this link.

The ESnet interactive timeline has also been updated — the story continues!

3 Questions with Katrina Turner

Three questions with a new staff member! Aloha, Katrina!

Katrina hails from Kāne’ohe, Hawai’i where she was born and raised. She recently graduated from the University of Hawai’i at Mānoa with an M.S. in Computer Science and is now with ESnet’s Software Engineering Management and Analysis Group.  Katrina loves her island life and enjoys dancing hula, hiking, and going to the beach.  She also loves both playing and making video games in her spare time.

Katrina Turner

What brought you to ESnet?

During my time as a Research Assistant at UH Mānoa, I had the opportunity to work with some of ESnet’s team members and I really admired both the work they did as well as the work culture they were a part of.  When I heard there were openings at ESnet, I jumped at the chance to continue working with such awesome people!

What is the most exciting thing going on in your field right now?

In recent years, Data Visualization has become more popular with the general public, being shared through social media and used by the masses instead of only scientists and analysts.  As a result, we are seeing really creative and interesting ways of showing data beyond the standard charts.  Also, the integration of machine learning to allow us to easily visualize large amounts of data is really exciting.

What book would you recommend?

If you like Fantasy Fiction, the Sword of Truth series by Terry Goodkind is great, but definitely a time commitment. I also just started reading The Windup Girl.

High Energy Physics Requirements Review Now Available: The Data Deluge Shows no Sign of Cresting!

Lauren Rotman and Jason Zurawski


Across the physical sciences, new instruments and capabilities are continuing a relentless growth in data production and need for high speed networking and analysis resources. 

ESnet stays on-top of these trends via the Network Requirements Review process, which for the past 15 years has been a remarkable and useful collaboration between the DOE Office of Advanced Supercomputing Research (ASCR), ESnet and science programs across the DOE Office of Science.

The latest Network Requirements Review for the Office of Science High Energy Physics program office (HEP) is now available — among many other findings, this review confirms that the exponential growth of scientific data generation will continue unabated as we proceed into what may well be a new golden age for high energy physics research. Some samples include:

The upcoming High Luminosity era for the Large Hadron Collider (beyond 2027, or Run-4) will require multi-Tbps network speeds to support globally dispersed “Tier 1” HPC resources.  Scientists will use the LHC to uncover how the Higgs-Boson interacts and gives mass to other particles, and explore emerging evidence for particle behaviors not explained by current physics models. Each data-taking year, the experiments, ATLAS and CMS combined, are expected to accumulate roughly 1 EB of new data and it is estimated that complete data set sizes may routinely exceed 100 PB.  

Expected maximum luminosity and integrated luminosity for the LHC as a function of calendar year, data produced tracks with improved luminosity and resolution

Scientists at the Deep Underground Neutrino Experiment (DUNE) in South Dakota and at Fermilab in Illinois, will use high speed data transfer to identify supernova events, as part of ongoing measurement of neutrino interactions. Supernovae measured by DUNE will generate over 200TB of compressed data per event, and Research and Educational Networks (REN) must be able to supply highly reliable, predictable data transfer capabilities to provide telescope targeting data to global arrays.

10kt DUNE Far-Detector SP module, showing the alternating 58 m long (into the page), 12 m high anode (A) and cathode (C) planes, as well as the field cage that surrounds the drift regions between the anode and cathode planes. The blank area on the left side was added to show the profile of a single anode plane assembly (APA). Person included for scale.

The Cosmic Microwave Background, Stage 4 (CMB-S4) experiment will require data management and transfer capabilities in some of the most demanding locations on earth. Operating two observational locations, and multiple telescopes with a combined total of 500,000 cryogenically-cooled superconducting detectors at the South Pole and in the Chilean Atacama Desert, CMB-S4 will provide an unprecedented picture back into the start of the Universe. Operating for seven years in these conditions, 22 TB (~8 TB at the South Pole and ~14 TB in Chile) of data will be generated daily, leading to an accrual of 3 PB annually, and as much as 100 TB over the full program lifecycle.

Two Cross-Dragone (CD) telescopes (one is pictured above) with six meter diameter input apertures will be deployed at the Chilean site to map roughly 70% of the sky every day to support the dark universe, matter-mapping, and time-varying mm-wave sky science goals. Image and caption courtesy of the CMB-S4 Project

Network Requirements Reviews analyze the current, near, and long-term needs of the HEP community, providing a network and data-centric understanding of the scientific process used by the researchers and scientists. These requirements reviews drive ESnet’s investments in new services and capabilities, and enable ESnet to build strong partnerships with Office of Science (SC) programs, PIs, and user facilities. More information on this ESnet requirements review process can be found here.

We would like to thank the 13 HEP projects, and all of the HEP & DOE Office of Science collaborators who generously gave of their time, expertise, and most importantly, their enthusiasm for the future of high energy physics, as part of creating this report.

We want to especially thank the entire Science Engagement team plus Kate Robinson, and Dale Carder with our Network Engineering group who all provided outstanding support and technical expertise.

Programmable Per-Packet Network Telemetry: From Wire to Kafka at Scale

High-speed intelligent Research and Educational Networks (RENs),  such as the one we’re building as part of the ESnet 6 program, will require a greater ability to understand and manage traffic flows. One research program underway to provide this capability is the High Touch effort,  a programmable, scalable, and expressive hardware and software solution that produces and analyzes per-packet telemetry information with nanosecond-accurate timing. Along with Zhang Liu, Bruce Mah, Yatish Kumar, and Chin Guok, I have just released a presentation for the Proceedings of the 2021 Virtual Meeting on Systems and Network Telemetry and Analytics, describing work underway to create a programmable, very high speed, packet monitoring, and telemetry capability as part of bringing High-Touch to life.

Richard Cziva presenting at the SNTA ’21: Proceedings of the 2021 on Systems and Network Telemetry and Analytics

For more information on this talk, please see this link.

Fatema Bannat Wala named Zeek Community Champion!

Fatema Bannat Wala

Fatema Bannat Wala with our Cyber Security team was recognized with the 2021 Zeek Community Champion award by Corelight! More information on the award and her work with Zeek can be found here.

Zeek is an open source network security monitoring software extensively used by ESnet. Zeek (formally called Bro) was initially developed by researchers at Berkeley Lab, and more information on ESnet’s use of Zeek can be found in Fatema’s October Light Bytes post.

CONGRATULATIONS, Fatema!

Three Questions with a new staff member: John Amerkhanian

Please meet our newest Network Operations Center Engineer, John Amerkhanian. John comes to us from Richmond, CA, and grew up locally in Albany, CA. He graduated from UC Berkeley in 2015 with a degree in Political Science.

What brought you to ESnet?

As a kid growing up in the Berkeley area, you always heard about how there is exciting research happening in the LBNL buildings up on the hill. When my friend got a job with ESnet in 2016, I knew I’d like to join them there someday. I’m very excited to support some of the best energy researchers in the world and can’t wait to see how they’re improving the ways we produce, consume, and store energy. 

What is the most exciting thing going on in your field right now?

Without a doubt it’s the leaps and bounds made in computer processor development, these days you can get a processor that is a fraction of the size of a Pentium 4 with nearly double the processing power and very low energy usage. The computing applications for these processors in my field are very exciting.

What book would you recommend?

Steve Wozniak’s autobiography “iWoz”

Deploying ZoMbis at ESnet – Part II

In the previous post we discussed deploying ZoMbis (Zeek on Management based information system) for ESnet6’s management network to monitor the traffic traversing the network and to provide visibility into what’s happening on our management network. This blog post will discuss how we use traffic sinkholes, which are a way of redirecting traffic so that it can be captured and analyzed. As opposed to our usual passive data collection system (e.g., tapping or port mirroring), traffic is being actively redirected to network monitoring systems such as Zeek. Network sensors can then perform various levels of in-depth analysis on the traffic, which can help detect misconfigurations, identify hostile traffic, or even perform automated mitigations for certain attacks.

Sinkholes are an important tool in the arsenal of network operators—they support network cyber defense by providing a way to redirect packets sent to or from unallocated (so-called “bogon” addresses) or other unexpected IP addresses. Additionally, they can help protect against reconnaissance or vulnerability scanning. If an attack does slip through these defenses, the damage could be limited, or the malicious traffic could be analyzed by network defenders to determine the source and methods being used.

As part of the ESnet6 security architecture, a sinkhole service will be deployed on the production management network, to redirect internal management traffic as well as externally sourced internet traffic destined to the management network. Using the Border Gateway Protocol (BGP), the sinkholes will advertise routes to the destination gateway for IP ranges of the management network to redirect the traffic to the target sinkhole. In our network, the management plane address set fits within a “supernet” (a collection of subnets) which can then advertise the sinkhole address as a destination. We will use this advertised supernet to redirect all traffic from external sources on the Internet away from the management network and to the external sinkhole.

An internal sinkhole will also advertise this management supernet for “inside” resources, but in this case, legitimate traffic will have a more specific route for the destination and not go to the sinkhole. This way, only traffic destined to an invalid subnet will be redirected to the internal sinkhole. This design should be extremely useful in identifying possible misconfigurations or other unexpected behaviors in the ESnet6 management network. if everything is behaving as expected, we should never see any traffic to the catch-all destination of the sinkhole.

The following diagram, taken from a ZeekWeek 2020 presentation by ESnet security engineer Scott Campbell, shows the basic design of the two kinds of sinkholes:

Example External Sinkhole

In the external sinkhole conceptual diagram above, routers R1 and R2 will be advertising the management address ranges to external sources. If any traffic destined to the management network is received from the Internet, it will instead be redirected to the sinkhole. 

The external use case is a bit simpler than the internal sinkhole, which is diagrammed below. In the latter case, there will be some legitimate connections, such as between two ESnet points of presence (POPs), or between a POP and our data center. Any unwanted, misconfigured, or hostile scanning traffic will end up in the internal sinkhole. Hence internal sinkholes can be thought of both as network “garbage cans” and intrusion sensors helping to detect changes in normal management traffic patterns. 

Example Internal Sinkhole

The ESnet Security Team will use Zeek, to analyze traffic at the application level, for both types of sinkholes. The logs generated by Zeek will then be collected centrally and will provide useful insights into what kind of unwanted traffic is being directed at our management plane, both from internal or external sources, and help better protect ESnet6 from attackers. 

ESnet6 Achieves 2021 Annual Review Milestone – the future research and education network is one step closer!

The ESnet6 2021 Annual Status Review was a great success, and the Review Committee, led by DOE, concluded that the ESnet6 Project is being managed and executed well!

Given that the project’s budget, scope, and schedule were approved in February 2020, this was the first official Annual Status Review – and what a year it has been! The 2021 Review was a major milestone, allowing the Project to formally present the project performance over the past year and, consequently, during the COVID-19 pandemic. I continue to be amazed by the entire project team, and I felt very honored to be the one to introduce the astounding progress we made during an extremely challenging year. Not only that, it was all done while operating the current ESnet5 production network at the same time.

The project execution continued at full speed while some of us started carving out time over the past several months to prepare for the Review. Pulling together all of the information required, synthesizing it into a clear and concise set of briefings and documents, and presenting it to leaders in our field is a monumental task under any circumstances, but the pandemic made this especially difficult. However, the project team, backed by strong support across LBNL (Procurement, Project Management Office, Project Management Advisory Board members, and many others) made everything appear seamless. The impressive level of teamwork did not go unnoticed and was specifically mentioned repeatedly during the Closeout session. I am grateful for and proud of, all of the members of the team who contributed to this terrific success.

The Review Committee consisted of three Subcommittees (Technical, Cost & Schedule, Project Management & Environment, Safety & Health), all charged with answering a set of questions to determine if we were on schedule, achieving scope, within budget, and performing all tasks safely. The answer to every charge question: Yes! It was an all-encompassing couple of days, but we really couldn’t have asked for a better result. In short, there were no formal recommendations, so we’ll be considering how best to implement several of the Review Committee’s extremely helpful comments as we proceed onward. Our hard work, not only on the Review itself, paid off!

With the formal Review complete for the year, we’re all back to our daily project plan of execution, while keeping the network “lights on” in the process, of course.

Three questions with a new staff member –James Kafader with Software Engineering.

Please welcome James Kafader to ESnet! James comes to us from Internet Archive (IA), where he worked on the Archive-It team, which develops and maintains a turnkey archiving platform. Archive-It partners with external institutions and national libraries to capture data on their behalf. It is essentially the project incubator at IA and focused on high-quality and large-scale archiving. The data collected by Archive-It represents about 30% of the available captures in the global wayback machine.

Question 1: What brought you to ESnet?

In 2020, I spent a lot of time thinking about the interconnectedness of natural systems, and how they relate to the earth’s climate. It strikes me that it’s imperative, as a planet and nation, to focus on reducing the impact of climate change in short order. This line of thinking led me to dedicate my time to science, which could have a positive impact on the global climate.

Question 2: What is the most exciting thing going on in your field right now?

This is a good question. I consider myself very much a generalist in terms of how I approach software development, as well as in my overall view of reality. My view of computational systems is very conservative as well — I like to understand the algorithms involved with any new technology as intimately as possible before selecting it for use. I’d say in many ways that the most exciting thing going on in my field is renewed interest in how large-scale systems affect equitability for their participants; that is, how the networks, systems, and structures that we build affect outcomes for each of us.

Question 3: What book would you recommend?

I recently read Breath by James Nestor. It was an engaging read and helped a lot with my mood and stability, if not the most scientifically accurate thing I’ve ever read. Another favorite is Difficult Conversations by Sheila Heen, Douglas Stone, and Bruce Patton.