Programmable Per-Packet Network Telemetry: From Wire to Kafka at Scale

High-speed intelligent Research and Educational Networks (RENs),  such as the one we’re building as part of the ESnet 6 program, will require a greater ability to understand and manage traffic flows. One research program underway to provide this capability is the High Touch effort,  a programmable, scalable, and expressive hardware and software solution that produces and analyzes per-packet telemetry information with nanosecond-accurate timing. Along with Zhang Liu, Bruce Mah, Yatish Kumar, and Chin Guok, I have just released a presentation for the Proceedings of the 2021 Virtual Meeting on Systems and Network Telemetry and Analytics, describing work underway to create a programmable, very high speed, packet monitoring, and telemetry capability as part of bringing High-Touch to life.

Richard Cziva presenting at the SNTA ’21: Proceedings of the 2021 on Systems and Network Telemetry and Analytics

For more information on this talk, please see this link.

Fatema Bannat Wala named Zeek Community Champion!

Fatema Bannat Wala

Fatema Bannat Wala with our Cyber Security team was recognized with the 2021 Zeek Community Champion award by Corelight! More information on the award and her work with Zeek can be found here.

Zeek is an open source network security monitoring software extensively used by ESnet. Zeek (formally called Bro) was initially developed by researchers at Berkeley Lab, and more information on ESnet’s use of Zeek can be found in Fatema’s October Light Bytes post.


Three Questions with a new staff member: John Amerkhanian

Please meet our newest Network Operations Center Engineer, John Amerkhanian. John comes to us from Richmond, CA, and grew up locally in Albany, CA. He graduated from UC Berkeley in 2015 with a degree in Political Science.

What brought you to ESnet?

As a kid growing up in the Berkeley area, you always heard about how there is exciting research happening in the LBNL buildings up on the hill. When my friend got a job with ESnet in 2016, I knew I’d like to join them there someday. I’m very excited to support some of the best energy researchers in the world and can’t wait to see how they’re improving the ways we produce, consume, and store energy. 

What is the most exciting thing going on in your field right now?

Without a doubt it’s the leaps and bounds made in computer processor development, these days you can get a processor that is a fraction of the size of a Pentium 4 with nearly double the processing power and very low energy usage. The computing applications for these processors in my field are very exciting.

What book would you recommend?

Steve Wozniak’s autobiography “iWoz”

Deploying ZoMbis at ESnet – Part II

In the previous post we discussed deploying ZoMbis (Zeek on Management based information system) for ESnet6’s management network to monitor the traffic traversing the network and to provide visibility into what’s happening on our management network. This blog post will discuss how we use traffic sinkholes, which are a way of redirecting traffic so that it can be captured and analyzed. As opposed to our usual passive data collection system (e.g., tapping or port mirroring), traffic is being actively redirected to network monitoring systems such as Zeek. Network sensors can then perform various levels of in-depth analysis on the traffic, which can help detect misconfigurations, identify hostile traffic, or even perform automated mitigations for certain attacks.

Sinkholes are an important tool in the arsenal of network operators—they support network cyber defense by providing a way to redirect packets sent to or from unallocated (so-called “bogon” addresses) or other unexpected IP addresses. Additionally, they can help protect against reconnaissance or vulnerability scanning. If an attack does slip through these defenses, the damage could be limited, or the malicious traffic could be analyzed by network defenders to determine the source and methods being used.

As part of the ESnet6 security architecture, a sinkhole service will be deployed on the production management network, to redirect internal management traffic as well as externally sourced internet traffic destined to the management network. Using the Border Gateway Protocol (BGP), the sinkholes will advertise routes to the destination gateway for IP ranges of the management network to redirect the traffic to the target sinkhole. In our network, the management plane address set fits within a “supernet” (a collection of subnets) which can then advertise the sinkhole address as a destination. We will use this advertised supernet to redirect all traffic from external sources on the Internet away from the management network and to the external sinkhole.

An internal sinkhole will also advertise this management supernet for “inside” resources, but in this case, legitimate traffic will have a more specific route for the destination and not go to the sinkhole. This way, only traffic destined to an invalid subnet will be redirected to the internal sinkhole. This design should be extremely useful in identifying possible misconfigurations or other unexpected behaviors in the ESnet6 management network. if everything is behaving as expected, we should never see any traffic to the catch-all destination of the sinkhole.

The following diagram, taken from a ZeekWeek 2020 presentation by ESnet security engineer Scott Campbell, shows the basic design of the two kinds of sinkholes:

Example External Sinkhole

In the external sinkhole conceptual diagram above, routers R1 and R2 will be advertising the management address ranges to external sources. If any traffic destined to the management network is received from the Internet, it will instead be redirected to the sinkhole. 

The external use case is a bit simpler than the internal sinkhole, which is diagrammed below. In the latter case, there will be some legitimate connections, such as between two ESnet points of presence (POPs), or between a POP and our data center. Any unwanted, misconfigured, or hostile scanning traffic will end up in the internal sinkhole. Hence internal sinkholes can be thought of both as network “garbage cans” and intrusion sensors helping to detect changes in normal management traffic patterns. 

Example Internal Sinkhole

The ESnet Security Team will use Zeek, to analyze traffic at the application level, for both types of sinkholes. The logs generated by Zeek will then be collected centrally and will provide useful insights into what kind of unwanted traffic is being directed at our management plane, both from internal or external sources, and help better protect ESnet6 from attackers. 

ESnet6 Achieves 2021 Annual Review Milestone – the future research and education network is one step closer!

The ESnet6 2021 Annual Status Review was a great success, and the Review Committee, led by DOE, concluded that the ESnet6 Project is being managed and executed well!

Given that the project’s budget, scope, and schedule were approved in February 2020, this was the first official Annual Status Review – and what a year it has been! The 2021 Review was a major milestone, allowing the Project to formally present the project performance over the past year and, consequently, during the COVID-19 pandemic. I continue to be amazed by the entire project team, and I felt very honored to be the one to introduce the astounding progress we made during an extremely challenging year. Not only that, it was all done while operating the current ESnet5 production network at the same time.

The project execution continued at full speed while some of us started carving out time over the past several months to prepare for the Review. Pulling together all of the information required, synthesizing it into a clear and concise set of briefings and documents, and presenting it to leaders in our field is a monumental task under any circumstances, but the pandemic made this especially difficult. However, the project team, backed by strong support across LBNL (Procurement, Project Management Office, Project Management Advisory Board members, and many others) made everything appear seamless. The impressive level of teamwork did not go unnoticed and was specifically mentioned repeatedly during the Closeout session. I am grateful for and proud of, all of the members of the team who contributed to this terrific success.

The Review Committee consisted of three Subcommittees (Technical, Cost & Schedule, Project Management & Environment, Safety & Health), all charged with answering a set of questions to determine if we were on schedule, achieving scope, within budget, and performing all tasks safely. The answer to every charge question: Yes! It was an all-encompassing couple of days, but we really couldn’t have asked for a better result. In short, there were no formal recommendations, so we’ll be considering how best to implement several of the Review Committee’s extremely helpful comments as we proceed onward. Our hard work, not only on the Review itself, paid off!

With the formal Review complete for the year, we’re all back to our daily project plan of execution, while keeping the network “lights on” in the process, of course.

Three questions with a new staff member –James Kafader with Software Engineering.

Please welcome James Kafader to ESnet! James comes to us from Internet Archive (IA), where he worked on the Archive-It team, which develops and maintains a turnkey archiving platform. Archive-It partners with external institutions and national libraries to capture data on their behalf. It is essentially the project incubator at IA and focused on high-quality and large-scale archiving. The data collected by Archive-It represents about 30% of the available captures in the global wayback machine.

Question 1: What brought you to ESnet?

In 2020, I spent a lot of time thinking about the interconnectedness of natural systems, and how they relate to the earth’s climate. It strikes me that it’s imperative, as a planet and nation, to focus on reducing the impact of climate change in short order. This line of thinking led me to dedicate my time to science, which could have a positive impact on the global climate.

Question 2: What is the most exciting thing going on in your field right now?

This is a good question. I consider myself very much a generalist in terms of how I approach software development, as well as in my overall view of reality. My view of computational systems is very conservative as well — I like to understand the algorithms involved with any new technology as intimately as possible before selecting it for use. I’d say in many ways that the most exciting thing going on in my field is renewed interest in how large-scale systems affect equitability for their participants; that is, how the networks, systems, and structures that we build affect outcomes for each of us.

Question 3: What book would you recommend?

I recently read Breath by James Nestor. It was an engaging read and helped a lot with my mood and stability, if not the most scientifically accurate thing I’ve ever read. Another favorite is Difficult Conversations by Sheila Heen, Douglas Stone, and Bruce Patton.

Creating the Tokamak Superfacility: Fusion with the ScienceDMZ

5.5 Questions with Eli Dart (ESnet), C.S. Chang, and Michael Churchill (PPPL)

In 2025, when the International Thermonuclear Experimental Reactor (ITER) generates “first plasma”, it will be the culmination of almost 40 years of effort.  First started in 1985, the project has grown to include the scientific talents of seven members (China, EU, India, Japan, Korea, Russia, and the US, with EU membership bringing the total to 35 countries) and if successful, will mark the first time that a large scale fusion reactor generates more thermal power than is used to heat isotopes of hydrogen gas to a plasma state.

ESnet is supporting this international scientific community as this dream of limitless, clean energy is pursued. When operational at full capacity, ITER will generate approximately a petabyte-per-day of data, much of which will need to be analyzed and fed back in near real-time to optimize the fusion reaction and manage distribution of data to a federated framework of geographically distributed “remote control rooms” or RCR.  To prepare for this demanding ability to distribute both data and analytics, recently ESnet’s Eli Dart and the Princeton Plasma Physics Laboratory’s (PPPL) Michael Churchill and  C.S. Chang were co-authors on a test exercise performed with collaborators at Pacific Northwest National Laboratory (PNNL), PPPL, Oak Ridge National Laboratory (ORNL), and with the Korean KREONET, KSTAR, National Fusion Research Institute, and the Ulsan National Institute of Science and Technology. This study ( successfully demonstrated the use of ESnet and the ScienceDMZ architecture as part of trans-Pacific large data transfer, and near real-time movie creation and analysis of the KSTAR electron cyclotron emission images, via links between multiple paths at high sustained speeds.

Q 1: This was a complex test, involving several sites and analytic workflows.  Can you walk our readers through the end-to-end workflow? 

End-to-end workflow of the demonstration comparing real-time streaming data from the KSTAR ECEI diagnostic to side-by-side movie from XGC1 gyrokinetic turbulence code.
End-to-end workflow of the demonstration comparing real-time streaming data from the KSTAR ECEI diagnostic to side-by-side movie from XGC1 gyrokinetic turbulence code.

Eli Dart: The data were streamed from a system at KSTAR, encoded into ADIOS format, streamed to PPPL, rendered into movie frames, and visualized at PPPL. One of the key attributes of this workflow is that it is a streaming workflow. Specifically, this means that the data passes through the workflow steps (encoding in ADIOS format, transfer, rendering movie frames, showing the movie) without being written to non-volatile storage. This allows for performance improvements, because no time is spent on storage I/O. It also removes the restriction of storage allocations from the operation of the workflow – only the final data products need to be stored (if desired). 

Q 2: A big portion of this research supports the idea of federated, near real-time analysis of data.  In order to make these data transfers performant, flexible, and adaptable enough to meet the requirements for a future ITER RCR, you had to carefully engineer and coordinate with many parties.  What was the hardest part of this experiment, and what lessons does it offer ITER?

Eli Dart: It is really important to ensure that the network path is clean. By “clean” I mean that the network needs to provide loss-free IP service for the experiment traffic. Because the fusion research community is globally distributed, the data transfers cover long distances, which greatly magnifies the negative impact of packet loss on transfer performance. Test and measurement (using perfSONAR) is very important to ensure that the network is clean, as is operational excellence to ensure that problems are fixed quickly if they arise. KREONET is an example of a well-run production network – their operational excellence contributed significantly to the success of this effort.

Q 3: One of the issues you had to work around was a firewall at one institution.  What was involved in working with their site security, and how should those working with Science DMZ work through these issues?

Eli Dart: Building and operating a Science DMZ involves a combination of technical and organizational work. Different institutions have different policies, and the need for different levels of assurance depending on the nature of the work being done on the Science DMZ. The key is to understand that security policy is there for a reason, and to work with the parties involved in the context that makes sense from their perspective. Then, it’s just a matter of working together to find a workable solution that preserves safety from a cybersecurity perspective and also allows the science mission to succeed. 

Q 4: How did you build this collaboration and how did you keep everyone on the same page, any advice you can offer other experiments facing the same need to coordinate multi-national efforts?

Eli Dart: From my perspective, this result demonstrates the value of multi-institution, multi-disciplinary collaborations for achieving important scientific outcomes. Modern science is complex, and we are increasingly in a place where only teams can bring all the necessary expertise to bear on a complex problem. The members of this team have worked together in smaller groups on a variety of projects over the years – those relationships were very valuable in achieving this result.

Q 5: In the paper you present a model for a federated remote framework workflow. Looking beyond ITER, are there other applications you can see for the lessons learned from this experiment?

C.S. Chang: Lessons learned from this experiment can be applied to many other distributed scientific, industrial, and commercial applications which require collaborative data analysis and decision making.  We do not need to look too far.  Expensive scientific studies on exascale computers will most likely be collaborative efforts among geographically distributed scientists who want to analyze the simulation data and share/combine the findings in near-real-time for speedy scientific discovery and for steering of ongoing or next simulations.  The lessons learned here can influence the remote collaboration workflow used in high energy physics, climate science, space physics, and others.

Q 5.5: What’s next? You mention quite a number of possible follow on activities in the paper? Which of these most interest you, and what might follow?

Michael Churchill: Continued work by this group has led to the recently developed  open-source Python framework, DELTA, for streaming data from experiments to remote compute centers, using ADIOS for streaming over wide-area networks, and on the receiver side using asynchronous Message Passing Interface to do parallel processing of the data streams. We’ve used this for streaming data from KSTAR to the NERSC Cori supercomputer and completing a spectral analysis in parallel in less than 10 minutes, which normally in serial would take 12 hours. Frameworks such as this, enabling connecting experiments to remote high-performance computers, will open up the quality and quantity of analysis workflows that experimental scientists can run. It’s exciting to see how this can help accelerate the progress of science around the world.

Congratulations on your success! This is a significant step forward in building the data management capability that ITER will need.  

Graduate students publish on network telemetry with ESnet

Two graduate students working with ESnet have published their papers recently in IEEE and ACM workshops.

Bibek Shrestha, a graduate student at the University of Nevada, Reno, and his advisor Engin Arslan worked with Richard Cziva from ESnet to publish a work on “INT Based Network-Aware Task Scheduling for Edge Computing”. In the paper, Bibek investigated the use of in-band network telemetry (INT) for real-time in-network task scheduling. Bibek’s experimental analysis using various workload types and network congestion scenarios revealed that enhancing task scheduling of edge computing with high-precision network telemetry can lead up to a 40% reduction in data transfer times and up to 30% reduction in total task execution times by favoring edge servers in uncongested (or mildly congested) sections of the network when scheduling tasks. The paper will appear in the 3rd Workshop on Parallel AI and Systems for the Edge (PAISE) co-conducted with IEEE IPDPS 2021 conference to be held on May 21st, 2021, in Portland, Oregon. 

Zhang Liu, a former ESnet intern and a current graduate student at the University of Colorado at Boulder, worked with the ESnet High Touch Team – Chin Guok, Bruce Mah, Yatish Kumar, and Richard Cziva – on fastcapa-ng, ESnet’s telemetry processing software. In the paper “Programmable Per-Packet Network Telemetry: From Wire to Kafka at Scale,” Zhang showed the scaling and performance characteristics of fastcapa-ng, and highlighted the most critical performance considerations that allow the pushing of 10.4 million telemetry packets per second to Kafka with only 5 CPU cores, which is more than enough to handle 170 Gbit/s of original traffic with 1512B MTU. This paper will appear in the 4th International Workshop on Systems and Network Telemetry and Analytics (SNTA 2021) held at the ACM HPCD 2021 conference in Stockholm, Sweden between 21-25 June 2021.

Congratulations Bibek and Zhang!

If you are a networked systems research student looking to collaborate with us on network measurements, please reach out to Richard Cziva. If you are interested in a summer internship with ESnet, please visit this page.

Arecibo Data Recovery: Behind the Scenes with Jason Zurawski

The dramatic collapse of the Arecibo Observatory Radio Telescope in Puerto Rico in December was a terrible loss for global science. The 305-meter dish had served for over 50 years, supporting a wide range of cosmic and earth science applications, including transmission of the famed “Arecibo Message” to globular star cluster M13 by a team led by Frank Drake and Carl Sagan in 1974.

When the 900-ton instrument platform crashed onto the observatory dish, the National Science Foundation was faced with a variety of challenges. Most immediately, how to ensure that several petabytes of historic (and now irreplaceable) data at the Arecibo Observatory (AO) data center, in the form of tapes, hard drives, and other physical media, could be preserved and moved off-site as an approximately $50M site cleanup and environmental remediation project begins to demobilize the iconic observatory.

This data recovery effort has required rapid mobilization of a team from the University of Central Florida (UCF), the Texas Advanced Computing Center  (TACC), the University of Puerto Rico (UPR), the University of Chicago, and others. A more detailed description of this overall effort has just been released here. In this blog, I will describe the key role that ESnet and the Engagement and Performance Operations Center (EPOC) played in this effort to save valuable scientific data.

My colleagues Hans Addleman (Indiana University International Networks), George Robb, and I became part of science use case discussions with AO and UCF as part of an ESnet requirements review and EPOC Deep Dive support to Arecibo in early 2020. In the summer of 2020, these efforts became much more active after the first suspension cable failed and AO began activities to migrate data storage and processing to a commercial cloud. We provided support to the Arecibo team for data movement hardware and software deployment.

With the failure of a second cable in November 2020, it became apparent that the facility had become unstable; this increased pressure on the team to find a faster solution. The UCF site management team decided that migration to the commercial cloud over the available 1Gbps connection (a previous 10Gbps connection was damaged by Hurricane Maria in 2017) would not meet requirements, so another data migration strategy was needed.

By December the  team developed an alternative data migration approach leveraging a timely offer of storage capacity at TACC. Because of the urgency, the team decided to move data using physical Network Attached Storage (NAS) appliances; data on tapes and other original sources were loaded onto the NAS at Arecibo. The NAS were then driven to data centers on the island: either at the UPR campuses located at Mayagüez or Río Piedras, or at a commercial data center on the island, each of which were connected to the global R&E network at 10Gbps. Using Globus data transfer software, the AO team then began the process of transferring the data to TACC. Using multiple devices, and by setting up a constantly moving supply line, they were able to fill a disk, transport to the better connected locations, start a transfer, take back a completed disk, and return to the AO data center to start the process all over. 

EPOC team members (specifically George) spent a lot of time working with AO and Globus technical support to tune the NAS appliances (which are usually used in commercial/enterprise settings) to be able to transfer the data at higher rates of performance than the factory settings allowed. EPOC, AO, UCF, TACC, and UPR staff also ran perfSONAR tests to ensure the entire path was able to deliver on these faster speeds that were necessary. George will be presenting a talk at Globus World in May, and those interested in more information about how this networking and disk NAS tuning was done should plan to attend.

The data transfer operation started in late December 2020 and is expected to continue through the spring of 2021, as stored data (on disks and tapes at AO) is transferred to TACC. As data flows into TACC’s storage cluster from Arecibo’s holdings, ESnet and the entire collaboration team will ensure that it is made widely available to the scientific community to perform new studies with this valuable research data. 

The destruction of the AO Radio Telescope was a catastrophe for global science; however, the quick response of the entire data recovery team helped prevent the loss of much of the valuable data collected by Arecibo over its lifetime. I’m very proud of this accomplishment: the work of the entire ESnet team and our data infrastructure ensured that we had the right capabilities at the right time to make a difference for science.

Three Questions with Joseph Nasal

Three questions with a new staff member!  Today, Joseph (Joe) Nasal, who has joined our Business Office as a Project Manager.

After graduating from Temple University, Joe began his career designing broadband Radio Frequency-hybrid fiber networks and management software for some of the first residential cable modem deployments in the country.  Early on, he also worked in defense and designed and operated private secure communications networks for federal contractors.  He spent the past two decades supporting higher education through roles in engineering, technical architecture, project management, and leadership. His work helped transform data communication at Pennsylvania State University, preparing the campus for tremendous growth in teaching and research. 

What brought you to ESnet?

I’ve been architecting and managing very large communication network design and implementation projects for most of my career.  After nearly 20 years at Penn State, it was time for a career change.  One of my close colleagues recently came to ESnet in support of Science Engagement, and when I learned through him of an opportunity to help with such exciting and important growth on a national scale I was very happy to find a place in the organization.  I’ll be operating out of my home office in State College, PA.

What is the most exciting thing going on in your field right now?

In data communications, it’s about getting more for less—more throughput, more distance, more fidelity, for less cost.  Cost is measured in units like dollars, or time, or energy, or human effort, and those of us who work in this space are always trying to optimize these resources. This is an exciting time because it seems like we’re on the cusp of training machines to give us a magnitude leap forward in efficiencies via automated processes and learning algorithms. But it’s going to take clear human vision to get us to where we want to be, which means as engineers, we will continue to have fun solving big problems. 

What book would you recommend?

The Man Who Loved Only Numbers, a biography of Paul Erdős.  Paul was one of the great mathematicians of the 20th century whose work has implications for both computer science and information theory.  He was an eccentric genius and his personal story is a fascinating one to follow. As engineers, I think it’s important to be aware of and appreciate the great thinkers who exist at the very base level of abstraction with respect to the technologies we use and build upon.