Defending ESnet with ZoMbis!

Zeek is a powerful open source network security monitoring software extensively used by ESnet. Zeek (formally called Bro) was initially developed by researchers at Berkeley Lab; it allows users to identify & manage cyber threats by tracking and logging network traffic activity. Zeek operates as a passive monitor, providing a holistic view of what is transpiring in the network and on all network traffic. 

In a previous post, I presented some of our efforts in approaching the WAN security using Zeek for general network monitoring, with successes and challenges found during the process. In this blog post I’ll focus on our efforts in using Zeek as  part of security monitoring for the ESnet6 management network – ZoMbis (Zeek on Management based information system).

ZoMbis on the ESnet6 management network:

Most research and educational networks employ a dedicated management network as a best practice. The management network provides a configuration command and control layer, as well as conduits for all of the inter-routing communications between the devices used to move our critical customer data. Because of the sensitive nature of these communications, the management network needs to be protected from external and general user network traffic (websites, file transfers, etc.), and our staff needs to have detailed visibility on management network activity.

At ESnet, we typically use real IP addresses for all internal network resources, and our management network is allocated a fairly large address space block advertised in our global routing table, to help protect against opportunistic hijacking attacks. By isolating our management network from user data streams, the amount of routine background noise is vastly reduced making the use of Zeek, or any network monitoring security capability, much more effective. 

The above diagram shows an overview of the deployment strategy of Zeek on the ESnet6 management network. The blue dots in the diagram show the locations that will have equipment running Zeek instances for monitoring the network traffic on the management network. The traffic from the routers on those locations is mirrored to the Zeek instances using a spanning port, and the Zeek logs generated are then aggregated in our central security information and event logging and management system (SIEM).

Scott Campbell presented ‘Using Zeek in ESnet6 Management Network Security Monitoring’ during virtual Zeek Week held last year that explained the overall strategy for deployment of Zeek on the management network in greater detail. Some ZoMbis deployment highlights are:

ESnet 6’s new management network will use only IPv6. From a monitoring perspective this change from the traditional IPv4 poses a number of interesting challenges; In particular, IPv6 traffic employs more multicast and link-local traffic for local subnet communications. Accordingly, we are in the process of adjusting and adding to Zeek’s policy based detection scripts to support these changes in network patterns. These new enhancements and custom scripts being written by our cybersecurity team to support IPv6 will be of interest to other Zeek users and we will release them to the entire Zeek community soon. 

The set of Zeek policy created for this project can be broken out into two general groups. The first of these is protocol mechanics – particularly looking closer between layer 2 and 3 where there are a number of interesting security behaviors with IPv6.  A subset of notices that these protocol mechanic policies will provide are:

  • ICMP6_RtrSol_NotMulticat – Router solicitation not multicast
  • ICMP6_RtrAnn_NotMulticat – Router announcement should be a multicast request
  • ICMP6_RtrAnn_NewMac – Router announcement from an unknown MAC
  • ICMP6_MacIPChange – If the MAC <-> IP mapping changes
  • ICMP6_NbrAdv_NotRouter – Advertisement comes from non-router
  • ICMP6_NbrAdv_UnSolicit – Advertisement is not solicited
  • ICMP6_NbrAdv_OverRide – Advertisement without override
  • ICMP6_NbrAdv_NoRequest – Advertisement without known request

The second set of Zeek policies that have been developed in support for ZoMbis involves taking advantage of predictable management network behavioral patterns – we build policy to model anticipated behaviors and let us know if something is amiss. For example looking at DNS and NTP behavior we can identify unexpected hosts and data volumes, since we know which systems are supposed to be communicating with one another, and what patterns traffic between these components should follow.

Stay tuned for the part II of this blogpost, where I will discuss ways of using Sinkholing, together with ZoMbis, to provide better understanding and visibility of unwanted traffic upon the management network.

Zeek and stream asymmetry research at ESnet

In my previous post, we discussed use of the open-source Zeek software to support network security monitoring at ESnet.  In this post, I’ll talk a little about work underway to improve Zeek’s ability to support network traffic monitoring when faced with stream asymmetry.

This comes from recent work by two of my colleagues on the ESnet Security team.

Scott Campbell and Sam Oehlert presented ‘Running Zeek on the WAN: Experiences and solutions for large scale flow asymmetry’ during a workshop held last year at CERN Geneva that explained the phases and deployment of the Zeek-on-the-WAN (ZoW) pilot in detail.

Scott Campbell at CERN presenting ‘Running Zeek on the WAN’
The asymmetry problem on a WAN (example)

Some of the significant findings and results from this presentation are highlighted below:

  • Phase I: Initial Zeek Node Design Considerations 
    • Select locations that provide an interesting network vantage point – in the case of our ESnet network, we deployed Zeek nodes on our commodity internet peerings (eqx-sj, eqx-chi, eqx-ash) since they represent the interface to the vast majority of hostile traffic.
    • Identifying easy traffic to test with and using spanning ports to forward traffic destined to the stub network on each of the routers used for collection.
  • Phase I: Initial Lessons learned from testing and results
    • Some misconfigurations were found in the ACL prefix lists. 
    • We increased visibility into our WAN side traffic through implementation of new background methods.
    • Establishing a new process for end-to-end testing, installing and verifying Zeek system reporting. 
  • Phase II:  Prove there is more useful data to be seen
    • For phase II we moved towards collection of full peer connection records, from statistical sampling based techniques. Started running Zeek on traffic crossing the interfaces which connect ESnet network peers to the internet from the AS (Autonomous system) responsible for most notices. .
    • To get high fidelity connection information without being crushed by data volume, define a subset of packets that are interesting – zero length control packets (Syn/Syn-Ack/Fin/Rst) from peerings.
  • Phase II: Results
    • A lot of interesting activity got discovered like information leakage in syslogs, logins (and attempted logins) using poorly secure authentication protocols, and analysis of the amount of asymmetric traffic patterns gave valuable insights to understand better the asymmetric traffic problems.
  • Ongoing Phase III: Expanding the reach of traffic collection on WAN
    • We are currently in the process of deploying Zeek nodes at another three WAN locations for monitoring commodity internet peering – PNWG (peering at Seattle WA), AM-SIX (peering at Amsterdam) and LOND (peering at London)
Locations for the ZoW systems, the pink shows ongoing Phase III deployment

As our use of Zeek on the WAN side of ESnet continues to grow, the next phase to the ZoW pilot is currently being defined.  We’re working to incorporate these lessons learned on how to handle traffic asymmetry into these next phases of effort. 

Some (not all) solutions being taken into consideration include: 

  • Aggregating traffic streams at a central location to make sense out of the asymmetric packet streams and then run Zeek on the aggregated traffic, or
  • Running Zeek on the individual asymmetric streams and then aggregating these Zeek streams @ 5-tuple which will be aggregation of connection metadata rather than the connection stream itself. 

We are currently exploring these WAN solutions as part of providing better solutions to both ESnet, and connected sites.

Zeekurity at ESnet

Zeek is an open source network security monitoring software extensively used by ESnet.  Zeek (formally called Bro) was initially developed by researchers at Berkeley Lab. Zeek allows users to identify & manage cyber threats by monitoring network traffic. It acts as a passive monitoring software (NSM – Network Security Monitor), that gives a holistic view of what is transpiring in the network and gives visibility into the network traffic. 

In order to better understand network behavior and provide flexible security services, we use Zeek as an important part of our data center security architecture and are experimenting with placing Zeek clusters on various WAN high value points. This is providing technical insights as well as significant challenges. 

In this post we would present some of our efforts in approaching the WAN security using Zeek for network monitoring, with successes and challenges hit during the process and interesting things learned.

Zeek on the ESnet LAN:

Monitoring local area and data center networks is a familiar and less complex network traffic monitoring design, and ESnet is no different. The traffic flowing through the LAN networks is currently monitored using two Zeek clusters, one at Brookhaven National Lab and another for the west coast at Berkeley Lab. We have implemented BHR (black hole routing) functionality on our data center routers to block external actors which violate our established policies based on Zeek detections on both IPv4 and IPv6 protocol stacks. 

Apart from network security monitoring using “standard” Zeek detection scripts, many enhancements and custom scripts written by the ESnet Security team members serve a vital role in detecting various kinds of suspicious activity. Recently, a Zeek package – Zeek-Known-outbound contributed by Michael “Dop” Dopheide won the first prize in the Zeek Package Contest-2 held in May 2020. The package provides the ability to track and alert on outbound service usage to a list of ‘watched’ countries, and also adds the country codes for the origin and recipient hosts in one of the log files that Zeek generates called conn.log, to log all the connection attempts seen on the network. The motivation behind this work came from the discovery of few systems contacting hosts in foreign countries for package updates, and DNS services found during routine log analysis. 

Zeek on the ESnet WAN:

To augment our LAN efforts on a wider scale, we have been experimenting with monitoring the network traffic on the WAN side of the network using Zeek in order to get more visibility and to provide improved security/network services. Most of this work is experimental: iterative design changes as we use what we learn from stage 1 to stage 3 and beyond.

  • Some notable differences and challenges from typical LAN network: 
    • Data Volume: There are a large number of WAN links that run at 1-400Gb/s
    • Data Encapsulation: Data with variable length headers is problematic, so we have been employing a load balancer to address this problem. 
    • Asymmetric Data Flows: This is a hard problem to solve, especially when the network is distributed across the country. When the packets corresponding inbound and outbound flows between two network nodes follow different paths, it can be challenging to reconcile conversation activities as part of network monitoring.
    • Technical Integration: Coordinating activities between teams distributed geographically  introduces challenges, which we are developing ways to overcome.

At ESnet we thrive to push the boundaries and try innovative ways to address challenges, Zeek on the WAN is an example of that and in my next article I will discuss some ways we have been experimenting with to address above noted complex problems and specifically going into details of the research been done in addressing Asymmetric Data Flows on WAN.

ESnet cybersecurity staff awarded the first-ever “ESNET”

Three members of the Energy Sciences Network’s Cybersecurity team – Michael Dopheide, Vlad Grigorescu, and Sam Oehlert – were recently honored with an Extra Special Noteworthy Exemplary Trophy Award for their SANS 2019 Holiday Hack Challenge entry.

“SANS is the premier security training organization for our profession. Their annual contest usually has 15,000-20,000 entries. For the past couple of years, our solutions have been used as the official answers and documentation, helping students and professionals around the world to hone their security skills. There is a real impact, and I am very proud of the team,” said Adam Slagell, Energy Sciences Network Chief Security Officer.

Dop Vlad Sam
From left to right: Michael Dopheide, Sam Oehlert, and Vlad Grigorescu

This eponymous award category called the “ESNET,” was created in reverence to the fact that all three team members asked to forgo any competition prizes. Instead, they asked that their prize be awarded to another winner.

“We felt that forgoing the prize was our small way of giving back to the community and rewarding one of the other participants, whose work should not be neglected. Honestly, this is something we look forward to every year. All three of us relish the puzzle-solving element, and our participation is really a labor of love,” said Grigorescu.  “We still felt it was important to share our report and the new techniques and tools for both offensive and defensive security we developed.”

The team notes that the SANS Holiday Hack competition is unique because the goal is to gamify cybersecurity challenges, all of which must be based on current threats. The end result is one of the best, most realistic security training, and it’s available for free, in perpetuity.

“By really pushing the envelope, we felt like we walked away from this year’s competition having significantly advanced our skills and being better able to safeguard the Energy Sciences Network. We’re incredibly honored to have an eponymous award, and to win the inaugural ‘ESNET’ award,” said Grigorescu.

This was Grigorescu’s fifth SANS Holiday Hack Competition and the fourth for Dopheide and Oehlert. All three were part of the Energy Sciences Network team that won the 2018 Grand Prize. Grigorescu’s teams also won an Honorable Mention in 2015, Best Technical Answer in 2016, and a Grand Prize in 2017.

Written by Linda Vu, Berkeley Lab Computing Sciences


Working Group on Open Science Cybersecurity Risks Releases First Document Draft for Public Comment

Michael Dopheide (left) and Sean Peisert (right).

Over the past several months, ESnet and the NSF Cybersecurity Center of Excellence collaborated with research and education community leaders to develop a risk profile for open science to formally capture and benchmark this expertise, allowing other organizations to apply these best practices more broadly.

Today, the group is releasing its draft Open Science Cyber Risk Profile (OSCRP) and inviting comment from the research community. The OSCRP is designed to help principal investigators and their supporting information technology professionals assess cybersecurity risks related to open science projects. The draft document, along with information on how to comment, can be found at

Managing the security risks to scientific instruments, data and cyberinfrastructure is a priority  for creating a trustworthy environment for science. Assessing, understanding and managing concerns of open science to explicitly capture risks to its integrity and availability, and sometimes also privacy issues, involves making judgments on the likelihood and consequences of risks. Deep experience in understanding cybersecurity and the science being supported is needed to achieve these goals.

The group invites comments on the document prior to final publication in early 2017.  Longer-term, the document is intended to be a living, community document, being updated as open science computing evolves, and also as new approaches to security arise.

About the OSCRP Working Group

Organized by Sean Peisert and Michael Dopheide from ESnet, and Von Welch and Andrew Adams from the NSF Cybersecurity Center of Excellence, the working group consists of: RuthAnne Bevier (Caltech), Rich LeDuc (Northwestern), Pascal Meunier (HUBzero), Stephen Schwab (USC Information Sciences Institute) and Karen Stocks (Scripps Institution of Oceanography), Ilkay Altintas (San Diego Supercomputer Center), James Cuff (Harvard), Reagan Moore (iRods), and Warren Raquel (NCSA/UIUC). To follow the activities of the working group, please follow

About the NSF Cybersecurity Center of Excellence •  

The Center for Trustworthy Scientific Cyberinfrastructure (CTSC) is funded as the National Science Foundation’s Cybersecurity Center of Excellence. The mission of CTSC is to improve the cybersecurity of NSF science and engineering projects, allowing those projects to focus on their science endeavors. This mission is accomplished through one-on-one engagements with projects to address their specific challenges; education, outreach, and training to raise the state of security practice across the scientific enterprise; and leadership on bringing the best and most relevant cybersecurity research to bear on the NSF cyberinfrastructure research community.

About ESnet •

The Energy Sciences Network (ESnet) is an international, high-performance, unclassified network built to support scientific research. Funded by the U.S. Department of Energy’s Office of Science (SC) and managed by Lawrence Berkeley National Laboratory, ESnet provides services to more than 40 DOE research sites, including the entire National Laboratory system, its supercomputing facilities, and its major scientific instruments. ESnet also connects to over 140 research and commercial networks, permitting DOE-funded scientists to collaborate productively with partners around the world.