Cybersecurity

3Q with Alex Withers, ESnet’s New Deputy Chief Information Security Officer

~ ESnetComms
Alex Withers in a blue t-shirt in front of green bushes
Alex Withers at home in Urbana, Illinois.

Network cybersecurity must strike a delicate balance between openness and safety. ESnet has long focused more on the first, in order to facilitate scientific communication, data sharing, and collaboration. But in today’s Wild West of ransomware, phishing, and other threats, safeguarding this vital network is equally critical. 

Deputy Chief Information Security Officer (CISO) Alex Withers oversees a reorganized, two-part structure for security at ESnet: the Security Engineering group, which he heads, and a new Threat & Vulnerability Management group, which Chief Security Officer Adam Slagell is leading during the search for a senior threat hunter/DOE community coordinator. For Security Engineering, among Alex’s responsibilities are overseeing ESnet’s effort to comply with the federal Zero Trust requirements — an approach to cybersecurity that goes beyond “trust but verify” and treats all networks and traffic as potential threats. Alex will be making sure that any new security policies, procedures, and architecture do not impede ESnet’s vision of enabling scientific progress that is completely unconstrained by the physical location of instruments, people, computational resources, or data.

Alex has deep experience in threat intelligence sharing, policy and compliance, and security architecture. Most recently he was the CISO and cybersecurity division manager at the National Center for Supercomputing Applications at the University of Illinois. While at NCSA, Alex oversaw groups responsible for security operations, applied cybersecurity research, cybersecurity engagement, and scientific computing in the HIPAA and Controlled Unclassified Information [CUI] space. He was a PI or co-PI on several National Science Foundation awards for projects focused on intrusion detection, threat intelligence dissemination, and capabilities-based authorization for access to scientific computing resources. Before NCSA, Alex worked for Brookhaven National Laboratory as a security and systems engineer for over 10 years. 

Alex grew up in Alaska and now lives in Urbana, Illinois, where he works out of ESnet’s Champaign office with Adam and Security Engineering team members Kapil Agrawal, Michael Dophelde, and Sam Oehlert, as well as about a dozen other ESnetters. He’s an avid long-distance runner – as are his wife and two of his four children. In the last few years, Alex has completed around a dozen ultra-marathons, or “ultras,” ranging from 50 km to 100 miles – something he’s “always reluctant to tell people about because it sounds crazy.”   

What brought you to ESnet?

Really it was the opportunity for growth — both to tackle new challenges in cybersecurity architecture and for me professionally, to try something new. ESnet is responsible for connecting a massive portion of the scientific computing infrastructure that supports not just this country’s scientific investments but also international collaborations. It’s growing extremely rapidly, and it’s a giant target for all sorts of reasons, whether from state-sponsored attacks or cyber criminals or anything in between. And so it looks like an immense challenge, and that’s very attractive to me.

What is the most exciting thing going on in your field right now?

There’s been a shift in how people view cybersecurity that’s making it easier for us to collaborate and innovate with users. 

Traditionally, cybersecurity has had a bad reputation as being the people who say “No, you can’t do this; no, you can’t do that.” And in the research and education sector, the culture tends to be much more open, much more about getting done what has to be done, whether that’s moving data around or access to computing for scientists and their students. That culture has often bumped up against cybersecurity, which tends to want to wall things off. But cybersecurity is now much more about enabling science. As I tell people, “Listen, the funding agencies, the government, have invested billions of dollars in science and in this infrastructure that you rely on for your particle accelerator, electron microscope, whatever. And we want to protect that investment, because at the end of the day, things like cybersecurity incidents, they can disrupt your work. They can stop it dead in its tracks, and that’s money that’s lost.” 

So today’s cybersecurity is about understanding how researchers use these systems and devices, how they access them through the network – and working together to make sure that we enable their use and make it available while at the same time very secure. At ESnet, we want to ensure the integrity of the data so that researchers can be productive on their computational systems and networks. That frame of approach is easier than traditional cybersecurity, which is more focused on things like confidentiality and privacy.

What book, movie, or podcast would you recommend?

A podcast I’ve been really enjoying has been “Some Work, All Play” by David Roche and Megan Roche. It’s an excellent inclusive running podcast for all runners, especially trail running, which is a hobby I enjoy.

Running ultras sounds like a little more than just a “hobby.” Tell us more about why you do it?

Well, a lot of people think of it as a very physical sort of endeavor. And I mean…that’s true, and I don’t want to downplay that, but you’d be surprised to find out that it’s not as difficult as you think it is. The real challenge is the mental challenge. It is extremely difficult mentally to go out on a trail and run for hours and hours and hours and hours. You’re really fighting against the urge to drop out and call it a day. And sometimes you’re not successful. 

What’s great about it is pushing yourself up against the limits of what you can do. There are people who seriously race these things, and they win. I’m not in any danger of doing that, I assure you. For me, it’s racing against various aspects of yourself. Racing against yourself mentally. Racing against your past self. Maybe you’re going to do better on a race you’ve done before. Sometimes you’re racing against your own stomach, because you have to eat during these things—but it’s not a pleasant task to eat while you’re doing all this running! 

It’s very challenging, but it’s also a lot of fun. It’s very rewarding when it works out. 

ESnet Highlights from ZeekWeek’21

Fatema Bannat Wala presenting at ZeekWeek21
~ ESnetComms

Slides and videos from ZeekWeek have just been made available — here are links to ESnet highlights.

ZeekWeek, an annual Fall conference organized by the Zeek Project, took place online from October 13-15 this year. The conference had over 2000 registered participants from the open source user community this year, who got together to share the latest and greatest about this cyber-security and network monitoring software tool.

Berkeley Lab staff member Vern Paxson developed the precursor to the Zeek intrusion detection software, then called Bro, in 1994. As an early adopter, ESnet’s cybersecurity team has strong relationships with the Zeek community, and this ZeekWeek was an opportunity to showcase advances and uses made by the software by ESnet and the entire Research and Educational Networking Community.

The talk “DNS and Spoofed traffic investigation with Zeek,” presented by Fatema Bannat Wala, discussed how Zeek is being used to do network traffic analysis/investigations at ESnet by triaging abnormal activities when these occur on our network.

The talks “A Better Way to Capture Packets with DPDK” and “Details for DPDK plugin development and performance measurement” presented by Vlad Grigorescu and Scott Campbell, detailed the development process of the plugin and the performance enhancements it brings to the network packet capture technology.

Fatema Bannat Wala also did a training session on “Introduction to Zeek,” which provided hands-on experience with Zeek tools and information about how to get involved with the collaboration.

ESnet’s cybersecurity team looks forward to continued collaboration with the Zeek community, attending next year’s ZeekWeek, and to contributing future code enhancements to this great software ecosystem.

3 Questions with Michael Haberman

~ ESnetComms

Michael comes to ESnet’s Cybersecurity group after working as a software engineer at the National Center for Supercomputing Applications (NCSA), and in the Automated Learning Group at the University of Illinois, Champaign/Urbana (UIUC). Recently, he has also been an instructor for a data science and machine learning course within the School of Informatics (iSchool).

Michael Haberman
Michael Haberman

What brought you to ESnet?
The classes I taught at UIUC were designed around mastery-based learning and evidence-based teaching. I built a framework that instrumented the assignments (similar to observability) so that I could get a good pulse on where students were struggling and where they weren’t. Creating the end-to-end workflows for the students made me realize how much I missed architecting (and building) software. I knew several great ESnet people and it was just perfect timing that the security group had an opening where they were receptive to bringing on someone with a software design background and also enthusiastic about letting me continue climbing the data analytics and machine learning mountain (I’m at the base). I also love that ESnet’s mission enables science.

What’s the most exciting thing happening in your field?
There’s a lot going on and staying current is a challenge. If I had to pick a topic that is ripe for potential (or hype) it’s using blockchain “decentralized ledger” technology (now being used for databases, voting, and electronic currencies), to create applications in digital identity, and remove unnecessary intermediaries from transactions. It seems like there are new application ideas for blockchain every day.

Although I do not know much about cryptocurrency (or its future), the idea of using their decentralized ‘bookkeeping’ architecture for secure transactions with provenance seems intriguing.

What book would you recommend?
I remember reading The Cuckoo’s Egg in high school and it’s one of the books that got me interested in both computer science and security. When I saw this question I remembered that the main character is from LBL! Perhaps the security group will want me to look into an accounting discrepancy?

ESnet Highlights from the National Science Foundation’s Cybersecurity Summit ’21

~ Fatema Bannat Wala

The National Science Foundation (NSF) Cybersecurity Center of Excellence, Trusted CI Project hosts a yearly cybersecurity summit, inviting people from various NSF-funded research organizations to share innovations and ideas. Here are some videos of ESnet presentations.

Scott Campbell presented “ESnet Security Group Impact on Network Architecture” where he discussed some of the social, technical, and architectural outcomes of the ESnet6 network upgrade that were beneficial to the organization. By being involved early, security design elements were incorporated into workflows at early stages and were both tightly integrated and vetted during the core design process. This early involvement also heightened the security group’s visibility, which led to a better understanding of how the various groups interact and their different methods of problem-solving and time management.

Eli Dart and Fatema Bannat Wala presented “Best practices for securing Science DMZ,” focusing on disentangling security policies and enforcement for science flows from traditional security approaches for business systems, and use of the Science DMZ model to protect high-performance science flows. They discussed thinking of the Science DMZ as a security architecture that provides useful and implementable security controls without impacting performance. 

Making the Research and Educational Community SAFER: Adam Slagell on the creation of a new global collaboration to combat cyberthreats.

~ ESnetComms

Adam Slagell is ESnet’s Chief Security Officer and a founding member of the newly formed Security Assistance For Education & Research (SAFER) trust group.

SAFER is an operational security entity focused on fighting computer misuse and defending the academic, research, and education (R&E) mission globally.  SAFER brings together expertise and resources from organizations across the Research and Educational cybersecurity community, including CERN, DFN-CERT, ESET, ESnet, LBNL, STFC, and WLCG.

More information can be found here https://www.safer-trust.org/.

What motivates the creation of SAFER and what do you think success will look like for the community?

There are many cybersecurity trust groups out there, some even dedicated to R&E like REN-ISAC or XSEDE’s trust group consisting of current and former Teragrid and XSEDE site  members. However, there really isn’t anything like this—both permanent and truly international— even though attacks are almost always transnational. So each time there is a new, major campaign, an international group connecting all these regional responders must be created again. What we are trying to do is create that permanent backbone with a core set of highly connected individuals who are a part of these regional and project-specific trust groups around the world.

If we are successful, we will see several things. First, I believe we will see more international cooperation and information sharing, leading to an earlier notice of new attack campaigns. Second, we will be able to activate a response more quickly, pulling in the expertise needed from a broad pool of SAFER members and their trusted colleagues. Finally, it is our hope that we can provide surge capabilities when a member is under attack. Many R&E organizations have limited resources and small teams. It is a tremendous asset if they can get help from their peers, maybe with unique expertise as they are facing a disruptive attack.

What kind of security resources will SAFER provide?

I alluded to some of the services when discussing what success will look like. But ultimately, our security resources will be determined by community needs. The founding members will serve as the steering committee for the first year until we elect the next steering committee. 

One of our  first-steps is  setting up a Malware Information Sharing Platform (MISP) instance to share Indicators of Compromise, e.g., IP addresses, file hashes, domain names, etc. Usually, there is no requirement for members to share such data as the rules and regulations differ so much across organizations. But even on day one, we will have enough organizations that can contribute to making this service useful.

There is also a secure messaging and chat service using decentralized cryptography that all of our members can participate in. These ad hoc conversations about what people are seeing on their networks will hopefully help detect trends early.

Finally, many of the founding members have more resources from these large institutions, and I believe we can quickly help those projects and institutions that might struggle with an attack by providing our expertise while helping to train the next generation of security professionals.

What excites you most about this effort and what is the opportunity to do the most good?

I love the community-building aspect. In a past life, I created the Bro (now Zeek) Leadership Team and really worked hard to build a vibrant community around that software. I think this expertise is where I can be most helpful as I am less technical in my roles today.

I will also say, I am excited about getting young people involved, too. Organizations who contribute time from their teams will really benefit. There is no training for an incident response like jumping in, and I expect the variety of issues we will see will prove very useful just from a training and development perspective.

LBL has a long history supporting cybersecurity research, from the early days of Clifford Stoll and The Cuckoo’s Egg to the creation of Bro.  What does the future of cybersecurity look like, and how will that shape the REN community?

Indeed, LBL’s security team is also a SAFER founding member. One of the things I love about working here and at ESnet is that our mission is outward-focused and when we help the community we raise all boats so to speak.

Fortune telling however is a dangerous game. We have anticipated some things, like cryptocurrency mining coming to HPCs. However, the threat landscape and tools available keep changing. That is part of what makes this job interesting. The important thing that I hope we keep in mind is that security is not done for its own sake, but to enable our mission of scientific research. To me, this means that we must always work to make risk-based security decisions, even when that might challenge pushes for compliance and simple one-size-fits-all solutions. 

Fatema Bannat Wala named Zeek Community Champion!

~ Andrew Wiedlea
Fatema Bannat Wala

Fatema Bannat Wala with our Cyber Security team was recognized with the 2021 Zeek Community Champion award by Corelight! More information on the award and her work with Zeek can be found here.

Zeek is an open source network security monitoring software extensively used by ESnet. Zeek (formally called Bro) was initially developed by researchers at Berkeley Lab, and more information on ESnet’s use of Zeek can be found in Fatema’s October Light Bytes post.

CONGRATULATIONS, Fatema!

Deploying ZoMbis at ESnet – Part II

~ Fatema Bannat Wala

In the previous post we discussed deploying ZoMbis (Zeek on Management based information system) for ESnet6’s management network to monitor the traffic traversing the network and to provide visibility into what’s happening on our management network. This blog post will discuss how we use traffic sinkholes, which are a way of redirecting traffic so that it can be captured and analyzed. As opposed to our usual passive data collection system (e.g., tapping or port mirroring), traffic is being actively redirected to network monitoring systems such as Zeek. Network sensors can then perform various levels of in-depth analysis on the traffic, which can help detect misconfigurations, identify hostile traffic, or even perform automated mitigations for certain attacks.

Sinkholes are an important tool in the arsenal of network operators—they support network cyber defense by providing a way to redirect packets sent to or from unallocated (so-called “bogon” addresses) or other unexpected IP addresses. Additionally, they can help protect against reconnaissance or vulnerability scanning. If an attack does slip through these defenses, the damage could be limited, or the malicious traffic could be analyzed by network defenders to determine the source and methods being used.

As part of the ESnet6 security architecture, a sinkhole service will be deployed on the production management network, to redirect internal management traffic as well as externally sourced internet traffic destined to the management network. Using the Border Gateway Protocol (BGP), the sinkholes will advertise routes to the destination gateway for IP ranges of the management network to redirect the traffic to the target sinkhole. In our network, the management plane address set fits within a “supernet” (a collection of subnets) which can then advertise the sinkhole address as a destination. We will use this advertised supernet to redirect all traffic from external sources on the Internet away from the management network and to the external sinkhole.

An internal sinkhole will also advertise this management supernet for “inside” resources, but in this case, legitimate traffic will have a more specific route for the destination and not go to the sinkhole. This way, only traffic destined to an invalid subnet will be redirected to the internal sinkhole. This design should be extremely useful in identifying possible misconfigurations or other unexpected behaviors in the ESnet6 management network. if everything is behaving as expected, we should never see any traffic to the catch-all destination of the sinkhole.

The following diagram, taken from a ZeekWeek 2020 presentation by ESnet security engineer Scott Campbell, shows the basic design of the two kinds of sinkholes:

Example External Sinkhole

In the external sinkhole conceptual diagram above, routers R1 and R2 will be advertising the management address ranges to external sources. If any traffic destined to the management network is received from the Internet, it will instead be redirected to the sinkhole. 

The external use case is a bit simpler than the internal sinkhole, which is diagrammed below. In the latter case, there will be some legitimate connections, such as between two ESnet points of presence (POPs), or between a POP and our data center. Any unwanted, misconfigured, or hostile scanning traffic will end up in the internal sinkhole. Hence internal sinkholes can be thought of both as network “garbage cans” and intrusion sensors helping to detect changes in normal management traffic patterns. 

Example Internal Sinkhole

The ESnet Security Team will use Zeek, to analyze traffic at the application level, for both types of sinkholes. The logs generated by Zeek will then be collected centrally and will provide useful insights into what kind of unwanted traffic is being directed at our management plane, both from internal or external sources, and help better protect ESnet6 from attackers. 

Defending ESnet with ZoMbis!

~ Fatema Bannat Wala

Zeek is a powerful open source network security monitoring software extensively used by ESnet. Zeek (formally called Bro) was initially developed by researchers at Berkeley Lab; it allows users to identify & manage cyber threats by tracking and logging network traffic activity. Zeek operates as a passive monitor, providing a holistic view of what is transpiring in the network and on all network traffic. 

In a previous post, I presented some of our efforts in approaching the WAN security using Zeek for general network monitoring, with successes and challenges found during the process. In this blog post I’ll focus on our efforts in using Zeek as  part of security monitoring for the ESnet6 management network – ZoMbis (Zeek on Management based information system).

ZoMbis on the ESnet6 management network:

Most research and educational networks employ a dedicated management network as a best practice. The management network provides a configuration command and control layer, as well as conduits for all of the inter-routing communications between the devices used to move our critical customer data. Because of the sensitive nature of these communications, the management network needs to be protected from external and general user network traffic (websites, file transfers, etc.), and our staff needs to have detailed visibility on management network activity.

At ESnet, we typically use real IP addresses for all internal network resources, and our management network is allocated a fairly large address space block advertised in our global routing table, to help protect against opportunistic hijacking attacks. By isolating our management network from user data streams, the amount of routine background noise is vastly reduced making the use of Zeek, or any network monitoring security capability, much more effective. 

The above diagram shows an overview of the deployment strategy of Zeek on the ESnet6 management network. The blue dots in the diagram show the locations that will have equipment running Zeek instances for monitoring the network traffic on the management network. The traffic from the routers on those locations is mirrored to the Zeek instances using a spanning port, and the Zeek logs generated are then aggregated in our central security information and event logging and management system (SIEM).

Scott Campbell presented ‘Using Zeek in ESnet6 Management Network Security Monitoring’ during virtual Zeek Week held last year that explained the overall strategy for deployment of Zeek on the management network in greater detail. Some ZoMbis deployment highlights are:

ESnet 6’s new management network will use only IPv6. From a monitoring perspective this change from the traditional IPv4 poses a number of interesting challenges; In particular, IPv6 traffic employs more multicast and link-local traffic for local subnet communications. Accordingly, we are in the process of adjusting and adding to Zeek’s policy based detection scripts to support these changes in network patterns. These new enhancements and custom scripts being written by our cybersecurity team to support IPv6 will be of interest to other Zeek users and we will release them to the entire Zeek community soon. 

The set of Zeek policy created for this project can be broken out into two general groups. The first of these is protocol mechanics – particularly looking closer between layer 2 and 3 where there are a number of interesting security behaviors with IPv6.  A subset of notices that these protocol mechanic policies will provide are:

  • ICMP6_RtrSol_NotMulticat – Router solicitation not multicast
  • ICMP6_RtrAnn_NotMulticat – Router announcement should be a multicast request
  • ICMP6_RtrAnn_NewMac – Router announcement from an unknown MAC
  • ICMP6_MacIPChange – If the MAC <-> IP mapping changes
  • ICMP6_NbrAdv_NotRouter – Advertisement comes from non-router
  • ICMP6_NbrAdv_UnSolicit – Advertisement is not solicited
  • ICMP6_NbrAdv_OverRide – Advertisement without override
  • ICMP6_NbrAdv_NoRequest – Advertisement without known request

The second set of Zeek policies that have been developed in support for ZoMbis involves taking advantage of predictable management network behavioral patterns – we build policy to model anticipated behaviors and let us know if something is amiss. For example looking at DNS and NTP behavior we can identify unexpected hosts and data volumes, since we know which systems are supposed to be communicating with one another, and what patterns traffic between these components should follow.

Stay tuned for the part II of this blogpost, where I will discuss ways of using Sinkholing, together with ZoMbis, to provide better understanding and visibility of unwanted traffic upon the management network.

Zeek and stream asymmetry research at ESnet

~ Fatema Bannat Wala

In my previous post, we discussed use of the open-source Zeek software to support network security monitoring at ESnet.  In this post, I’ll talk a little about work underway to improve Zeek’s ability to support network traffic monitoring when faced with stream asymmetry.

This comes from recent work by two of my colleagues on the ESnet Security team.

Scott Campbell and Sam Oehlert presented ‘Running Zeek on the WAN: Experiences and solutions for large scale flow asymmetry’ during a workshop held last year at CERN Geneva that explained the phases and deployment of the Zeek-on-the-WAN (ZoW) pilot in detail.

Scott Campbell at CERN presenting ‘Running Zeek on the WAN’
The asymmetry problem on a WAN (example)

Some of the significant findings and results from this presentation are highlighted below:

  • Phase I: Initial Zeek Node Design Considerations 
    • Select locations that provide an interesting network vantage point – in the case of our ESnet network, we deployed Zeek nodes on our commodity internet peerings (eqx-sj, eqx-chi, eqx-ash) since they represent the interface to the vast majority of hostile traffic.
    • Identifying easy traffic to test with and using spanning ports to forward traffic destined to the stub network on each of the routers used for collection.
  • Phase I: Initial Lessons learned from testing and results
    • Some misconfigurations were found in the ACL prefix lists. 
    • We increased visibility into our WAN side traffic through implementation of new background methods.
    • Establishing a new process for end-to-end testing, installing and verifying Zeek system reporting. 
  • Phase II:  Prove there is more useful data to be seen
    • For phase II we moved towards collection of full peer connection records, from statistical sampling based techniques. Started running Zeek on traffic crossing the interfaces which connect ESnet network peers to the internet from the AS (Autonomous system) responsible for most notices. .
    • To get high fidelity connection information without being crushed by data volume, define a subset of packets that are interesting – zero length control packets (Syn/Syn-Ack/Fin/Rst) from peerings.
  • Phase II: Results
    • A lot of interesting activity got discovered like information leakage in syslogs, logins (and attempted logins) using poorly secure authentication protocols, and analysis of the amount of asymmetric traffic patterns gave valuable insights to understand better the asymmetric traffic problems.
  • Ongoing Phase III: Expanding the reach of traffic collection on WAN
    • We are currently in the process of deploying Zeek nodes at another three WAN locations for monitoring commodity internet peering – PNWG (peering at Seattle WA), AM-SIX (peering at Amsterdam) and LOND (peering at London)
Locations for the ZoW systems, the pink shows ongoing Phase III deployment

As our use of Zeek on the WAN side of ESnet continues to grow, the next phase to the ZoW pilot is currently being defined.  We’re working to incorporate these lessons learned on how to handle traffic asymmetry into these next phases of effort. 

Some (not all) solutions being taken into consideration include: 

  • Aggregating traffic streams at a central location to make sense out of the asymmetric packet streams and then run Zeek on the aggregated traffic, or
  • Running Zeek on the individual asymmetric streams and then aggregating these Zeek streams @ 5-tuple which will be aggregation of connection metadata rather than the connection stream itself. 

We are currently exploring these WAN solutions as part of providing better solutions to both ESnet, and connected sites.

Zeekurity at ESnet

~ Fatema Bannat Wala

Zeek is an open source network security monitoring software extensively used by ESnet.  Zeek (formally called Bro) was initially developed by researchers at Berkeley Lab. Zeek allows users to identify & manage cyber threats by monitoring network traffic. It acts as a passive monitoring software (NSM – Network Security Monitor), that gives a holistic view of what is transpiring in the network and gives visibility into the network traffic. 

In order to better understand network behavior and provide flexible security services, we use Zeek as an important part of our data center security architecture and are experimenting with placing Zeek clusters on various WAN high value points. This is providing technical insights as well as significant challenges. 

In this post we would present some of our efforts in approaching the WAN security using Zeek for network monitoring, with successes and challenges hit during the process and interesting things learned.

Zeek on the ESnet LAN:

Monitoring local area and data center networks is a familiar and less complex network traffic monitoring design, and ESnet is no different. The traffic flowing through the LAN networks is currently monitored using two Zeek clusters, one at Brookhaven National Lab and another for the west coast at Berkeley Lab. We have implemented BHR (black hole routing) functionality on our data center routers to block external actors which violate our established policies based on Zeek detections on both IPv4 and IPv6 protocol stacks. 

Apart from network security monitoring using “standard” Zeek detection scripts, many enhancements and custom scripts written by the ESnet Security team members serve a vital role in detecting various kinds of suspicious activity. Recently, a Zeek package – Zeek-Known-outbound contributed by Michael “Dop” Dopheide won the first prize in the Zeek Package Contest-2 held in May 2020. The package provides the ability to track and alert on outbound service usage to a list of ‘watched’ countries, and also adds the country codes for the origin and recipient hosts in one of the log files that Zeek generates called conn.log, to log all the connection attempts seen on the network. The motivation behind this work came from the discovery of few systems contacting hosts in foreign countries for package updates, and DNS services found during routine log analysis. 

Zeek on the ESnet WAN:

To augment our LAN efforts on a wider scale, we have been experimenting with monitoring the network traffic on the WAN side of the network using Zeek in order to get more visibility and to provide improved security/network services. Most of this work is experimental: iterative design changes as we use what we learn from stage 1 to stage 3 and beyond.

  • Some notable differences and challenges from typical LAN network: 
    • Data Volume: There are a large number of WAN links that run at 1-400Gb/s
    • Data Encapsulation: Data with variable length headers is problematic, so we have been employing a load balancer to address this problem. 
    • Asymmetric Data Flows: This is a hard problem to solve, especially when the network is distributed across the country. When the packets corresponding inbound and outbound flows between two network nodes follow different paths, it can be challenging to reconcile conversation activities as part of network monitoring.
    • Technical Integration: Coordinating activities between teams distributed geographically  introduces challenges, which we are developing ways to overcome.

At ESnet we thrive to push the boundaries and try innovative ways to address challenges, Zeek on the WAN is an example of that and in my next article I will discuss some ways we have been experimenting with to address above noted complex problems and specifically going into details of the research been done in addressing Asymmetric Data Flows on WAN.