Fatema Bannat Wala with our Cyber Security team was recognized with the 2021 Zeek Community Champion award by Corelight! More information on the award and her work with Zeek can be found here.
Zeek is an open source network security monitoring software extensively used by ESnet. Zeek (formally called Bro) was initially developed by researchers at Berkeley Lab, and more information on ESnet’s use of Zeek can be found in Fatema’s October Light Bytes post.
In the previous post we discussed deploying ZoMbis (Zeek on Management based information system) for ESnet6’s management network to monitor the traffic traversing the network and to provide visibility into what’s happening on our management network. This blog post will discuss how we use traffic sinkholes, which are a way of redirecting traffic so that it can be captured and analyzed. As opposed to our usual passive data collection system (e.g., tapping or port mirroring), traffic is being actively redirected to network monitoring systems such as Zeek. Network sensors can then perform various levels of in-depth analysis on the traffic, which can help detect misconfigurations, identify hostile traffic, or even perform automated mitigations for certain attacks.
Sinkholes are an important tool in the arsenal of network operators—they support network cyber defense by providing a way to redirect packets sent to or from unallocated (so-called “bogon” addresses) or other unexpected IP addresses. Additionally, they can help protect against reconnaissance or vulnerability scanning. If an attack does slip through these defenses, the damage could be limited, or the malicious traffic could be analyzed by network defenders to determine the source and methods being used.
As part of the ESnet6 security architecture, a sinkhole service will be deployed on the production management network, to redirect internal management traffic as well as externally sourced internet traffic destined to the management network. Using the Border Gateway Protocol (BGP), the sinkholes will advertise routes to the destination gateway for IP ranges of the management network to redirect the traffic to the target sinkhole. In our network, the management plane address set fits within a “supernet” (a collection of subnets) which can then advertise the sinkhole address as a destination. We will use this advertised supernet to redirect all traffic from external sources on the Internet away from the management network and to the external sinkhole.
An internal sinkhole will also advertise this management supernet for “inside” resources, but in this case, legitimate traffic will have a more specific route for the destination and not go to the sinkhole. This way, only traffic destined to an invalid subnet will be redirected to the internal sinkhole. This design should be extremely useful in identifying possible misconfigurations or other unexpected behaviors in the ESnet6 management network. if everything is behaving as expected, we should never see any traffic to the catch-all destination of the sinkhole.
The following diagram, taken from a ZeekWeek 2020 presentation by ESnet security engineer Scott Campbell, shows the basic design of the two kinds of sinkholes:
In the external sinkhole conceptual diagram above, routers R1 and R2 will be advertising the management address ranges to external sources. If any traffic destined to the management network is received from the Internet, it will instead be redirected to the sinkhole.
The external use case is a bit simpler than the internal sinkhole, which is diagrammed below. In the latter case, there will be some legitimate connections, such as between two ESnet points of presence (POPs), or between a POP and our data center. Any unwanted, misconfigured, or hostile scanning traffic will end up in the internal sinkhole. Hence internal sinkholes can be thought of both as network “garbage cans” and intrusion sensors helping to detect changes in normal management traffic patterns.
The ESnet Security Team will use Zeek, to analyze traffic at the application level, for both types of sinkholes. The logs generated by Zeek will then be collected centrally and will provide useful insights into what kind of unwanted traffic is being directed at our management plane, both from internal or external sources, and help better protect ESnet6 from attackers.
In my previous post, we discussed use of the open-source Zeek software to support network security monitoring at ESnet. In this post, I’ll talk a little about work underway to improve Zeek’s ability to support network traffic monitoring when faced with stream asymmetry.
This comes from recent work by two of my colleagues on the ESnet Security team.
Some of the significant findings and results from this presentation are highlighted below:
Phase I: Initial Zeek Node Design Considerations
Select locations that provide an interesting network vantage point – in the case of our ESnet network, we deployed Zeek nodes on our commodity internet peerings (eqx-sj, eqx-chi, eqx-ash) since they represent the interface to the vast majority of hostile traffic.
Identifying easy traffic to test with and using spanning ports to forward traffic destined to the stub network on each of the routers used for collection.
Phase I: Initial Lessons learned from testing and results
Some misconfigurations were found in the ACL prefix lists.
We increased visibility into our WAN side traffic through implementation of new background methods.
Establishing a new process for end-to-end testing, installing and verifying Zeek system reporting.
Phase II: Prove there is more useful data to be seen
For phase II we moved towards collection of full peer connection records, from statistical sampling based techniques. Started running Zeek on traffic crossing the interfaces which connect ESnet network peers to the internet from the AS (Autonomous system) responsible for most notices. .
To get high fidelity connection information without being crushed by data volume, define a subset of packets that are interesting – zero length control packets (Syn/Syn-Ack/Fin/Rst) from peerings.
Phase II: Results
A lot of interesting activity got discovered like information leakage in syslogs, logins (and attempted logins) using poorly secure authentication protocols, and analysis of the amount of asymmetric traffic patterns gave valuable insights to understand better the asymmetric traffic problems.
Ongoing Phase III: Expanding the reach of traffic collection on WAN
We are currently in the process of deploying Zeek nodes at another three WAN locations for monitoring commodity internet peering – PNWG (peering at Seattle WA), AM-SIX (peering at Amsterdam) and LOND (peering at London)
As our use of Zeek on the WAN side of ESnet continues to grow, the next phase to the ZoW pilot is currently being defined. We’re working to incorporate these lessons learned on how to handle traffic asymmetry into these next phases of effort.
Some (not all) solutions being taken into consideration include:
Aggregating traffic streams at a central location to make sense out of the asymmetric packet streams and then run Zeek on the aggregated traffic, or
Running Zeek on the individual asymmetric streams and then aggregating these Zeek streams @ 5-tuple which will be aggregation of connection metadata rather than the connection stream itself.
We are currently exploring these WAN solutions as part of providing better solutions to both ESnet, and connected sites.