perfSONAR is a set of tools that help engineers actively monitor & troubleshoot multi-domain network issues. It provides the ability to pinpoint and solve service problems immediately to restore the highest levels of network performance
ESnet is proud to welcome Rede Nacional de Ensino e Pesquisa (RNP), the national research and education network of Brazil, as an official collaboration partner on the perfSONAR project. The official announcement on the perfSONAR website is here. RNP joins five other organizations (ESnet, GEANT, Indiana University, Internet2, and the University of Michigan) committed to providing dedicated resources that develop and maintain the perfSONAR software.
Even though RNP is now becoming an official project member, they have been part of the perfSONAR community for the past 15 years. RNP has used their own perfSONAR fork for eight years and recently moved to the baseline version of perfSONAR across RNP’s 27 points of presence. The use of the perfSONAR branch code will provide network engineers and customers with improved abilities to maintain their network and validate on-demand circuit use.
Going forward, RNP will be a key contributor in several areas. Iara Machado will be working in conjunction with perfSONAR’s existing steering committee members to provide executive level guidance for the project. Additionally, Marcos Schwarz will join the perfSONAR leadership team to not only spearhead the RNP development team, but also help manage the day-to-day development of the project with existing partner institutions. Initial interests include containerization, perfSONAR as microservices, standard display and analysis packages, and possibly making their circuit validation tool available to the broader community. RNP’s experience and insight will be invaluable to the perfSONAR project going forward.
ESnet and the entire perfSONAR collaboration are excited to officially have them on the team. Having partners like RNP strengthens the perfSONAR initiative and helps ensure a bright future in its continued role as a critical piece of infrastructure for high-performance scientific networks.
Scientific discovery increasingly relies on the ability to perform large data transfers across networks operated by many different providers (including ESnet) around the globe. But what happens when a researcher initiates one of these large data transfers and data movement is slow? What does “slow” even mean? These can be surprisingly complex questions and it is important to have the right tools to help answer them. perfSONAR is an open source software tool designed to measure network performance and pinpoint issues that occur as data travels across many different networks on the way to a destination.
perfSONAR has been around for more than 15 years and is primarily maintained today by a collaboration of ESnet, GEANT, Indiana University, Internet2, and the University of Michigan. perfSONAR has an active community that extends well beyond the five core organizations that maintain the software with more than 2000 public deployments that span six continents and hundreds of organizations. perfSONAR deployments are capable of scheduling and running tests that calculate metrics including (but not limited to) how fast a transfer can be performed (throughput), if a unit of information makes it to a desired destination (packet loss), if so how long did it take (latency) and what path did it take to get there (traceroute). What is novel about perfSONAR is not just these metrics, but the set of tools to feature these metrics in dashboards built by multiple collaborating organizations. These dashboards aim to clearly identify patterns that signify potential issues and provide the means to drill-down into graphs that give more information.
While perfSONAR has had great success in providing the current set of capabilities, there is more that can be done. For example, perfSONAR is very good at correlating metrics it collects with the other perfSONAR metrics with at least one similar endpoint. But what if we want to correlate the metrics by location, intermediate network or with non-perfSONAR collected statistics like flow statistics and interface counters? These are all key questions the perfSONAR project is looking to answer.
Building upon a strong foundation
PerfSONAR has the ability to add analytics from other software tools using a plug-in framework. Recently, we have begun to use Elastic Search via this framework, to ingest log data and enable improved search and analytics on perfSONAR data.
For example, traditionally perfSONAR has viewed an individual measurement as something between a pair of IP addresses. But what do these IP addresses represent and where are they located? Using off-the-shelf tools Elastic Search in combination with Logstash, perfSONAR is able to answer questions like “What geographic areas are showing the most packet loss?”.
Additionally, we can apply this same principle to traceroute (and similar tools) that yield a list of IP addresses giving an idea of the path a measurement takes between source and destination. Each IP address is a key to more information about the path including not only geographic information but also the organization at each point. This means you can ask questions such as “What is the throughput of all results that transit a given organization?”. Previously a user would not only have to know the exact address of the IPs, but it would have to be the first (source) or last (destination) address in the path.
Integration with non-perfSONAR data is another area the project is looking to expand. By putting perfSONAR data in a well established data store like Elasticsearch, the door is open to leverage other off-the-shelf open source tools like Grafana to display results. What’s interesting about this platform is not only its ability to build new visualizations, but also the diverse set of backends it is capable of querying. If data such as host metrics, network interface counters and flow statistics are kept in any of the supported data stores, then there is a means to present this information along perfSONAR data.
These efforts are very much still in their early stages of development, but initial indicators are promising. Leveraging the perfSONAR architecture in conjunction with the wealth of off-the-shelf open source tools available on the market today create opportunities to gain new insights from the network, like those described above, not previously possible with the traditional perfSONAR tools.
Getting involved and learning more
The perfSONAR project will continue to provide updates as this work progresses. You can also see the perfSONAR web site for updates and more information on keeping in touch through our mailing lists. The perfSONAR project looks forward to working with the community to provide exciting new network measurement capabilities.
ESnet, Network Startup Resource Center Combine Expertise to Spread the Word
For members of the established research and education (R&E) networking community, attending conferences or sitting in on workshop sessions is the normal way to learn about the latest equipment, architecture, tools and technologies.
But for network engineers striving to establish basic R&E infrastructure where bandwidth and other resources are scarce, the University of Oregon’s Network Startup Resource Center (NSRC) is often the primary information conduit. NSRC staff travel to emerging nations in Africa, Asia-Pacific, Middle East and South America where they hold intensive hands-on training courses combined with direct engineering assistance to bring institutions up to speed.
And for the second time in a year, ESnet and the NSRC have produced and released a library of short explanatory videos to help network engineers around the world gain basic knowledge, set up basic systems and drill down into areas of specific interest. In December, 15 videos detailing the Science DMZ network architecture were posted, covering the background and structure, specific designs, and techniques and technology.
“The goal is to make the information more accessible to networking staff, in the U.S. and particularly in emerging economic areas where institutions are trying to bootstrap a research network,” said ESnet Network Engineer Eli Dart, who developed the Science DMZ concept with Brent Draney of the National Energy Research Scientific Computing Center (NERSC). Both ESnet and NERSC are DOE Office of Science User Facilities managed by Lawrence Berkeley National Laboratory.
Created in 1986, the U.S. Department of Energy’s (DOE’s) Energy Sciences Network (ESnet) is a high-performance network built to support unclassified science research. ESnet connects more than 40 DOE research sites—including the entire National Laboratory system, supercomputing facilities and major scientific instruments—as well as hundreds of other science networks around the world and the Internet.
Step 9: Why just rent fiber? Pick up your own dark fiber network at a bargain price for future expansion. In the meantime, boost your bandwidth to 100G for everyone. (2012)
Step 10: Here’s a cool idea, come up with a new network design so that scientists moving REALLY BIG DATASETS can safely avoid institutional firewalls, call it the Science DMZ, and get research moving faster at universities around the country. (2012)
Step 12: 100G is fast, but it’s time to get ready for 400G. To pave the way, ESnet installs a production 400G network between facilities in Berkeley and Oakland, Calif., and even provides a 400G testbed so network engineers can get up to speed on the technology. (2015)
Step 13: Celebrate 30 years as a research and education network leader, but keep looking forward to the next level. (2016)
A joint effort between ESnet, Internet2, Indiana University, and GEANT, the pan-European research network, perfSONAR is a tool for end-to-end monitoring and troubleshooting of multi-domain network performance. In January 2014, perfSONAR reached a milestone with 1,000 instances of the diagnostic software installed on networking hosts around the U.S. and in 13 other countries. perfSONAR provides network engineers with the ability to test and measure network performance, as well as to archive data in order to pinpoint and solve service problems that may span multiple networks and international boundaries.
At the workshop, Tierney will give an introduction to perfSONAR and present a session on debugging using the software. Zurawski will talk about maintaining a perfSONAR node, describe some user case studies and success stories, discuss “Pulling it All Together – perfSONAR as a Regional Asset” and conclude with “perfSONAR at 10 Years: Cleaning Networks & Disrupting Operation.”
Since it was first released about five years ago, the perfSONAR network measurement toolkit has provided the research and education networking community with tools for end-to-end monitoring and troubleshooting of multi-domain network performance. And over the years, this ability to diagnose network problems has become increasingly important as research is increasingly collaborative and dependent on sharing large data sets.
The latest release of perfSONAR, version 3.4, gives network engineers access to more data about network performance as well as increased security protections. The new version will be discussed in a session called “perfSONAR 3.4: Not Just another Incremental Update” at the Technical Exchange conference being held from Oct. 19-25 in Indianapolis. perfSONAR is developed by a collaboration between the Department of Energy’s ESnet, Internet2, Indiana University and GEANT, the pan-European research network.
perfSONAR provides network engineers with the ability to test and measure network performance, as well as to archive data in order to pinpoint and solve performance problems that may span multiple networks and international boundaries.
Berkeley Lab staff from ESnet, NERSC and the IT Division will be among the presenters at the 2014 Technology Exchange, a leading technical event in the global research and education networking community. The annual meeting is co-organized by ESnet and Internet2. The conference will be held Oct. 27-30 in Indianapolis.
Among the topics to be addressed by Berkeley Lab staff are ESnet’s recently announced 100 gigabits-per-second connections to Europe, the newest release of the perSONAR network measurement software library, Science DMZs and cyber security.
The annual meeting brings together a wide range of technical experts to address the challenges facing the research and education networking community as it supports data-intensive research. The conference will be hosted this year by Indiana University.
The perfSONAR-PS project celebrated a milestone by surpassing 1000 deployed software instances in December of 2013. The perfSONAR software is designed to assist network operators and end users with the task of monitoring end-to-end performance, and assisting with debugging tasks in the event that problems arise. As the number of deployments grows, the software becomes more effective by offering better coverage across more network paths, including ESnet and the connectivity to Department of Energy and NSF funded resources. perfSONAR-PS is a joint effort between ESnet, Fermilab, Georgia Tech, Indiana University, Internet2, SLAC, and the University of Delaware.
Need to troubleshoot some network performance problems? perfSONAR-PS is an open source development effort to create a colletion of easy-to-use and easy-to-install perfSONAR network performance monitoring services and tools. Version 3.2.1 of the pS-Performance Toolkit is now available for download. This update contains: new throughput graphs, new delay/loss graphs, scheduled traceroute tests and numerous bug fixes. Release notes are available to denote all changes since the 3.2 release. Please visit the pS Performance Toolkit page for more information.