Although ESnet is well known for its expertise in supporting the transfer of datasets across the country and around the globe, for the past four years the facility’s staff has also been transferring their networking expertise to staff at other research and education organizations.

Partnering with Internet2 and Indiana University, ESnet co-led 23 workshops as part of the Operating Innovative Networks (OIN) series. Through both in-person and online workshops, the organizers reached and estimated 750 network employees at 360 institutions in 39 states and 38 other nations.

Each workshop was held at a different location and sites were usually chosen by working with regional research and education (R&E) networks. This allowed smaller organizations to tap into the combined expertise of the of the workshop leaders and also made the workshops more accessible to staff at institutions without large travel budgets, said Jason Zurawski, ESnet’s lead for the workshops.

The workshops followed a standard format. The first day covered the Science DMZ architecture, data transfer node tuning and the perfSONAR measurement software. The second day was devoted to some of the concepts behind software-defined networking.

“The OIN workshops have a good balance of lecture and hands-on practical experience. I got the most out of the exercises where the instructors, like Jason Zurawski, took real-world performance issues from the attendees’ live perfSONAR servers and demonstrated how to analyze, explain, and, in several cases resolve, throughput limitations, said Network Manager Brian Jemes of Information Technology Services at the University of Idaho. “In addition, the OIN workshops provided an opportunity to meet and share information with research information technology teams outside our institution.”

Zurawski said that on a couple of occasions, the instructors diverted from the planned instruction to address real-time issues, such as debugging network performance problems and rethinking a planned network architecture.

According to Zurawski, the workshop leads conducted surveys after every session and universally, the responses said the workshops were very timely and very useful. While the larger organizations could more quickly apply the information to their operations, the smaller schools were more likely to use the content for planning and then implement as resources were made available.

“We really tried to include as many institutions as we could in each session and optimize the location to reduce travel,” said Zurawski, who convened all 23 workshops. “We really saw this as an opportunity to talk to the people on the ground who were trying to implement these technologies.”

The value of the workshops was recognized by both the DOE Office of Science, which funded ESnet to participate, and the National Science Foundation, which provided funding under award number 1541421 for the last 10 workshops as part of its Campus Cyberinfrastructure program.

Jemes said his network team had found significant throughput limiting issues, such as microbursts of packet loss due to a faulty optical card in our service provider’s network, that none of their network management tools could identify. At the workshop, they learned that perfSONAR servers “are an indispensable tool for maintaining a high throughput network path for the hosts in our Science DMZ.”

“But despite being well-designed with good documentation and packaged for easy install, perfSONAR is not a turn-key solution,” Jemes said. “To set up and use perfSONAR effectively, you need to do network and server tuning.”

In all, the University of Idaho sent five people to OIN workshops over the past four years and “we found the workshops to be extremely valuable in quickly getting experienced network engineers and server administrators up-to-speed on the effective tuning and operation of a perfSONAR server and the science DMZ network,” Jemes said.

In September 2015, Clemson University hosted one of the workshops at the invitation of Kate Petersen Mace, then director of External Partnership Management at Clemson. In that role, she was project manager for the university’s NSF Campus Cyberinfrastructure award which funded installation of a Science DMZ and implementation of software-defined networking tools.

“As we completed the project, there was interest by both our network engineers and university researchers in learning more about the work and I also wanted to share this deeper knowledge about the Science DMZ with surrounding universities,” said Mace, who joined ESnet’s Science Engagement Team in December 2015. “It was very well attended and all of the people I talked to said it was very beneficial.”

After joining ESnet, Mace began helping teach at the workshops, in particular talking about the importance of science engagement and security best practices, “not to dictate exactly what to do, but to give them ideas on what to think about when implementing new technologies and capabilities. We discussed that the Science DMZ is meant to serve as a security architecture, not just a way to speed up data transfers.”

Although the workshop series is now on hiatus. Zurawski said the team is considering whether to continue them, adapting the content based on feedback from attendees. And while many universities have gotten up to speed after attending an OIN workshop, there is still a need for new information.

Damian Clarke, the CIO for South Carolina State University (SCSU), was the first representative from his university to attend one the workshops, was also the final one held in December 2017.

“As the CIO of SCSU, I was impressed by the knowledge base of the presenters and the right balance of lecture and hands-on exercises,” Clarke said. “I felt that many topics were covered without feeling overwhelmed or confused. I hope that the workshops continue to be funded as more HBCUs (Historically Black Colleges and Universities) and MSIs (Minority Serving Institutions) need to attend.”