ESnet and SURF Offer Hands-On Network Orchestration Lessons at TNC23

~ ESnetComms
In a TNC23 workshop organized by ESnet’s Chris Cummings and SURFnet’s Hans Trompert, NREN administrators hailing from six continents took the Workflow Orchestrator for a test drive.

If the world’s scientific research networks transported people instead of data packets, in early June you would have seen a traffic spike transiting Tirana, Albania – the site of TNC23, the prestigious research and education networking conference put on by GÉANT. More than 800 participants from 70-plus countries, representing regional and national research and education networks (NRENs), schools and universities, technology providers, and world-changing scientific projects, came together in southeastern Europe for three days of discussion and collaboration.

A sizable delegation from the Department of Energy’s Energy Sciences Network (ESnet) was there, both to share with and learn from their peers. As the United States’ foremost scientific research network, ESnet partners with GÉANT, a federation of European NRENs, as well as with multiple individual NRENs such as SURF in the Netherlands. They’re united by a similar goal: to provide innovative networking infrastructure and services that support and advance scientific research all over the world. 

Cummings (standing) and ESnet’s Nemi McCarter-Ribakoff (seated, center) kick off the Workflow Orchestrator workshop.

Sharing Lessons Learned and Best Practices

In that vein, ESnet Orchestration and Core Data Software Engineer Chris Cummings, with help from ESnet colleagues Nemi McCarter-Ribakoff and Brian Eschen, teamed up with SURFnet Senior Network Architect & Innovation Hans Trompert and Peter Boers for the well-received session “From Zero to Orchestrated — A Workflow Orchestrator Workshop.” This was the first time that ESnet and SURF have together shared the in-depth learnings and hard-earned knowledge gained by their network and software engineers.

Network orchestration and intent-based networking refers to the design and centralized coordination of network resources that allows higher-level services to be realized on the network. This is in contrast to the legacy approach of individual configuration and provisioning of routers, switches, firewalls, and other network devices to deliver a network service. The open-source Workflow Orchestrator tool developed by SURF and ESnet helps network administrators both automate (execute repetitive tasks reliably and easily) and orchestrate (adding a layer of intelligence to tasks being automated and a complete audit log of changes).

Many NRENs would like to add more orchestration, but getting started can be a daunting task requiring a lot of forethought and domain knowledge. Representatives from more than 20 NRENs from six continents attended the all-day, interactive workshop at TNC, which began with introductions to product and workflow modeling, followed by interactive development sessions, and ending with an open discussion around tailoring the Workflow Orchestrator to theoretical use cases. The goal was for attendees to get a locally running version of the Workflow Orchestrator on their laptops as well as some example workflows, provide guided troubleshooting, and show how to make code changes to fix bugs.  

While the attendees appreciated having a working environment to take home with them, “it was also very beneficial for us – learning how to think in an orchestration-forward manner by spending time planning out theoretical product designs with other R&E community members,” says Cummings.

There were challenges: some had trouble getting the workshop running, due to unfamiliarity with the docker containerization platform, as well as not having administrative rights to install docker on their systems. Pulling resources over the hotel wifi was also difficult, but Cummings reports that “Karl Newell from Internet2 came up with some really clever solutions to help his fellow workshop-mates access the images locally – a great example of cross-R&E teamwork!” Cummings and Trompert, Nemi McCarter-Ribakoff, and other ESnet engineers will be applying these lessons learned to the next edition of the Workflow Orchestrator workshop, planned for Internet2’s TechEx conference in late September.

ESnet’s Tom Lehman explained the advantages of SENSE orchestration to the TNC23 audience.

ESnet at TNC23

Among ESnet’s other speakers and presenters were Chief Technology Officer and Planning & Innovation Group Lead Chin Guok and ESnet/Berkeley Lab Networked Systems Researcher and Developer Tom Lehman, who took the massive concert hall stage to share an overview of Managed Network Services for Large Data Transfers, focusing on the integration work between the SENSE [SDN for End-to-End Networking at Exascale] orchestration and Rucio data management systems. Their goal was to demystify the often opaque role of the network in science workflow processes by showing how advanced wide area network traffic engineering, end site infrastructure awareness/control, and domain science workflow intelligence can improve research results and planning abilities.

Science Engagement Acting Group Lead Eli Dart gave an update on the high-performance network design pattern Science DMZ.

ESnet Science Engagement Acting Group Lead Eli Dart presented on The Strategic Future of the Science DMZ, the science-focused high-performance network design pattern created by ESnet, highlighting new environments and applications such as Streaming DTNs, Zero Trust, and Exascale HPC, and workflows that couple experimental and computing facilities to achieve previously unachievable results. Dart also teamed up with Karl Newell from Internet2 to talk about Identifying and Understanding Scientific Network Flows, in particular the effort underway from the High-Energy Physics (HEP) and Worldwide LHC Computing Grid (WLCG) communities to mark packets/flows so they can be correlated with specific research projects. This approach, which allows identification of flows for troubleshooting and gives network providers visibility into the research flows they support, can be leveraged by any research organization and network provider willing to participate in packet marking.

Chin Guok shared an assessment of the effectiveness of ESnet’s pilot cache system.

Planning & Architecture Computer Systems Engineer Nick Buraglio chaired a session titled “If It Was Easy, We’d Have Done it By Now” about innovations in networking that included Guok summarizing the findings of ESnet’s In-Network Caching Pilot. Guok also co-chaired a workshop, Planning and Development in R&E Networks, that included strategy discussion for approaches to intercontinental connectivity, packet layer renewal, automation, and Big Science requirements.

It was an intense couple of days of networking. Another half-dozen ESnetters, including Executive Director Inder Monga, were also in Albania to attend TNC23. A group of them unwound after the conference ended by going on a challenging hike above stunning Lake Bovilla.

“We all learned a lot,” said Cummings. “And it was great to be able to contribute in a concrete way to the workflow orchestration community.”   

After TNC ended, a group of ESnetters hiked up above Lake Bovilla, a reservoir northeast of Tirana within Mount Dajt National Park. Photo: Brian Eschen.

ESnet’s EinR Summer Interns Are Visualizing Their Path to a STEM Career

~ ESnetComms

Among this summer’s cohort of 53 Experiences in Research high school interns are three from Hawai’i and two from the Bay Area who are working on similar but different network data visualization projects for ESnet.

“Much of the things I am doing in the project were things I could not have imagined were in my ability to try a month ago,” said Ella Jeon, a rising junior in Pleasanton, CA. “One significant new mindset I have experienced over the course of this internship is the whole ‘being able to try things that I didn’t think were really possible or something I was really capable of’ type of realization. The boost of guidance and support in this internship has made me realize how much more I could go on to try and achieve on my own as well.”

Read the full story at ES.net/news

Experience in Research (EinR) Program students Ella Jeon and Sahil Hayak meeting with their mentors Sartaj Baveja and Chris Tracy of ESnet.

ESnet Launches Cloud Connect Service to Support Scientific and Enterprise Workloads 

~ ESnetComms

Diagram of ESnet6’s peering points for the new Cloud Connect Service

By Joshua Stewart, ESnet

Part of managing a network dedicated to handling vast swaths of scientific data is also ensuring it adapts to trends for how data is being created, stored, and computed. A pattern has emerged in recent years allowing for access to elastic and scalable systems on demand. Nebulously titled “The Cloud,” it refers to software and services that run over the public internet. For ESnet, this is just another place where science intends to happen. 

To drill down more on the nebulosity of the term “The Cloud,” there are different flavors of how the services/software are consumed. “Public Cloud” refers to services and software that are open for all users and subscribers around the world: for example, those provided by Dropbox, Slack, Salesforce, and Office 365. Meanwhile, as its name suggests, a Virtual Private Cloud (VPC) is an environment in which all virtualized hardware and software resources are dedicated exclusively to, and accessible only by, a single organization. The intention of a VPC is to emulate the on-premise data centers of old while removing the headaches of managing their physicality (space and power constraints), and offering the added benefit of instantaneous access to scale when needed. Although some organizations decided to go all-in on the new virtual environments by harnessing a cloud-native posture, some took a more measured approach by seamlessly blending their on-premises infrastructure with the new virtualized territory, in a format also known as a hybrid cloud. 

As usage of virtual private clouds grew, it became apparent that connectivity over the public internet was too unreliable, slow, and insecure: dedicated, high-bandwidth connectivity was a must-have. In response, every major Cloud Service Provider (CSP) launched an offering. Amazon Web Services (AWS) was first, launching “Direct Connect” in 2012; Azure followed in 2014 with its “ExpressRoute”; and in 2017, Google launched Cloud Interconnect. (Read more about the history.)

These virtual circuits are the driver behind the new ESnet Cloud Connect service aimed at supporting both scientific and enterprise workloads. The goal is to carve out a dedicated, high-bandwidth path (up to 10 Gbps) across ESnet’s 400GE-capable backbone from any supported user facility to the nearest cloud on-ramp by utilizing two interim network service providers: Packet Fabric and Equinix. From there, ESnet would help provision the major CSPs’ (Azure, AWS, GCP) aforementioned flavor of dedicated connectivity into your Virtual Private Cloud.

This solution is designed to scale from simple dedicated connectivity and a singular cloud provider to a virtual routed network utilizing multiple cloud providers, onramps, and interconnecting user facilities.  This series of blog posts will focus on a few suggested use cases for utilizing ESnet’s new service offering. For questions or to learn more, email Joshua Stewart.

3Qs with Measurement & Analysis Intern Felix Renken

~ ESnetComms
Felix Renken sitting in chair in front of monitor displaying project
Felix Renken at his desk in ESnet’s Building 59

ESnet Measurement & Analysis Intern Felix Renken is a senior student from Technische Universität Berlin, majoring in Computer Science with a focus on Media Technologies and Signal Processing. Originally from a rural area near Hamburg in northern Germany, he moved to Berlin to pursue his college education. He arrived in Berkeley in March and will be going home in early July.

During his internship, Felix worked on developing an open-source Grafana plugin for visualizing network data that can be used in ESnet’s Stardust system, which collects precise network measurement data and allows users to retrieve information about specific equipment over a given time range. (Learn more about Stardust via this talk by Ed Balas and Andy Lake.) Felix’s plugin enables users to visualize various data collected by Stardust, revealing the relationship between pairs of data from different destinations for a single source and showcasing common attributes in nodes and links along with the option to visualize AS paths. The plugin is currently undergoing the Grafana community plugin review process; the source code is available on GitHub. It is installed on Stardust too, for anyone who wants to check it out.

Want to know more? On June 30, Felix gave an ESnet Policy, Innovation, Practices, and Engineering (PIPE) talk about the Arc Diagram Plugin and his internship experience.

Diagram of the Arc plugin
Arc Diagram: Click to enlarge

What brought you to ESnet?

During my search for interesting internship opportunities, I came across ESnet’s student program and contacted Marc Körner and Katrina Turner to get more information on the projects they supervise. I eventually applied for the “Data Visualization of Network Measurement Data” project. It encompasses the development of an open-source tool that visualizes network data in an exciting way. The opportunity of getting work experience in a research environment greatly appealed to me. And, of course, the chance to spend time in California!

What is the most exciting aspect of your field right now?

The cross-disciplinary nature of visualizing data is particularly interesting to me. It utilizes principles from design, statistics, and computer science, offering opportunities to learn from diverse perspectives.

How was Berkeley different from Berlin? What fun things did you do here?

Berkeley and Berlin are distinct in so many aspects. Berkeley is, of course, much smaller in size than Berlin, and I really enjoyed being in a city that is less hectic. People here seem more relaxed. And the fact that Berkeley is somewhat shaped by its university was also something that I’m not used to from Berlin or any other German city. Cycling here was scarier than in Berlin though. Another thing is the accessibility to the fantastic nature around Berkeley. I went hiking a lot and will definitely miss being in close proximity to beautiful trails when going back to Germany. Other fun things I did were camping and eating a lot of burritos.

3Q with Alex Withers, ESnet’s New Deputy Chief Information Security Officer

~ ESnetComms
Alex Withers in a blue t-shirt in front of green bushes
Alex Withers at home in Urbana, Illinois.

Network cybersecurity must strike a delicate balance between openness and safety. ESnet has long focused more on the first, in order to facilitate scientific communication, data sharing, and collaboration. But in today’s Wild West of ransomware, phishing, and other threats, safeguarding this vital network is equally critical. 

Deputy Chief Information Security Officer (CISO) Alex Withers oversees a reorganized, two-part structure for security at ESnet: the Security Engineering group, which he heads, and a new Threat & Vulnerability Management group, which Chief Security Officer Adam Slagell is leading during the search for a senior threat hunter/DOE community coordinator. For Security Engineering, among Alex’s responsibilities are overseeing ESnet’s effort to comply with the federal Zero Trust requirements — an approach to cybersecurity that goes beyond “trust but verify” and treats all networks and traffic as potential threats. Alex will be making sure that any new security policies, procedures, and architecture do not impede ESnet’s vision of enabling scientific progress that is completely unconstrained by the physical location of instruments, people, computational resources, or data.

Alex has deep experience in threat intelligence sharing, policy and compliance, and security architecture. Most recently he was the CISO and cybersecurity division manager at the National Center for Supercomputing Applications at the University of Illinois. While at NCSA, Alex oversaw groups responsible for security operations, applied cybersecurity research, cybersecurity engagement, and scientific computing in the HIPAA and Controlled Unclassified Information [CUI] space. He was a PI or co-PI on several National Science Foundation awards for projects focused on intrusion detection, threat intelligence dissemination, and capabilities-based authorization for access to scientific computing resources. Before NCSA, Alex worked for Brookhaven National Laboratory as a security and systems engineer for over 10 years. 

Alex grew up in Alaska and now lives in Urbana, Illinois, where he works out of ESnet’s Champaign office with Adam and Security Engineering team members Kapil Agrawal, Michael Dophelde, and Sam Oehlert, as well as about a dozen other ESnetters. He’s an avid long-distance runner – as are his wife and two of his four children. In the last few years, Alex has completed around a dozen ultra-marathons, or “ultras,” ranging from 50 km to 100 miles – something he’s “always reluctant to tell people about because it sounds crazy.”   

What brought you to ESnet?

Really it was the opportunity for growth — both to tackle new challenges in cybersecurity architecture and for me professionally, to try something new. ESnet is responsible for connecting a massive portion of the scientific computing infrastructure that supports not just this country’s scientific investments but also international collaborations. It’s growing extremely rapidly, and it’s a giant target for all sorts of reasons, whether from state-sponsored attacks or cyber criminals or anything in between. And so it looks like an immense challenge, and that’s very attractive to me.

What is the most exciting thing going on in your field right now?

There’s been a shift in how people view cybersecurity that’s making it easier for us to collaborate and innovate with users. 

Traditionally, cybersecurity has had a bad reputation as being the people who say “No, you can’t do this; no, you can’t do that.” And in the research and education sector, the culture tends to be much more open, much more about getting done what has to be done, whether that’s moving data around or access to computing for scientists and their students. That culture has often bumped up against cybersecurity, which tends to want to wall things off. But cybersecurity is now much more about enabling science. As I tell people, “Listen, the funding agencies, the government, have invested billions of dollars in science and in this infrastructure that you rely on for your particle accelerator, electron microscope, whatever. And we want to protect that investment, because at the end of the day, things like cybersecurity incidents, they can disrupt your work. They can stop it dead in its tracks, and that’s money that’s lost.” 

So today’s cybersecurity is about understanding how researchers use these systems and devices, how they access them through the network – and working together to make sure that we enable their use and make it available while at the same time very secure. At ESnet, we want to ensure the integrity of the data so that researchers can be productive on their computational systems and networks. That frame of approach is easier than traditional cybersecurity, which is more focused on things like confidentiality and privacy.

What book, movie, or podcast would you recommend?

A podcast I’ve been really enjoying has been “Some Work, All Play” by David Roche and Megan Roche. It’s an excellent inclusive running podcast for all runners, especially trail running, which is a hobby I enjoy.

Running ultras sounds like a little more than just a “hobby.” Tell us more about why you do it?

Well, a lot of people think of it as a very physical sort of endeavor. And I mean…that’s true, and I don’t want to downplay that, but you’d be surprised to find out that it’s not as difficult as you think it is. The real challenge is the mental challenge. It is extremely difficult mentally to go out on a trail and run for hours and hours and hours and hours. You’re really fighting against the urge to drop out and call it a day. And sometimes you’re not successful. 

What’s great about it is pushing yourself up against the limits of what you can do. There are people who seriously race these things, and they win. I’m not in any danger of doing that, I assure you. For me, it’s racing against various aspects of yourself. Racing against yourself mentally. Racing against your past self. Maybe you’re going to do better on a race you’ve done before. Sometimes you’re racing against your own stomach, because you have to eat during these things—but it’s not a pleasant task to eat while you’re doing all this running! 

It’s very challenging, but it’s also a lot of fun. It’s very rewarding when it works out. 

ESnet Team Showcases New Services, Pushes Networking Boundaries at OFC2023

~ ESnetComms ~ 1 Comment
ESnet software engineers Sarah Larsen, Dan Doyle, and Bruce Mah at ESnet’s High Touch demonstration booth

After a pandemic-related in-person hiatus, the Optical Fiber Communication Conference and Exhibition (OFC), sponsored by Optica, IEEE Communication Society, and IEEE Photonics Society, resumed operations with a sold-out event in March 2023 at the San Diego Convention center. More than 11,500 participants and 515 exhibitors attended this global event for optical communications and networking, including almost two dozen from ESnet. Planning & Architecture Acting Group Lead Chris Tracy led ESnet’s multifaceted involvement at OFC23, which ranged from a booth demonstrating ESnet’s High Touch project and panel discussion to helping implement OFCnet, an unconventional high-speed network connecting the show floor to a research center in Chicago. 

Staffed by ESnet software engineers Bruce Mah, Sarah Larsen, and Dan Doyle, the ESnet booth presented a high-level technical overview and showed examples of data and analysis from the High-Touch system being deployed in ESnet6, the latest version of ESnet’s backbone network for supporting scientific collaborations and research around the globe.​ The High-Touch project uses a combination of software and programmable, off-the-shelf hardware to deliver new network services. Its first applications provide high-precision network telemetry, including summarization of network flows and capture of packet headers, which are computed from unsampled streams of packets from multiple 100GE and 400GE links. This demonstration relied heavily on the efforts of ESnet’s Infrastructure team to install and configure dozens of data collection servers across ESnet’s network footprint.

ESnet High-Touch Architecture and Design: This diagram shows the flow of packets and network measurements through ESnet’s High-Touch system, which uses a combination of programmable, off-the-shelf hardware and software to provide high-precision network telemetry.

ESnet Executive Director Inder Monga and Chris also realized that OFC2023 offered potential for demonstrating network capabilities that went beyond the exhibition floor. Prior to OFC2022, there was no high-speed, “external” network connectivity at the event suitable for data-intensive demonstrations. The conference consisted of technical talks about papers that were being published and vendor booths. At OFC2022, Optica, Lumen, CENIC, Ciena and Smart City successfully showed in a modest proof of concept that external fiber could be brought into the convention center so that a live demonstration could be run on the show floor. For OFC2023, Ciena’s office of the CTO – who was leading the OFCnet effort – approached ESnet about demonstrating high-performance networking applications as well as emerging technologies, and more broadly, bringing some networking focus into the conference.

Working with Ciena staff, ESnet Network Services Optical Network Group Lead Patrick Dorn and Network Engineers Michael Blodgett, Kate Robinson, and Nathan Miller helped build an un-regenerated 400 Gbps link between the OFC show floor in San Diego and the StarLight Data Center in Chicago. “Un-regenerated” means the signal remains solely in the optical domain, e.g. as wavelengths of light, not an electrical signal, for transcontinental distances (more than 4,600 kilometers). 

Another interesting feature of this demonstration was that the ESnet team connected ESnet6’s production Infinera FlexILS line system to a Cisco NCS 1010 line system (provided by Cisco to support OFCnet), effectively bridging the purpose-built OFC exhibition network to a live, nation-scale infrastructure. In addition to the Infinera and Cisco line systems, Ciena provided the ultra-long-haul transponder equipment necessary to communicate over such distances, plus the engineering expertise – along with staff from Cisco, ESnet and CENIC – to ensure it all worked. 

Enlarge for more detail

Using the high-speed channel ESnet established between San Diego and Chicago, researchers from Northwestern University’s International Center for Advanced Internet Research (iCAIR) could showcase data transfer applications being used to move massive scientific datasets. By helping to implement this somewhat unconventional infrastructure, the ESnet team sought to show what might someday be possible when networks can transport 400 Gigabit Ethernet over such long distances without relying on bonding two 200 Gbps wavelengths using inverse muxing. 

Some of the ESnet team at OFC23

In addition to the two demonstrations, ESnet staff participated in multiple panel discussions and a bird of a feather (BoF) event at OFC23. For a panel on how high performance research networks continue to drive fundamental science and innovation, Chris Tracy and others used OFCnet and its connection to an external Research & Education network to discuss data transfer for data intensive science, detailed monitoring of science flows within the network, network security considerations in the research network environment, and applications like distributed computing that take advantage of these networks. At the BoF event, Inder presented, while Chris, ESnet staff, and other OFCnet volunteers brainstormed ideas for how OFCnet might evolve as a next-generation optical photonic network for OFC2024. One recommendation: a Sunday workshop titled: “How Can OFC with a Real-Life Testbed Accelerate Innovation in the Design and Operation of Next Generation Optical Photonic Networks?” The BoF participants believe this would provide an opportunity to invite speakers and publish papers within the context of the workshop for these kinds of networking-related topics.

Planning for next year’s iteration of OFCnet (March 24-28, 2024) has already kicked off, with ESnet once again participating in a leadership role. The goals for OFCnet24 are ambitious. The volunteer team hopes to attract attendees from different communities, such as networking science (academia and research labs); make it possible to showcase high performance networking application use cases and other emerging technologies – turning the exhibits floor as a science accelerator; and bridge the exhibit and technical programs by offering the opportunity to present advanced technical papers with live demos. 

“It was great to be able to demonstrate some of the innovative services we’re delivering through the High Touch project,” said Chris. “And of course we welcome any opportunity for ESnet to participate in something such as OFCnet that advances the state of the art for networking and allows us to showcase emerging technologies on our network. Next year is going to be even more exciting.” 

Marc Lyonnais (right), OFCnet chair and director of external research at Ciena, presented Planning & Architecture Acting Group Lead Chris Tracy (left) and Executive Director Inder Monga (center) with a plaque in thanks for ESnet’s OFCnet efforts

The Science Conversation Continues at Confab23, Oct. 16-18 in D.C.

~ ESnetComms
confab23 logo

Registration is now open for Confab23, to be held October 16 to 18 in Gaithersburg, Maryland! 

ESnet’s second annual Confab gathering is designed for scientists across all disciplines who want to vastly improve their workflows and collaborations to accelerate time to discovery; for network engineers from national labs and universities who support science IT services for researchers on their campuses; and for the research networking professionals who partner with ESnet to move data across the world.

Last year’s inaugural Confab in Berkeley, held concurrently with the unveiling of ESnet6, was a fun and resounding success — and we believe Confab23 will continue and broaden the conversation we started. 

Confab23 will showcase scientists who use ESnet today to perform real-time data analysis, leverage multiple supercomputers in parallel for large-scale simulations, and collaborate with colleagues on experiments as if side by side while thousands of miles apart – among many other applications.

Together we can chart the future of scientific data management and integrated scientific infrastructure.  

In addition to lively conversation and informal technical discussions between our ESnet, DOE, and scientific community attendees, the program includes:

  • Updates and discussions with the Department of Energy on major initiatives, such as the Integrated Research Infrastructure initiative and the High Performance Data Facility, that support our shared vision — that scientific progress will be completely unconstrained by the physical location of instruments, people, computational resources, or data.
  • An overview of quantum networking activities by Inder Monga, part of a lively evening program

We will also hold a meeting of the ESnet Site Coordinator Committee at the same venue on October 19 and 20.

If you have any questions or suggestions – or would like to offer a presentation or session topic – please email our workshop team directly at confab@es.net.

Register now >

ESnet6 Honored with DOE Project Assessment Award

~ ESnetComms
Man presenting award to woman flanked by flags
DOE Office of Project Assessment Director Kurt W. Fisher and ESnet Network Services Group Lead Kate Petersen Mace, ESnet6 project director, who accepted the award on behalf of the ESnet6 team

It’s rare for any technology project to be completed early and under budget — let alone a massively complex one involving extensive hardware and software upgrades across many states. Yet Energy Sciences Network’s (ESnet) ESnet6 project was finished more than two years ahead of schedule and for less than it was estimated. In recognition of this unusual feat, the Department of Energy (DOE) recently presented ESnet with a special Project Assessment Award. (As an IT project, ESnet6 is not eligible for the DOE’s Project Management Awards.) 

ESnet6 is the newest iteration of the DOE’s high-performance network, also known as the “data circulatory system” for the DOE science complex. Not only did ESnet6 boost bandwidth to more than 46 Terabits per second — a significant increase – it also automated network operations for scalability and reliability, improved security services, and replaced aging equipment. In addition, ESnet6 offers greater programmable network flexibility that will support evolving computation and data models in the emerging exabyte data era. 

Six years in the making, ESnet6 was completed well under budget six months before the forecasted early finish date of January 2023 – and more than two years ahead of the forecasted CD-4 date in January 2025. 

DOE Office of Project Assessment Director Kurt W. Fisher presented the award in a private ceremony at the DOE Project Management Workshop in Washington, DC, in April. ESnet Network Services Group Lead Kate Petersen Mace, ESnet6’s project director, accepted on behalf of the ESnet6 team.

“ESnet6 represents the culmination of several years of extraordinary commitment and tireless dedication by all of ESnet’s staff,” said Inder Monga, ESnet’s executive director. “We’re grateful to Berkeley Lab for its support and to DOE for recognizing the collective efforts of the team behind this critical piece of scientific infrastructure.” 

Rotating GIF of two zoom galleries of ESnet6 project staff
Part of the ESnet6 team smiling at the CD-4 Review meeting, joined by reviewers and DOE representatives.

At ESnet, Innovation and Collaboration Build Solutions for Today and Tomorrow

~ Inder Monga ~ 1 Comment

Inder Monga reflects on 2022’s highlights and looks ahead to the future.

ESnet Executive Director Inder Monga at the launch event celebrating the unveiling of ESnet6, the sixth generation of the Department of Energy’s (DOE’s) dedicated high-speed scientific network.

Dear Friends, Well-wishers, Colleagues, and all of ESnet,

It’s been less than a year since ESnet formally introduced ESnet6, the latest iteration of the U.S. Department of Energy’s Energy Sciences Network. And we’ve already made much progress in enhancing research capabilities and data sharing across a broad spectrum of scientific applications.

For more than 35 years, ESnet – headquartered at Lawrence Berkeley National Laboratory – has served as the data circulatory system for the DOE, connecting all of its national laboratories, tens of thousands of DOE-funded researchers, and DOE’s premier scientific instruments and supercomputing centers. This interconnected system enables data to move quickly between sites and collaborators, accelerating time-to-discovery.

ESnet6, unveiled in October 2022 in conjunction with Confab, our first user meeting, takes the network’s capabilities to the next level. ESnet6 represents a transformational change in the way networks are built for research, with improved capacity, resiliency, and flexibility. With more than 46 Terabits per second of aggregate bandwidth deployed, it features a significant increase over prior generations of the network. This boost in capacity enables scientists to more quickly process, analyze, visualize, share, and store the mountains of research data produced by experiments, modeling, and simulations.

But the new network – which was completed under budget and ahead of schedule – does more than just increase capacity. With ESnet6, our engineers have developed smart, programmable, and automated services uniquely built to support the multi-petabyte dataflows typical of science research today. In addition, they are future-proofed to manage the emerging exabyte data era, streaming data from instruments and high-impact digital twins that require predictability and low latency. 

For example, ESnet is a critical component of Berkeley Lab’s Superfacility Project, which offers researchers seamless analysis of their experimental data in real time and regardless of their location. Additionally, with the recent ‘Superlab’ demonstration of the ARIES project by the National Renewable Energy Laboratory (NREL), we demonstrated how these new capabilities can be used to “address large-scale emergent challenges to meet the nation’s clean energy goals and to reinforce the energy security needs of every community,” as Rob Hovsapian, ARIES research lead in hybrid energy systems at NREL, noted in a collaborative news release announcing this project. With this in mind, we’re already looking to what users and stakeholders would like to see next.

Four strategic thrusts will define our efforts: 

  1. Transform Operations: While priority one is to operate a highly performant and robust network, we are also exploring new architectures, infrastructure enhancements, improvements to business processes, additional orchestration and automation capabilities, and ways to integrate new technologies like AI/ML – all to improve the resiliency, efficiency, and effectiveness of the user facility.
  2. Expand Services Portfolio: Our current services are foundational to the national labs and science communities. As we enter into an exascale era, with data-intensive instruments and widely distributed experiments, the network will play a key role in providing critical data services and supporting distributed data workflows, both for our scientists and the sites. The staff continue to innovate, experiment, prototype, and transition to production new data and network services. In addition, we actively look to expand the modalities through which scientists acquire data, from private 5G to low-Earth-orbiting satellites in remote locations, and potentially through quantum networks.
  3. Increase Stakeholder Value: As high-speed and big-data networking experts, we can co-design solutions based on upcoming requirements with our scientific and site user community to ensure that ESnet provides the most value to all of DOE as well as the worldwide research and education community stakeholders.
  4. Build Accountability and Transparency: We will foster the culture of accountability and transparency that provides the right environment for our users and our employees to perform at their personal best. 

ESnet exemplifies the team science value of Berkeley Lab. Our partnerships with all of the DOE national labs, vendors, global research and education networks, and academia have been essential to the design and build of ESnet6 and our future endeavors. Our integration of experimental, networking, and computational facilities gives scientists the ability to take a giant leap forward in gaining insight from massive datasets produced by experiments that use large-scale instruments such as genome sequencers, telescope observatories, X-ray light sources, and particle accelerators, among many others. We know we cannot do this alone. Participating in community-based collaborative initiatives better positions us to address future needs for all users and stakeholders. Some examples include:

  1. Co-design with science collaborations: SENSE/Rucio integration (collaboration with U.S. CMS [Compact Muon Solenoid experiment]) and GRETA networking (collaboration with Nuclear Physics) in co-designing data/science workflows with scientists.
  2. Open source contributions: Collaborating with and contributing to the SURFnet Workflow Orchestrator for network automation. (Please see the “From Zero to Orchestrated—A Workflow Orchestrator Beginners’ Workshop” at TNC, June 2023, co-organized with SURFNet.) Contributions to perfSONAR, iperf3, Grafana, and many others are part of ESnet’s work with the larger networking community.
  3. Strategic collaborations with worldwide R&E partners: Transatlantic MOU with ANA (Advanced North Atlantic) collaboration partners to make “gap on oceans” irrelevant when it comes to scientists.
  4. Enabling impactful networking research through multi-organization collaboration: Research collaborations on the FABRIC Testbed to supercharge network and distributed systems research within the U.S. and internationally. The Berkeley Lab–led Quantum Testbed (QUANT-NET) will accomplish the same for quantum communications and computing.

We are applying the same thoughtfulness to our staffing efforts. People want to work in organizations that have meaningful impact and contribute to humanity, and we are building the foundation to support this. Between 2018 and 2022, ESnet grew by 200%, hiring and adding a diverse array of skillsets to realize a dedicated staff of more than 100. As we look to the future, we strive to build a balanced workplace that represents a diversity of backgrounds, skillsets, regions, and states. 

Ultimately, ESnet’s success depends on the sum of its people – those who work in or with our organization have ample opportunity to have a meaningful impact on humanity and science.

Ultimately, ESnet’s success depends on the sum of its people – those who work in or with our organization have ample opportunity to have a meaningful impact on humanity and science. In addition to our commitment to next-generation enabling technologies, this is a key focus for ESnet over the next 10 years and beyond. ESnet6 is designed to support the DOE’s multi-billion dollars of investments in scientific research that touches our everyday lives, and we will continue to invest in these and related technologies, services, and people to support the needs of the DOE, HPC, and global science communities. 

The ESnet Portal: Visualizing the Network

~ ESnetComms

As a facility that provides reliable, high bandwidth interconnectivity to scientists at national laboratories, universities, and research institutions, it is important for ESnet to share timely and accessible information about the network and its current status. The my.es.net portal has been an innovative place for sharing this information since its inception. 

Following a six-month upgrade project, ESnet’s public-facing portal now provides improved visualization capabilities. It allows users to easily view the core network and its connections to national lab sites. With this updated version, users can now see the network in a more detailed view that also provides the ability to zoom in and pan. By utilizing an in-house developed network visualization library, this enhanced version synthesizes a longer time range of data to provide faster, more accurate, and more detailed network topology updates.

You can experience the new version and look at all its new capabilities at https://my.es.net

Building on a Treasure Trove of Measurement Data

In 2021, ESnet rolled out its Stardust system, which collects precise network measurement data and allows users to retrieve information about specific equipment over a given time range. This updated release builds on Stardust’s capabilities, giving users a window into measurement data over the entire network topology.

A New View of the Network

Historically, ESnet’s portal has only offered a logical view of the network, visualizing the connections between sites but only approximating the network footprint. The updated version offers two visualization options: an updated and expanded logical “subway-style” view and a geographically referenced view. While building both views of the network, care was taken to strike a balance between providing a visualization that is as rich and accurate as possible while minimizing visual clutter. The approach uses interactive layers to help target the most important network information.

ESnet pictured in “logical” view

ESnet pictured in “geographic” view

Understanding Network Utilization

One of the most important things to understand when building and maintaining a network is measurements of bandwidth utilization, particularly at peak and near-peak conditions. To give users a clearer understanding of these key peak bandwidth utilization measurements, the map sends Stardust queries for traffic aggregations that display the “95th percentile,” or near-peak traffic, and a “maximum,” or peak measurement for a given period of time. These measurements give a sense of the “high water marks” for the network, letting ESnet know when a “flood” might occur and helping visualize the available headroom and plan for times of highest utilization.

Map (with options) picturing the “high water marks” for the last week

Tools for the Future

In conjunction with ESnet’s Stardust system, this update to the portal allows for much more responsiveness to changes in the network topology in the future. When a new router or site is added, ESnet can bring a visualization of it online in minutes rather than days. 

With this added capability and flexibility, ESnet may enhance site-centric capabilities, providing tailored views or new network layers (e.g., layers of university sites or peering points with commercial and R&E networks) to better inform users.
The portal’s new visualization of ESnet is a significant upgrade that provides researchers, network engineers, and other stakeholders with a more comprehensive, detailed, and accurate view of our network. With new statistical methods, more extensive time-based analysis, and a greater range of visualization capabilities, our portal update provides valuable insights into network behavior and performance. If you’re an ESnet user, check out the new tool and see how it can help you understand the network in greater detail.