ESnet6

ESnet Launches Cloud Connect Service to Support Scientific and Enterprise Workloads 

~ ESnetComms

Diagram of ESnet6’s peering points for the new Cloud Connect Service

By Joshua Stewart, ESnet

Part of managing a network dedicated to handling vast swaths of scientific data is also ensuring it adapts to trends for how data is being created, stored, and computed. A pattern has emerged in recent years allowing for access to elastic and scalable systems on demand. Nebulously titled “The Cloud,” it refers to software and services that run over the public internet. For ESnet, this is just another place where science intends to happen. 

To drill down more on the nebulosity of the term “The Cloud,” there are different flavors of how the services/software are consumed. “Public Cloud” refers to services and software that are open for all users and subscribers around the world: for example, those provided by Dropbox, Slack, Salesforce, and Office 365. Meanwhile, as its name suggests, a Virtual Private Cloud (VPC) is an environment in which all virtualized hardware and software resources are dedicated exclusively to, and accessible only by, a single organization. The intention of a VPC is to emulate the on-premise data centers of old while removing the headaches of managing their physicality (space and power constraints), and offering the added benefit of instantaneous access to scale when needed. Although some organizations decided to go all-in on the new virtual environments by harnessing a cloud-native posture, some took a more measured approach by seamlessly blending their on-premises infrastructure with the new virtualized territory, in a format also known as a hybrid cloud. 

As usage of virtual private clouds grew, it became apparent that connectivity over the public internet was too unreliable, slow, and insecure: dedicated, high-bandwidth connectivity was a must-have. In response, every major Cloud Service Provider (CSP) launched an offering. Amazon Web Services (AWS) was first, launching “Direct Connect” in 2012; Azure followed in 2014 with its “ExpressRoute”; and in 2017, Google launched Cloud Interconnect. (Read more about the history.)

These virtual circuits are the driver behind the new ESnet Cloud Connect service aimed at supporting both scientific and enterprise workloads. The goal is to carve out a dedicated, high-bandwidth path (up to 10 Gbps) across ESnet’s 400GE-capable backbone from any supported user facility to the nearest cloud on-ramp by utilizing two interim network service providers: Packet Fabric and Equinix. From there, ESnet would help provision the major CSPs’ (Azure, AWS, GCP) aforementioned flavor of dedicated connectivity into your Virtual Private Cloud.

This solution is designed to scale from simple dedicated connectivity and a singular cloud provider to a virtual routed network utilizing multiple cloud providers, onramps, and interconnecting user facilities.  This series of blog posts will focus on a few suggested use cases for utilizing ESnet’s new service offering. For questions or to learn more, email Joshua Stewart.

ESnet6 Honored with DOE Project Assessment Award

~ ESnetComms
Man presenting award to woman flanked by flags
DOE Office of Project Assessment Director Kurt W. Fisher and ESnet Network Services Group Lead Kate Petersen Mace, ESnet6 project director, who accepted the award on behalf of the ESnet6 team

It’s rare for any technology project to be completed early and under budget — let alone a massively complex one involving extensive hardware and software upgrades across many states. Yet Energy Sciences Network’s (ESnet) ESnet6 project was finished more than two years ahead of schedule and for less than it was estimated. In recognition of this unusual feat, the Department of Energy (DOE) recently presented ESnet with a special Project Assessment Award. (As an IT project, ESnet6 is not eligible for the DOE’s Project Management Awards.) 

ESnet6 is the newest iteration of the DOE’s high-performance network, also known as the “data circulatory system” for the DOE science complex. Not only did ESnet6 boost bandwidth to more than 46 Terabits per second — a significant increase – it also automated network operations for scalability and reliability, improved security services, and replaced aging equipment. In addition, ESnet6 offers greater programmable network flexibility that will support evolving computation and data models in the emerging exabyte data era. 

Six years in the making, ESnet6 was completed well under budget six months before the forecasted early finish date of January 2023 – and more than two years ahead of the forecasted CD-4 date in January 2025. 

DOE Office of Project Assessment Director Kurt W. Fisher presented the award in a private ceremony at the DOE Project Management Workshop in Washington, DC, in April. ESnet Network Services Group Lead Kate Petersen Mace, ESnet6’s project director, accepted on behalf of the ESnet6 team.

“ESnet6 represents the culmination of several years of extraordinary commitment and tireless dedication by all of ESnet’s staff,” said Inder Monga, ESnet’s executive director. “We’re grateful to Berkeley Lab for its support and to DOE for recognizing the collective efforts of the team behind this critical piece of scientific infrastructure.” 

Rotating GIF of two zoom galleries of ESnet6 project staff
Part of the ESnet6 team smiling at the CD-4 Review meeting, joined by reviewers and DOE representatives.

At ESnet, Innovation and Collaboration Build Solutions for Today and Tomorrow

~ Inder Monga ~ 1 Comment

Inder Monga reflects on 2022’s highlights and looks ahead to the future.

ESnet Executive Director Inder Monga at the launch event celebrating the unveiling of ESnet6, the sixth generation of the Department of Energy’s (DOE’s) dedicated high-speed scientific network.

Dear Friends, Well-wishers, Colleagues, and all of ESnet,

It’s been less than a year since ESnet formally introduced ESnet6, the latest iteration of the U.S. Department of Energy’s Energy Sciences Network. And we’ve already made much progress in enhancing research capabilities and data sharing across a broad spectrum of scientific applications.

For more than 35 years, ESnet – headquartered at Lawrence Berkeley National Laboratory – has served as the data circulatory system for the DOE, connecting all of its national laboratories, tens of thousands of DOE-funded researchers, and DOE’s premier scientific instruments and supercomputing centers. This interconnected system enables data to move quickly between sites and collaborators, accelerating time-to-discovery.

ESnet6, unveiled in October 2022 in conjunction with Confab, our first user meeting, takes the network’s capabilities to the next level. ESnet6 represents a transformational change in the way networks are built for research, with improved capacity, resiliency, and flexibility. With more than 46 Terabits per second of aggregate bandwidth deployed, it features a significant increase over prior generations of the network. This boost in capacity enables scientists to more quickly process, analyze, visualize, share, and store the mountains of research data produced by experiments, modeling, and simulations.

But the new network – which was completed under budget and ahead of schedule – does more than just increase capacity. With ESnet6, our engineers have developed smart, programmable, and automated services uniquely built to support the multi-petabyte dataflows typical of science research today. In addition, they are future-proofed to manage the emerging exabyte data era, streaming data from instruments and high-impact digital twins that require predictability and low latency. 

For example, ESnet is a critical component of Berkeley Lab’s Superfacility Project, which offers researchers seamless analysis of their experimental data in real time and regardless of their location. Additionally, with the recent ‘Superlab’ demonstration of the ARIES project by the National Renewable Energy Laboratory (NREL), we demonstrated how these new capabilities can be used to “address large-scale emergent challenges to meet the nation’s clean energy goals and to reinforce the energy security needs of every community,” as Rob Hovsapian, ARIES research lead in hybrid energy systems at NREL, noted in a collaborative news release announcing this project. With this in mind, we’re already looking to what users and stakeholders would like to see next.

Four strategic thrusts will define our efforts: 

  1. Transform Operations: While priority one is to operate a highly performant and robust network, we are also exploring new architectures, infrastructure enhancements, improvements to business processes, additional orchestration and automation capabilities, and ways to integrate new technologies like AI/ML – all to improve the resiliency, efficiency, and effectiveness of the user facility.
  2. Expand Services Portfolio: Our current services are foundational to the national labs and science communities. As we enter into an exascale era, with data-intensive instruments and widely distributed experiments, the network will play a key role in providing critical data services and supporting distributed data workflows, both for our scientists and the sites. The staff continue to innovate, experiment, prototype, and transition to production new data and network services. In addition, we actively look to expand the modalities through which scientists acquire data, from private 5G to low-Earth-orbiting satellites in remote locations, and potentially through quantum networks.
  3. Increase Stakeholder Value: As high-speed and big-data networking experts, we can co-design solutions based on upcoming requirements with our scientific and site user community to ensure that ESnet provides the most value to all of DOE as well as the worldwide research and education community stakeholders.
  4. Build Accountability and Transparency: We will foster the culture of accountability and transparency that provides the right environment for our users and our employees to perform at their personal best. 

ESnet exemplifies the team science value of Berkeley Lab. Our partnerships with all of the DOE national labs, vendors, global research and education networks, and academia have been essential to the design and build of ESnet6 and our future endeavors. Our integration of experimental, networking, and computational facilities gives scientists the ability to take a giant leap forward in gaining insight from massive datasets produced by experiments that use large-scale instruments such as genome sequencers, telescope observatories, X-ray light sources, and particle accelerators, among many others. We know we cannot do this alone. Participating in community-based collaborative initiatives better positions us to address future needs for all users and stakeholders. Some examples include:

  1. Co-design with science collaborations: SENSE/Rucio integration (collaboration with U.S. CMS [Compact Muon Solenoid experiment]) and GRETA networking (collaboration with Nuclear Physics) in co-designing data/science workflows with scientists.
  2. Open source contributions: Collaborating with and contributing to the SURFnet Workflow Orchestrator for network automation. (Please see the “From Zero to Orchestrated—A Workflow Orchestrator Beginners’ Workshop” at TNC, June 2023, co-organized with SURFNet.) Contributions to perfSONAR, iperf3, Grafana, and many others are part of ESnet’s work with the larger networking community.
  3. Strategic collaborations with worldwide R&E partners: Transatlantic MOU with ANA (Advanced North Atlantic) collaboration partners to make “gap on oceans” irrelevant when it comes to scientists.
  4. Enabling impactful networking research through multi-organization collaboration: Research collaborations on the FABRIC Testbed to supercharge network and distributed systems research within the U.S. and internationally. The Berkeley Lab–led Quantum Testbed (QUANT-NET) will accomplish the same for quantum communications and computing.

We are applying the same thoughtfulness to our staffing efforts. People want to work in organizations that have meaningful impact and contribute to humanity, and we are building the foundation to support this. Between 2018 and 2022, ESnet grew by 200%, hiring and adding a diverse array of skillsets to realize a dedicated staff of more than 100. As we look to the future, we strive to build a balanced workplace that represents a diversity of backgrounds, skillsets, regions, and states. 

Ultimately, ESnet’s success depends on the sum of its people – those who work in or with our organization have ample opportunity to have a meaningful impact on humanity and science.

Ultimately, ESnet’s success depends on the sum of its people – those who work in or with our organization have ample opportunity to have a meaningful impact on humanity and science. In addition to our commitment to next-generation enabling technologies, this is a key focus for ESnet over the next 10 years and beyond. ESnet6 is designed to support the DOE’s multi-billion dollars of investments in scientific research that touches our everyday lives, and we will continue to invest in these and related technologies, services, and people to support the needs of the DOE, HPC, and global science communities. 

Under Budget and Ahead of Schedule, ESnet6 Project Receives Final CD-4 Approval

~ ESnetComms

The Department of Energy’s Office of Project Assessment recently issued its final CD-4 Review Report on the Energy Sciences Network (ESnet)’s ESnet6 upgrade project. The review, held on July 12 – 13, 2022, and conducted at the request of Barbara Helland, Associate Director of Science for Advanced Scientific Computing Research (ASCR), assessed the project’s readiness to proceed to the approval of project completion. The project completed all threshold key performance parameters (KPPs) six months ahead of the early finish date, two years ahead of the CD-4 Level 1 milestone date, and well under budget. The committee assessed that the project was ready to proceed to CD-4 approval, which was achieved on July 29, 2022. The Final Closeout report and Lessons Learned are being submitted next week within the specified 90-day window.

“I want to congratulate the entire ESnet organization especially the ESnet6 project director, Kate Mace, and the project team,” said Inder Monga, executive director of ESnet. “When the team set out to deliver on a project scope as vast as the ESnet6 launch, we did not imagine a global pandemic would interrupt the process. Despite that, the team delivered the entire project ahead of the deadline and, even with supply-chain issues, managed to complete the scope below the projected budget. Most of the team attended the ESnet6 Unveiling Event on October 11 and heard their accomplishments praised by the lab directorate as well as congresspeople and DOE staff.”

The committee commended the project team for their “unique and innovative approach” in completing the project objectives and complimented ESnet for their agility in following through with the project scope while dealing with the difficult environment generated by the COVID-19 pandemic. The report stated that COVID-19 restrictions, limitations, and supply-chain issues presented “no significant impact” on the project’s critical path. The report also identified the distributed nature of operations and ESnet’s support for a remote workforce as an “invaluable” approach and a best practice to be shared beyond the DOE complex.

The final reports required for the official Project Closeout will be submitted to DOE this week. The ESnet team has continued to keep up the pace as they work toward additional enhancements to the ESnet6 Facility. “The #ESnet6Week festivities the week of October 9 energized the team. Not only were the project accomplishments celebrated at the ESnet6 Unveiling event, but the team also heard firsthand about the impact the project has already had on scientific discovery,” said Kathryn Mace, the ESnet6 project director, and Network Engineering group lead. “Hearing about the expansion of scientific collaborations made easier with the ESnet6 network and automated operations provided the team with newfound motivation to keep moving full speed ahead. ESnet6 sets the foundation for global scientific innovation over the next 10 years.”

The ESnet6 project team, DOE staff, IPR Committee, and members of the Berkeley Lab directorate during the final CD-4 IPR closeout session.

ESnet6 Unveiled Tomorrow!

~ Andrew Wiedlea

We’re getting things set up for the ESnet6 Unveiling tomorrow – our tent has gone up, we’re holding final rehearsals for the presentations, printing badges, and doing a thousand other small things.  

Photo: Lauren Rotman, LBL
Photo: Lauren Rotman, LBL

The only thing missing from our pictures is you! 

See you tomorrow for the big day, if you are visiting in person, travel safe, and if you are joining us virtually, the show starts at 9:00 AM on https://streaming.lbl.gov.

ESnet6 Investment Supports Next Generation Exascale Earth System Model

~ ESnetComms

Scientists at Oak Ridge, Argonne, and Lawrence Livermore National Laboratories are collaborating on the next generation of integrated Earth climate models using Exascale Computing Project computers and simulation models. The Earth System Grid Federation program is building vast simulation models using data collected about our planet at all levels, from space to far below the surface. Predictions from these models are vital to our understanding of climate, ocean, and other complex systems that make life possible. Read more about this and ESnet’s role in this important international science conversation in a new phys.org article from Oak Ridge National Laboratory.

Visualization from the Earth System Model, one component of the Earth System Grid Federation program. ESnet provides the data connectivity necessary to stitch teams and computers at different labs together. Credit: LLNL, U.S. Dept. of Energy

The ESnet6 Unveiling Ceremony is 4 days away!  Come celebrate our new network and the great science we support, like the Earth System Grid Federation. Join us from 9 a.m. – 12 a.m., 11 October on https://streaming.lbl.gov.

Why We Designed and Deployed ESnet6: It is All About the Science!

~ ESnetComms

We’re just a few days away from the ESnet6 unveiling and Confab22!

Here’s a great video interview with Ann Almgren, Senior Scientist in CCSE and the Department Head of the Applied Mathematics Department in the Applied Mathematics and Computational Research Division at Berkeley Lab. In it she discusses her research into wind power generation/distribution, and how she will use ESnet6.

Ann Almgren, Berkeley Lab

To watch the unveiling of ESnet6 and learn more about Ann’s research, join us 11 October from 900AM – 12 PM PT at streaming.lbl.gov!

ESnet6 Unveiling in Seven Days!

~ ESnetComms

On October 11, 2022, we will welcome the newest generation of our high-performance scientific network, ESnet6, at an unveiling ceremony hosted by Berkeley Lab.

ESnet6 marks a new era of our high-performance network supporting the needs of scientists. We’re able to handle massive flows of data in a reliable, nimble way, and we can specifically configure our setup to match the needs of individual experiments. The upgrade ensures that ESnet is ready to support the future of science today, including the significant increase in the amount of data produced by scientific experiments and the increasingly complex needs of scientists and the way they interact with our network. 

Come watch the ESnet6 unveiling ceremony 9AM -12 PM PT, October 11, at streaming.lbl.gov!

At the speed of ESnet!

Watch our latest video: Serving Conversations That Matter

Title card from the Serving Conversations That Matter video
~ ESnetComms

ESnet exists to support research into some of the most important questions of our time. The traffic that travels over our network on a daily basis contains data from tens of thousands of researchers – data that could lead to the next major discovery or scientific breakthrough.

In our latest video, learn about just a few of the revolutionary research collaborations we support, and the questions they’re working to answer.

Making the Research and Educational Community SAFER: Adam Slagell on the creation of a new global collaboration to combat cyberthreats.

~ ESnetComms

Adam Slagell is ESnet’s Chief Security Officer and a founding member of the newly formed Security Assistance For Education & Research (SAFER) trust group.

SAFER is an operational security entity focused on fighting computer misuse and defending the academic, research, and education (R&E) mission globally.  SAFER brings together expertise and resources from organizations across the Research and Educational cybersecurity community, including CERN, DFN-CERT, ESET, ESnet, LBNL, STFC, and WLCG.

More information can be found here https://www.safer-trust.org/.

What motivates the creation of SAFER and what do you think success will look like for the community?

There are many cybersecurity trust groups out there, some even dedicated to R&E like REN-ISAC or XSEDE’s trust group consisting of current and former Teragrid and XSEDE site  members. However, there really isn’t anything like this—both permanent and truly international— even though attacks are almost always transnational. So each time there is a new, major campaign, an international group connecting all these regional responders must be created again. What we are trying to do is create that permanent backbone with a core set of highly connected individuals who are a part of these regional and project-specific trust groups around the world.

If we are successful, we will see several things. First, I believe we will see more international cooperation and information sharing, leading to an earlier notice of new attack campaigns. Second, we will be able to activate a response more quickly, pulling in the expertise needed from a broad pool of SAFER members and their trusted colleagues. Finally, it is our hope that we can provide surge capabilities when a member is under attack. Many R&E organizations have limited resources and small teams. It is a tremendous asset if they can get help from their peers, maybe with unique expertise as they are facing a disruptive attack.

What kind of security resources will SAFER provide?

I alluded to some of the services when discussing what success will look like. But ultimately, our security resources will be determined by community needs. The founding members will serve as the steering committee for the first year until we elect the next steering committee. 

One of our  first-steps is  setting up a Malware Information Sharing Platform (MISP) instance to share Indicators of Compromise, e.g., IP addresses, file hashes, domain names, etc. Usually, there is no requirement for members to share such data as the rules and regulations differ so much across organizations. But even on day one, we will have enough organizations that can contribute to making this service useful.

There is also a secure messaging and chat service using decentralized cryptography that all of our members can participate in. These ad hoc conversations about what people are seeing on their networks will hopefully help detect trends early.

Finally, many of the founding members have more resources from these large institutions, and I believe we can quickly help those projects and institutions that might struggle with an attack by providing our expertise while helping to train the next generation of security professionals.

What excites you most about this effort and what is the opportunity to do the most good?

I love the community-building aspect. In a past life, I created the Bro (now Zeek) Leadership Team and really worked hard to build a vibrant community around that software. I think this expertise is where I can be most helpful as I am less technical in my roles today.

I will also say, I am excited about getting young people involved, too. Organizations who contribute time from their teams will really benefit. There is no training for an incident response like jumping in, and I expect the variety of issues we will see will prove very useful just from a training and development perspective.

LBL has a long history supporting cybersecurity research, from the early days of Clifford Stoll and The Cuckoo’s Egg to the creation of Bro.  What does the future of cybersecurity look like, and how will that shape the REN community?

Indeed, LBL’s security team is also a SAFER founding member. One of the things I love about working here and at ESnet is that our mission is outward-focused and when we help the community we raise all boats so to speak.

Fortune telling however is a dangerous game. We have anticipated some things, like cryptocurrency mining coming to HPCs. However, the threat landscape and tools available keep changing. That is part of what makes this job interesting. The important thing that I hope we keep in mind is that security is not done for its own sake, but to enable our mission of scientific research. To me, this means that we must always work to make risk-based security decisions, even when that might challenge pushes for compliance and simple one-size-fits-all solutions.