Science begins as a Conversation! See how ESnet creates a world where conversations become discovery. Watch our new video now!

Ever want to know how big research data moves around the globe? ESnet plays a significant role in supporting the great scientific conversations, collaborations, and experiments underway, wherever and whenever they occur. We move Exabytes of data around the world creating a global laboratory that accelerates scientific discovery.

In order to meet these needs of scientists, we are constantly looking for opportunities to expand our capabilities with our next generation network ESnet6, intelligent edge analytics, advanced network testbeds, 5G wireless, quantum networking and more.

https://www.es.net/scienceconversation/

ESnet’s Data Mobility Exhibition: Moving to petascale with the research community

Research and Education Networks (REN) capacity planning and user requirements differ from those faced by commodity internet service providers for home users. One key difference is that scientific workflows can require the REN to move large, unscheduled, high-volume data transfers, or “bursts” of traffic. Experiments may be impossible to duplicate and even one underperforming network link can cause the entire data transfer to fail.  Another set of challenges stem from the federated nature of scientific collaboration and networking. Because network performance standards cannot be centrally enforced, performance is obtained as a result of the entire REN community working together to identify best practices and resolve issues.  For example:

  • Data Transfer Nodes (DTN), which connect network endpoints to local data storage systems are owned by individual institutions, facilities, or labs. DTNs can be deployed with various equipment configurations, with local or networked storage configurations, and connected to internal networks in many different ways. 
  • Research institutions have diverse levels of resources and varied data transfer requirements; DTNs and local networks are maintained and operated based on these local considerations.
  • Devising performance benchmarks for “how fast a data transfer should be” is difficult as capacity, flexibility, and general capabilities of networks linking scientists and resources constantly evolve and are not consistent across the entire research ecosystem.

ESnet has long been focused on developing ways to streamline workflows and reduce network operational burdens on the scientific programs, researchers, and others both those we directly serve and on behalf of the entire R&E network community.  Building on the successful Science DMZ design pattern and the Petascale DTN project, the Data Mobility Exhibition (DME) was developed to improve the predictability of data movement between research sites and universities. Many sites use perfSONAR to test end-to-end network performance. The DME allows sites to take this a step farther and test end to end data transfer performance.

DME is a resource that enables the calibration of data transfer performance for a site’s DTNs to ensure that they are performing well by using ESnet’s own test environment, at scale. As part of the DME, system/storage administrators and network engineers have a wide variety of resources available to analyze data transfer performance against ESnet’s standard DTNs, obtain help from ESnet Science Engagement (or from universities, Engagement and Performance Operation Centers) to tune equipment, and to share performance data and network designs with the community to help others.  For instance, a 10Gbps DTN should be capable of – at a minimum – transferring one Terabyte per hour. However, we would like to see DTNs > 10G or a cluster of 10G DTNs transfer at PetaScale rates of 6TB/hr or 1PB/week.

Currently, the DME has geographically dispersed benchmarking DTNs in three research locations:

  • Cornell Center for Advanced Computing in Ithaca, NY, connected through NYSERnet
  • NCAR GLADE in Boulder, CO, connected through Front Range Gigapop
  • Petrel system at Argonne National Lab, connected through ESnet

Benchmarking DTNs are also deployed in two commercial cloud environments: Google Drive and Box.  All five DME DTN can be used for both upload and download testing allowing users to calibrate and compare their network’s data transfer performance. Additional DTNs are being considered for future capacity. Next generation ESnet6 DTNs will be added in FY22-23, supporting this data transfer testing framework.

DME provides calibrated data sets ranging in size from 100MB to 5TB, so that performance of different sized transfers can be studied. 

DOE scientists or infrastructure engineers can use the DME testing framework, built from the Petascale DTN model, with their peers to better understand the performance that institutions are achieving in practice. Here are examples of how past Petascale DTN data mobility efforts have helped large scientific data transfers:

  1. 768 TB of DESI data sent via ESnet, between OLCF and NERSC automatically via Globus over 20 hours. Despite the interruption of a maintenance activity at ORNL, the transfer was seamlessly reconnected without any user involvement.
  2. Radiation-damage-free high-resolution SARS-CoV-2 main protease SFX structures obtained at near-physiological-temperature offer invaluable information for immediate drug-repurposing studies for the treatment of COVID19. This Work required near-real-time collaboration and data movement between LCLS, NERSC via ESnet.

To date, over 100 DTN operators have used DME benchmarking resources to tune their own data transfer performance. In addition, the DME has been added to the NSF-funded Engagement and Performance Operations Center (EPOC) program’s six main scientific networking consulting support services, bringing this capability to a wide set of US Research Universities. 

As the ESnet lead for this project, I invite you to contact me for more info (consult@es.net). We also have information up on our knowledge-base website fasterdata.es.net. DME is an easy, effective way to ensure your network, data transfer, and storage resources are operating at peak efficiency! 

Three questions with a new staff member! Please meet Rémy Doucet

Rémy comes to us from ByteDance/TikTok where they worked as a Systems Engineer responsible for large-scale server allocation and bare-metal OS deployment.  They have worked as a systems engineer for five years, with experience both in the Telecom industry and for large social media companies.  Rémy began their career as a software developer in Python but shifted when they realized a passion for infrastructure and systems.  

Rémy Doucet

What brought you to ESnet?

I have a long history of activism and also worked in the nonprofit sector prior to my engineering career. I became dissatisfied working only for social media giants and began seeking a career that married my passion for technology with my drive to make a positive impact on the world. Climate change is the most pressing issue humans are facing today, so I am excited to begin contributing to a place that not only has an impressive legacy of scientific discovery, but is continuing to make strides in areas such as renewable and clean energy.

What is the most exciting thing going on in your field right now?

Although it is not exactly under my purview, I have always been fascinated by artificial intelligence. Not only will it continue to transform our society in unimaginable ways, but I am also curious to see how it will come to be used for systems administration tasks such as monitoring and deployment. Currently, these processes are still largely human and automation driven but I think we will start to see more AI incorporated into the process in the future. For my personal interests, I enjoy experiencing art or music created by AI.

What book would you recommend?

Simulacra and Simulation by Jean Baudrillard. It is a philosophical treatise that I think will become increasingly relevant in our society.

ESnet’s Nick Buragilio Wins Prize at the Annual LBNL IPO Pitch Competition for Hecate

Three questions with Nick Buragilio about the Pitch Competition

Nick Buragilio took third prize at the LBNL Intellectual Property Office’s Annual Pitch Competition on September 9 for his talk on “Hecate: Directing happiness to internet service provider customers.” Hecate is a software tool that leverages machine learning to automate complex network traffic engineering.  The prize includes $1000 for the Hecate team, supporting continued lab-to-market progress.


How did you develop the technology? It’s a team of three: myself, Scott Campbell, and Mariam Kiran. Scott is handling the data collection and curation and pipelining the data into AI algorithms being developed by Mariam. Mariam is also working on porting the algorithms to GPUs. I’m handling the overall technology and product strategy, plus network elements supporting large-scale traffic engineering. 

The idea came from traffic engineering and segment routing conversations; Mariam had some ideas about bringing in machine learning from the SENSE project, so we sat down over Zoom and sketched things out – it was a natural meeting of minds and very much a virtual “mapping out a project on a napkin” moment, despite the pandemic.

What was it like pulling together the pitch? I enjoy public speaking, and I like to challenge myself. The Pitch Competition seemed like a good opportunity to test the waters and experiment to see what would work and what might not. The challenge was to fit a complicated technical topic into a 5-minute elevator pitch. The Intellectual Property Office supplied coaching as well.

What’s next? We continue on with our testing, and we are looking for more opportunities to use the demonstration software on real data, especially research and educational network partners who can give access to their network data. I’m at buraglio@es.net if a reader is interested in learning more!

Summer at ESnet: student notes, part 3

Eashan Adhikarla is pursuing a Ph.D. at Lehigh University and joined my group this summer to work on our “DTN as a Service” project. He contributed a lot of energy and novel insights into our work over the summer and I hope we have the opportunity to collaborate again in the near future. Here are some thoughts from Eashan on the summer student experience at ESnet.

This is my second internship at Berkeley Lab and my first at the Scientific Networking Division (SND). It has been full of excitement, thrills, challenges, and surprises, and it is a dream place to be.

This summer, I have been working on the intersection of machine learning and high-performance computing in data transfer nodes (DTNs). ESnet connects 40 DOE sites to 140 other networks and therefore has a high demand for data transfers ranging from megabytes to petabytes. The team is designing DTN-as-a-Service (DTNaaS), where the goal is to deploy and optimize the performance of the data movement across various sites. Managing the transmission control protocol (TCP) flows is a key factor in achieving good performance of transfers over a wide range of network infrastructure. My research helps automate DTN performance via machine learning – thus improving the overall DTNaaS framework.

At present, most DTN software is deployed on bare metal servers, limiting the flexibility for operational configuration changes and automation of transfer configurations. Manually inferring best tuning parameters for a dynamic network is a challenge. To optimize the throughput over TCP flow, we currently often use a pacing rate-function to control packet inter-arrival time. A part of my work proposes two different alternative approaches (supervised or sparse regression-based models) to better predict pacing rate, as well as automate change of related DTN settings based on the nature of the transfers.

Overall, my summer research involved getting experience with a wide set of networking areas of interest:

  • Improving the DTN-as-a-Service agent traffic control API with profiles and setting pacing
  • Creating a method for statistics retrieval for the harness toolkit for dynamic data visualization and analysis, and preparing these statistics to train the pacing model
  • Developing a pacing prediction approach that reduces much of the effort for manual pacing rate configuration.

I was also able to contribute to a separate team’s project on exploring the use of network congestion control algorithms for DTNs; the resulting paper will be submitted to an SC21 workshop.

For me, one of the best things at ESnet is that the summer interns get to work directly with quintessential research scientists and research engineers in the lab and learn a variety of skills to tackle the most challenging problems on a real-world scale. It’s a place from which I always come out as a better version of myself.


If you are interested in learning more about future summer opportunities with ESnet, please see this link (https://cs.lbl.gov/careers/summer-student-and-faculty-program/). We typically post notices and application information starting in January or February.

Meeting the Challenge of High Availability through the HASS

Operating a highly optimized network across two continents that meets the needs of very demanding scientific endeavors requires a tremendous amount of automation, orchestration, security, and monitoring.  Any failure to provide these services can create serious operational challenges. 

As we enter the ESnet6 era, ESnet is dedicated to ensuring that we continue to relentlessly push the boundaries of operational excellence and obsessively seek out and improve upon operational risks. Our new High Availability Services Site (HASS) in San Jose, CA. will be a critical component to realizing those goals in our computing platforms and services. ESnet’s HASS will soon provide fully redundant network operations platforms, allowing us to seamlessly maintain services if our operations at LBL are disrupted.

For about a decade, ESnet has augmented its data center operations at Berkeley Lab in California with a small footprint at Brookhaven National Laboratory in New York.  This has allowed us to synchronize important information across two sites and to run multiple instances of important services to ensure operational continuity in the case of a failure.  While this has provided great stability and reliability, there are limitations.  In particular, the 2,500 mile gap across a continent does not let ESnet restore operations without some degree of delay as some key services must be manually transitioned.  HASS will enable seamless operational continuity, since the shorter distance between Berkeley and San Jose will let us automatically maintain the active synchronization of operational platforms.

Deployment of HASS involves a team effort of our ESnet Computing Infrastructure, Network Engineering, and Security teams, working together to architect and deploy the next evolution in our computing and service reliability strategy.  After finalizing our requirements, we are now working with Equinix, a commercial colocation provider, to deploy a site adjacent to the ESnet6 network.  Equinix was able to provide a secure suite in their San Jose facility and this location gives us the capacity, and physical adjacency we require to directly connect this suite to ESnet6 and reach our Berkeley data center comfortably within our demanding latency goals (10ms or less).  

As part of standing up HASS , we’ll be installing a new routing platform with a 100G upstream connection to ESnet6 in both San Jose and Berkeley.  We’ll also be installing new high performance switching platforms, security services (high throughput firewalls, tapping, black hole routing, etc.), virtualization resources, and several other redundant internal operational platforms.  Our existing virtualization platform (ESXi/vSAN) will “stretch” into the new space as part of the same logical cluster we operate in Berkeley.  Once this is deployed, even networking services that lack native high availability capabilities will be able to simply “float” between the two physical data centers with data mirrored and striped across both sites.  

We’re very excited by the addition of the San Jose HASS, and HASS, in combination with existing reliability resources at Brookhaven, will continue to ensure that ESnet6 has the ability to meet scientific networking community needs for service hosting, disaster recovery, and offsite data replication.

Summer at ESnet: The view from our students, Part 2

Pratyush Muthukumar is a student intern with our Prototypes & Testbed Team. He is a 2021 Goldwater Scholar at UC Irvine, working toward a BS in Computer Science. 

Pratyush Muthukumar

My time as an intern at ESnet has been thrilling and rewarding. Over the last seven months as an intern here, I have learned many valuable research, scholarly, and professional skills that I will undoubtedly use in the future. 

I have primarily been working on the SENSE (Software Defined Networking [SDN] for End-to-End Networked Science at the Exascale) project with my amazing mentors, Xi Yang and Tom Lehman, and this has been exciting and fascinating. I’ve learned a plethora of new research skills and frameworks, including multi-domain monitoring, software-defined networking, network polling, scripting, and virtualization. My project mentors have provided great guidance and supported me throughout every stage of the project. What I enjoy best about being an intern at ESnet is the resources, guidance, and tools available to support my research. I’m excited to work on such a challenging yet rewarding tool that will soon have real-world impact when deployed on ESnet.

The environment as an ESnet intern has been extremely friendly and inviting. I feel a part of the community within ESnet and I really enjoy attending the frequent seminars and brown bag lunches held multiple times weekly. Learning about new research developments from other teams at ESnet and guest lecturers from the Computer Science and Mathematics department at UC Berkeley has broadened my knowledge and interests greatly.

While my experiences may have been different during an in-person internship at ESnet, I appreciate the numerous networking and social activities that ESnet has facilitated to make remote interns feel connected. I have really enjoyed my time as an ESnet intern thus far, and I’m looking forward to the rest of my time at ESnet!

If you are interested in learning more about future summer opportunities with ESnet, please see this link (https://cs.lbl.gov/careers/summer-student-and-faculty-program/). We typically post notices and accept applications for the next summer starting in January or February.

Five Women Nab a WIN-ing Opportunity for SC21

Women in IT Networking at SC (WINS) Program Selects SC21 SCinet Cohort

Five new participants have been chosen to join the Women in IT Networking program at SC (WINS) program for SC21 – the annual International Conference for High Performance Computing, Networking, Storage, and Analysis (SC) which will take place this year from  November 14-19 in St. Louis, Missouri.

WINS was developed in 2015 to help address the gender gap that exists in information technology, particularly in the fields of network engineering and high performance computing (HPC). Each year WINS welcomes a cohort of early-to-mid career women to join the team that builds SCinet, a multi-terabit network built specially for high-bandwidth demonstrations SC attendees and exhibitors who perform high-bandwidth demonstrations. The program seeks women of all ages, races, backgrounds, IT-related professions, and geographical areas. 

“Lack of diversity in IT particularly with women and minorities is significant and an ongoing challenge and concern.  WINS is a small effort to encourage women in their early and mid-careers to stay in IT and find a professional network to engage and gain technical and non-technical support. Underrepresented minority groups are defined as African-American/Black, Latino/ Hispanic, and Native American/Native Hawaiians. The problem is particularly acute for women of color, which represent less than 2 percent of high-level technical positions,” said Marla Meehl (Manager Networking and Front Range GigaPoP (FRGP), University Corporation for Atmospheric Research (UCAR)) and WINS-PI.

This year, 19 women applied for WINS. Leaders in the research and education (R&E) network and HPC communities reviewed and selected the following awardees:

  • Mary Bull, College of William & Mary, SCinet Wireless Team
  • Karen Lopez, National Renewable Energy Lab (NREL), SCinet DevOps Team
  • Deshon Miguel, Tohono O’odham Community College, SCinet WAN Team
  • Stacie Nixon, North Carolina Central University, SCinet Wireless Team
  • Kimberly Schjang, University of Nevada – Las Vegas (UNLV), SCinet Edge Team

SC21 participant Kimberly Schjang, Associate Network Engineer from UNLV shared her thoughts on being selected. “I am honored to represent UNLV and WINS by participating in the SC Conference this year. I feel that this opportunity will not only help me grow as a network engineer, but it will also give me the opportunity to represent Black women, the LGBT community, and women in STEM.”

As a core component of the program, the WINS awardees are deeply immersed in a multi-month-long, intense engineering experience before the event begins and during the SC conference while SCinet is operated live. They join one of over fifteen SCinet teams, are paired with a mentor, and are involved in all phases of the SCinet process. 

Kimball Sekaquaptewa, Chief Technology Director for the Santa Fe Indian School and SC19 WINS participant can attest to the experience. “In the span of three weeks, amazing highly-skilled and intelligent professionals who are generous with their knowledge come together to build an incredible 4.2 Terabit wide area network spanning the continental United States to Asia and beyond. You won’t know until you do it, how profound the experience really is.” 

When WINS began in 2015, 14% of SCinet professional volunteers were women. By 2017, the number had climbed to 21%, and in 2020, 40 of the 115 SCinet volunteers – 19 from the WINS program – and 40% of the SCinet leadership were women. 

“WINS has had a tremendous impact on the diversity of SCinet,” said Lauren Rotman, ESnet Science Engagement Group Lead, and a co-lead of the WINS program. “Not only has the male-to-female volunteer ratio doubled since the program started – the program has added another dimension of diversity by bringing in participants from community colleges, tribal colleges, and other smaller institutions that are traditionally underrepresented.” 

To that end, Sekaquaptewa added, “Because I had just completed building two fiber optic backbones connecting six tribes in New Mexico for my home institution, I could now see how to grow them to connect to a national backbone more meaningfully than just access to commodity Internet. I saw and better yet met the people behind the networks with whom I now collaborate to build connections that empower Tribal nations in the digital space.”

The WINS program has also had a significant impact on the entire SCinet volunteer team which speaks to the broad benefits of a diverse workforce. 

Lance Hutchinson, Manager, Space Control Applications, Sandia National Laboratories and SC21 SCinet Chair said, “WINS is not just creating a more inclusive environment for SCinet but it is also improving the quality of learning opportunities for all of our volunteers as part of their participation. All of our team leads, management and others involved in supporting the WINS awardees are building brand new hands-on mentoring, teaching and leadership skills. The program has exponential value as it improves the SCinet community on multiple levels.”

Jennifer Kim, IT Manager at the Montgomery County Community College, an SC19 WINS awardee shared her thoughts. “I am looking forward to the much anticipated event that is SC21. Originally awarded for SC19, I deferred attendance as I was about to be a first-time Mom around the same time. I think this is important to note as this offering demonstrates WINS’ support that is unique to women.” 

Kim added, “My gratitude only grew as the following year, SC20, was virtual, and WINS made sure to offer us awardees the opportunity to participate in  SC21. I am excited to experience the energy and networking, in every sense of the word, of all facets of the SCinet event. I can hardly wait to work with fellow awardees, teammates and learn of their journeys to SC and what keeps them coming back. And of course, eager to build and service the fastest network I will have the pleasure of working on, which happens to be one of the fastest networks in the world.”

Carlos Rojas Torres, Network Engineer at UCAR and team lead for the SCinet Inclusivity team added, “The SCinet team are thrilled to welcome the SC21 WINS participants to become part of this growing ecosystem as well as the SC19 and SC20 deferred participants – making this the largest WINS cohort ever! WINS is a special and significant part of the SC21 Inclusivity Program which aims to provide opportunities for people of diverse backgrounds.”

In addition to the SCinet experience, WINS provides ongoing support and career development opportunities for the awardees before, during, and after the conference. This even includes monthly calls for the community of WINS alumni participants which now tops 40 women. 

Wendy Huntoon, co-PI for the WINS NSF grant said, “The WINS program provides significant opportunities for technical skill development, broad professional development, and career advancement for its participants. WINS participants broaden and deepen their technical skills, gain exposure to technologies or equipment outside their standard work environment, expand their professional network, and improve their collaboration and communication skills. All skills the women are able to apply at their home institution or in the broader networking community.”

WINS is a joint effort between the University Corporation for Atmospheric Research (UCAR),the  Department of Energy’s Energy Sciences Network (ESnet), and the Keystone Initiative for Network Based Education and Research (KINBER). The multi-year program is funded primarily by grants from the National Science Foundation and direct funding from ESnet with support from the Department of Energy. Due to COVID travel restrictions, ESnet will provide programmatic support for SC21. Members of the WINS coordinating team include PI Marla Meehl, Belinda Green, Susan Guastella, Carlos Rojas Torres (SCinet Inclusivity Liaison), of UCAR; co-PI Wendy Huntoon of KINBER;  Jason Zurawski, Lauren Rotman, Kate Robinson of ESnet; and Kimball Sekaquaptewa with Santa Fe Indian School (SCinet Co-Chairs of WINS).

3 Questions with Jeremy Randolph

Please meet Jeremy Randolph, the newest software engineer in ESnet’s Software Measurement and Analysis Group!

Jeremy Randolph

Jeremy has an extensive background in distributed systems, working for companies like DataDog, Fitbit, and Google, to build resilient backends to power customer data visualization and real user interactions. Before that, he also worked in the video game industry at 2K Sports, LucasArts, “The Force Unleashed” franchise, and number of other sports titles.   

What brought you to ESnet?

Recently, I’ve been spending chunks of my free time watching math and physics channels. I’ve also tried some publicly available lecture series with varying degrees of success. In the recent past, working on distributed systems has been rewarding, but I’ve never felt passionate about the wider mission statement of the various companies I’ve worked at. I see ESnet as a chance to build interesting systems while also contributing to our scientific understanding of the universe.

What is the most exciting thing going on in your field right now?

The slow, but steady migration to cloud-based environments and virtual systems. Software Engineers tell horror stories about how our vocation used to have to write our programs on punch cards and would get the program’s output the next day (including things like syntax errors). Real-time syntax highlighting of compile errors in my  IDE (integrated development environment) allows me to focus on the bigger picture and more complicated systems. I suspect the next generation of software engineers will also tell horror stories about DevOps and how we had to have intimate knowledge about what hardware our code was running on and where specifically in the world it was running.

What book would you recommend?

Rendezvous with Rama, by Arthur C. Clarke.