You may have noticed that the masthead has changed. After almost a decade, we have finally decided that the “ESnet Blog” deserves a less literal name. “Light Bytes” was selected to better embody two things about which we at ESnet are especially proud.
First, getting to build the world’s greatest research and education network, and to support global science is a great honor and a technical challenge. Through ESnet6 and our continuing research, we are advancing our mission of making scientific data free of geographical constraints, to “make bytes light” in terms of fast transport, and to deploy state of the art optical network for “bytes being transported by light.”
Second, while ESnet is officially a DOE User Facility, it is most importantly a remarkable group of people. We hope that this website will show a bit about the great people who make our mission happen and the interesting problems we get to work on. In that sense, “Light Bytes” is a small written offering, a collection of features about things that are happening while the cause of “networking for science” and “science of networking” progresses.
As a Network Engineer at ESnet, I am no stranger to the importance of designing and maintaining a robust fiber-optic network. To operate a network that will “enable and accelerate scientific discovery by delivering unparalleled network infrastructure, capabilities, and tools,” ESnet has acquired an impressive US continental footprint of more than 21,000 kilometers of leased fiber-optic cable. We spend a great deal of effort designing and sourcing redundant fiber-optic paths to support network data connectivity between scores of DOE Office of Science facilities and research collaborators across the country.
But network data transfer is only one of the uses for fiber-optic cable. What about using buried fiber-optic cable for some truly “ground-shaking” science? The answer is “Yes, absolutely!” – and I was fortunate to play a part in exploring new uses for fiber-optic cable networks this past year.
Back in 2017, the majority of our 21,000 km fiber footprint was still considered “dark fiber,” meaning it was not yet in use. At that time, ESnet was actively working on the design to upgrade from our current production network “ESnet5” to our next-generation network “ESnet6,” but we hadn’t yet put our fiber into production.
At the same time, Dr. Jonathan Ajo-Franklin, then graduate students Nate Lindsey and Shan Dou, and the Berkeley Lab’s Earth and Environmental Science Area (EESA) were exploring the use of distributed acoustic sensing (DAS) technology to detect seismic waves by using laser pulses across buried fiber optic cable. The timing was perfect to try and expand on the short-range tests that Dr. Ajo-Franklin and his team had been performing at the University of California’s Richmond Field Station by using a section of the unused ESnet dark fiber footprint in the West Sacramento area for more extensive testing. ESnet’s own Chris Tracy worked with Dr. Ajo-Franklin and team to demonstrate how the underground fiber-optic cables running from West Sacramento northwest toward Woodland in California’s Central Valley made an excellent sensor platform for early earthquake detection, monitoring groundwater, and mapping new sources of potential geothermal energy.
Fast forward to May 2019, and Dr. Ajo-Franklin was heading up a new collaborative scientific research project for the DOE’s Geothermal Technology Office based on his prior DAS experimentation successes using ESnet fiber. The intent was to map potential geothermal energy locations in the California Imperial Valley south of the Salton Sea, near Calipatria and El Centro. The team, including scientists in EESA, Lawrence Livermore National Laboratory (LLNL), and Rice University needed a fiber path to conduct the experiment. It would make sense to assume that ESnet’s fiber footprint, which runs through that area, would be an excellent candidate for this experiment. Fortunately for ESnet’s other users, but unfortunately for the DAS team, by 2018 the ESnet6 team was already “lighting” this previously dark fiber.
However, just because ESnet fiber in the Imperial Valley was no longer a candidate for DAS-based experiments, that didn’t mean there weren’t ways to gain access to unused dark fiber. For every piece of fiber that has been put into production to support ESnet6, there are dozens if not hundreds of other fibers running right alongside it. When fiber-optic providers install new fiber paths, they pull large cables consisting of many individual fibers to lease or sell to as many customers as possible. Because the ESnet fiber footprint was running right through the Imperial Valley, we knew that there was likely unused fiber in the ground, and only had to find a provider that would be willing to lease a small section to Berkeley Lab for Dr. Ajo-Franklin’s experiment.
Making the search a little more complicated, the DAS equipment utilized for this experiment has an effective sensing range that is limited to less than 30 kilometers. Most fiber providers expect to lease long sections of fiber connecting metropolitan areas. For example, the fiber circuits that run through the Imperial Valley are actually intended to connect metropolitan areas of Arizona to large cities in Southern California. Finding a provider that would be willing to break up a continuous 600 km circuit connecting Phoenix to Los Angeles just to sell a 30 km piece for a year-long research project would be a difficult task.
One of my contributions to the ESnet6 project was sourcing new dark fiber circuits and data center colocation spaces to “fill out” our existing footprint and get ready for our optical system deployments. Because of those efforts, I knew that there were often entire sections of fiber that had been damaged across the country and would likely not be repaired until there was a new customer that wanted to lease the fiber. I was asked to assist Dr. Ajo-Franklin and his team to engineer a new fiber solution for the experiment. I just had to find someone willing to lease us one of these small damaged sections.
After speaking with many providers in the area, the communications company Zayo was able to find a section of fiber starting in Calipatria, heading south through El Centro and then west to Plaster City, that was a great candidate for DAS use. This section of fiber had been accidentally cut near Plaster City and was considered unusable for networking purposes. Working with Zayo, we were able to negotiate a lease on this “broken” fiber span along with a small amount of rack space and power to house the DAS equipment that Dr. Ajo-Franklin’s team would need to move forward with their research.
This cut fiber segment was successfully “turned up” for the project on November 10, 2020 by a team including Co-PI Veronica Rodriguez Tribaldos, Michelle Robertson, and Todd Wood (EESA/LBNL), and seismic data collection equipment is now up and running. The figure above (D) shows some great initial data recorded on the array, a small earthquake many miles to the north. There will be many more articles and reports from the Imperial Valley Dark Fiber Team as they continue to gather data and perform their experiments, and I’m sure we’ll begin to see fiber across the country put to use for this type of sensing and research.
I’ve had a great experience working with the different groups that were assembled for this project. By seeing how new technologies and methods are being developed to use fiber-optic cable for important research outside of computing science, I’ve developed a greater appreciation for how our labs and universities are tackling some of our biggest energy and public safety challenges.
ESnet is proud to welcome Rede Nacional de Ensino e Pesquisa (RNP), the national research and education network of Brazil, as an official collaboration partner on the perfSONAR project. The official announcement on the perfSONAR website is here. RNP joins five other organizations (ESnet, GEANT, Indiana University, Internet2, and the University of Michigan) committed to providing dedicated resources that develop and maintain the perfSONAR software.
Even though RNP is now becoming an official project member, they have been part of the perfSONAR community for the past 15 years. RNP has used their own perfSONAR fork for eight years and recently moved to the baseline version of perfSONAR across RNP’s 27 points of presence. The use of the perfSONAR branch code will provide network engineers and customers with improved abilities to maintain their network and validate on-demand circuit use.
Going forward, RNP will be a key contributor in several areas. Iara Machado will be working in conjunction with perfSONAR’s existing steering committee members to provide executive level guidance for the project. Additionally, Marcos Schwarz will join the perfSONAR leadership team to not only spearhead the RNP development team, but also help manage the day-to-day development of the project with existing partner institutions. Initial interests include containerization, perfSONAR as microservices, standard display and analysis packages, and possibly making their circuit validation tool available to the broader community. RNP’s experience and insight will be invaluable to the perfSONAR project going forward.
ESnet and the entire perfSONAR collaboration are excited to officially have them on the team. Having partners like RNP strengthens the perfSONAR initiative and helps ensure a bright future in its continued role as a critical piece of infrastructure for high-performance scientific networks.
Shawn comes to us from Oak Ridge National Laboratory where he most recently held the role of a Senior Research Network Engineer. One of his major research projects was to develop and implement a Precision Timing Protocol (PTP) timing network, to keep data from the nation’s power grid substations synchronized and improve security by removing the dependency on GPS timing signals. Another fun project, for the United States Forestry Service involved developing low-cost (sub $100) durable, yet “disposable” fixed-wing drones, able to detect airborne asbestos fibers released during wildfires.
When did you first become interested in networking and what brought you to ESnet?
My first introduction to networking occurred while enlisted in the US Army. I will forever remember learning subnetting while attending a voluntary class on a rather hot summer Saturday morning held inside of a windowless trailer, without air conditioning, located in the Southern AZ desert. From that point on, I was hooked.
The desire to work with some of the brightest minds on one of the fastest networks around brought me to ESnet. As a network engineer working at another DOE Lab, I had a few chances to work with ESnet engineers as well as being a customer of ESnet. Over the years, I was exposed to the team, their wonderful attitudes, and their first-class customer service. Transitioning to ESnet feels like a natural fit.
What is the most exciting thing going on right now?
Software Defined Networking and 400 Gbps transport speeds are perhaps two of the most exciting things in networking right now, and ESnet is in the middle of both of those. PTP synchronization providing nanosecond timestamping and accuracy is also something that I find exciting. Equally exciting was working on Quantum Key Distribution over WAN networks.
What book would you recommend?
For me, it is a toss-up between “The River of Doubt: Theodore Roosevelt’s Darkest Journey” by Candice Millard or the “One Second After” trilogy by William R. Forstchen.
Please meet Minori Telang. Minori Telang comes to us from a variety of network engineering activities. She has worked for a telecom, in the health sector, and at a SAAS based company.
Outside work, she enjoys travel, hiking, and exploring new coffee shops with her husband. She is very keen on photography and loves to capture nature.
What brought you to ESnet?
During my interview process, I was told that ESnet’s main focus is to build networks for Scientific research and development. Hearing that I felt excited because I could be part of such a big community and can help support various researches across the world. I also think that this will help me grow professionally.
What is the most exciting thing going on right now?
SDN and Cloud Networking is the most exciting technology right now.
What book would you recommend?
Instead of a book, I can recommend a spectacular hike, the Pipiwai trail in Hawaii 🙂
ESnet has recently completed an experiment testing high-performance, file-based data transfers using Data Transfer Nodes (DTNs) on the 100G ESnet Testbed. Within ESnet, new ways to provide optimized, on-demand data movement tools to our network users are being prototyped. One such potential new data movement tool is offered by Zettar, Inc. Zettar’s “zx” product integrates with several storage technologies with an API for automation. This ESnet data movement experiment allowed us to test the use of tools like zx on our network.
Two 100Gbps capable DTNs were deployed on the ESnet Testbed for this work, each with 8 x NVMe SSDs for fast disk-to-disk transfers, and connected using an approximately 90ms round trip time network path. As many readers are aware, this combination of fast storage and fast networking requires careful tuning from both a file I/O and network protocol standpoint to achieve expected end-to-end transfer rates, and this evaluation was no exception. With the help of a storage throughput baseline achieved using the freely available elbenchotool, a single tuning profile for zx was found that struck an impressive performance balance when moving a sweep of hyperscale data sets (>1TB total size or >1M total files or both, see figure below) between the testbed DTNs.
To keep things interesting, the DTN software under evaluation was configured and launched within Docker containers to understand any performance and management impacts, and to establish a potential use case for more broadly deploying DTNs as-a-Service using containerization approaches. Spoiler: the testing was a great success! When configured appropriately, our evaluation has shown that modern container namespaces using performance-oriented Linux networking impart little to no impact on achievable storage and network performance at the 100Gbps scale while enabling a great deal of potential for distributed deployment of DTNs. More critically, the problem of service orchestration and automation becomes the next great challenge when considering any large-scale deployment of dynamic data movement endpoints.
When properly provisioned and configured, a containerized environment has a high potential to provide an optimized, on-demand data movement service.
Data movers such as zx demonstrate that when modern TCP is used efficiently to move data at scale and speed, network latency becomes less of a factor – the same level of data rates are attainable over LAN, Metro, and WAN as long as packet loss rates can be effectively kept low
Finally, creating a holistic data movement solution demands integrated consideration of storage, computing, networking, and highly concurrent and intrinsically scale-out data mover software that incorporates a proper understanding of the variety in data movement scenarios.
For more information, a project report detailing the testing environment, performance comparisons, and best practices may be found here.
Three questions with a new staff member on our Networking Engineering Team!
Asma earned her Bachelors in Computer Science from Higher Colleges of Technology in Dubai, where she majored in Network Sciences and Engineering. In Dubai, she was a member of IEEE Women in Engineering (WIE), one of the first organizations to recognize women’s presence in Engineering in the UAE, and participated in many WIE summits. Asma left Dubai, immigrating to California where she has worked as a Network Engineer in the San Francisco Bay Area for multinational corporations such as Google and Amazon, in Los Angeles for AT&T, and in the Central Valley as a Technical Infrastructure Lead for the Turlock Irrigation District. Beyond network engineering, Asma is also keenly interested in scripting, virtualization, automation, building databases, and working with open-source operating systems.
In her personal time, Asma enjoys reading, traveling, hiking and baking vegan goods.
Question 1: What brought you to ESnet?
I was introduced to ESnet and LBNL through my professor who also happens to work at LLNL. After hours of research on the ESnet public website, I was impressed by the lab’s accomplishments and future projects, specifically the ones that are focused on providing network services for national labs and some international research facilities. At this stage of my career, I wanted to be part of an organization which has an impactful mission that is beyond the bottom line. ESnet seems to satisfy both my professional and personal interests, and I am thrilled about this opportunity!
Question 2: What is the most exciting thing going on right now?
Automation! The vast majority of networking tasks are still executed manually, which can be time and effort taxing for network engineers. Incorporating automation into network services will assist in managing repetitive tasks and consequently improve network availability.
Question 3: What book do you recommend?
Big Farms make Big Flu by Rob Wallace. Looking at our current situation with this deadly pandemic, it’s very important to educate ourselves of how we got here. Apart from the fact that I learned a lot from Rob Wallace’s extraordinary analysis of our current agricultural practices, I also incorporated significant changes in my daily life (plant-based diet, awareness of ethical trades and supporting sustainable energy).
Advancing our strategy and shaping our position on the board. Some thoughts from Inder on the year-that-was.
Dear Friends, Well-wishers, Colleagues, and all of ESnet,
Chess! 2020 has been much more challenging than this game. It’s also been a year where we communicated through the squares on our zoom screens, filled with faces of our colleagues, collaborators, and loved ones.
In January, Research and Education leaders came together in Hawaii at the Pacific Telecommunications Council meeting to discuss the future of networking across the oceans. It was impossible to imagine then that we would not be able to see each other again for such a long time. Though thanks to those underwater cables, we have been able to communicate seamlessly across the globe.
Looking back at 2020, we not only established a solid midgame position on our ESnet chessboard, but succeeded in ‘winning positions’ despite the profound challenges. The ESnet team successfully moved our network operations to be fully remote (and 24/7) and accomplished several strategic priorities.
ESnet played some really interesting gambits this year:
Tackled COVID-related network growth and teleworking issues for the DOE complex
We saw a 4x spike in remote traffic and worked closely across several Labs to upgrade their connectivity. We continue to address the ever-growing demand in a timely manner.
As we all shifted to telework from home, ESnet engineers developed an impromptu guide that was valuable to troubleshoot our home connectivity issues.
Progressed greatly on implementing our next-generation network, ESnet6
We deployed and transitioned to the ESnet6 optical backbone network, with 300 new site installations, 100’s of 100G waves provisioned, with just six months of effort, and while following pandemic safety constraints. I am grateful to our partners Infinera (Carahsoft) and Lumen for working with our engineers to make this happen. Check out below how we decommissioned the ESnet5 optical network and lit up the ESnet6 network.
Installed a brand new management network and security infrastructure upgrades along with significant performance improvements.
We awarded the new ESnet6 router RFP (Congratulations Nokia and IMPRES!); the installs start soon.
Issued another RFP for optical transponders, and will announce the winner shortly.
Took initiative on several science collaborations to address current and future networking needs
We brainstormed new approaches with the Rubin Observatory project team, Amlight, DOE and NSF program managers to meet the performance and security goals for traffic originating in Chile. We moved across several countries in South America before reaching the continental U.S. in Florida (Amlight), and eventually the U.S. Data Facility at SLAC via ESnet.
Drew insights through deep engagement of ESnet engineers with the High Energy Physics program physicists, for serving the data needs of their current and planned experiments expediently. Due to the pandemic, a two-day immersive in-person meeting turned into a multi-week series of Zoom meetings, breakouts, and discussions.
When an instrument produces tons of data, how do you build the data pipeline reliably? ESnet engineers took on this challenge, and worked closely with the GRETA team to define and develop the networking architecture and data movement design for this instrument. This contributed to a successful CD 2/3 review of the project—a challenging enough milestone during normal times, and particularly tough when done remotely.
Exciting opening positions were created with EMSL, FRIB, DUNE/SURF, LCLS-II…these games are still in progress, more will be shared soon.
Innovated to build a strong technology portfolio with a series of inspired moves
We demonstrated Netpredict, a tool using deep learning models and real-time traffic statistics to predict when and where the network will be congested. Mariam’s web page showcases some of the other exciting investigations in progress.
Richard and his collaborators published Real-time flow classification by applying AI/ML to detailed network telemetry.
High-touch ESnet6 project
Ever dream of having the ability to look at every packet, a “packetscope”, at your fingertips? An ability to create new ways to troubleshoot, performance engineer, and gain application insights? We demonstrated a working prototype of that vision at the SC20 XNET workshop.
We deployed a beta version of software that provides science applications the ability to orchestrate large data flows across administrative domains securely. What started as a small research project five years ago (Thanks ASCR!) is now part of the AutoGOLE project initiative in addition to being used for Exascale Computing Project (ECP) project, ExaFEL.
Initiated the Q-Factor project this year, a research collaboration with Amlight, funded by NSF. The project will enable ultra-high-speed data transfer optimization by TCP parameter tuning through the use of programmable dataplane telemetry: https://q-factor.io/
Executed on the vision and design of a nationwide @scale research testbed working alongside a superstar multi-university team.
With the new FAB grant, FABRIC went international with plans to put nodes in Bristol, Amsterdam, Tokyo and Geneva. More locations and partners are possibilities for the future.
Created an prototype FPGA-based edge-computing platform for data-intensive science instruments in collaboration with the Computational Research Division and Xilinx. Look for exciting news on the blog as we complete the prototype deployment of this platform.
What are the benefits of widespread deployment of 5G technology on science research? We contributed to the development of this important vision at a DOE workshop. New and exciting pilots are emerging that will change the game on how science is conducted. Stay tuned.
Growth certainly has its challenges. But, as we grew, we evolved from our old game into an adept new playing style. I am thankful for the trust that all of you placed in ESnet leadership, vital for our numerous, parallel successes. Our 2020 reminds me of the scene in Queen’s Gambit where the young Beth Harmon played all the members of a high-school chess team at the same time.
Several achievements could not make it to this blog, but are important pieces on the ESnet chess board. They required immense support from all parts of ESnet, CS Area staff, Lab procurement, Finance, HR, IT, Facilities, and Communications partners.
I am especially grateful to the DOE Office of Science, Advanced Scientific Computing Research leadership, NSF, and our program manager Ben Brown, whose unwavering support has enabled us to adapt and execute swiftly despite blockades.
All this has only been possible due to the creativity, resolve, and resilience of ESnet staff — I am truly proud of each one of you. I am appreciative of the new hires that trusted their careers with us and joined us remotely—without shaking hands or even stepping foot at the lab.
My wish is for all to stay safe this holiday season, celebrate your successes, and enjoy that extra time with your immediate family. In 2021, I look forward to killer moves on the ESnet chessboard, while humanity checkmates the virus.
Marc Körner is the most recent addition to the ESnet Software Engineering – Orchestration and Core Data team (OCD). He comes to us from Join Digital in San Jose where he was Lead Engineer in their Network Services team.
Marc has a PhD in Computer Science from the Technical University of Berlin, and has spent a number of years working as a researcher in both the Berlin and Berkeley areas. He’ll be working on the automation side of OCD getting familiar with the network services orchestrator platform and helping us achieve our ESnet6 deliverables. Marc is onboarding virtually this week but resides full time in the San Jose area.
What brought you to ESnet?
I was always very passionate about computer networks. The idea of having a global technology for the data and knowledge exchange was always very fascinating for me. It started with the LAN sessions I had with my friends and ended up with the tremendous opportunity to build the first SDN research network in Europe. After my time as a research fellow at the UC Berkeley Netsys lab group and my startup experiences in the access network provider business, the open position for the network automation at ESnet was the ultimate opportunity to take it to the next level.
What is the most exciting thing going on in your field right now?
This question is not easy to answer, there are so many things going on in computer networks. I think one of the biggest innovations in the last decade is the virtualization in general and the centralization of network management and control. However, one of the more recent trends which correlates with this particular network development is edge computing, or the slightly more generalized concept of fog computing and its seamless orchestration. It’s basically a fine granular fusion of the compute and network control plan, which we also observed in cloud computing.
What book would you recommend?
It has been a while since I read a book. As an EECS guy people are probably expecting something to hear like: “The programming language C by Brian Kernighan and Dennis Ritchie”. However, if you are interested in science in general I would probably recommend: “The Universe in a Nutshell by Stephen Hawking”. The book provides some interesting insights about modern physics and has the potential to open up interesting views on the world around us.
ESnet’s first 40 Gb/s public data transfer node (DTN) has been deployed and is now available for community testing. This new DTN is the first of a new generation of publicly available networking test units, provided by ESnet to the global research and engineering network community as part of promoting high-speed scientific data mobility. This 40G DTN will provide four times the speed of previous-generation DTN test units, as well as the opportunity to test a variety of network transfer tools and calibrated data sets.
The 40G DTN server, located at ESnet’s El Paso location, is based on an updated reference implementation of our Science DMZ architecture. This new DTN (and others that will soon follow in other locations) will allow our collaborators throughout the global research and engineering network community to test high speed, large, demanding data transfers as part of improving their own network performance. The deployment provides a resource enabling the global science community to reach levels of data networking performance first demonstrated in 2017 as part of the ESnet Petascale DTN project.
The El Paso 40G DTN has Globus installed for gridFTP and parallel file transfer testing. Additional data transfer applications may be installed in the future. To facilitate user evaluation of their own network capabilities ESnet Data Mobility Exhibition (DME), test data sets will be loaded on this new 40G DTN shortly.
All ESnet DTN public servers can be found at https://app.globus.org/file-manager. ESnet will continue to support existing 10G DTNs located at Sunnyvale, Starlight, New York, and CERN.
The full 40G DTN Reference architecture and more information on the design of these new DTN can be found here:
A second 40G DTN will be available in the next few weeks, and will be deployed in Boston. It will feature Google’s bottleneck bandwidth and round-trip propagation time (BBR2) software, allowing improved round-trip-time measurement and the ability for users to explore BBR2 enhancements to standard TCP congestion control algorithms.
In an upcoming blog post, I will describe the Boston/BBR2-enabled 40G DTN and perfSONAR servers. In the meantime, ESnet and the deployment team hope that the new El Paso DTN will be of great use to the global research community!