Author: Inder Monga

At ESnet, Innovation and Collaboration Build Solutions for Today and Tomorrow

~ Inder Monga ~ 1 Comment

Inder Monga reflects on 2022’s highlights and looks ahead to the future.

ESnet Executive Director Inder Monga at the launch event celebrating the unveiling of ESnet6, the sixth generation of the Department of Energy’s (DOE’s) dedicated high-speed scientific network.

Dear Friends, Well-wishers, Colleagues, and all of ESnet,

It’s been less than a year since ESnet formally introduced ESnet6, the latest iteration of the U.S. Department of Energy’s Energy Sciences Network. And we’ve already made much progress in enhancing research capabilities and data sharing across a broad spectrum of scientific applications.

For more than 35 years, ESnet – headquartered at Lawrence Berkeley National Laboratory – has served as the data circulatory system for the DOE, connecting all of its national laboratories, tens of thousands of DOE-funded researchers, and DOE’s premier scientific instruments and supercomputing centers. This interconnected system enables data to move quickly between sites and collaborators, accelerating time-to-discovery.

ESnet6, unveiled in October 2022 in conjunction with Confab, our first user meeting, takes the network’s capabilities to the next level. ESnet6 represents a transformational change in the way networks are built for research, with improved capacity, resiliency, and flexibility. With more than 46 Terabits per second of aggregate bandwidth deployed, it features a significant increase over prior generations of the network. This boost in capacity enables scientists to more quickly process, analyze, visualize, share, and store the mountains of research data produced by experiments, modeling, and simulations.

But the new network – which was completed under budget and ahead of schedule – does more than just increase capacity. With ESnet6, our engineers have developed smart, programmable, and automated services uniquely built to support the multi-petabyte dataflows typical of science research today. In addition, they are future-proofed to manage the emerging exabyte data era, streaming data from instruments and high-impact digital twins that require predictability and low latency. 

For example, ESnet is a critical component of Berkeley Lab’s Superfacility Project, which offers researchers seamless analysis of their experimental data in real time and regardless of their location. Additionally, with the recent ‘Superlab’ demonstration of the ARIES project by the National Renewable Energy Laboratory (NREL), we demonstrated how these new capabilities can be used to “address large-scale emergent challenges to meet the nation’s clean energy goals and to reinforce the energy security needs of every community,” as Rob Hovsapian, ARIES research lead in hybrid energy systems at NREL, noted in a collaborative news release announcing this project. With this in mind, we’re already looking to what users and stakeholders would like to see next.

Four strategic thrusts will define our efforts: 

  1. Transform Operations: While priority one is to operate a highly performant and robust network, we are also exploring new architectures, infrastructure enhancements, improvements to business processes, additional orchestration and automation capabilities, and ways to integrate new technologies like AI/ML – all to improve the resiliency, efficiency, and effectiveness of the user facility.
  2. Expand Services Portfolio: Our current services are foundational to the national labs and science communities. As we enter into an exascale era, with data-intensive instruments and widely distributed experiments, the network will play a key role in providing critical data services and supporting distributed data workflows, both for our scientists and the sites. The staff continue to innovate, experiment, prototype, and transition to production new data and network services. In addition, we actively look to expand the modalities through which scientists acquire data, from private 5G to low-Earth-orbiting satellites in remote locations, and potentially through quantum networks.
  3. Increase Stakeholder Value: As high-speed and big-data networking experts, we can co-design solutions based on upcoming requirements with our scientific and site user community to ensure that ESnet provides the most value to all of DOE as well as the worldwide research and education community stakeholders.
  4. Build Accountability and Transparency: We will foster the culture of accountability and transparency that provides the right environment for our users and our employees to perform at their personal best. 

ESnet exemplifies the team science value of Berkeley Lab. Our partnerships with all of the DOE national labs, vendors, global research and education networks, and academia have been essential to the design and build of ESnet6 and our future endeavors. Our integration of experimental, networking, and computational facilities gives scientists the ability to take a giant leap forward in gaining insight from massive datasets produced by experiments that use large-scale instruments such as genome sequencers, telescope observatories, X-ray light sources, and particle accelerators, among many others. We know we cannot do this alone. Participating in community-based collaborative initiatives better positions us to address future needs for all users and stakeholders. Some examples include:

  1. Co-design with science collaborations: SENSE/Rucio integration (collaboration with U.S. CMS [Compact Muon Solenoid experiment]) and GRETA networking (collaboration with Nuclear Physics) in co-designing data/science workflows with scientists.
  2. Open source contributions: Collaborating with and contributing to the SURFnet Workflow Orchestrator for network automation. (Please see the “From Zero to Orchestrated—A Workflow Orchestrator Beginners’ Workshop” at TNC, June 2023, co-organized with SURFNet.) Contributions to perfSONAR, iperf3, Grafana, and many others are part of ESnet’s work with the larger networking community.
  3. Strategic collaborations with worldwide R&E partners: Transatlantic MOU with ANA (Advanced North Atlantic) collaboration partners to make “gap on oceans” irrelevant when it comes to scientists.
  4. Enabling impactful networking research through multi-organization collaboration: Research collaborations on the FABRIC Testbed to supercharge network and distributed systems research within the U.S. and internationally. The Berkeley Lab–led Quantum Testbed (QUANT-NET) will accomplish the same for quantum communications and computing.

We are applying the same thoughtfulness to our staffing efforts. People want to work in organizations that have meaningful impact and contribute to humanity, and we are building the foundation to support this. Between 2018 and 2022, ESnet grew by 200%, hiring and adding a diverse array of skillsets to realize a dedicated staff of more than 100. As we look to the future, we strive to build a balanced workplace that represents a diversity of backgrounds, skillsets, regions, and states. 

Ultimately, ESnet’s success depends on the sum of its people – those who work in or with our organization have ample opportunity to have a meaningful impact on humanity and science.

Ultimately, ESnet’s success depends on the sum of its people – those who work in or with our organization have ample opportunity to have a meaningful impact on humanity and science. In addition to our commitment to next-generation enabling technologies, this is a key focus for ESnet over the next 10 years and beyond. ESnet6 is designed to support the DOE’s multi-billion dollars of investments in scientific research that touches our everyday lives, and we will continue to invest in these and related technologies, services, and people to support the needs of the DOE, HPC, and global science communities. 

A word from Inder Monga: The Road to ESnet6 (Part 1)

~ Inder Monga
Inder Monga, Executive Director of ESnet.

Dear Friends, Well-wishers, Colleagues, and all of ESnet,

In October of this year we will launch ESnet6, a next-generation network featuring an entirely new, software-driven network design that enhances the ability to rapidly invent, test, and deploy new innovations to meet the data needs of the Office of Science/DOE.

We put forth the vision for ESnet6 in 2016. Since then, this $151M project (total project cost – DOE 413.3 parlance including contingency) has overcome pandemic-induced issues like site lockdowns, differing vaccination and inter-state travel policies, and variable supply chain delays, and is now in its final stages of implementation. As I prepare this historic unveiling, I can’t help but look back at what the team accomplished last year.

This is the first post in a series of blog posts about the people, partnerships, and innovations that have paved the road to ESnet6.

2021 was a year for growth within ESnet. We have 100+ people in the organization now—a 30% increase from last year—and it has been great to have new employees on-boarded, integrated, and productive in this challenging environment. 

A diagram showing the dimensions of growth within ESnet: Foundations, Innovation, Co-design, and Culture. Foundations, Innovation, and Co-design all point outward in separate directions, while Culture lies alongside all three Axes, growing in tandem with them.
The dimensions of growth for ESnet

Looking towards the future, we think of ESnet growing around four dimensions. The three spatial axes are: 

  • Foundations: Next Generation Network and Services 
  • Innovation: Testbeds and Advanced Research and Development
  • and Co-design: Partnerships with Science for new data and network solutions. 

The fourth axis, Culture, is pervasive across all three dimensions. 

The main reason for choosing this very technical representation is to illustrate that these are not independent thrusts—success in each of these dimensions depends on the capabilities of the other.

In this post, I’d like to focus on that first axis: Foundations. In the next few posts, I will focus on the Innovation and Co-Design dimensions and share more thoughts about our focus for 2022 and beyond.

Major capacity improvements

In 2021, we installed a brand new routing infrastructure on our network backbone, while decommissioning a portion of the previous generation packet processors in parallel. We seamlessly transitioned all ESnet customers and peers onto the forty new backbone routers before the holidays, and the remaining router upgrades at our customer sites are in progress and scheduled through 2022.

The greenfield optical infrastructure (installed at 300 locations in 2020— another noteworthy accomplishment) is getting a wonderful upgrade: 400G wavelengths are being standardized across our national backbone as we complete the second phase of optical upgrades.

In addition to our team’s intricate efforts to decommission the existing network, we added another 100G on the ring in Europe (thanks to our collaboration with GEANT). This ensured that the first Large Hadron Collider Data Challenge had enough bandwidth to accommodate both ESnet scientific data and LHC data challenge (test) streams. We also established a new point of presence in Dallas to support new peerings and the FABRIC project. 

ESnet network map showing LHC data challenge traffic sending nearly 100Gbps from Amsterdam to Boston
ESnet network map showing LHC data challenge traffic sending nearly 100Gbps from Amsterdam to Boston.

Creating a smarter network

The vision laid out in 2016 focused not only on capacity, but also on improving the essential framework of how we operate with the network. 

We made a significant investment in building out a high-availability site within 10ms of our main data center, in addition to our disaster-recovery site on the east coast. So any planned or unplanned power outages will be handled without a scramble. While the supply chain issues prevented the site from being ready for operations, we are making steady progress and look forward to completing it this year. 

The software orchestration team made tremendous progress in laying down the vision and framework for automation. They were supported by strong internal collaboration with the engineering team. Many repetitious deployments were automated, and I know it took diligent effort to make these tools available in the right time frame, aligned with evolving constraints of the deployments. A few examples of where automation was used include:

  • Deployment of optical wavelengths on our backbone
  • Deployment of routers and base configurations, and service provisioning
  • Customer migration configurations from old network to the new equipment automatically generated from ESnet Database (ESDB)
  • Virtualized test environment was developed to test out new tools and services before actual in-field deployment.

This year, we prepare to bring the official DOE 413.3 ESnet6 project to a close, but as you know the network never sleeps, data never stops growing, and we have to constantly evolve the network. I can proudly say that we have the core foundations of the enduring ESnet user facility ready to handle the next big challenges of Data, AI, and Integrated multi-facility research that the scientists and National Labs are actively pursuing.

Wishing you all a very Happy New Year from ESnet. 

Inder

This post is part of a series of posts reflecting on the road to ESnet6. Check back soon to see upcoming posts from Inder focusing on innovation, co-design, and his vision for ESnet6 and beyond.

Light Bytes – a new name for the ESnet Blog

~ Inder Monga

You may have noticed that the masthead has changed.  After almost a decade, we have finally decided that the “ESnet Blog” deserves a less literal name. “Light Bytes” was selected to better embody two things about which we at ESnet are especially proud.  

First, getting to build the world’s greatest research and education network, and to support global science is a great honor and a technical challenge.  Through ESnet6 and our continuing research, we are advancing our mission of making scientific data free of geographical constraints, to “make bytes light” in terms of fast transport, and to deploy state of the art optical network for “bytes being transported by light.”

Second, while ESnet is officially a DOE User Facility, it is most importantly a remarkable group of people. We hope that this website will show a bit about the great people who make our mission happen and the interesting problems we get to work on.  In that sense, “Light Bytes” is a small written offering, a collection of features about things that are happening while the cause of “networking for science” and “science of networking” progresses. 

Looking back at ESnet’s 2020

~ Inder Monga

Advancing our strategy and shaping our position on the board.
Some thoughts from Inder on the year-that-was.

Miniature from Alfonso X’s Libro del axedrez dados et tablas (Book of chess, dices and tables), c. 1283. , Public domain, via Wikimedia Commons

Dear Friends, Well-wishers, Colleagues, and all of ESnet,

Chess! 2020 has been much more challenging than this game. It’s also been a year where we communicated through the squares on our zoom screens, filled with faces of our colleagues, collaborators, and loved ones.

In January, Research and Education leaders came together in Hawaii at the Pacific Telecommunications Council meeting to discuss the future of networking across the oceans. It was impossible to imagine then that we would not be able to see each other again for such a long time. Though thanks to those underwater cables, we have been able to communicate seamlessly across the globe.

Looking back at 2020, we not only established a solid midgame position on our ESnet chessboard, but succeeded in ‘winning positions’ despite the profound challenges. The ESnet team successfully moved our network operations to be fully remote (and 24/7) and accomplished several strategic priorities. 

ESnet played some really interesting gambits this year: 

  1. Tackled COVID-related network growth and teleworking issues for the DOE complex
    • We saw a 4x spike in remote traffic and worked closely across several Labs to upgrade their connectivity. We continue to address the ever-growing demand in a timely manner. 

    • As we all shifted to telework from home, ESnet engineers developed an impromptu guide that was valuable to troubleshoot our home connectivity issues. 
  2. Progressed greatly on implementing our next-generation network, ESnet6
    • We deployed and transitioned to the ESnet6 optical backbone network, with 300 new site installations, 100’s of 100G waves provisioned, with just six months of effort, and while following pandemic safety constraints. I am grateful to our partners Infinera (Carahsoft) and Lumen for working with our engineers to make this happen. Check out below how we decommissioned the ESnet5 optical network and lit up the ESnet6 network.
    • Installed a brand new management network and security infrastructure upgrades along with significant performance improvements.
    • We awarded the new ESnet6 router RFP (Congratulations Nokia and IMPRES!); the installs start soon.
    • Issued another RFP for optical transponders, and will announce the winner shortly.
  3. Took initiative on several science collaborations to address current and future networking needs
    • We brainstormed new approaches with the Rubin Observatory project team, Amlight, DOE and NSF program managers to meet the performance and security goals for traffic originating in Chile. We moved across several countries in South America before reaching the continental U.S. in Florida (Amlight), and eventually the U.S. Data Facility at SLAC via ESnet.
    • Drew insights through deep engagement of ESnet engineers with the High Energy Physics program physicists, for serving the data needs of their current and planned experiments expediently.
      Due to the pandemic, a two-day immersive in-person meeting turned into a multi-week series of Zoom meetings, breakouts, and discussions.
    • When an instrument produces tons of data, how do you build the data pipeline reliably? ESnet engineers took on this challenge, and worked closely with the GRETA team to define and develop the networking architecture and data movement design for this instrument. This contributed to a successful CD 2/3 review of the project—a challenging enough milestone during normal times, and particularly tough when done remotely. 
    • Exciting opening positions were created with EMSL, FRIB, DUNE/SURF, LCLS-II…these games are still in progress, more will be shared soon. 
  4. Innovated to build a strong technology portfolio with a series of inspired moves
    • AI/ML
      • We demonstrated Netpredict, a tool using deep learning models and real-time traffic statistics to predict when and where the network will be congested. Mariam’s web page showcases some of the other exciting investigations in progress. 
      • Richard and his collaborators published Real-time flow classification by applying AI/ML to detailed network telemetry.
    • High-touch ESnet6 project
      • Ever dream of having the ability to look at every packet, a “packetscope”, at your fingertips? An ability to create new ways to troubleshoot, performance engineer, and gain application insights? We demonstrated a working prototype of that vision at the SC20 XNET workshop. 
    • SENSE
      • We deployed a beta version of software that provides science applications the ability to orchestrate large data flows across administrative domains securely. What started as a small research project five years ago (Thanks ASCR!) is now part of the AutoGOLE project initiative in addition to being used for Exascale Computing Project (ECP) project, ExaFEL.
    • TCP
      • Initiated the Q-Factor project this year, a research collaboration with Amlight, funded by NSF. The project will enable ultra-high-speed data transfer optimization by TCP parameter tuning through the use of programmable dataplane telemetry: https://q-factor.io/
      • We testbed thoroughly the interactions between TCP congestion control algorithms, BBRv2 and CUBIC. A detailed conversation with Google, the authors of the BBRv2 implementation, is in progress.
  5. Initiated strategic new games, with a high potential for impact
    • FABRIC/FAB
      • Executed on the vision and design of a nationwide @scale research testbed working alongside a superstar multi-university team.
      • With the new FAB grant, FABRIC went international with plans to put nodes in Bristol, Amsterdam, Tokyo and Geneva. More locations and partners are possibilities for the future.  
    • Edge Computing
      • Created an prototype FPGA-based edge-computing platform for data-intensive science instruments in collaboration with the Computational Research Division and Xilinx. Look for exciting news on the blog as we complete the prototype deployment of this platform.
    • Quantum
    • 5G
      • What are the benefits of widespread deployment of 5G technology on science research? We contributed to the development of this important vision at a DOE workshop. New and exciting pilots are emerging that will change the game on how science is conducted. Stay tuned. 

Growth certainly has its challenges. But, as we grew, we evolved from our old game into an adept new playing style. I am thankful for the trust that all of you placed in ESnet leadership, vital for our numerous, parallel successes. Our 2020 reminds me of the scene in Queen’s Gambit where the young Beth Harmon played all the members of a high-school chess team at the same time. 

Several achievements could not make it to this blog, but are important pieces on the ESnet chess board. They required immense support from all parts of ESnet, CS Area staff, Lab procurement, Finance, HR, IT, Facilities, and Communications partners.

I am especially grateful to the DOE Office of Science, Advanced Scientific Computing Research leadership, NSF, and our program manager Ben Brown, whose unwavering support has enabled us to adapt and execute swiftly despite blockades. 

All this has only been possible due to the creativity, resolve, and resilience of ESnet staff — I am truly proud of each one of you. I am appreciative of the new hires that trusted their careers with us and joined us remotely—without shaking hands or even stepping foot at the lab.

My wish is for all to stay safe this holiday season, celebrate your successes, and enjoy that extra time with your immediate family. In 2021, I look forward to killer moves on the ESnet chessboard, while humanity checkmates the virus. 

Signing off for the year, 

Inder Monga

Faucet – Enterprise SDN from an office to multi-terabit SCinet at SC18

~ Inder Monga

This post is written by ESnet Director, Inder Monga, and shared through a personal lens, other perspectives may vary

I wanted to share a remarkable effort happening in the field of networking that is going to be shown in production at the Supercomputing 2018 as part of SCinet. Most of the people working in the SC booths (or even at SCinet) may never realize the role they are playing in moving the field of networking to the next level. Let me share some history [and you can skip the Background section to go straight to the main essence of the blog], and talk about the international, collaborative effort at play here.

Background: my software-driven journey

This is being shared by the perspective of the writer’s journey, other perspectives may vary

From the early 2000’s, the dream that excited me was building software layer that would manage networks and expose the network black-box as a set of ‘knobs and dials’ that applications could use to request and customize for their purposes. A small group of us at Nortel Networks came up with an idea of CO2 aka Content over Optics in 2001 – see the slide below.

This excitement was very new and led to my most productive set of patents filed during that time. Just as the concept was taking hold, an innovative research and education network, SURFnet, decided to deploy this concept on their newly built network, and the name evolved from CO2 to DRAC aka Dynamic Resource Allocation Controller.

Independently, ESnet was on a similar journey and started working on a concept called OSCARS aka ‘On-demand Secure Circuits and Reservation System’. As the community coalesced on the same direction, multiple initiatives like OSCARS, UCLP, DRAC, MANTICORE and others started merging towards an aligned software-driven network journey for the WAN. In the meantime, the Openflow effort from Stanford driven by an enterprise-based architecture, Ethane, captured the hearts of many, and the ‘Software-Defined Network’ (SDN) wave was born.


I am going to skip discussing the twists and turns of the evolution of SDN in industry, where there have been many recent articles on whether OpenFlow is dead. In fact there is a grassroots and under-the-radar SDN/Openflow effort that started in New Zealand called Faucet that proves these concepts are still alive and well, and are poised to thrive! In fact this work recently won the award in the New Zealand Open Source Software Project Award.

IMG_5690

Faucet at ESnet

Ever since I was invited to join the Faucet Foundation Board, I was insistent that I ‘eat my own dog food’. With the help of Josh Bailey and Simeon Miteff, I began running a faucet controlled switch (OpenFlow is not dead) in my office as my only bridge into the Lab’s network and the Internet, i.e. if faucet did not work, I could not get connected! I am proud to say that other than one small self-inflicted upgrading glitch, the system has been working seamlessly over the past year no matter what I do or use in the office. You can see the couple of raspberry PIs running the SDN controller working with a commercial off-the-shelf switch in the image below.

[I wish it was cleaner picture, but I had to get everything together to click a photograph without my desk in the way.]

Then my colleague, Nick Buraglio, and I decided, why not wire our satellite branch office with faucet based networking than buy traditional switches? With Josh (Bailey)’s help, Nick has now wired the lives of eight of ESnet’s key employees to be dependent on faucet and he has been having a blast (no downtime there either)! Read more about his journey on his own private blog: http://www.forwardingplane.net/2018/11/faucet-enterprise-openflow-in-production/

Faucet at SCinet, SC18

I may have led you on and will keep the suspense on for a bit longer….Faucet is being showcased in SCinet as production SDN controller managing a portion of the booth networks. SC18 is in two weeks and I encourage all of you to stop by SCinet to see it working! Reach out to the faucet champions – Josh Bailey, Brad Cowie, Richard Nelson, Nick Buraglio and Kate Mace who can talk you through the intricacies of the faucet deployment there. Poseidon, as you see printed on one of the raspberry pi’s in the image above, is also being featured at SC as it integrates learning and security with faucet.

More details on this topic will be available after SC is over. In the meantime, do check out Nick’s blog for the engineering details. An image from the recent multi-vendor plugfest at SCinet staging in Dallas to – yes, vendors support this.

Look forward to seeing you all in Dallas as all of us continue on this journey

Inder

ESnet’s Monga Keynotes Two R&E Network Workshops

~ Inder Monga
I’ll be busy in Brazil next week. Sharing the newsletter article about it with the Blog readers:

ESnet’s Monga Keynotes Two R&E Network Workshops

ESnet’s Chief Technology Officer Inder Monga will keynote two workshops and participate in a panel focusing on research and education (R&E) networks in Brazil next week.

 On May 18, Monga opens the National Research and Education Network Workshop (WRNP) hosted by ESnet’s Brazilian counterpart. RNP logo imagesIn his talk, entitled “R&E Networks: Imagining the next generation,” Monga will focus on new ideas in R&E networks, from technologies like software defined networking (SDN) and named data networking (NDN) to collaborative architectures to build an internet of different capabilities for global science collaborations. He will also showcase the challenges R&E networks face and focus on enabling end-to-end architectures, including concepts like the Science DMZ.

On May 22, Monga opens the Experimental Research Workshop of the Future Internet (WPIEF). His keynote is entitled “Moving from SDN demo to operations: Challenges.”

Both workshops are held as part of the Brazilian Symposium on Computer Networks  and Distributed Systems (SBRC) conference where Monga will contribute to a May 19 panel discussing “Challenges in the Development of Network Infrastructure, Testbeds for Software Defined Networks.”

Vote for ESnet5 – a good steward of Petabytes

~ Inder Monga

It has almost been a year since we turned 25, and transferred a “whole universe of data” at Supercomputing 2011 – and that was over a single 100G link between NERSC and Seattle. Now we are close to the end of building out the fifth generation of our network, ESnet5.

In order to minimize the downtime for the sites, we are building ESnet5 parallel to ESnet4, with just a configuration-driven switch of traffic from one network to the other. Since the scientific community we serve depends on the network to be up, it’s important to have assurance that the transition is not disruptive in anyway. The question we have heard over and over again from some of our users – when you switch the ESnet4 production traffic to ESnet5, how confident are you that the whole network will work, and not collapse?

In this blog post, I’d like to introduce an innovative testing concept the ESnet network engineering team (with special kudos to Chris Tracy) developed and implemented to address this very problem.

The goal of our testing was to ensure that the entire set of backbone network ports would perform solidly at full 100 Gbps saturation with no packet loss, over a 24 hour period. However we had some limitations. With only one Ixia test-set with 100 GE cards at hand to generate and receive packets and not enough time to ship that equipment to every PoP and test each link, we had to create a test scenario that would generate confidence that all the deployed routers and optical hardware, optics, the fiber connections, and the underlying fiber would performing flawlessly in production.

This implied creating a scenario where the 100 Gbps traffic stream being generated by the Ixia would be carried bi-directionally over every router interface deployed in ESnet5, traverse it only once and cover the entire ESnet5 topology before being directed back to the test hardware. A creative traffic loop was created that traversed the entire footprint, and we called it the ‘Snake Test’. Even though the first possible solution was used to create the ‘snake’, I am wondering if this could be framed as a NP-hard theoretical computer science and optimization approach known as the traveling salesman problem for more complex topologies?

The diagram below illustrates the test topology:

So after sending around 1.2 petabytes of data in 24 hours, and accounting for surprise fiber maintenance events that caused the link to flap, the engineering team was happy to see a zero loss situation.

Here’s a sample portion of the data collected:

Automation is key – utility scripts had been built to do things like load/unload the config from the routers, poll the firewall counters (to check for loss ingress/egress at every interface), clear stats, parse the resulting log files and turn them into CSV (a snapshot you see in the picture) for analysis.

Phew! – the transition from ESnet4 to ESnet5 continues without a hitch. Watch out for the completion news, it may come quicker than you think…..

On behalf of the ESnet Engineering team

ESnet unveils new dashboard showcasing IPv6 status of its connected sites

~ Inder Monga

Image

Just last month our resident IPv6 expert, Mike Sinatra, discussed the Risks of not deploying IPv6 in the R&E Community. On World IPv6 Launch, ESnet is happy to unveil a simple dashboard that tracks the status of IPv6 deployment across its sites. This page is updated based on summary of tests performed by a v6 connected host within ESnet.

Do visit the page at – https://my.es.net/sites/ipv6

Our policy board member, Vint Cerf, also released a compelling video on why we need IPv6, a must-watch.

Happy World IPv6 Launch!

ECSEL leverages OpenFlow to demonstrate new network directions

~ Inder Monga

ESnet and its collaborators successfully completed three days of demonstrating its End-to-End Circuit Service at Layer 2 (ECSEL) software at the Open Networking Summit held at Stanford a couple of weeks ago. Our goal is to build “zero-configuration circuits” to help science applications seamlessly use networks for optimized end-to-end data transport. ECSEL, developed in collaboration with NEC, Indiana University, and the University of Delaware builds on some exciting new conceptual thinking in networking.

Wrangling Big Data 

To put ECSEL in context, the proliferating tide of scientific data flows – anticipated at 2 petabytes per second as planned large-scale experiments get in motion – is already challenging networks to be exponentially more efficient. Wide area networks have vastly increased bandwidth and enable flexible, distributed, scientific workflows that involve connecting multiple scientific labs to a supercomputing site, a university campus, or even a cloud data center.

Heavy network traffic to come

The increasing adoption of distributed, service-oriented computing means that resource and vendor independence for service delivery is a key priority for users. Users expect seamless end-to-end performance and want the ability to send data flows on demand, no matter how many domains and service providers are involved.  The hitch is that even though the Wide Area Network (WAN) can have turbocharged bandwidth, at these exponentially increasing rates of network traffic even a small blockage in the network can seriously impair the flow of data, trapping users in a situation resembling commute conditions on sluggish California freeways. These scientific data transport challenges that we and other R&E networks face are just a taste of what the commercial world will encounter with the increasing popularity of cloud computing and service-driven cloud storage.

Abstracting a solution

One of the key feedback from application developers, scientists and end-users is that they do not want to deal with the complexity at the infrastructure level while still accomplishing their mission. At ESnet, we are exploring various ways to make networks work better for users. A couple of concepts could be game-changers, according to Open Network Summit conference presenter and Berkeley professor Scott Shenker: 1) using abstraction to manage network complexity, and 2) extracting and exposing simplicity out of the network. Shenker himself cites Barbara Liskov’s Turing Lecture as inspiration.

ECSEL is leveraging OSCARS and OpenFlow within the Software Defined Networking (SDN) paradigm to elegantly prevent end-to-end network traffic jams.  OpenFlow is an open standard to allow application-driven manipulation of network flows. ECSEL is using OSCARS-controlled MPLS virtual circuits with OpenFlow to dynamically stitch together a seamless data plane delivering services over multi-domain constructs.  ECSEL also provides an additional level of simplicity to the application, as it can discover host-network interconnection points as necessary, removing the requirement of applications being “statically configured” with their network end-point connections. It also enables stitching of the paths end-to-end, while allowing each administrative entity to set and enforce its own policies. ECSEL can be easily enhanced to enable users to verify end-to-end performance, and dynamically select application-specific protocol forwarding rules in each domain.

The OpenFlow capabilities, whether it be in an enterprise/campus or within the data center, were demonstrated with the help of NEC’s ProgrammableFlow Switch (PFS) and ProgrammableFlow Controller (PFC). We leveraged a special interface developed by them to program a virtual path from ingress to egress of the OpenFlow domain. ECSEL accessed this special interface programmatically when executing the end-to-end path stitching workflow.

Our anticipated next step is to develop ECSEL as an end-to-end service by making it an integral part of a scientific workflow. The ECSEL software will essentially act as an abstraction layer, where the host (or virtual machine) doesn’t need to know how it is connected to the network–the software layer does all the work for it, mapping out the optimum topologies to direct data flow and make the magic happen. To implement this, ECSEL is leveraging the modular architecture and code of the new release of OSCARS 0.6.  Developing this demonstration yielded sufficient proof that well-architected and modular software with simple APIs, like OSCARS 0.6, can speed up the development of new network services, which in turn validates the value-proposition of SDN. But we are not the only ones who think that ECSEL virtual circuits show promise as a platform for spurring further innovation. Vendors such as Brocade and Juniper, as well as other network providers attending the demo were enthusiastic about the potential of ECSEL.

But we are just getting started. We will reprise the ECSEL demo at SC11 in Seattle, this time with a GridFTP application using Remote Direct Memory Access (RDMA) which has been modified to include the XSP (eXtensible Session Protocol) that acts as a signaling mechanism enabling the application to become “network aware.”  XSP, conceived and developed by Martin Swany and Ezra Kissel of Indiana University and University of Delaware,  can directly interact with advanced network services like OSCARS – making the creation of virtual circuits transparent to the end user. In addition, once the application is network aware, it can then make more efficient use of scalable transport mechanisms like RDMA for very large data transfers over high capacity connections.

We look forward to seeing you there and exchanging ideas. Until Seattle, any questions or proposals on working together on this or other solutions to the “Big Data Problem,” don’t hesitate to contact me.

–Inder Monga

imonga@es.net

ECSEL Collaborators:

Eric Pouyoul, Vertika Singh (summer intern), Brian Tierney: ESnet

Samrat Ganguly, Munehiro Ikeda: NEC

Martin Swany, Ahmed Hassany: Indiana University

Ezra Kissel: University of Delaware

Idea Power: Two ESnet Projects are Honored With Internet2 IDEA Awards

~ Inder Monga

We are proud to announce that two of ESnet’s projects have received IDEA (Internet2 Driving Exemplary Applications) awards in Internet2’s 2011 annual competition for innovative network applications that have had the most positive impact and potential for adoption within the research and education community. (see: Internet2’s press release).

Internet2 recognized OSCARS (On-Demand Secure Circuits and Advance Reservation System), developed by the ESnet team led by Chin Guok, including Evangelos Chaniotakis, Andrew Lake, Eric Pouyoul and Mary Thompson. Contributing partners also included Internet2, USC ISI and DANTE.

ESnet’s MAVEN (Monitoring and Visualization of Energy consumed by Networks) proof of concept application was also recognized with an IDEA award in the student category. MAVEN was prototyped by Baris Aksanli during his summer internship at ESnet. Baris is a Ph.D student at the University of California, San Diego conducting research at the System Energy Efficiency Lab with his thesis advisor, Dr. Tajana Rosing. Baris worked closely with his summer advisor, Inder Monga, and Jon Dugan to implement MAVEN as part of ESnet’s new Green Networking Initiative.

The idea behind OSCARS

OSCARS enables researchers to automatically schedule and guarantee end-to-end delivery of scientific data across networks and continents. For scientists, being able to count on reliable data delivery is critical as scientific collaborations become more expansive, often global. Meanwhile, in disciplines ranging from high-energy physics to climate, scientists are using powerful, geographically dispersed instruments like the Large Hadron Collider that are producing increasingly massive bursts of data, challenging the capabilities of traditional IP networks.

OSCARS virtual circuits can reliably schedule time-sensitive data flows – like those from the LHC – round the clock across networks, enabling research and education networks to seamlessly meet user needs. OSCARS code is also being deployed by R&E networks worldwide to support an ever-growing user base of researchers with data-intensive collaboration needs. Internet2, U.S. LHCnet, NORDUNet, RNP in Brazil as well as over 10 other regional and national networks have currently implemented OSCARS for virtual circuit services. Moreover, Internet2’s NSF-funded DyGIR and DYNES projects will in 2012 deploy over 60 more instances of OSCARS at university campuses and regional networks to support scientists involved in LHC, Laser Interferometer Gravitational-Wave Observatory (LIGO), Large Synoptic Survey Telescope (LSST) and Electronic Very-Long Baseline Interferometry (eVLBI) programs.

We are proud of the hard work and dedication the OSCARS development team has demonstrated since the start of this project. Just as importantly we are proud to see this work paying off in with new science collaboration and discoveries.

The potential of MAVEN

The Monitoring and Visualization of Energy consumed by Networks (MAVEN) project is a brand new prototype portal that will help network operators and researchers better track live network energy consumption and environmental conditions. MAVEN – implemented by Baris during his summer internship – is a first major step for ESnet in instrumenting our network with the tools to understand these operational dynamics. As networks continue to get bigger and faster, they will require more power and cooling in an era of decreased energy resources. To address this pressing challenge, ESnet is leading a new generation of research aimed at understanding how networks can operate in a more energy-efficient manner. We are grateful for Baris’ significant contributions in leading the development of MAVEN and glad to see that his talent is being recognized by the R&E networking community through this award.

Baris is now back in school at UCSD, completing his Ph.D in computer science. http://cseweb.ucsd.edu/~baksanli/. Congratulations Baris!