Berkeley Lab and ESnet Document Flow, Performance of 56 Terabytes Climate Data Transfer

Visualization by Prabhat (Berkeley Lab).
The simulated storms seen in this visualization are generated from the finite volume version of NCAR’s Community Atmosphere Model. Visualization by Prabhat (Berkeley Lab).

In a recent paper entitled “An Assessment of Data Transfer Performance for Large‐Scale Climate Data Analysis and Recommendations for the Data Infrastructure for CMIP6,” experts from Lawrence Berkeley National Laboratory (Berkeley Lab) and ESnet (the Energy Sciences Network, document the data transfer workflow, data performance, and other aspects of transferring approximately 56 terabytes of climate model output data for further analysis.

The data, required for tracking and characterizing extratropical storms, needed to be moved from the distributed Coupled Model Intercomparison Project (CMIP5) archive to the National Energy Research Supercomputing Center (NERSC) at Berkeley Lab.

The authors found that there is significant room for improvement in the data transfer capabilities currently in place for CMIP5, both in terms of workflow mechanics and in data transfer performance. In particular, the paper notes that performance improvements of at least an order of magnitude are within technical reach using current best practices.

To illustrate this, the authors used Globus to transfer the same raw data set between NERSC and Argonne Leadership Computing Facility (ALCF) at Argonne National Lab.

Read the Globus story:
Read the paper:

Submissions Now Open for Workshop on Improving Data Management for Climate Science

ESnet and Internet2 are now accepting submissions for the Focused Technical Workshop on Improving Data Mobility & Management for International Climate Science. The workshop, hosted by the National Oceanic and Atmospheric Administration (NOAA), will be held July 14-16, 2014, in Boulder, Colo.

The program committee is seeking presentation proposals for these focus areas:

  • Technology Best Practices
  • Scientific Drivers
  • Facility Overviews

If you are interested in submitting a proposal, please send an email to that includes the title, length of talk, abstract, name of speaker and contact information (title, organization, and email for any speakers other than the submitter).

Submissions are due by Wednesday, April 30. Accepted speakers will be notified by May 15.

ESnet and Internet2 – in collaboration with partners Indiana University (IU), the National Center for Atmospheric Research (NCAR), and NOAA – are hosting the workshop to bring together network, software and data management experts with scientists from the international climate community. The goal is to initiate a dialogue among climate scientists, climate data manager, and cyberinfrastructure engineers with the aim of building longer-term relationships. The workshop also seeks to provide climate researchers with immediately useful tools and resources for improved data transport and management.

The workshop will include a slate of invited speakers and panelists in a format designed to encourage lively, interactive discussions with the goal of developing a set of tangible next steps for supporting this data-intensive science community. The workshop intends to include breakout sessions as well as many opportunities for professional networking.

Find more information at:


International Climate Community Kicks-Off a Year of Networking!

Starting this January, the Earth System Grid Federation (ESGF) has started a new working group—the International Climate Network Working Group—to help set up and optimize network infrastructures for their climate data sites around the world. They need network connections that can deal with petabytes of modeling and observational data, which will traverse more than 13,000 miles of networks (more than half the circumference of the Earth!), spanning two oceans.

By the end of 2014, this working group will aim to obtain at least 4Gbps of data transfer throughput at five of their climate data centers at PCMDI/LLNL (US), NCI/ANU (AU), CEDA/SFTC (UK), DKRZ (DE), and KNMI (NE). This goal runs in parallel with the Enlighten Your Research Global international networking program award that ESGF received this last November 2013. This initiative is lead by Dean Williams of Lawrence Livermore National Lab and ESnet’s Science Engagement Team, along with collaborating international network organizations in Australia (AARnet), Germany (DFN), the Netherlands (SURFnet), and the UK (Janet). We are helping to shepherd ESGF’s project and working group to make sure all their climate sites get up and running at proficient network speeds for the future peta-scale climate data that is expected within the next 5 years.

As we work closely with ESGF to pave the way for climate science, we look forward to developing a new set of networking best practices to help future climate science collaborations. In all, we are excited to get this started and see their science move forward!

The Enlighten Your Research Global program will set up, optimize and/or troubleshoot 5 ESGF locations in different countries throughout 2014.
The Enlighten Your Research Global program will set up, optimize and/or troubleshoot 5 ESGF locations in different countries throughout 2014.