network monitoring

The simulated storms seen in this visualization are generated from the finite volume version of NCAR’s Community Atmosphere Model. Visualization by Prabhat (Berkeley Lab).

In a recent paper entitled “An Assessment of Data Transfer Performance for Large‐Scale Climate Data Analysis and Recommendations for the Data Infrastructure for CMIP6,” experts from Lawrence Berkeley National Laboratory (Berkeley Lab) and ESnet (the Energy Sciences Network, document the data transfer workflow, data performance, and other aspects of transferring approximately 56 terabytes of climate model output data for further analysis.

The data, required for tracking and characterizing extratropical storms, needed to be moved from the distributed Coupled Model Intercomparison Project (CMIP5) archive to the National Energy Research Supercomputing Center (NERSC) at Berkeley Lab.

The authors found that there is significant room for improvement in the data transfer capabilities currently in place for CMIP5, both in terms of workflow mechanics and in data transfer performance. In particular, the paper notes that performance improvements of at least an order of magnitude are within technical reach using current best practices.

To illustrate this, the authors used Globus to transfer the same raw data set between NERSC and Argonne Leadership Computing Facility (ALCF) at Argonne National Lab.

Read the Globus story: https://www.globus.org/user-story-lbl-and-esnet
Read the paper: https://arxiv.org/abs/1709.09575

In the November issue of IEEE Communications, ESnet’s Brian Tierney and Jason Zurawski served as co-editors (with four others) of a special section on network measurement and monitoring. According to the authors, writing in a guest editorial, even though more and more services (including the cloud) are increasingly important for both the scientific and corporate worlds, network performance has not kept up. And the network problems are becoming more subtle and detrimental, while also being more challenging to troubleshoot across domains. Addressing the problem requires both sophisticated measurement and monitoring tools that are interoperable and also meet policy requirements.

To examine novel approaches and standardization efforts, the team of guest editors solicited research papers and received 18 submissions, of which eight were selected for publication in two issues of IEEE Communications, with four appearing in the November 2013 issue and four in May 2014. In addition to Tierney and Zurawski, the editors were Prasad Calyam, University of Missouri—Columbia; Constantine Dovrolis, Georgia Tech; Loki Joergenson, Lionsgate Technologies; and Raj Kettimuthu, Argonne National Laboratory and University of Chicago.

Access to the research papers is limited to IEEE members and subscribers.