ESnet, the Department of Energy’s high-bandwidth network, is working with Caltech and DOE’s National Energy Research Scientific Computing Center (NERSC) to move large data files at an average 90 gigabits per second (Gbps) to the SC13 conference in Denver.

ESnet’s Eric Pouyoul, who set up the data transfers, said that the transfer rate is both more impressive and more realistic as it is being transferred from disc to memory.  “That’s reality – moving files from memory to memory is faster, but it’s not real life,” Pouyoul said.
To set up the demo, Pouyoul created 24 files, each containing 130 gigabytes of data and stored them at NERSC in Oakland, Calif. He configured them to transfer in loops over ESnet’s link to Denver and into the convention center – and over 15 hours achieved an average of 90 Gbps. The demo used the FDT (Fast Data Transfer) tool forr transferring the files, with support from the FDT team.ESnet is supporting four 100 Gbps links into SC13 at the Colorado Convention Center in Denver.

Working in the Caltech booth in the SC13 Exhibition, network engineer Azher Mughal said the demonstration was created to help people understand how large datasets are becoming and how to efficiently transfer the data. Caltech manages the trans-Atlantic link that brings data from the Large Hadron Collider in Switzerland to the United States.Caltech-portal

“We want to showcase how you can efficiently manage large data sets, the challenges you’re likely to see, and how to optimize the transfer,” Mughal said. By sending the data across a single 100 Gbps line rather than multiple 10 Gbps links, there are fewer transfer problems and  it requires fewer CPUs on the computer systems, making them more efficient in the process.