ESnet publishes design guide for high-performance data movers

As science becomes more and more data-intensive, demand for the capability of moving large data sets between sites over high-performance networks keeps increasing.  Now ESnet engineers Eric Pouyoul and Roberto Morelli have designed a powerful, yet inexpensive data transfer host that can function as a test server for network troubleshooting or as a data mover for scientific applications.

Assemble your own machine for that D.I.Y. glow of accomplishment

“We have right now a very fast network, but some users have difficulty realizing its full potential,” said Pouyoul. “We are in the process of increasing network bandwidth from 10Gbps to 100Gbps. But with computers, any time you multiply speed by a factor of 10, something is going to break. We also need to provide enough compute power to simulate the large data transfers that occur in normal scientific research, so we are locating boxes at different places on the ESnet network.”

So far, ESnet has built three test hosts based on the design, and deployed them at different points in the network so that users can test their own installations using the ESnet hosts as a reference. “This way, ESnet and the community can experiment with data transfers over different parts of the network.” Pouyoul says.  The test hosts are available to resources located on networks devoted to scientific research.

Pouyoul and Morelli’s box is powerful, yet affordable.  Anybody can build one using off-the-shelf components. Pouyoul points out that while the I/O speed of their creation is comparable to a half million-dollar machine, the footprint is much smaller. And parts, excluding labor, cost only around 10 grand. “We wanted to make it cheap and easy for do-it-yourself deployments.” Pouyoul said. “Our next step is to publish documentation to encourage people to build and install one on their own network. This is close to a production level machine for the R&D community.” Pouyoul has overseen the building and deployment of several machines by students.  You can find more about Pouyoul and Morelli’s innovation at http://fasterdata.es.net/dtn.html. Instructions on how to run your own tests on our three public test hosts is at: http://fasterdata.es.net/disk_pt.html