The ESnet tools team is pleased to announce that they are launching a brand new interface to http://my.es.net that will showcase real time statistics on the 100 Gbps demos running at SC11.
This interface is designed to provide the community with access to multiple live visualization tools of 100 Gbps network utilization and topology. The tools team continues to build out the MyESnet portal to meet the evolving needs of the community—we will keep you posted on new developments.
ESnet is supporting eight 100 Gbps community projects at SC11:
· Brookhaven National Laboratory with Stony Brook University: Booth 2443
· Indiana University with collaborators Brocade, Ciena, Data Direct Networks, IBM, Whamcloud, and ZIH: Booth 2239
· Lawrence Berkeley National Laboratory, Booth 512
· NASA with collaborators from MAX, International Center for Advanced Research (iCAIR) at Northwestern University, Laboratory for Advanced Computing at University of Chicago, Open Cloud Consortium, Ciena, Alcatel, StarLight, MREN, Fujitsu, Brocade, Force 10, Juniper, Arista: Booths 615, 2615, 635
· Orange Silicon Valley with the InfiniBand Trade Association and OpenFabrics Alliance: Booth 6010
· The California Institute of Technology: Booth 1223
· Fermi National Accelerator Laboratory and University of California, San Diego: Booth 1203 and 1213.
We have been pretty quiet about the MyESnet Portal lately. That’s because we have been busy working on some exciting new features! We are happy to announce that we have rolled out a new version of the portal. We’ll start out discussing the new features and then go into the development road map.
New, improved widgets
New Features
The graphing widget used on the site summary, flow and site interface pages has been significantly improved:
The user can zoom in on a region of any graph by clicking and sweeping out a time range on the graph;
All controls have been consolidated on the right hand side of the graph;
The time range of the displayed data is shown in the controls section on the right;
The tracker is updated for all visible graphs;
All visualizations are now done using the d3.js toolkit which provides greater interactivity. You can see many interesting examples of d3.js based visualizations here.
Other improvements include:
The details of an outage trouble ticket are accessible in the “site outage” and “availability” displays by clicking on the ticket number
Duplicate site peering entries were removed (although the speed is still wrong in some cases)
Development Road Map
We have discussed our plans with many people both inside and outside of ESnet and have produced a list of features and an approximate timeline for the completion of those features. This timeline is by no means set in stone – we will most likely rearrange things as priorities and goals are updated. We would love to hear your comments on this plan, both in terms of timing and features. Are there features you would like to see that aren’t listed? Do you think something should be given higher priority? Let us know.
Data Improvements (29 Sep 2011)
General improvements to the quality and freshness of the data presented in the portal. The MyESnet Portal gathers information from a wide variety of data sources inside ESnet. When one data source has a different idea of how things work than another data source, we will be working to reconcile these differences and address bugs we have found. In a few cases data is imported into the portal manually and will be automated.
Topology Visualization (12 Oct 2011)
Build a clean, simple, data driven widget to visualize topology information. We will not dive into solving the difficult problem of automatic layout of large complex topologies. Instead the widget will focus on a generalization of the subway diagram to allow a two dimensional grid of nodes where the relative position of each node is specified. This visualization widget will be used to create several visualizations:
Advanced Networking Initiative (ANI) 100G Prototype Network Visualization
This will use the topology visualization widget to show the details of the ANI network including current buildout status and the full network. A stretch goal is to include some support for mobile devices in this visualization.
SC11 Demo Visualizations
A series of visualizations showing the extent of all the SC11 demos that ESnet is supporting.
OSCARS Details
We’ll use the visualization widget to display details about any given OSCARS circuit. This will allow visibility into the details of OSCARS circuit for each ESnet site.
Power Stats and Visualizations (22 Oct 2011)
The ANI network is being instrumented to measure power consumption. This will
use the existing graph widget to show power data for ANI. We will also look to integrate some portion of the visualizations developed by our summer Ph.D student through our Monitoring and Visualization of Energy consumed by Networks (MAVEN) project.
Ideation Tool (04 Nov 2011)
This tool is intended to be a simple system to collect and develop ideas for ESnet as a whole. People can share ideas for the portal and eventually all ESnet services, and comment on them. They will also be able to “Like on Facebook”, “+1 Google Plus” and tweet about the specific ideas. More sophisticated social media integration will be explored but will not be a blocker to the deployment timeline.
Monthly Stats (03 Dec 2011)
ESnet collects statistics about the utilization of the network. Each month these statistics are analyzed to produce a report of the traffic volume the ESnet carried for that month. Currently these reports are emailed to specific users, but are not available on the web. This will provide an interface that will allow users to explore historical data and see projections of anticipated growth. This includes not only vanilla IP connections but OSCARS and optical connections as well.
Path of Interest (17 Dec 2011)
This tool will allow users to get statistics on a particular path of interest within ESnet. Users will be able to examine utilization and error rates which will be visualized using the topology widget. This will set the stage for visualizing paths that cross into other networks once perfSONAR integration is complete.
Shibboleth / OpenID (14 Jan 2012)
Allow users to login with identities provided by other organizations using Shibboleth or OpenID. This will allow, for example, users to login in using their credentials from their home organization.
Site Health (21 Jan 2012)
The site summary page has a minimal set of health related displays for ESnet sites. We will develop more in depth displays including details on open trouble tickets for the site, prefixes announced to and received from the site and secondary DNS delegations. If possible this will integrate with the new ServiceNow deployment.
Looking Glass (28 Jan 2012)
This tool allows users to run commands on our routers and see real time results. This includes running traceroute, ping and route lookups and will allow users to do some preliminary troubleshooting on their own.
perfSONAR Integration (11 Feb 2012)
perfSONAR is an open-source software package that helps users to quickly identify problems on any given path – and can do so across any set of networks that is perfSONAR-enabled. The portal will be extended to allow the Path of Interest widget to visualize perfSONAR’s network performance data in a more user-friendly display to help users better understand and troubleshoot network issues from end to end.
Peering Visualization (18 Feb 2012)
The topology widget will be extended to give users deeper insight into ESnet’s peering relationships. For each site there will be a visualization that shows how that site is connected to ESnet. There may also be visualizations that show how ESnet connects to various other networks.
Mobile Device Support (24 Feb 2012)
Currently the MyESnet portal can be used on many mobile devices. We will analyze the site and make improvements to the mobile experience where necessary.
The mission of ESnet is to serve the needs of our diverse user community. We work very hard to deliver the fastest, most reliable and best supported science network on the planet. However, our users have at times expressed that ESnet is a “black box” to them. We’ve tried to address this in the past by building a variety of user-facing tools, but up until now, those tools were not well integrated with each other.
Today that changes. Today we publicly launch the MyESnet portal.
What’s a portal?
The term portal or web portal is a widely used but often poorly defined term. It’s a bit of a buzzword like “web-scale” or “the cloud.” When we talk about a portal we have some very specific ideas about what it is. At its core the MyESnet portal is about two things: information and community.
The MyESnet portal provides a coherent interface to information about ESnet to our community.
ESnet collects a huge amount of data as part of operating and monitoring the network. This data is processed and aggregated in various ways to distill useful information. The MyESnet portal provides a cohesive view of that information organized through a single unified interface.
Who, then, is our community? The most narrow definition of our community would simply be the sites we are mandated to support for the DOE Office of Science. But in our view that is far too narrow a definition, as the ESnet community expands far beyond these sites and includes people such as the IT staff at peer organizations and major science facilities, the scientists who depend on ESnet to move their data, the DOE program managers, policy makers and even the general public.
What does the portal do today?
The portal currently implements the following features:
A high-level view of site status
Visualizations of the total traffic that enters and exits a given ESnet site with breakdowns by:
protocol (such as TCP or UDP)
application (such as SSH, HTTP or FTP)
source and destination autonomous system (such as Internet2, GEANT or Google)
source and destination country
source and destination ESnet site
Information on how each site connects to ESnet (both in terms of physical interfaces and BGP peerings)
Ranked “top talkers” list, with the final octet of the IP address obscured
The ESnet maintenance calendar for viewing past and upcoming outages
“Sites of Interest” which allows quick access to information about those sites
A matrix showing the availability for all ESnet sites for the last 12 months
The initial incarnation of the portal is aimed at ESnet site coordinators. However, that doesn’t mean it is only useful to site coordinators. The public can see nearly all the information listed above for each site and can learn a great deal about how ESnet serves that site.
The only item that will not be visible to the public is the very specific “top-talkers” reports as it may contain somewhat sensitive information. Everything else is available for the public to see. (Note that most of this data has been publicly available for some time, but the portal consolidates it in one place.)
Where is it going?
We also have some ambitious goals of what the MyESnet portal might become. We would like to see it expand the information it presents — we are already working on additional functionality. We are very interested to know what the community would like to see from ESnet. Our next task is to develop a tool to make it easy for the ESnet user community to suggest, discuss and rank suggestions for ESnet (including changes to MyESnet). Following that we will develop visualizations of topology information. These visualizations will include tools to show how sites connect to ESnet, what paths traffic takes from a site to user defined endpoints, and visualizations of OSCARS circuits and the overall topology of ESnet. These efforts will include integration with more internal data sources as well as external data sources via perfSONAR.
Giving our user community a view into the network is only part of the equation when it comes to enabling science to use networks effectively. We have also developed an extensive knowledge base about how to build, tune and troubleshoot networks to meet the needs of modern, data intensive science. Be sure to take a look at fasterdata.es.net for more information.
MyESnet is your own personal, customized view of ESnet. What else would you like to see?
We look forward to hearing what you think, and your suggestions and comments. Please send any and all feedback to the ESnet Tools Team at tools@es.net. If you would like to be notified when changes are made to the portal you can join the portal-announce@es.net email list by visiting https://listserv.es.net/mailman/listinfo/portal-announce.
By autumn most folks are thinking about the holidays, but for me, fall is filled with thoughts of something different. Since 1998, I’ve had the privilege of working with the SCinet committee of Supercomputing, a.k.a. the International Conference for High Performance Computing, Networking, Storage, and Analysis, if you are not into the whole brevity thing. SCinet is the team of people that builds the local area and wide area network for the Supercomputing conference.
This year’s conference, SC10 is in New Orleans, LA, from November 13 until November 19. Planning for each year’s show starts a few years ahead of time. Not long after one year’s show ends, the serious planning for the next year’s SCinet begins. It’s a cycle that I’ve been through many times now, and it’s a bit like an old friend at this point. Most of the time we enjoy each other’s company immensely but when things get stressful, we can really irritate each other.
SCinet is a pretty amazing network. After a year of planning, there are three weeks of concentrated effort to set it up. It’s operational for about one week and it takes about two days to tear down. This year we will have 270 Gbps of wide area network connectivity with dedicated circuits to ESnet, Internet2, NLR, and NOAA. We will deliver over 200 network connections to the various booths on the show floor.
As amazing as the network is, the people who build it are even more amazing. They are drawn from universities, national laboratories, network equipment vendors and nationwide research and education networks. It’s not just Americans; there are people from several different countries with strong showings from the Netherlands and Germany most years. Many of these folks are leaders in their areas of expertise and all of them are bright, capable people. Each of them has given up a fair bit of their own time to participate (while most have some sponsorship from their employers, it’s not unheard of for people to take vacation time to participate).
Why would people give up many evenings and weekends every fall to be a part of SCinet? Because it’s an amazing opportunity to learn about the state of the art in computer networking and to expand your professional network as welll. I consider myself extremely fortunate to work with each of the people that make up SCinet.
So what is ESnet doing for with SCinet this year? Glad you asked. First off, we are bringing three 10G circuits to the show floor. As of Friday, October 29th all three were up and operational. One of these circuits will be used for general IP traffic, but the other two will be used to carry dynamic circuits managed by the OSCARS IDC software.
These circuits will provide substantial support for various demonstrations by exhibitors, connecting scientific and computational resources at labs and universities to the booths on the show floor.
Finally, ESnet has four people who are volunteering in various capacities within SCinet. Evangelos Chaniotakis and myself will be working with the routing team. The routing team provides IP, IPv6, IP multicast service, manages the wide area peerings, manages wide area layer 2 circuits, configures the interfaces that face the booths on the show floor and works closely with several other teams to provide a highly scalable and reliable network. John Christman is working with the fiber team, building the optical fiber infrastructure to support SCinet (all booth connections are delivered over optical fiber, which allows booths to be connected to the network using the highest-speed interfaces available.) Brian Tierney will be working with the SCinet measurement team collecting network telemetry, and using it to provide useful and meaningful visualizations of what’s happening inside SCinet as well as providing tools and hosts to allow making active network measurement such as Iperf, nuttcp, and OWAMP. The measurement data is also made accessible using the perfSONAR suite of tools. They’re also using the SNMP polling software I wrote for ESnet called ESxSNMP.
Important spots to visit:
If you are coming to SC10 this year, be sure to come by the SCinet NOC in booth 3351. I’d be happy to meet anyone who’s read this; feel free to ask for me at the SCinet help desk at the same booth. LBNL (ESnet’s parent organization) is located in booth 2448. Finally, I am hosting a Bird’s of a Feather (BOF) session on network measurement during the show, the details are here.
And check out the other ESnet demos: You can download a map of ESnet at SC10: SC 2010_floormapFL
LBNL Booth 2448, ESnet roundtable discussions
Inder Monga, Advanced Network Technologies Group, ESnet, will lead a roundtable discussion on: On-demand Secure Circuits and Advance Reservation System (OSCARS), 1-2 p.m., Wednesday, Nov. 17
Many of the demos at SC10 are being carried by OSCARS virtual circuits developed by ESnet with DOE support. OSCARS enables networks to reserve and schedule virtual circuits that provide bandwidth and service guarantees to support large-scale science collaborations. In the first quarter of 2011, ESnet expects to unveil OSCARS 0.6, which will offer vastly expanded capabilities, such as a modular architecture allowing for easy plug and play of the various functional modules and a flexible path computation engine (PCE) workflow architecture. Adoption of OSCARS has been accelerating as 2010 has seen deployments at Internet2 and other domestic and international research and education networks. Since last year, ESnet saw a 30% increase in the use of virtual circuits. OSCARS virtual circuits now carry over 50% of ESnet’s monthly production traffic. Increased use of virtual circuits was a major factor enabling ESnet to easily handle a nearly 300% rise in traffic from June 2009 to May 2010.
Brian Tierney, Advanced Network Technologies Group, ESnet, will lead a roundtable discussion on: ARRA-funded Advanced Networking Initiative (ANI) Testbed, 2- 3 p.m. Wednesday, Nov. 17
The research and education community’s needs for managing and transferring data are exploding in scope and complexity. In 2009 the DOE Office of Science awarded ESnet $62 million in Recovery Act funds to create the Advanced Networking Initiative (ANI). This next-generation, 100 Gbps network will connect DOE’s largest unclassified supercomputers. ANI is also establishing a high performance, reconfigurable network testbed for researchers to experiment with advanced networking concepts and protocols. ESnet has now opened the testbed to researchers. A variety of experiments pushing the boundaries of current network technology are underway. Another round of proposals are in the offing. The testbed will be moving from Lawrence Berkeley National Laboratory to ESnet’s dark fiber ring at Long Island (LIMAN: Long Island Metropolitan Area Network) in January 2011 and eventually the 100 Gbps national prototype network ESnet is building to accelerate deployment of 100 Gbps technologies and provide a platform for the DOE experimental facilities at Oak Ridge National Laboratory and the Magellan resources at at the National Energy Research Scientific Computing Center (NERSC) and Argonne National Laboratory.
You must be logged in to post a comment.