The Open Science Grid

This week’s Harvard Medical School's Structural Biology Grid (SBGrid) group is hosting the  Open Science Grid annual all hands meeting.  Think of the Open Science Grid as a way to harness the unused computing cycles of high performance computing centers for the benefit of all - a kind of SETI at home  for science. It's worth learning more about.

The Open Science Grid (OSG) is an open consortium of science and research communities, university and Department of Energy (DOE) laboratory IT facilities,  and software developers. Together they have built and are now operating a broad distributed high throughput computing cyber infrastructure in the US. Staff are funded by the DOE SciDAC-2 program and NSF.

The OSG is designed to be effective for job runs of between one hour and a few of days, jobs that can be check-pointed, jobs that require management of large scale data movement and storage, and ensembles of jobs that can effectively run across a large number of resources.

On a typical day, the OSG supports 1.2 million CPU hours of computation, the launch of more than half a million jobs, and the transfer of more than a quarter of a petabyte of data, across more than sixty sites. Over the past year the Harvard Medical School SBGrid group has used more than 6 million CPU hours.

20% of the usage of OSG Is non-physics – across more than five different scientific disciplines. The Large Hadron Collider (LHC)  uses 50% and the existing Tevatron experiments at Fermilab, Laser Interferometer Gravitational Wave Observatory (LIGO), STAR, and other physics experiments use the final 30%.

The Open Science Grid provides the US contribution to the World Wide LHC Computing Grid – most recently presented by Bob Jones.

OSG provides an engagement effort that helps new users, ranging from individuals and small groups to large communities.  It engages teams to help them adapt their software (and their culture) to use a distributed set of computing and storage resources which they don’t manage directly themselves.

The OSG is built on a set of underlying principles of distributed high throughput computing. Professor Miron Livny, the lead of the Condor project at the University of Wisconsin-Madison, serves as Technical Director.

OSG partners with the NSF TeraGrid and new XD program. It has a long and productive history of collaboration with peer European projects, continuing with the European Grid Initiative (EGI-InSPIRE) and European Middleware Initiative (EMI). It works closely with ESNet and Internet2 in understanding the networking needs of high performance computing communities, testing and integration of advanced networks.

The OSG brings distributed high throughput computing services into and across campuses themselves by working with groups of faculty and researchers to leverage local and remote resources. OSG is currently working with communities at Clemson, Nebraska, Notre Dame, Purdue and Wisconsin-Madison on a prototyping effort that includes enabling the formation of local partnerships and dynamic access to shared resources using campus identities.

Here's an architecture diagram that shows the scope of OSG.  The image above illustrates the usage over the past few years.

OSG is a very worthwhile application of technology - a grid computing initiative that captures millions of unused CPU hours for general use.