StarCluster brings cloud computing to the classroom and beyond

By Justin Riley, Software Developer, Software Tools for Academics and Researchers, OEIT

In the “Introduction to Modeling and Simulation” course (3.021) at MIT students learn to use both atomistic and quantum mechanical computer modeling simulations to study materials properties. For this course students need to run many computationally intensive simulations in order to experiment and analyze various systems for their homework assignments. These simulations can take anywhere from 30 minutes to 6 hours or more and generate up to several gigabytes of data in each run. Needless to say, with approximately 50 students enrolled, this course has a serious need for computational resources in the classroom.

So how do you go about supporting such a course? Traditionally, a university would have to locate a set of servers on campus, either shared or dedicated, that could be used within the time-frame of the course. If a faculty member is lucky they might be allowed exclusive access to the servers. More than likely these servers would also be in use by another faculty member for another course or even a research lab. In the worst case either the university or faculty member would be faced with purchasing and housing a new set of dedicated servers. Regardless, these servers must be configured with all of the software needed for the course and maintained either by a faculty member, student, or staff member.

The long-term implications of using local resources to support the computational requirements of a particular course present a system administration nightmare. With shared resources it's hard to guarantee priority access for a given course which can cause a significant performance penalty for the students using the resources. Maintaining the course software and providing the proper environment can be a hassle from year to year given that software updates will sooner or later break the course software in some way. To top it off, an unrecoverable hardware failure could render students unable to complete their homework. Aside from all of these issues it's also possible that dedicated resources would go under-utilized when the course is not in session.

The Software Tools for Academics and Researchers (STAR) group at MIT has experienced all of these issues with local computational resources over the last 2 years while supporting the 3.021 and the 12.950 courses at MIT. We decided to try to improve this situation by utilizing cloud computing via Amazon's Elastic Compute Cloud (EC2) web service to support the course's computational resource requirements. To this end, the STAR group created StarCluster (http://web.mit.edu/starcluster), an open-source cloud computing toolbox for education and research.

StarCluster allows anyone with an Amazon EC2 account to fill in their account credentials and in five to ten minutes have a fully functioning computer cluster configured for distributed and parallel computing out-of-the-box on Amazon's data-center servers. With Amazon EC2's pay-by-the-hour pricing model, you only pay for what you use. Combining this pricing model with StarCluster's ability to fully automate the process of requesting and configuring the machines from Amazon to form a distributed computing environment presents a very scalable and on-demand solution for both the classroom and research.

In order to significantly lower the costs for MIT of using Amazon EC2 in the classroom the STAR group worked with Professor Markus Buehler and Dean Daniel Hastings to apply for two Amazon Teaching grants for supporting cloud-computing enabled courses at MIT. Professor Buehler and Dean Hastings acted as the principal investigators for these proposals which were funded by Amazon in July of 2009. The money received from these grants was used to support the 12.950 IAP 2010 course for the duration of IAP and is currently being used to support 3.021 for the entire Spring semester of 2010. The remaining funds will be used to support future courses and also for further development of StarCluster.

StarCluster is gaining popularity and we've counted over 500 downloads from the Python Package Index (PyPI), Freshmeat.net, and Softpedia.com. StarCluster's user community, centered around the StarCluster mailing list, is also growing with currently 23 registered members. These users have used StarCluster to fulfill a variety of computing needs at MIT, Harvard, and beyond in both academic and research settings. We asked a few of our users to provide some examples of how they're using StarCluster to improve their academic and research environments:

Mark Pearrow - Brain and Cognitive Sciences at MIT

The McGovern Institute for Brain Research has an in-house compute cluster that has a total of 56 CPU cores. Although it is often sufficient for day-to-day neuro-imaging analysis, we increasingly have high-memory and high-CPU requiring jobs that would tie up the entire cluster for an extended period of time. For such jobs, we are using StarCluster and Amazon's EC2 and EBS systems to temporarily extend our capacity to process information. We have, to date, processed several large batch jobs using this "cloud" system. It is likely that given our lack of an in-building data center, we will increasingly rely on StarCluster and off-site processing-on-demand.

Dan Yamins – Harvard and MIT

My most rewarding StarCluster experience almost certainly has to be the time when, having found out about an urgent need to re-run a massive analysis pipeline while on an Internet-enabled bus ride from Boston to New York, I was able to start up a twenty-node cluster with a single command, activate my work-flow engine software to do the computation with one more single command, put my laptop away and take a nap -- and by the time I woke up in New York, my computation had finished successfully.

Seppo Sahrakorpi - SEAS IRCS Harvard

The Harvard University School of Engineering and Applied Sciences (SEAS) Instructional and Research Computing Services (IRCS) uses StarCluster as a part of the SEAS Cloud offering. In particular, StarCluster is used in the CS262 “Introduction to Distributed Computing” course to allow the students to launch small on-demand clusters for their Java-based distributed computing homework assignments. StarCluster also provides a scalable on-demand research computing resource for our whole research community.