Develop domain-specific web portals for submitting and managing corresponding compute jobs on the HECToR National Supercomputing Facility (http://www.hector.ac.uk/) in order to reduce the current failure rates and lower the barrier of uptake to new user groups.
Who am I?:
Jano van Hemert (Informatics) and Andrew Turner (EPCC)
Submitted by jvanhem2 on Tue, 09/14/2010 - 08:02
Project Outcomes:
A portal for the national supercomputing facility HECToR has been created. Please see http://research.nesc.ac.uk/node/615 for a demo of the portal.
Do the outcomes match the objectives:
Benefits for the future:
This is a stepping stone to show Edinburgh is the ideal candidate to host the National Facility for Computational Chemistry as we have chosen chemistry applications for the prototype.
Opportunities for further research:
A proposal is under review with EPSRC entitled "A COMPUTATIONAL CHEMISTRY FACILITY AND SERVICE".
Has the project created any new shared resources?:
Yes, a portal now exists that allows easier access to HECToR.
Future research plans:
Further automate the way portals are constructed based on analysis of execution logs.
Publications and presentations:
Collaborations:
This is a funded collaboration with EPCC that relies heavily on cooperation with the School of Chemistry. It demonstrates Informatics can be of benefit to the School of Chemistry, something that has never happened before.
Use of funding:
Funding was used to fund Gary McGilvary and Jos Koetsier instead of just Jos Koetsier as he was already committed to several projects.
How is it novel? What is exciting about it?:
Large computational simulations or analysis are becoming essential tools for a wide variety of scientists who would not previously have needed to use high-end computational facilities for their research. Although there is a huge amount of computational time available on the HECToR National Supercomputer (hosted and administered by the University of Edinburgh) it is difficult for novice users to utilise the resources effectively due to a lack of both technical expertise and familiarity with the interface and job submission system used. This problem effects every user of HECToR as up to 30% of all jobs fail due to errors in the jobs' batch script; sometimes after waiting as long as 3 days to run. A browser-based interface can remove these hurdles by providing canned solutions and placing constraints on values.
Although web applications are routinely used for a wide variety of tasks, submitting jobs to the UK National Supercomputing facility is not one of them. A web interface is easily accessible to all researchers---whether they have a background in HPC or not---and has the potential to open up access to these powerful computing tools to scientists who cannot currently make use of them. This, in turn, enables new science to be performed.
A major reason for not creating a browser-based solution is the cost and time associated, as well as the problem that different groups of users may want different interfaces to work with. Some projects have reported 12 or even 24 person months of effort dedicated to a single portal development. The UK National e-Science Centre has developed a technology that can deliver web-portals cost effectively and in a fraction of the time (weeks rather than months) compared to manually developed portals.
What will I do next? What opportunities will it open up?:
This exemplar project will be demonstrated to the HECToR user base in order to generate collaborations. We will also demonstrate it to Research Councils and promote it to potential HECToR user groups on the HECToR website via on-line videos (examples can be seen here http://research.nesc.ac.uk/taxonomy/term/77).
The dialogue with specific research groups around the UK will open up avenues for future collaboration and funding bids between University of Edinburgh and these other research institutions for the development of more domain-specific portals.
The reputation of the University of Edinburgh as an innovative host of large-scale HPC facilities will be enhanced across the UK, strengthening our bids for future hosting contracts.
We will be able to promote the University of Edinburgh as a centre of excellence in providing web-based access to large HPC facilities. This will allow us to foster collaborations and apply for funding with other HPC facilities world-wide that aim to have their own web-based interfaces.
What constitutes success? How risky is it?:
The success of the project will be measured by the feedback on the web portals received from HECToR users and also the number of future collaborations that result from this project.
The HECToR service provider (EPCC) will continue to monitor the number of failed jobs, and we aim for a significant reduction by web-enabling applications that are used in jobs that have the highest failure rates.
The major risk associated with this project is that there will be little uptake of the web interface from current HECToR users due to unforeseen social elements. We will include users from the start in the design of the portals to prevent inertia from current users of the facility.
What resources do I bring to the project?:
The UK National e-Science Centre brings in the expertise in scientific gateway development and deployment (see http://research.nesc.ac.uk/node/469 http://research.nesc.ac.uk/node/335 http://research.nesc.ac.uk/node/423). Moreover, we have encapsulated much of our expertise in a method and technology called Rapid (http://research.nesc.ac.uk/rapid) to speed up and improve these processes.
EPCC brings in its expertise in HPC: hosting HPC facilties; enabling access for researchers; and running applications on the faciltity. More specifically, we provide the technical expertise needed to run applications on HECToR; access to HECToR usage information; and provide the links into the HECToR user base. EPCC will also provide us with a route to accessing the HECToR facility.
Via the Middleware team (previously in NESC, now in IS) we will provide the necessary resources to host the web portal.
What resources and expertise do I need?:
2 months of Jos Koetsier (Lead Architect of Rapid technology, NeSC/Informatics)---will design, develop and deploy several portlets as part of the scientific gateway.
1 month of Andrew Turner (User Support Consultant, EPCC)---will extract the relevant monitoring data, liaison with the user groups of HECToR and coordinate the feedback.
Total cost: £11,131
What shared resources, if any, will the project create?:
The scientific gateway will be the most important output of the project, it will be a shared resource for the HECToR user community.
The portal itself will be released freely under an Open Source license to the benefit of other HPC facilities and communities using HPC facilities as it allows them to redeploy the solution.
The requirements put forward in this project for the scientific gateway may require improvements to the Rapid methodology/technology, which in turn will benefit other communities that rely on it for generating their scientific gateways.
What is the timescale?:
We would like to start this on 1 March 2010.
