UZH.13

SwissEx integration

Long Title: SwissEx integration in SMSCG
Leading
Organization:
Universität Zürich
Participating
Organizations:
Universität Bern
Eidg. Forschungsanstalt für Wald, Schnee und Landschaft (WSL)
Domain: Grid
Status: finished
Start Date: 10.07.2012
End Date: 30.04.2013
Project Leader: S. Maffioletti

This integration project enables SwissEx to lever and integrate SMSCG infrastructure as a means to access the centralized computing and data facility for resource demanding statistical and analysis pipelines.

Results

Component Description
gsnwoapck Gsnowpack has been deployed at the university of Zurich in production mode to periodically fetch and generate plots from data collect from a set of authorized IMIS stations. It is running in production since April 2013. The access to the automatically generated plots is only available to SLF members, for data privacy reasons
gregrid gregrid (and gsnwoapck) is part of the GC3Pie library package
Application of large-scale computing infrastructure for diverse environmental research applications using GC3Pie Publication by S. Maffioletti, N. Dawes, M. Bavay, S. Sarni, and M. Lehning.Presented at European Geosciences Union General Assembly 2013 Vienna, Austria. 07-12 April 2013.

The SwissEx integration project has established a first collaborative step between the SwissExperiment community and the SMSCG infrastructure. It has showed and demonstrated that the SMSCG infrastructure, as well as the Academic Cloud infrastructure, could be lever by the SwissExperiment community to support a large variety of environmental science computational and data processing usecases. In particular, the project has established a first simulation model that allows to analyze and plot meteorological data from a set of geographically distributed stations. Another implemented usecase is the re-gridding model that allows to generate plots from a set of CFD data.

The implemented usecases gsnowpack and gregrid are available for the SwissExperiment user communities; these are tools that allows to easily analyze and generate plot images from various type of reference data while leveraging a scalable cloud-based computational infrastructure.
End-users are no-longer restricted by the availability of their local resources and can also plan more fine-grained analysis, that normally requires a substantial number of computational resources. Research groups that are now levering the services from the SwissExperiment project, will also have the possibility of enabling additional usecases taking full benefit from this collaborative model.

Application support will be provided by UZH/GC3 as part of their support to the GC3Pie application collection while user support will be provided by the SwissExperiment team.


Goals

SwissExperiment aims to provide the common access point for Swiss environmental data and state-of-the-art analysis tools. In a world of rapidly progressing data access technology, SwissExperiment will keep the Swiss environmental sciences at the forefront of data collection, but more importantly of data management and exploitation.

The SwissEx integration project aims to evaluate how the SwissExperiment project can lever the SMSCG infrastructure for centralized data archival and for computing pipelines; for this two selected usecases from the SwissExperiment project will be enabled and ported on the SMSCG infrastructure. The project will provide a reliable and scalable model for future exploitations.

Benefits

  • SwissEx will benefit from offloading the operational and management aspect of a centralized data and computing infrastructure.
  • The business model that may emerge from this collaboration (SMSCG presented as the Swiss academic cloud infrastructure) may be an interesting usecase for other disciplines.
  • SMSCG will profit from such an intensive data-driven use-case, enabling potential consolidation of data management solutions.
  • SMSCG has the objective to become one of the reference platforms for high-throughput scientific computing. Collaboration with the SwissEx project will further consolidate the infrastructure, services, and competences within SMSCG.

Steps

Effort for integration of the SwissExperiment infrastructure will be divided in the following tasks:

  1. Deployment of the central repository for hosting public data gathered from distributed GSN stations:
    Data collected from the meteorological stations are archived and classified in distributed sites connected with a proprietary component called GSN. The publicly available data will be made available to either a centralized database or deployed on an available SMSCG Storage server and made available to the software appliances through the SMSCG supported data transfer protocols (http, GridFTP).
  2. Prepare the Virtual appliances for the "statistics and plotting" and for the "data analysis" pipelines:
    Two initial pipelines will be considered:
    1. Execution of R scripts on the TRAM, RECORD and MOUNTLAND datasets to produce either plot images or statistical information.
    2. Alpine3D execution of SNOWPACK on data from IMIS stations.
    Both pipelines will be implemented on SMSCG using the AppPot virtualization mechanism.
  3. Implement the pipeline execution control logic:
    both execution pipelines will be implemented using the GC3Pie framework developed by UZH. This will allow systematic and automatic execution of the pipelines to be described in terms of software appliances.
  4. Integration with the SwissEx web interface:
    Another demonstration usecase will be to provide access to the results of the TRAM, RECORD and MOUNTLAND datasets through a web interface. In this case, the pipelines will be activated automatically on a periodic base using a defined data-set.

All the software components developed in the proposed project will be placed under public licences (e.g. GPL or apache2).

Back