Cardiff University | Prifysgol Caerdydd ORCA
Online Research @ Cardiff 
WelshClear Cookie - decide language by browser settings

Bundle and pool architecture for multi-language, robust, scalable workflow executions

Rogers, David Mckendrick, Harvey, Ian Charles, Huu, Tram Truong, Evans, Kieran ORCID: https://orcid.org/0000-0003-0414-0812, Glatard, Tristan, Kallel, Ibrahim, Taylor, Ian James ORCID: https://orcid.org/0000-0001-5040-0772, Montagnat, Johan, Jones, Andrew Clifford and Harrison, Andrew 2013. Bundle and pool architecture for multi-language, robust, scalable workflow executions. Journal of Grid Computing 11 (3) , pp. 457-480. 10.1007/s10723-013-9267-2

Full text not available from this repository.

Abstract

In this paper, we leverage the previous work on the SHIWA bundling format and expand on this specification in order to facilitate workflow execution within a multi-workflow environment. We introduce a scalable and robust execution pool environment that supports workflows consisting of sub-workflows built upon a multitude of different workflow engines and environments, and also provide a common workflow representation for seamless connectivity through serialization to workflow bundles. We also present a meta-workflow scenario based upon this system. Workflow bundles employ the lightweight Open Archives Initiative Object Reuse and Exchange (ORE) Web-based standard, to provide a common format for representing and sharing workflows and the associated metadata required for their execution. This generalized bundling approach is already available within five workflow engines and has proven a useful environment for inter-workflow experimentation. The execution pool facilitates federated access to multiple distributed computing infrastructures supported by the underlying workflow engines subscribed to the pool. Workflow bundles are exposed using the eXtensible Messaging and Presence Protocol (XMPP), which provides the necessary communication backbone to enable multiple workflow engine agents to asynchronously publish and subscribe to bundles in meta-workflow pipelines. We present experiments showing the scalability and robustness of the pool execution approach with results showing that overheads remain controlled for up to 150 workflow agents, and that agent failures have very limited impact. We then demonstrate the applicability of our architecture by describing how a Java-based music analysis workflow can be distributed within such a multi-workflow environment consisting of the Triana and MOTEUR workflow engines.

Item Type: Article
Date Type: Publication
Status: Published
Schools: Computer Science & Informatics
Subjects: Q Science > QA Mathematics > QA75 Electronic computers. Computer science
Uncontrolled Keywords: Scientific workflows; Distributed computing infrastructure; Grid computing; Cloud computing; Interoperability; Data modelling
Publisher: Springer
ISSN: 1570-7873
Last Modified: 05 Jan 2024 06:39
URI: https://orca.cardiff.ac.uk/id/eprint/48590

Citation Data

Cited 17 times in Scopus. View in Scopus. Powered By Scopus® Data

Actions (repository staff only)

Edit Item Edit Item