Measuring the effectiveness of throttled data transfers on data-intensive workflows

Rodríguez, Ricardo J., Tolosana-Calasanz, Rafael and Rana, Omer Farooq ORCID: https://orcid.org/0000-0003-3597-2646 2012. Measuring the effectiveness of throttled data transfers on data-intensive workflows. Presented at: KES-AMSTA 2012: 6th KES International Symposium on Agent and Multi-Agent Systems: Technologies and Applications, Dubrovnik, Croatia, 25-27 June 2012. Published in: Jezic, G., Kusek, K., Nguyen, N.-T., Howlett, R. J. and Jain, L. C. eds. Agent and Multi-Agent Systems. Technologies and Applications: 6th KES International Conference, KES-AMSTA 2012,Dubrovnik, Croatia, June 25-27, 2012. Proceedings. Lecture Notes in Computer Science. , vol.7327 Berlin and Heidelberg: Springer, pp. 144-153. 10.1007/978-3-642-30947-2_18

Full text not available from this repository.

Abstract

In data intensive workflows, which often involve files, transfer between tasks is typically accomplished as fast as the network links allow, and once transferred, the files are buffered/stored at their destination. Where a task requires multiple files to execute (from different previous tasks), it must remain idle until all files are available. Hence, network bandwidth and buffer/storage within a workflow are often not used effectively. In this paper, we are quantitatively measuring the impact that applying an intelligent data movement policy can have on buffer/storage in comparison with existing approaches. Our main objective is to propose a metric that considers a workflow structure expressed as a Directed Acyclic Graph (DAG), and performance information collected from historical past executions of the considered workflow. This metric is intended for use at the design-stage, to compare various DAG structures and evaluate their potential for optimisation (of network bandwidth and buffer use).

Item Type:	Conference or Workshop Item (Paper)
Date Type:	Publication
Status:	Published
Schools:	Computer Science & Informatics
Subjects:	Q Science > QA Mathematics > QA75 Electronic computers. Computer science
Publisher:	Springer
ISSN:	0302-9743
Last Modified:	21 Oct 2022 09:13
URI:	https://orca.cardiff.ac.uk/id/eprint/35616

Citation Data

Cited 1 time in Scopus. View in Scopus. Powered By Scopus® Data

Actions (repository staff only)

Edit Item