SparkFlow: Towards high-performance data analytics for spark-based genome analysis

Filgueira, Rosa, Awaysheh, Feras M., Carter, Adam, White, Darren J. and Rana, Omer ORCID: https://orcid.org/0000-0003-3597-2646 2022. SparkFlow: Towards high-performance data analytics for spark-based genome analysis. Presented at: IEEE/ACM International Symposium on Cluster Computing and the Grid (CCGRID), Taormina, Italy, 16-19 May 2022. Proceedings: 22nd IEEE/ACM International Symposium on Cluster, Cloud and Internet Computing. IEEE, pp. 1007-1016. 10.1109/CCGrid54584.2022.00123

Preview

PDF - Accepted Post-Print Version Download (665kB) | Preview

Abstract

The recent advances in DNA sequencing technology triggered next-generation sequencing (NGS) research in full scale. Big Data (BD) is becoming the main driver in analyzing these large-scale bioinformatics data. However, this complicated process has become the system bottleneck, requiring an amal-gamation of scalable approaches to deliver the needed performance and hide the deployment complexity. Utilizing cutting-edge scientific workflows can robustly address these challenges. This paper presents a Spark-based alignment workflow called SparkFlow for massive NGS analysis over singularity containers. SparkFlow is highly scalable, reproducible, and capable of parallelizing computation by utilizing data-level parallelism and load balancing techniques in HPC and Cloud environments. The proposed workflow capitalizes on benchmarking two state-of-art NGS workflows, i.e., Base Recalibrator and ApplyBQSR. SparkFlow realizes the ability to accelerate large-scale cancer genomic analysis by scaling vertically (HyperThreading) and horizontally (provisions on-demand). Our result demonstrates a trade-off inevitably between the targeted applications and proces-sor architecture. SparkFlow achieves a decisive improvement in NGS computation performance, throughput, and scalability while maintaining deployment complexity. The paper's findings aim to pave the way for a wide range of revolutionary enhancements and future trends within the High-performance Data Analytics (HPDA) genome analysis realm.

Item Type:	Conference or Workshop Item (Paper)
Date Type:	Published Online
Status:	Published
Schools:	Computer Science & Informatics
Publisher:	IEEE
ISBN:	9781665499576
Date of First Compliant Deposit:	20 July 2022
Last Modified:	26 Oct 2022 14:33
URI:	https://orca.cardiff.ac.uk/id/eprint/151376

Actions (repository staff only)

Edit Item