Cardiff University | Prifysgol Caerdydd ORCA
Online Research @ Cardiff 
WelshClear Cookie - decide language by browser settings

SparkFlow: Towards high-performance data analytics for spark-based genome analysis

Filgueira, Rosa, Awaysheh, Feras M., Carter, Adam, White, Darren J. and Rana, Omer ORCID: 2022. SparkFlow: Towards high-performance data analytics for spark-based genome analysis. Presented at: IEEE/ACM International Symposium on Cluster Computing and the Grid (CCGRID), Taormina, Italy, 16-19 May 2022. Proceedings: 22nd IEEE/ACM International Symposium on Cluster, Cloud and Internet Computing. IEEE, pp. 1007-1016. 10.1109/CCGrid54584.2022.00123

[thumbnail of 995600b007.pdf]
PDF - Accepted Post-Print Version
Download (665kB) | Preview


The recent advances in DNA sequencing technology triggered next-generation sequencing (NGS) research in full scale. Big Data (BD) is becoming the main driver in analyzing these large-scale bioinformatics data. However, this complicated process has become the system bottleneck, requiring an amal-gamation of scalable approaches to deliver the needed performance and hide the deployment complexity. Utilizing cutting-edge scientific workflows can robustly address these challenges. This paper presents a Spark-based alignment workflow called SparkFlow for massive NGS analysis over singularity containers. SparkFlow is highly scalable, reproducible, and capable of parallelizing computation by utilizing data-level parallelism and load balancing techniques in HPC and Cloud environments. The proposed workflow capitalizes on benchmarking two state-of-art NGS workflows, i.e., Base Recalibrator and ApplyBQSR. SparkFlow realizes the ability to accelerate large-scale cancer genomic analysis by scaling vertically (HyperThreading) and horizontally (provisions on-demand). Our result demonstrates a trade-off inevitably between the targeted applications and proces-sor architecture. SparkFlow achieves a decisive improvement in NGS computation performance, throughput, and scalability while maintaining deployment complexity. The paper's findings aim to pave the way for a wide range of revolutionary enhancements and future trends within the High-performance Data Analytics (HPDA) genome analysis realm.

Item Type: Conference or Workshop Item (Paper)
Date Type: Published Online
Status: Published
Schools: Computer Science & Informatics
Publisher: IEEE
ISBN: 9781665499576
Date of First Compliant Deposit: 20 July 2022
Last Modified: 26 Oct 2022 14:33

Actions (repository staff only)

Edit Item Edit Item


Downloads per month over past year

View more statistics