CloudFlow: A data-aware programming model for cloud workflow applications on modern HPC systems
Author | Zhang, Fan |
Author | Malluhi, Qutaibah M. |
Author | Elsayed, Tamer |
Author | Khan, Samee U. |
Author | Li, Keqin |
Author | Zomaya, Albert Y. |
Available date | 2015-12-29T12:54:37Z |
Publication Date | 2015-10 |
Publication Name | Future Generation Computer Systems |
Resource | Scopus |
Identifier | http://dx.doi.org/10.1016/j.future.2014.10.028 |
Citation | Zhang F., Malluhi Q.M., Elsayed T., Khan S.U., Li K., Zomaya A.Y., CloudFlow: A data-aware programming model for cloud workflow applications on modern HPC systems, (2015) Future Generation Computer Systems, 51, pp. 98-110. |
ISSN | 0167-739X |
Abstract | Traditional High-Performance Computing (HPC) based big-data applications are usually constrained by having to move large amount of data to compute facilities for real-time processing purpose. Modern HPC systems, represented by High-Throughput Computing (HTC) and Many-Task Computing (MTC) platforms, on the other hand, intend to achieve the long-held dream of moving compute to data instead. This kind of data-aware scheduling, typically represented by Hadoop MapReduce, has been successfully implemented in its Map Phase, whereby each Map Task is sent out to the compute node where the corresponding input data chunk is located. However, Hadoop MapReduce limits itself to a one-map-to-one-reduce framework, leading to difficulties for handling complex logics, such as pipelines or workflows. Meanwhile, it lacks built-in support and optimization when the input datasets are shared among multiple applications and/or jobs. The performance can be improved significantly when the knowledge of the shared and frequently accessed data is taken into scheduling decisions. To enhance the capability of managing workflow in modern HPC system, this paper presents CloudFlow, a Hadoop MapReduce based programming model for cloud workflow applications. CloudFlow is built on top of MapReduce, which is proposed not only being data aware, but also shared-data aware. It identifies the most frequently shared data, from both task-level and job-level, replicates them to each compute node for data locality purposes. It also supports user-defined multiple Map- and Reduce functions, allowing users to orchestrate the required data-flow logic. Mathematically, we prove the correctness of the whole scheduling framework by performing theoretical analysis. Further more, experimental evaluation also shows that the execution runtime speedup exceeds 4X compared to traditional MapReduce implementation with a manageable time overhead. |
Sponsor | NPRP grant # 09-1116-1-172 from the Qatar National Research Fund (a member of Qatar Foundation). Ministry of Science and Technology of China under National 973 Basic Research Program (Grant No. 2013CB228206), National Natural Science Foundation of China (Grant Nos. 61472200 and 61233016). |
Language | en |
Publisher | Elsevier |
Subject | Concurrency Data aware HPC MapReduce Programming model |
Type | Article |
Pagination | 98-110 |
Volume Number | 51 |
Check access options
Files in this item
Files | Size | Format | View |
---|---|---|---|
There are no files associated with this item. |
This item appears in the following Collection(s)
-
Network & Distributed Systems [65 items ]