In-memory distributed matrix computation processing & optimization
المؤلف | Yu, Yongyang |
المؤلف | Tang, Mingjie |
المؤلف | Aref, Walid G. |
المؤلف | Malluhi, Qutaibah M. |
المؤلف | Abbas, Mostafa M. |
المؤلف | Ouzzani, Mourad |
تاريخ الإتاحة | 2020-09-20T08:35:38Z |
تاريخ النشر | 2017 |
اسم المنشور | Proceedings - International Conference on Data Engineering |
المصدر | Scopus |
الرقم المعياري الدولي للكتاب | 10844627 |
الملخص | The use of large-scale machine learning and data mining methods is becoming ubiquitous in many application domains ranging from business intelligence and bioinformatics to self-driving cars. These methods heavily rely on matrix computations, and it is hence critical to make these computations scalable and efficient. These matrix computations are often complex and involve multiple steps that need to be optimized and sequenced properly for efficient execution. This paper presents new efficient and scalable matrix processing and optimization techniques for in-memory distributed clusters. The proposed techniques estimate the sparsity of intermediate matrix-computation results and optimize communication costs. An evaluation plan generator for complex matrix computations is introduced as well as a distributed plan optimizer that exploits dynamic cost-based analysis and rule-based heuristics to optimize the cost of matrix computations in an in-memory distributed environment. The result of a matrix operation will often serve as an input to another matrix operation, thus defining the matrix data dependencies within a matrix program. The matrix query plan generator produces query execution plans that minimize memory usage and communication overhead by partitioning the matrix based on the data dependencies in the execution plan. We implemented the proposed matrix processing and optimization techniques in Spark, a distributed in-memory computing platform. Experiments on both real and synthetic data demonstrate that our proposed techniques achieve up to an order-of-magnitude performance improvement over state-ofthe-Art distributed matrix computation systems on a wide range of applications. |
راعي المشروع | This work was supported by an NPRP grant 4-1534-1-247 from the Qatar National Research Fund and by the National Science Foundation Grant IIS 1117766. |
اللغة | en |
الناشر | IEEE Computer Society |
الموضوع | Distributed computing Matrix computation Query optimization |
النوع | Conference Paper |
الصفحات | 1047-1058 |
الملفات في هذه التسجيلة
الملفات | الحجم | الصيغة | العرض |
---|---|---|---|
لا توجد ملفات لها صلة بهذه التسجيلة. |
هذه التسجيلة تظهر في المجموعات التالية
-
علوم وهندسة الحاسب [2288 items ]