Show simple item record

AuthorZhai, Yanlong
AuthorTchaye-Kondi, Jude
AuthorLin, Kwei Jay
AuthorZhu, Liehuang
AuthorTao, Wenjun
AuthorDu, Xiaojiang
AuthorGuizani, Mohsen
Available date2022-10-29T21:59:29Z
Publication Date2021-10-01
Publication NameJournal of Parallel and Distributed Computing
Identifierhttp://dx.doi.org/10.1016/j.jpdc.2021.05.011
CitationZhai, Y., Tchaye-Kondi, J., Lin, K. J., Zhu, L., Tao, W., Du, X., & Guizani, M. (2021). Hadoop perfect file: A fast and memory-efficient metadata access archive file to face small files problem in hdfs. Journal of Parallel and Distributed Computing, 156, 119-130.‏
ISSN07437315
URIhttps://www.scopus.com/inward/record.uri?partnerID=HzOxMe3b&scp=85108089031&origin=inward
URIhttp://hdl.handle.net/10576/35578
AbstractHDFS faces several issues when it comes to handling a large number of small files. These issues are well addressed by archive systems, which combine small files into larger ones. They use index files to hold relevant information for retrieving a small file content from the big archive file. However, existing archive-based solutions require significant overheads when retrieving a file content since additional processing and I/Os are needed to acquire the retrieval information before accessing the actual file content, therefore, deteriorating the access efficiency. This paper presents a new archive file named Hadoop Perfect File (HPF). HPF minimizes access overheads by directly accessing metadata from the part of the index file containing the information. It consequently reduces the additional processing and I/Os needed and improves the access efficiency from archive files. Our index system uses two hash functions. Metadata records are distributed across index files using a dynamic hash function. We further build an order-preserving perfect hash function that memorizes the position of a small file's metadata record within the index file.
SponsorThe authors thank the anonymous reviewers for their insightful suggestions. This work is supported by the National Natural Science Foundation of China (Grant No. 61602037 ).
Languageen
PublisherAcademic Press Inc.
SubjectDistributed file system
Fast access
HDFS
Massive small files
TitleHadoop Perfect File: A fast and memory-efficient metadata access archive file to face small files problem in HDFS
TypeArticle
Pagination119-130
Volume Number156
dc.accessType Open Access


Files in this item

Thumbnail

This item appears in the following Collection(s)

Show simple item record