Performance Evaluation and Optimization of Hybrid Indexes in Hive

Abstract

Big data technologies decrease corporate risks and expenses by delivering crucial and accurate analysis that leads to concrete decision-making and improved operational efficiencies. You will need a system to manage and handle massive amounts of categorized and uncategorized data in real time, as well as to safeguard sensitive information, in order for this data to be useful. Operational Big Data and Analytical Big Data are two broad categories of big data products offered by many vendors, such as Amazon, IBDM, and Microsoft. With Hive, an easy-to-use tool for query and analysis of data warehoused on Hadoop, we may recapitulate Big Data. Compression and Bitmap indexing are supported in order to increase query performance. Both indexes have been applied on the same partition in Hive and have seen significant speed improvements as a result. Analyzing Simultaneous Bitmap and Compact Indexing on a partition using text data has resulted in a significant speed improvement.