Article
Comparative Study of Adaptive Indexing Techniques for Performance Improvement in Dynamic Workloads
Abstract
Database optimization is highly influenced by indexing with performance parameters such as query execution time, storage usage, and responsiveness suffering degradation without it. Legacy indexing methods like B-Trees, Hash Indexing, and Bitmap Indexing generally fail to adjust to changing workloads in real-time which creates performance issues in modern databases. As the world moves towards cloud, NoSQL and distributed databases, the importance of adaptive indexing methods continues to grow. This research evaluates a set of adaptive indexing methods which include self-tuning B-Trees, adaptive hash indexing, AI learned indexes, and hybrid indexes against transactional (OLTP), analytical (OLAP), and hybrid workloads. Using a broad set of tests on PostgreSQL, MySQL, Apache Cassandra databases, as well as cloud databases like Google BigQuery and Amazon Redshift, we measure the performance of different indexes against the cost of executing the query, the time it takes to build the index, CPU and memory usage and disk I/O operations. These tests showed that adaptive indexing techniques exceeds query execution times in comparison to traditional methods using indexing by up to 45% while also optimizing memory and storage usage. In particular, learned AI based indexing and reinforcement learning assisted indexing showed the greatest flexibility in high variance workloads and were therefore the most suitable for cloud-native and distributed databases. The analysis draws attention to the intricacies involved in balancing index maintenance expenses against the level of a query’s execution efficiency, offering relevant advice and support to system admins and architecture designers. As a culmination, we suggest a new approach to indexing that leverages AI to automatically restructure indexes in response to changes in workload patterns, thereby enhancing scalability in database performance. This work offers fresh perspectives on the prospects of fully automated query tuning and the merging of artificial intelligence with database indexing systems.
