Article

Vol. 1 (2025)
DOI : https://doi.org/10.66096/JIGBP.V1.5
Published : Jan 8, 2025

Experimental Evaluation of Data Partitioning and Sharding Approaches for Scalability in Distributed Databases

person Ryan Diaz (1), Uma Gomez (2), Tom Adams (3)

(1) AI Automation Analyst IndustriX Dynamics, Germany
(2) Digital Twin Developer IndustriX Dynamics, Germany
(3) Process Optimization Engineer IndustriX Dynamics, Germany
picture_as_pdf Fulltext View | Download
Abstract

Performance is proportional to the productivity of applications in distributed databases and is dependent on how data is managed. In this research, we analysed different methods of data partitioning and sharding in MongoDB, Cassandra, PostgreSQL, and MySQL Cluster in terms of their impacts on query latency, transaction throughput, resource use efficiency, and fault tolerance. A variety of real-world (TPC-C, TPC-H) and custom workloads were used to test range-partitioning, hash-partitioning, list-partitioning, round-robin-partitioning, and dynamic and horizontal versus vertical sharding. The outcomes indicate that load balancing and fault tolerance is maximized with hash-based partitioning, while read-intensive workloads benefit from lower query latency due to key-based sharding. Dynamic sharding outperforms static approaches by 27% due to its superior ability to accommodate fluctuating workloads. This research describes the effectiveness of various partitioning and sharding methods and their relevance in addressing the issues of resource allocation in large distributed systems. Our findings assist database architects and engineers in devising more efficient data distribution policies to improve scalability, performance, and efficiency. Further development will focus on the use of AI for adaptive sharding and the use of edge computing to enhance performance.