Database Scaling Strategies

What is database scaling?

When thinking about a software system’s scalability, we must consider the performance of one of the most common system resources: a database. Databases play a crucial role in the overall performance of any system that is dependent on the retrieval and storage of information. If a database is responding to too many requests or runs out of storage capacity, a system may perform poorly (e.g., slow response speed). This is why it is important to consider database scaling to accommodate a system’s growing data storage and performance needs.

Database Scaling is the process of adding or removing from a database’s pool of resources to support changing demand. A database can be scaled up or down to accommodate the needs of the application that it’s supporting. In this article, we’ll explore two main ways to scale a database: sharding and replication.

Let’s dive in!

Sharding

First, let’s explore database sharding. It is the process of splitting a single (usually large) dataset into various smaller chunks (known as shards) that are stored across multiple databases. Sharding is considered to be a horizontal scaling solution since it increases the number of database instances in a system.

If, for example, we wanted to shard a large dataset of clothing inventory for an e-commerce application, it might look like this:

Note how the inventory table was broken up and spread across multiple machines hosting a database. In this case, the table was broken up by the size value, with all items of a particular size in its own database instance (aka a shard).

Now that we have a general overview of the concept of sharding, let’s explore the distinct advantages and disadvantages:

Advantages

• Increase in storage capacity: By increasing the number of shards, the overall total storage capacity of a system is increased.
• Increased Availability: Even if one shard goes offline, the majority of shards will still be available to retrieve and store data. This means only a portion of the overall dataset will be unavailable.

Disadvantages

• Query overhead: A database that has been sharded must have an independent machine or service that can properly route database queries to the appropriate shard. This increases latency and expense on every operation because if the query requires data from multiple shards, the router must query each shard and then merge the data.

• Administration complexity: A database that has been sharded requires more upkeep and maintenance since there are now multiple machines with their own databases.

• Increased cost: There is an inherent increase in cost because sharding requires more machines as well as computing power.

Replication

Replication is a scaling strategy where identical copies of a database are created on additional machines. If we return to our clothing inventory database, here is what the database architecture would look like using the replication strategy:

Now that we have a general overview of the concept of replication, let’s explore the distinct advantages and disadvantages:

Advantages

• Decreased load: Due to data being replicated, queries can be spread across multiple databases. This reduces the likelihood that any single database will be overwhelmed with queries.
• Increased Availability: With the same data being replicated on multiple servers; replication ensures that if one database goes down, the entire system can still be fully functional.

Disadvantages

• Increased write complexity: Write-focused queries (i.e., saving data to the database) increase complexity because the data must be copied to every replicated database instance to make sure each database stays in sync.
• Potential Data inconsistency: Data that has been replicated that is either incorrect or out of date can lead to other machines part of the system being out of sync.

Wrap Up

We’ve now learned some of the fundamentals of using database scaling strategies within our applications. Let’s take a moment to review what we’ve learned:

• Databases are a crucial part of any system architecture and must be considered as part of a scaling strategy.
• Sharding is a database scaling technique where a large dataset is broken into smaller chunks (shards) that live on separate machines.
• Replication is a database scaling technique where an entire database is replicated onto multiple machines.

Not every database needs to be scaled, so it is important to understand the needs of the application. This will help to determine what data is being stored, the approximate number of queries, as well as the most appropriate scaling strategy if one is needed.