Database Sharding

Why shard?

To exceed the limits of a single node: capacity, IOPS, memory, and to isolate failures.

Strategies

Range sharding: ordered key (e.g., timestamps). Great for range scans; hotspot risk.
Hash sharding: uniform distribution; harder for range queries.
Directory/lookup: routing table; flexible but extra component.
Geo sharding: keep data close to users, compliance (data residency).

Choosing a shard key

High cardinality, stable, and aligns with access patterns (e.g., tenant_id, user_id).

Rebalancing

Consistent hashing with virtual nodes, or move chunks; consider double‑write during migrations.

Cross‑shard queries

Avoid cross‑shard joins when possible; use denormalization and materialized views.

Code: routing helper

// language-typescript
const VIRTUAL_SHARDS = 256
function shardFor(tenantId: string) {
  const h = murmur3(tenantId) % VIRTUAL_SHARDS
  return physicalShardMap[h]
}

Code: Postgres table partitioning (range)

-- language-sql
CREATE TABLE events (
  id uuid PRIMARY KEY,
  created_at timestamptz NOT NULL,
  payload jsonb
) PARTITION BY RANGE (created_at);

CREATE TABLE events_2025_09 PARTITION OF events
  FOR VALUES FROM ('2025-09-01') TO ('2025-10-01');

Hotspot mitigation

Add randomness to keys (e.g., append hash suffix) to spread writes.
Use write queues per shard; cap concurrency per partition.

Operational concerns

Backup/restore per shard, schema migrations in waves, observability per shard.

Analogy

Sharding is like opening more checkout lanes in a supermarket. Choose a lane assignment (hash/range) that keeps lines even and lets regulars find their lane.

FAQ

Can I change the shard key later? Painful but possible with dual‑write and backfill.
Do I need sharding now? Not if one node meets SLOs. Design for it, don’t over‑engineer.

Pourquoi sharder

Dépasser les limites d’une base unique: capacité, IOPS, latence, isolement des pannes.

Stratégies

Range sharding: clé ordonnée (p. ex. timestamps). Simple pour scans, risque de hotspots.
Hash sharding: distribution uniforme, moins pratique pour scans range.
Directory/lookup: table de routage; flexible mais composant critique.
Geo-sharding: proximité utilisateur, conformité (résidence des données).

Clés de shard

Doivent être stables, à forte cardinalité, et corrélées aux patterns d’accès.

Rebalancing

Prévoir le resharding: consistent hashing, virtual nodes, double écriture pendant migration.

Pièges

Transactions cross-shards coûteuses; privilégier idempotence et agrégats locaux.
Jointures cross-shards → pensez à la dénormalisation et aux vues matérialisées.

*Database Sharding