Beyond Rsync: Scaling Large Data Migrations

Rsync's Strengths Are Hard to Argue With

Rsync remains the default tool for server migrations because it solves the core problem elegantly: sync files from source to destination, transmitting only the bytes that changed. For a 5 TB dataset with 90% already on the target, rsync sends roughly 500 GB, not 5 TB. That's incremental efficiency.

It's battle-tested. It works over SSH without additional server setup. Resume capability (with --partial) lets you restart from where you left off. Checksums catch corruption. For 90% of production workloads—fresh VM deployments, weekly backups, incremental syncs of code repositories—rsync is the right choice.

But rsync's design assumes something that hasn't been true for a decade: single-threaded I/O was the bottleneck, not the algorithm.

Where Rsync Hits the Wall

Problem 1: The Checksum Scan Pass

Before transferring any bytes, rsync walks the source filesystem, checksumming every file. On large datasets, this is where you lose hours.

Suppose you're syncing a 100 TB NAS with 50 million small files (typical for media serving or multi-tenant storage). Rsync must:

• Stat every file (filesystem call per file)
• Read and checksum every file (full file scan, even unchanged ones)
• Send the file list to the remote
• Wait for the remote to walk its own filesystem and return checksums

This initial phase—before any actual data transmission—can take 4-8 hours on slow NAS hardware. If the network hiccups and you restart, you repeat the entire checksum scan.

Problem 2: Single-Threaded Delta Calculation

Rsync uses a single thread to compute file deltas. On modern servers with 32+ CPU cores, you're using 3% of available compute. The protocol doesn't parallelize checksumming, delta calculations, or compression.

For a 50 GB file with scattered changes, rsync will take the same time whether you have 4 cores or 128 cores. If that delta calculation takes 2 hours, you wait 2 hours. Another 30 TB file? 2 more hours. Sequential.

Problem 3: Memory Scaling

Rsync's file list structure grows with filesystem size. On a 100 TB dataset with millions of files, rsync can consume 10-20 GB of RAM per process. At scale, this matters. On a shared server with other workloads, memory pressure will kill performance.

Problem 4: SSH Overhead

Rsync over SSH means encryption and authentication overhead per file. For millions of small files, that encryption per-file cost adds up. You're also limited to a single SSH connection unless you run multiple rsync instances in parallel (which creates coordination problems).

Comparison: Tools at Scale

When to Use Alternatives

Parallel Rsync (GNU Parallel + Rsync)

If you're doing incremental sync of very large datasets without a budget for new tools, parallel rsync is your next step. The idea is simple: split the file list into chunks and run multiple rsync jobs in parallel.

Example:

find /source -type f | parallel -j 8   rsync -avz {} remote:/dest/

This reduces a 12-hour sync to 2-3 hours on a 8-core system. Trade-off: you lose rsync's incremental logic per-file, so you're not saving bandwidth on partial updates. Use this for initial migrations, not ongoing syncs.

BBCP (Bulk Berkeley Copy)

BBCP is designed for LAN and WAN bulk transfers. It opens multiple TCP streams in parallel and distributes file chunks across them. Native resume. No encryption by default (faster), but you can tunnel over SSH if needed.

BBCP shines when:

• Network is the limiting factor, not disk I/O
• You have 10+ Gbps available per direction
• You want raw speed without incremental overhead
• Files are large (multi-GB files, not millions of tiny ones)

Downside: BBCP doesn't do incremental sync. It transfers what you tell it to. If 80% of the data already exists on the target, you still transfer everything. It's a bulk mover, not a sync tool.

Multi-Stream FTP Clients

Older than rsync, simpler than both. An FTP server + multi-threaded client (like FileZilla in parallel mode or custom Python) can shift data quickly over LAN. No incremental logic, but no encryption overhead either.

Use for: one-time bulk movement of data where both endpoints are under your control and on the same network.

The Right Tool for Your Scenario

Small Dataset, Incremental Syncs

Stick with rsync. A 5 TB dataset with weekly 5% changes? Rsync is perfect. Fast, reliable, no moving parts.

100+ TB, Large Files, No Incremental Need

Use BBCP or aspera. You want raw speed, not checksums. Expect 2-6 hour window for a 100 TB transfer.

100+ TB, Mixed File Sizes, Incremental

Your use case doesn't have a perfect match in the traditional toolkit. Either:

• Accept that rsync will take 12-24 hours (it's still reliable)
• Split the dataset: use parallel rsync for small files, bbcp for large files
• Consider newer P2P tools designed for exactly this problem

Multiple Migrations in Flight

If you're managing dozens of server migrations, the infrastructure cost of setting up FTP, BBCP, or SSH-tunneled sync solutions justifies custom tooling. Handrive, for example, handles parallel resumable transfers without requiring separate SSH tunnels or complex multi-threaded client logic.

The Real Lesson

Rsync isn't broken. It's specialized for a workload that was common in 2005 and is now niche. Small incremental syncs where the network is slow and CPU is abundant. That's still a real use case.

But a 2026 sysadmin managing multi-petabyte migrations on 100 Gbps networks with 64-core servers can't rely on a tool that uses one thread and assumes SSH is the bottleneck.

The pragmatic move: test your migration with rsync, measure the timeline, and if it's longer than your maintenance window allows, evaluate BBCP, parallel tools, or modern alternatives that parallelize by default.