Adapter Migration

Migrate data between adapters using the export/import pipeline without downtime.

Why migrate?

As your application grows, you may need to move from one adapter to another. Common migrations include SQLite to Postgres (scaling beyond a single machine), local SQLite to a remote Postgres sync server, or consolidating multiple SQLite files into a single Postgres database.

Export/import pipeline

Persist provides a built-in export/import pipeline that works across any combination of adapters. The pipeline reads all data from the source store and writes it to the destination store in batches.

import { createStore, migrate } from "@cuitty/persist";

const source = await createStore("app", {
  adapter: "sqlite",
  path: "./data/app.db",
});

const destination = await createStore("app", {
  adapter: "postgres",
  connectionString: "postgresql://user:pass@host:5432/persist",
});

const result = await migrate(source, destination, {
  batchSize: 500,
  onProgress: (progress) => {
    console.log(`${progress.completed}/${progress.total} records migrated`);
  },
});

console.log(`Migration complete: ${result.records} records, ${result.events} events`);

CLI migration

The CLI provides a simpler interface for common migrations:

# Export to a portable format
persist export my-app --format jsonl --output ./backup.jsonl

# Import into a new store
persist import my-app --from ./backup.jsonl \
  --adapter postgres \
  --connection "postgresql://user:pass@host:5432/persist"

The JSONL format stores one record per line, making it easy to inspect, filter, or transform with standard Unix tools.

Zero-downtime migration

For production workloads, use the dual-write pattern:

1. Set up the destination store with the new adapter.
2. Enable dual-write mode. Persist writes to both adapters simultaneously.
3. Run the backfill to copy existing data from source to destination.
4. Verify that both stores have identical data.
5. Switch reads to the destination.
6. Disable dual-write and decommission the source.

import { createStore, enableDualWrite } from "@cuitty/persist";

const store = await createStore("app", {
  adapter: "sqlite",
  path: "./data/app.db",
});

await enableDualWrite(store, {
  target: {
    adapter: "postgres",
    connectionString: "postgresql://...",
  },
  backfill: true,
  verify: true,
});

Data integrity

The migration pipeline checksums each batch and verifies that the destination matches the source. If a batch fails, the pipeline retries with exponential backoff. After migration, run persist verify to compare record counts and checksums between source and destination.

persist verify my-app \
  --source sqlite:./data/app.db \
  --target postgres:postgresql://user:pass@host/persist