CLI Reference

All commands (except init) discover fdl.toml by walking up from the current working directory, so they can be run from any subdirectory of your project. The same lookup is also used by the Python API.

Commands

init

Initialize a new project. Creates fdl.toml, the .fdl/ directory, and a DuckLake catalog.

fdl init [NAME] [--public-url URL] [--target-url URL] [--target-name NAME]

When flags are omitted, you'll be prompted interactively.

Generated files:

• fdl.toml — Project config (with target settings)
• .fdl/{target}/ducklake.sqlite — Local DuckLake catalog (SQLite, so multiple processes can read/write concurrently)

On failure, fdl.toml and .fdl/ are automatically rolled back.

pull

Download a catalog from target.

fdl pull TARGET

Requires prior initialization with fdl init.

push

Push catalog to target.

fdl push TARGET [--force]

Pushes the DuckLake catalog (ducklake.duckdb) and fdl.toml to the target. Data files are not included — they are written directly to the target via fdl run or fdl sql.

Requires an existing local catalog. fdl push does not create one; run fdl init or fdl pull TARGET first.

The local catalog is SQLite; push converts it to DuckDB format (the distribution format) before upload. Push also updates ducklake_metadata.data_path in the shipped catalog to match the current public_url in fdl.toml, so changing public_url and running fdl push is sufficient to redeploy at a new origin. Pull reverses the conversion, converting the downloaded DuckDB back to SQLite locally.

Snapshot expiration

Before converting the catalog, push runs automatic snapshot expiration, so the shipped catalog (and the conversion itself) stays bounded to the retention window.

Conflict detection (S3 targets)

For S3 (and S3-compatible) targets, fdl uploads the catalog with an HTTP If-Match precondition using the ETag recorded from the previous push or pull. If another client has pushed in the meantime, the S3 server rejects the upload with 412 Precondition Failed and fdl surfaces the conflict:

Remote catalog has been updated since the last pull. Run 'fdl pull' first, or use --force to override.

The precondition is evaluated atomically on the server, so there is no race window between the check and the write. The first push to an empty target uses If-None-Match: *, which succeeds only when no catalog is present.

Use --force to skip the precondition. Local (non-S3) targets are assumed single-user and skip conflict detection entirely.

sync

Pull, run a pipeline command, and push — all in one step.

fdl sync TARGET [--force]
fdl sync TARGET [--force] -- COMMAND [ARGS...]

When COMMAND is omitted, fdl reads command from fdl.toml — first from targets.<name>.command, then from the top-level command:

command = "python main.py"

Per-target override:

command = "python main.py"

[targets.local]
command = "python main.py --quick"

# These are equivalent:
fdl sync default
fdl sync default -- python main.py

Processing:

1. Auto-pull if local catalog is missing or stale (same as fdl run)
2. Run command with FDL_* environment variables injected
3. Push catalog to target (only on success)

If the command exits with a non-zero code, push is skipped and the exit code is propagated.

See Working with Data for details on injected environment variables and usage patterns.

run

Run a command with fdl environment variables injected.

fdl run TARGET
fdl run TARGET -- COMMAND [ARGS...]

When COMMAND is omitted, uses command from fdl.toml (same lookup as fdl sync).

The subprocess runs with the project root (the directory containing fdl.toml) as its working directory. Relative paths in your pipeline script resolve against the project root regardless of where fdl run was invoked from.

If the target has no local catalog and no remote catalog to pull from, fdl run exits with an error telling you to run fdl init or fdl pull TARGET first. It does not silently create an empty catalog.

See Working with Data for details on injected environment variables.

sql

Execute a SQL query against the DuckLake catalog.

fdl sql TARGET QUERY [--force]

Requires a local catalog at .fdl/{target}/ducklake.sqlite. Run fdl init or fdl pull TARGET first if it is missing — fdl sql does not auto-initialize.

Stale catalog check

Before executing, fdl verifies that the local catalog is up to date with the remote. If someone else has pushed since your last pull, the command is rejected:

Local catalog is stale (remote pushed at 2026-04-01T00:00:00+00:00).
Run 'fdl pull' first, or use --force to override.

This prevents queries against outdated data and, for maintenance operations like CHECKPOINT, accidental deletion of active files added by another user's push.

Examples:

# Create a table
fdl sql default "CREATE TABLE cities (name VARCHAR, population INTEGER)"

# Insert data
fdl sql default "INSERT INTO cities VALUES ('Tokyo', 14000000), ('Shanghai', 24900000)"

# Query
fdl sql default "SELECT * FROM cities ORDER BY population DESC"

# Skip stale catalog check
fdl sql default --force "SELECT * FROM cities"

See Working with Data for details on how the catalog connection works and caveats.

expire

Expire old snapshots and delete unreferenced data files.

fdl expire TARGET [--retention-days N] [--dry-run]

DuckLake never removes anything on its own: every write adds a snapshot, and dropped or replaced tables (including their inlined-data tables) stay in the catalog forever. Without expiration the catalog grows linearly with build count, which bloats the shipped catalog and slows down the SQLite ↔ DuckDB conversion in push/pull.

fdl expire expires every snapshot older than the retention period via DuckLake's ducklake_expire_snapshots (the latest snapshot is always kept, even when it is older than the cutoff) and deletes the data files that become unreferenced. It also deletes orphaned files left behind by crashed writes — even when no snapshot was old enough to expire — so an explicit run doubles as a storage prune. For S3 targets the file cleanup operates on remote storage.

With --dry-run, the reported file count is a lower bound: files that the expiration itself would schedule for deletion are only known once it actually runs.

Automatic expiration

You rarely need to run fdl expire by hand. Comparable to git gc --auto, fdl runs it automatically:

• after fdl run / fdl sql, when the command actually wrote to the catalog (read-only commands never trigger it), and
• before the catalog conversion in fdl push.

The automatic runs use maintenance.snapshot_retention_days from fdl.toml (default: 7 days); set it to false to disable them (see Configuration). Unlike explicit runs, they skip the file cleanup when nothing was expired (the cleanup lists DATA_PATH, a network LIST on S3, not worth paying on every write). The explicit fdl expire command works regardless of that setting.

Writers that bypass fdl commands entirely (e.g. a scheduler invoking your pipeline directly with FDL_* environment variables) are outside fdl's sight; run fdl expire periodically in that setup.

Caveats

Expiring snapshots removes time travel history older than the retention period, and clients still reading a catalog copy published before the retention window may fail to resolve deleted data files. Tune maintenance.snapshot_retention_days to the staleness you need to support.

duckdb

Launch an interactive DuckDB shell with the target's DuckLake catalog attached and selected.

fdl duckdb TARGET [--read-only] [--force] [--dry-run] [--duckdb-bin PATH]

fdl resolves the target, performs the stale catalog check, then execs into the duckdb CLI with INSTALL ducklake, ATTACH, and USE pre-applied via -cmd. Because fdl replaces itself with duckdb, TTY, signal handling, and the exit code are inherited normally — Ctrl-C, history, and .exit work as they do in a plain duckdb invocation.

For S3 targets, fdl also loads httpfs and issues a CREATE SECRET from the target's credentials in fdl.toml.

Requires the duckdb CLI on PATH (or pass --duckdb-bin). Install it from duckdb.org/docs/installation.

Examples:

# Open a shell on the default target
fdl duckdb default

# Read-only (safe for exploration on shared data)
fdl duckdb default --read-only

# Inspect the command fdl would run, without launching duckdb
fdl duckdb default --dry-run

# Use a specific duckdb binary
fdl duckdb default --duckdb-bin ~/bin/duckdb

config

Get or set configuration.

fdl config [KEY] [VALUE]

Reads and writes fdl.toml. Without arguments, lists all settings.

Examples:

# Set target URL
fdl config targets.default.url 's3://${FDL_S3_BUCKET}'
fdl config targets.default.public_url https://data.example.com

# Set S3 credentials with env var references
fdl config targets.default.s3_endpoint '${FDL_S3_ENDPOINT}'
fdl config targets.default.s3_access_key_id '${FDL_S3_ACCESS_KEY_ID}'
fdl config targets.default.s3_secret_access_key '${FDL_S3_SECRET_ACCESS_KEY}'

# List all settings
fdl config

See Configuration for details on targets and ${VAR} expansion.

serve

Start an HTTP server with CORS and Range request support.

fdl serve TARGET [--port PORT]

Examples:

# Serve the default target
fdl serve default

# Serve on a different port
fdl serve default --port 8080