CHANGELOG
v0.10.1 (2026-04-22)
Bug Fixes
- Require SQLite local catalog; ignore legacy ducklake.duckdb
(
9b88ccd)
catalog_path / catalog_url no longer fall back to ducklake.duckdb. pull_if_needed and do_push key
off ducklake.sqlite only, so a legacy v0.8 workspace surfaces the standard "Run 'fdl init' or 'fdl
pull
To migrate a legacy local catalog, run 'fdl pull
- dbt: Use FDL_CATALOG_PATH in profiles.yml attach path
(
829a833)
The DuckLake extension's ATTACH expects 'ducklake:
Use FDL_CATALOG_PATH (the bare absolute path) instead, matching how fdl sql builds its own ATTACH statements in build_attach_sql. FDL_CATALOG_URL remains the right variable for SQLAlchemy-style clients like dlt.
Chores
- Sync uv.lock with current package version
(
f2db44b)
Continuous Integration
- Trigger docs rebuild after successful release
(
1e09465)
The semantic-release step pushes the CHANGELOG.md update using GITHUB_TOKEN, so the downstream docs.yml workflow (which watches CHANGELOG.md) never fires. Kick docs.yml via workflow_dispatch at the end of release.yml; workflow_dispatch is exempt from the GITHUB_TOKEN recursion guard.
v0.10.0 (2026-04-22)
Chores
- Sync uv.lock with current package version
(
0534f4b)
Features
- Redesign FDL_* environment variables around URLs and components
(
40ef01b)
Replace path-oriented env vars with a URL-first, component-rich layout so pipeline code can pass values straight into DuckLake / dlt / dbt without scheme inference or urlparse.
Always-present keys: FDL_CATALOG_URL sqlite:///
S3-only keys: FDL_DATA_BUCKET parsed from target URL FDL_DATA_PREFIX ends with ducklake.duckdb.files/ FDL_S3_* (unchanged)
BREAKING CHANGE: FDL_STORAGE, FDL_DATA_PATH, and FDL_CATALOG are removed. Use FDL_CATALOG_URL / FDL_CATALOG_PATH / FDL_DATA_URL instead.
Breaking Changes
- Fdl_storage, FDL_DATA_PATH, and FDL_CATALOG are removed. Use FDL_CATALOG_URL / FDL_CATALOG_PATH / FDL_DATA_URL instead.
v0.9.2 (2026-04-22)
Bug Fixes
- Enable SQLite WAL mode and busy timeout on catalog attach
(
91e036a)
Pass META_JOURNAL_MODE 'WAL' and BUSY_TIMEOUT 5000 as DuckLake ATTACH options for every SQLite catalog. META_JOURNAL_MODE is honored both on catalog creation and on re-attach, so v0.9 catalogs still in the default delete journal mode auto-migrate to WAL on the next FDL command. BUSY_TIMEOUT is per-connection and waits out the short lock windows that occur during concurrent writes, avoiding SQLITE_BUSY surfaces.
This finishes the v0.9 "local catalog is SQLite" switch: reads (fdl serve) and writes (fdl sql) no longer block each other, which was the original motivation for moving off DuckDB's exclusive file lock.
Chores
- Sync uv.lock with current package version
(
5b4de27)
Documentation
- dagster: Rewrite integration guide around FDLResource and pool-based serialization
(
c8bf3d7)
Replace the pass-through ConfigurableResource example with an FDLResource exposing pull(), push(), and a get_connection() context manager that auto-pulls on entry and auto-pushes on successful exit. Assets now only describe the catalog transformation. Add a short section pointing S3-target users at Dagster concurrency pools (default_limit: 1) to avoid ETag races on parallel pushes. Drop the subprocess-style fdl.run example since Dagster prefers direct Python calls.
Testing
-
Pin regression guards for the pull review fixes (
b8ceb9a) -
test_pull_preserves_etag_when_conversion_fails: monkeypatches convert_duckdb_to_sqlite to raise, then verifies meta.json still holds the pre-pull ETag so the next pull retries instead of falsely reporting "Already up to date". - test_run_errors_when_target_has_no_catalog: additionally asserts that the misleading "pulled from
" log line does not appear above the catalog-missing error when the remote is empty. -
Pin WAL mode regression guards for SQLite catalogs (
9ebe304)
Assert journal_mode=wal on: - freshly created catalog from fdl init - sqlite catalog produced by the DuckDB -> SQLite conversion path - legacy v0.9 catalog forced back to delete mode, after any FDL command attaches it
Also assert that build_attach_sql() emits META_JOURNAL_MODE and BUSY_TIMEOUT for SQLite catalogs.
v0.9.1 (2026-04-22)
Bug Fixes
- Defer pull ETag update until catalog conversion succeeds
(
8d6b950)
Three review findings addressed together:
-
fetch_from_s3 previously wrote the new remote ETag before _convert_downloaded_catalog ran. If the conversion failed (corrupt catalog, disk full, SIGKILL after unlink), the user was left with no local ducklake.sqlite but an updated meta.json, and the next fdl pull would report "Already up to date" while the project was silently back on a DuckDB local catalog. Write the ETag only after the conversion succeeds so failures retry on next pull.
-
pull_if_needed always returned a truthy reason string after do_pull, even when the remote was empty and no catalog was materialized. The caller (fdl run) then printed "No local catalog, pulled from TARGET" immediately before the red "No catalog" error from the catalog-missing guard. Return None when no catalog landed locally.
-
Add
.tmp-shmto the leftovers list in _convert_ducklake_catalog. SQLite WAL mode creates -wal and -shm together; listing -wal without -shm was inconsistent with the stated intent of the cleanup block.
v0.9.0 (2026-04-22)
Documentation
-
Drop legacy duckdb backward-compat notes from cli/working-with-data (
fa32bd3) -
Update guides for v0.9 SQLite-only local catalog (
5daf2e5) -
configuration.md: drop the
catalogkey row and fdl.toml example - reference/cli.md: drop --sqlite from fdl init; describe SQLite-first local layout and push/pull format conversion + data_path rewrite - integrations/dlt.md: SQLite is now the default, so dlt users no longer need --sqlite; simplify the setup steps - getting-started/quickstart.md: ducklake.duckdb -> ducklake.sqlite, and point to the new push-time data_path rewrite instead of the removed known-issue section
Features
- Convert pulled catalog to local SQLite format
(
402d0d7)
After downloading the remote DuckDB catalog, transparently convert it to ducklake.sqlite so the
local workspace is always in the format that supports concurrent read/write. The
ducklake.duckdb copy is removed once the conversion succeeds.
pull_from_local now takes target_name and project_dir so the conversion helper can locate
the target directory consistently with fetch_from_s3.
- Require existing catalog before fdl push
(
fc7fe92)
Match the policy already applied to fdl run / fdl sql / fdl duckdb: refuse to operate
when the target has no local catalog, and point the user at fdl init or fdl pull TARGET.
Previously push would silently succeed-with-empty-payload: the SQLite -> DuckDB conversion was a
no-op, _rewrite_catalog_data_path then opened a fresh zero-byte DuckDB file via
duckdb.connect() and its UPDATE failed with a raw IOException, leaving a stray file behind.
BREAKING CHANGE: fdl push on an uninitialized target now exits 1 with a helpful error instead of
succeeding with an empty upload.
- Require explicit init/pull before fdl run and fdl sql
(
b8c9851)
Previously fdl run silently materialized an empty SQLite catalog when a target had neither local
nor remote state, hiding typos in the target name and obscuring the distinction between "data
lost" and "data never existed". fdl sql and fdl duckdb in the same state would surface the
raw FileNotFoundError traceback from DuckLake.
Drop the auto-init branch from run_command and raise FileNotFoundError with a message pointing
at fdl init or fdl pull TARGET. Have the CLI wrappers (run / sync / sql / duckdb)
translate that into a one-line red error and exit 1. Python API callers see the FileNotFoundError
directly.
BREAKING CHANGE: fdl run no longer creates a catalog as a side effect. Users relying on the
implicit init must now call fdl init or fdl pull before their first run/sync.
- Rewrite data_path on push from public_url
(
f9b8d25)
Changing public_url in fdl.toml used to leave ducklake_metadata.data_path inside the catalog
pointing at the old URL; deployments would then serve data files from the wrong origin unless
users ran a manual UPDATE on the catalog. Push now rewrites the DuckDB copy's data_path to
public_url/<datasource>/<ducklake.duckdb>.files/ immediately after the SQLite->DuckDB
conversion, so every upload is self-consistent with the current config.
Done on the DuckDB copy only: the local SQLite catalog is untouched, so a mid-push crash leaves local state intact.
Drop the corresponding entry from docs/resources/known-issues.md.
- Store local catalog as SQLite only
(
4718083)
The local DuckLake catalog is now always ducklake.sqlite. This lets multiple processes
read/write the catalog concurrently (DuckDB file locking rejects a second opener with LockError,
while SQLite uses OS-level locking with snapshot isolation).
Remote catalogs continue to ship as ducklake.duckdb: push converts SQLite to DuckDB before
upload, and (in a follow-up commit) pull will convert DuckDB back to SQLite locally.
BREAKING CHANGE: the --sqlite CLI flag, the catalog key in fdl.toml, and the sqlite
keyword argument on fdl.init are removed. Any catalog value left in existing fdl.toml is
silently ignored.
Refactoring
- Extract shared ducklake catalog conversion helper
(
3b7ac2e)
Introduce _convert_ducklake_catalog as a direction-agnostic core, and rebuild convert_sqlite_to_duckdb as a thin wrapper on top of it. Add convert_duckdb_to_sqlite (inverse direction) for use by the upcoming auto-migration and pull-side conversion paths.
Testing
- Add concurrent read/write tests for local SQLite catalog
(
12f4d26)
Spawn two fdl sql processes against the same SQLite catalog to verify the v0.9 improvement.
Under v0.8's DuckDB catalog the second opener would have failed with a LockError; SQLite's
OS-level file locking allows snapshot-isolated concurrency.
- Strengthen concurrency tests with barrier and DuckDB negative case
(
94c399c)
Spawn two attacher subprocesses behind a ready/go file barrier so that both land inside the ATTACH critical section simultaneously. Add a DuckDB-side test that exercises the v0.8 failure mode (DuckDB's exclusive file lock triggering "Conflicting lock"); if that stops failing, the justification for the v0.9 SQLite switch needs a fresh look.
The previous version only ran fdl sql SELECT twice with no synchronization, so overlap was
incidental and there was no evidence that DuckDB would actually have failed.
Breaking Changes
fdl pushon an uninitialized target now exits 1 with a helpful error instead of succeeding with an empty upload.
v0.8.0 (2026-04-17)
Chores
- Sync uv.lock with pyproject version 0.7.1
(
86d3a1c)
Documentation
- Describe ETag-based push conflict detection
(
67922dc)
Replace the .fdl/meta.json timestamp description in the push section with the new If-Match precondition flow and clarify that local (non-S3) targets skip conflict detection.
Features
- Use ETag + If-Match for push conflict detection
(
07aeaf0)
Replace the client-generated pushed_at JSON file with server-side ETag preconditions on the catalog object (put_object + If-Match / If-None-Match). The S3 server evaluates the precondition atomically, closing the TOCTOU race between check and write, removing client clock-drift sensitivity, and detecting direct catalog mutations such as set_option.
Local (non-S3) targets skip conflict detection entirely. The remote .fdl/meta.json object is no longer written or read. Legacy state files containing only {"pushed_at": ...} are treated as no record on read.
BREAKING CHANGE: existing users must run fdl pull once after upgrading, or pass --force on the
next push, to initialize the new ETag-based local state.
- cli: Add fdl duckdb for interactive shells
(
2ed08b0)
Add fdl duckdb TARGET [--read-only] [--force] [--dry-run] [--duckdb-bin PATH] that resolves the
target, performs the stale catalog check, builds the INSTALL / ATTACH / USE init SQL (including
INSTALL httpfs + CREATE SECRET for S3 targets), and execs into the DuckDB CLI via os.execvp so
that TTY, signals, and exit code are inherited. --dry-run prints shlex.join(argv) instead of
execing.
Extract the init SQL builder as build_attach_sql() in fdl.ducklake and reuse it from fdl.ducklake.connect(). Paths and credentials embedded in SQL literals are single-quote escaped. Move shared moto/s3_project fixtures to tests/integration/conftest.py.
Breaking Changes
- Existing users must run
fdl pullonce after upgrading, or pass--forceon the next push, to initialize the new ETag-based local state.
v0.7.1 (2026-04-16)
Bug Fixes
- Update connect function to use catalog_path and simplify ducklake_path logic
(
3f56968)
Refactoring
- Storage URL handling and add remote meta key function
(
5638d2f)
v0.7.0 (2026-04-16)
Bug Fixes
-
api: Propagate push conflicts, move init rollback into API (
c0206a3) -
fdl.push now raises PushConflictError instead of calling SystemExit, so Dagster/CI code can catch it. CLI wraps push/sync in try/except and translates to SystemExit(1). - fdl.init rolls back fdl.toml and the partially created .fdl/ directory itself when init_ducklake fails. CLI no longer needs its own rollback block. - Unified Raises sections across fdl.pull, fdl.push, fdl.run, fdl.sync, and fdl.connect docstrings. - Documented that fdl run / fdl sync execute the subprocess with the project root as cwd (was implicit in the feat/python-api series).
BREAKING CHANGE: fdl run (and fdl sync) now run the pipeline subprocess with the directory containing fdl.toml as its working directory, rather than the caller's cwd. Pipelines that relied on the caller's cwd for relative paths should adjust.
Chores
- Sync uv.lock with pyproject version bump
(
68868c7)
Documentation
- Document the Python API and Dagster integration
(
7b65815)
Add docs/reference/python-api.md (mkdocstrings-rendered reference for the six entry points) and docs/integrations/dagster.md (asset, ConfigurableResource, and run-script patterns). Mention the Python API in the home page, quickstart, README, and CLI reference, and note the fdl.toml walk-up lookup and fdl.init's non-idempotent behavior where relevant.
Features
- Add Python API mirroring CLI commands
(
ac70ff0)
Expose fdl.init, fdl.pull, fdl.push, fdl.run, fdl.sync, and fdl.connect at the top of the fdl package so pipelines (e.g. Dagster assets) can drive DuckLake catalogs without spawning a CLI subprocess. Each function mirrors its CLI counterpart and accepts an optional project_dir keyword that falls back to find_project_dir() when omitted. all is set so IDE completion and strict re-export checkers treat these six names as the public surface.
Refactoring
- Thread project_dir through internal modules
(
2516d66)
Add an optional project_dir keyword to do_pull, pull_if_needed, do_push, run_command, ducklake.connect, and the meta helpers so every internal call-chain can resolve fdl.toml against an explicit project root instead of Path.cwd(). run_command also inherits project_dir as the subprocess cwd. No behavior change when project_dir is not provided — find_project_dir returns the nearest ancestor with fdl.toml, matching the previous cwd assumption.
- cli: Delegate CLI handlers to the public Python API
(
1f58600)
Rewrite fdl pull, push, run, sync, and init as thin wrappers around the new fdl.pull / fdl.push / fdl.run / fdl.sync / fdl.init entry points. Keep CLI-only responsibilities (interactive prompts, ValueError -> BadParameter conversion, exit-code transport) in cli.py, and move the "--- pull: ... ---" / "Already up to date" / "skipping push" console messages into the API so Python callers see the same output.
- config: Auto-detect project_dir by walking up to find fdl.toml
(
2342b27)
Add find_project_dir() and switch default resolution in datasource_name, catalog_type, resolve_target, target_s3_config, target_public_url, target_command, and project_config_path from Path.cwd() to the nearest ancestor that contains fdl.toml. Enables running fdl from subdirectories and lets the upcoming Python API be called from any cwd.
Testing
- Cover Python API and find_project_dir walk-up behavior
(
b7818be)
Add tests for the six public entry points (init/pull/push/run/sync/connect), including a local roundtrip, cross-cwd use with explicit project_dir, required target arg, exit-code handling, sync-skip-on-failure, the fdl.toml command fallback, connection close on context exit, and the all surface. Cover find_project_dir with cwd, walk-up, closest-match, missing-file, and explicit-start cases.
Breaking Changes
- api: Fdl run (and fdl sync) now run the pipeline subprocess with the directory containing fdl.toml as its working directory, rather than the caller's cwd. Pipelines that relied on the caller's cwd for relative paths should adjust.
v0.6.0 (2026-04-04)
Features
- Add fdl sync command
(
777528c)
Combines run and push into a single command to prevent forgetting to push after running a pipeline.
Both fdl run and fdl sync read command from fdl.toml when no explicit -- COMMAND is given. Lookup
order: targets.
- run: Auto-pull catalog before command execution
(
4045f61)
fdl run でコマンド実行前にカタログの同期状態を確認し、 必要に応じて自動的に pull する。
判定条件: - ローカルカタログが存在しない - meta.json がない (pull/push されたことがない) - リモートの方が新しい (pushed_at 比較)
Refactoring
- pull: Extract do_pull/pull_if_needed and add --force flag
(
128d063)
pull コマンドのS3/ローカル分岐ロジックを do_pull() に抽出し、 同期判定ロジックを pull_if_needed() に切り出した。
pull はデフォルトで最新の場合スキップし、--force で強制再ダウンロード。
v0.5.7 (2026-04-02)
Bug Fixes
- Use CREATE SECRET for S3 configuration in DuckDB
(
410fac3)
v0.5.6 (2026-04-02)
Bug Fixes
-
Log catalog path on every fdl run (
1e04295) -
Set PYTHONUNBUFFERED environment variable in run command (
ea07bf7)
v0.5.5 (2026-04-02)
Bug Fixes
- Support sqlite catalog type in fdl run auto-init
(
eb7d6f7)
Read catalog type from fdl.toml to correctly initialize sqlite catalogs for dlt-based datasets. Also fall back to fdl.toml when auto-detecting catalog path for new targets.
Chores
- Sync uv.lock
(
171f4b9)
v0.5.4 (2026-04-02)
Bug Fixes
- Auto-initialize catalog on first fdl run for a target
(
71f2c49)
When fdl run is invoked for a target that has no catalog yet, initialize it automatically so pipelines and dbt can attach without requiring a separate fdl init step.
v0.5.3 (2026-04-02)
Bug Fixes
- Ensure target catalog directory exists in fdl run
(
66868df)
Create .fdl/{target}/ before subprocess execution so DuckLake ATTACH can create the catalog file on first run. Also respect FDL_CATALOG env var in connect() for pipelines that hardcode target_name.
- Respect FDL_CATALOG env var in connect()
(
577d924)
When running under fdl run, FDL_CATALOG is set to the correct target-specific path. Pipelines that hardcode target_name in connect() calls now use the env var instead.
v0.5.2 (2026-04-02)
Bug Fixes
- Isolate .fdl catalog directory per target
(
1b889bf)
Each target (default, local, etc.) now gets its own subdirectory under .fdl/ to prevent catalog state conflicts when switching between targets.
.fdl/ducklake.duckdb → .fdl/{target}/ducklake.duckdb .fdl/meta.json → .fdl/{target}/meta.json
Chores
- Add Google Analytics to documentation site
(
bac7f24)
Continuous Integration
- Fix docs workflow to watch zensical.toml instead of mkdocs.yml
(
6378ed3)
Documentation
-
Add semantic-release versioning rules to CLAUDE.md (
dc393ad) -
Refine "Why fdl" section for clarity and conciseness (
1f63113) -
Rewrite README with quick start workflow and feature overview (
8627c14) -
Update .fdl paths to reflect target-based layout (
82f4872)
Testing
- Update path assertions for target-based .fdl layout
(
a00c417)
v0.5.1 (2026-03-30)
Bug Fixes
- Correct formatting in CLI reference table
(
e6de3e8)
Chores
- Sync uv.lock with pyproject.toml version
(
1c4dbea)
Documentation
- Update README formatting for documentation link
(
1fb8e20)
Refactoring
- Replace checkpoint command with stale catalog check in fdl sql
(
677f23f)
v0.5.0 (2026-03-29)
Build System
- Simplify dependencies and add project metadata
(
946e2ba)
Remove dbt-core, pydantic, jinja2, pyyaml dependencies. Add MIT license, project URLs, and authors to pyproject.toml. Move zensical from docs dependency group to dev.
Chores
- Remove fdl_common dbt macros package
(
605a3d4)
Continuous Integration
Documentation
-
Add concept pages and global install instructions (
35180ad) -
Add "Why fdl" page explaining the Frozen DuckLake pattern and motivation - Add Roadmap page with vision toward a package manager for Frozen DuckLake - Add global install instructions (uv tool / pipx) to quickstart
-
Migrate from mkdocs to zensical (
5b8359c)
Replace mkdocs-material with zensical as the documentation engine. Replace changelog symlink with pymdownx.snippets include.
- Rewrite documentation for target-based architecture
(
e48b8e3)
Restructure docs to match new target-based config model. Migrate from mkdocs to zensical. Add guide, reference, and resources sections with new content.
-
Update CLAUDE.md and add CONTRIBUTING.md (
28d12f1) -
Update README, CLAUDE.md, and CONTRIBUTING.md (
0c48dff)
Features
-
Add --version flag, improve gc output (
2046a11) -
Add push conflict detection and --force flag (
3db0216) -
Track pushed_at timestamp in .fdl/meta.json (local + remote) - Push rejects when remote was updated since last pull - --force overrides conflict detection - Pull syncs meta.json from remote for next push check
-
Rewrite to target-based architecture (
a17d8ec)
Replace 3-layer config (env → workspace → user) with single fdl.toml and ${VAR} environment variable expansion.
BREAKING CHANGE: - remote renamed to target across all commands - FDL_ATTACH_PATH renamed to
FDL_CATALOG - gc command renamed to prune - fdl run now requires TARGET argument -
metadata command removed - create_destination() removed from ducklake module - User/workspace
config files (~/.fdl/config, .fdl/config) no longer read
New features: - fdl sql TARGET QUERY — execute SQL against the catalog - fdl init prompts
interactively for target config - S3Config dataclass for typed credential handling
Refactoring
-
Make load_toml strict and remove datasource_name fallback (
63ca8ed) -
load_toml → _load_toml: internal function, raises FileNotFoundError - set_value: catch FileNotFoundError for new file creation - datasource_name: require name in fdl.toml, no directory name fallback
-
Replace print with rich console output (
3958b6b) -
Replace prune with checkpoint command (
6433bcd) -
Use DuckLake CHECKPOINT statement via connect() context manager - Remove S3-only restriction, works on any target - Extract is_stale pure function and read_remote_pushed_at to meta.py - Add stale catalog check before maintenance (reuses push conflict detection)
-
Unify branding to Frozen DuckLake (
cd148ca) -
Rename "Frozen Data Lake" to "Frozen DuckLake" across all files - Normalize casing: FDL → fdl - Remove "Git-like" messaging - Update site_name to "fdl — Frozen DuckLake CLI"
Testing
- Add FDL_DATA_PATH injection test
(
2644ae9)
Verify fdl run injects FDL_DATA_PATH ending with ducklake.duckdb.files/
- Add integration and unit tests (89 tests)
(
1d01ffe)
Integration tests (tests/integration/): - test_init: default catalog, invalid name, existing toml, double init, rollback, sqlite - test_config: get, set, list-all, env var reference, s3 credentials, missing key, without init - test_sql: data persistence, invalid sql - test_run: env injection, no-overwrite, separator, exit code propagation - test_push: catalog copy, sqlite conversion, meta.json, conflict detection, force push - test_pull: catalog restore, meta.json sync, pull-then-push, empty target - test_prune: local target rejection, orphan deletion, dry-run - test_serve: CORS, range request, HEAD, OPTIONS, 404
Unit tests (tests/): - test_config: set_value, get_all, datasource_name, storage/data_path/ catalog_path, resolve_target, target_s3_config, ducklake_url, fdl_env_dict - test_init_module: ducklake_data_path, default_target_url - test_s3: S3Config endpoint_host - test_serve: CORSRangeHandler
-
Add Phase 1 pure function tests (
a118610) -
test_init_module.py: ducklake_data_path, default_target_url - test_s3.py: S3Config endpoint_host
-
Add S3 target integration tests for push and pull (
c6caf1d) -
push uploads catalog to S3 - push conflict detection on S3 - pull restores catalog from S3
v0.4.0 (2026-03-27)
Continuous Integration
- Add GitHub Pages deployment workflow
(
ebad8ba)
Build and deploy docs on push to main (docs/**, mkdocs.yml, CHANGELOG.md) and manual dispatch.
Documentation
-
Add MkDocs Material documentation site (
7f00206) -
mkdocs.yml with Material theme (light/dark, code copy, search) - 6 pages: index, quickstart, CLI reference, configuration, dlt integration, changelog - Changelog page symlinked to CHANGELOG.md - Update README.md with project overview and commands - Add site/ to .gitignore
-
Update GitHub Pages actions and set custom site URL (
5fe951b)
Features
- Add fdl gc command with --dry-run support
(
9e5d451)
fdl gc origin --dry-run # list orphaned files and sizes fdl gc origin # interactive deletion with confirmation fdl gc origin --force # delete without confirmation fdl gc origin --older-than 7 # only files older than 7 days
v0.3.3 (2026-03-27)
Bug Fixes
- Include fdl.toml in config value resolution
(
2e6c921)
_get_config_value now checks project config (fdl.toml) before workspace and user config, matching resolve_remote's 3-layer lookup.
v0.3.2 (2026-03-26)
Bug Fixes
- Expand env vars in remote URLs
(
38341e3)
resolve_remote() now calls os.path.expandvars() so that fdl.toml remotes like s3://${FDL_S3_BUCKET} are expanded.
v0.3.1 (2026-03-26)
Bug Fixes
- Remove .fdl/ pre-check from pull command
(
4e2023b)
pull.py already creates .fdl/ via mkdir(parents=True). The check broke CI where .fdl/ is gitignored and doesn't exist before pull.
v0.3.0 (2026-03-26)
Chores
- Bump frozen-ducklake to 0.2.1
(
b359533)
Features
- Require fdl init before pull, remove --sqlite fallback
(
b21369e)
pull no longer auto-initializes the catalog. Users must run fdl init first. Removes --sqlite option from pull.
-
Rewrite fdl init with fdl.toml scaffolding and rollback (
e5e6454) -
name argument required (like git init
) - generates fdl.toml with name and optional catalog type - auto-creates .gitignore entry for .fdl/ - rolls back fdl.toml and .fdl/ on failure - set_value now auto-detects top-level vs sectioned keys
Refactoring
- Move datasource/URL resolution from DatasetConfig to config module
(
6ff1ff2)
datasource_name(), public_url(), ducklake_url() are now in config.py with 3-layer resolution (env var → workspace → user config). DatasetConfig no longer holds public_url or ducklake_url.
-
Resolve storage in create_destination, add FDL_S3_ENDPOINT_HOST (
26bcec8) -
create_destination defaults to config.storage() instead of hardcoded .fdl - s3_env_dict now derives FDL_S3_ENDPOINT_HOST (scheme-less) for DuckDB
v0.2.1 (2026-03-26)
Bug Fixes
- Update outdated docstring in create_destination
(
ad31665)
Continuous Integration
- Merge publish into release workflow
(
22ba582)
GITHUB_TOKEN tags don't trigger other workflows. Run PyPI publish in the same job after semantic-release.
- Switch release to manual trigger (workflow_dispatch)
(
0186734)
v0.2.0 (2026-03-26)
Bug Fixes
Chores
- Update uv.lock
(
1331865)
Features
- Named remotes, fdl run/config/serve, FDL_* env vars
(
d749935)
BREAKING CHANGE: push/pull now require named remotes instead of URLs. s3_url removed from dataset.yml, replaced by fdl.toml remotes. DUCKLAKE_STORAGE renamed to FDL_STORAGE. S3 env vars prefixed with FDL_S3_.
Named remotes: - push/pull require explicit remote name (e.g. origin, local) - Remotes defined in fdl.toml (project), .fdl/config (workspace), ~/.fdl/config (user) - No built-in remotes; all user-defined via fdl config
fdl run: - Sets FDL_STORAGE, FDL_DATA_PATH, FDL_ATTACH_PATH for pipeline execution - S3 credentials loaded from config (env var → workspace → user) - Usage: fdl run [REMOTE] -- COMMAND
fdl config: - git config-like settings management - fdl config key value (user), fdl config --local key value (workspace)
fdl serve: - HTTP server with CORS + Range request support - Optional remote arg: fdl serve (project .fdl/) or fdl serve REMOTE
Other changes: - DIST_DIR renamed to FDL_DIR - S3 endpoint now stored with https:// scheme - gc.py: add OVERRIDE_DATA_PATH for v1.0 compatibility
Breaking Changes
- Push/pull now require named remotes instead of URLs. s3_url removed from dataset.yml, replaced by fdl.toml remotes. DUCKLAKE_STORAGE renamed to FDL_STORAGE. S3 env vars prefixed with FDL_S3_.