# FDL

> Versions of a DuckLake catalog on object storage, kept the way Git keeps commits

## Docs

- [FDL](https://fdl.flo8s.com/): Versions of a DuckLake catalog on object storage, kept the way Git keeps commits.
- [Getting started](https://fdl.flo8s.com/getting-started): Make a repository, write a first version with DuckDB, and read it back.

## Concepts

- [Repositories](https://fdl.flo8s.com/concepts/repositories): What a repository is, what it needs from the store it lives in, and what it leaves to the host.
- [Versions and refs](https://fdl.flo8s.com/concepts/versions-and-refs): A version is a frozen catalog with a record; a ref is a name that points at one. How they relate to DuckLake snapshots and to Git commits.
- [Working catalogs](https://fdl.flo8s.com/concepts/working-catalogs): Where writing happens: a DuckLake catalog in SQLite, checked out from a version and frozen into the next one.

## How it works

- [From write to read](https://fdl.flo8s.com/internals/from-write-to-read): One change followed from a working catalog to a reader, with the requests at each step, and what each part of FDL is in Git.
- [Inside a repository](https://fdl.flo8s.com/internals/inside-a-repository): The files a repository holds after each step, from init to a tag and a retention run, and the order they are written in.
- [DuckLake formats](https://fdl.flo8s.com/internals/ducklake-formats): The three format numbers a version carries, and how a repository serves readers on older DuckDB releases when DuckLake's format moves.

## Guides

- [Stores](https://fdl.flo8s.com/guides/stores): Using a local folder, S3, R2 or another S3-compatible store as a repository, and what credentials the fdl command reads.
- [Serving over HTTP](https://fdl.flo8s.com/guides/serving-over-http): Letting readers attach a repository from a plain HTTP server or a CDN, and the caching each kind of file allows.
- [Mirrors](https://fdl.flo8s.com/guides/mirrors): Copying a repository to another host, keeping the copy up to date, and reading from it.
- [Reading without DuckDB](https://fdl.flo8s.com/guides/reading-without-duckdb): Reading a table's Parquet files from JavaScript with files.json, in Workers, browsers and Node.
- [Public and private data](https://fdl.flo8s.com/guides/public-and-private): Publishing some tables of a dataset and keeping the rest private, with two repositories.
- [Building a host](https://fdl.flo8s.com/guides/building-a-host): Using the fdl Python package in a service that stores repositories for others, approves changes and moves refs itself.

## Specification

- [Specification](https://fdl.flo8s.com/spec): The normative description of FDL 0.1 (draft) - what it decides, what it leaves to hosts, and the terms it uses.
- [Layout](https://fdl.flo8s.com/spec/layout): The files of a repository, their paths, and the content of fdl.json.
- [Versions](https://fdl.flo8s.com/spec/versions): The version ID, the fields of version.json, the frozen catalog, and catalogs in other DuckLake formats.
- [Refs](https://fdl.flo8s.com/spec/refs): The content of HEAD and refs, the names they may take, and the conditions for creating and moving them.
- [Data files](https://fdl.flo8s.com/spec/data-files): Rules for data files and source files, when a data file may be deleted, and how a version is taken out of retention.
- [Reading](https://fdl.flo8s.com/spec/reading): How a reader resolves a ref, checks a version and attaches its catalog, and what may be cached.
- [Writing](https://fdl.flo8s.com/spec/writing): Checkout, commit and push - the working catalog, the order of writes, and when a push is refused.
- [Pushing over HTTP](https://fdl.flo8s.com/spec/http): The requests a host answers so that writers can push without keys to its storage.

## Reference

- [Command line](https://fdl.flo8s.com/reference/cli): The fdl commands, their options and when they refuse.
- [Python API](https://fdl.flo8s.com/reference/python): The fdl package - stores, refs, and the repository operations without a terminal.

## 日本語

### Docs

- [FDL](https://fdl.flo8s.com/ja): DuckLake のカタログのバージョンを、Git がコミットを残すのと同じ形でオブジェクトストレージに残します。
- [はじめかた](https://fdl.flo8s.com/ja/getting-started): リポジトリを作り、DuckDB で最初のバージョンを書いて、読み返します。

### 概念

- [リポジトリ](https://fdl.flo8s.com/ja/concepts/repositories): リポジトリとは何か、置き場に何が求められるか、何をホストに任せるか。
- [バージョンと ref](https://fdl.flo8s.com/ja/concepts/versions-and-refs): バージョンは固めたカタログと記録、ref はバージョンを指す名前です。DuckLake のスナップショットや Git のコミットとの関係も説明します。
- [作業カタログ](https://fdl.flo8s.com/ja/concepts/working-catalogs): 書き込みをする場所です。SQLite の DuckLake カタログで、バージョンからチェックアウトし、次のバージョンに固めます。

### 仕組み

- [書いてから読まれるまで](https://fdl.flo8s.com/ja/internals/from-write-to-read): 1 つの変更を作業カタログから読み手まで追い、各段階のリクエストと、FDL の各部分が Git の何に当たるかを示します。
- [リポジトリの中身](https://fdl.flo8s.com/ja/internals/inside-a-repository): init からタグと保持期間の処理までの各段階で、リポジトリにどのファイルがどの順に書かれるか。
- [DuckLake の形式](https://fdl.flo8s.com/ja/internals/ducklake-formats): バージョンが持つ 3 つの形式番号と、DuckLake の形式が上がったときに古い DuckDB の読み手へどう届けるか。

### ガイド

- [置き場](https://fdl.flo8s.com/ja/guides/stores): フォルダ、S3、R2、ほかの S3 互換ストレージをリポジトリの置き場にする方法と、fdl コマンドが読む認証情報。
- [HTTP で配信する](https://fdl.flo8s.com/ja/guides/serving-over-http): 普通の HTTP サーバーや CDN からリポジトリを ATTACH してもらう方法と、ファイルの種類ごとのキャッシュ。
- [ミラー](https://fdl.flo8s.com/ja/guides/mirrors): リポジトリを別のホストに写し、最新に保ち、写しから読む方法。
- [DuckDB を使わずに読む](https://fdl.flo8s.com/ja/guides/reading-without-duckdb): files.json を使い、表の Parquet ファイルを JavaScript で読む方法。Workers・ブラウザ・Node で動きます。
- [公開と非公開](https://fdl.flo8s.com/ja/guides/public-and-private): データセットの一部の表だけを公開し、残りを非公開にするには、リポジトリを 2 つにします。
- [ホストを作る](https://fdl.flo8s.com/ja/guides/building-a-host): ほかの人のリポジトリを預かり、変更を承認し、ref を自分で動かすサービスで fdl Python パッケージを使う方法。

### 仕様

- [仕様](https://fdl.flo8s.com/ja/spec): FDL 0.1（草案）の規範となる説明。何を決め、何をホストに任せ、どんな用語を使うか。
- [配置](https://fdl.flo8s.com/ja/spec/layout): リポジトリのファイル、そのパス、fdl.json の中身。
- [バージョン](https://fdl.flo8s.com/ja/spec/versions): バージョン ID、version.json の項目、固めたカタログ、dataset.json、ほかの DuckLake 形式のカタログ。
- [ref](https://fdl.flo8s.com/ja/spec/refs): HEAD と ref の中身、付けてよい名前、作るときと動かすときの条件。
- [データファイル](https://fdl.flo8s.com/ja/spec/data-files): データファイルと元のファイルの決まり、データファイルを消してよいとき、保持期間を終わらせる手順と取り下げる手順。
- [読む](https://fdl.flo8s.com/ja/spec/reading): 読み手が ref をたどり、バージョンを確かめ、カタログを ATTACH する手順と、キャッシュしてよいもの。
- [書く](https://fdl.flo8s.com/ja/spec/writing): チェックアウト、コミット、push。作業カタログ、書き込みの順番、push が断られるとき。
- [HTTP での push](https://fdl.flo8s.com/ja/spec/http): 書き手が置き場の鍵を持たずに push できるよう、ホストが応えるリクエスト。

### リファレンス

- [コマンドライン](https://fdl.flo8s.com/ja/reference/cli): fdl のコマンド、オプション、断られるとき。
- [Python API](https://fdl.flo8s.com/ja/reference/python): fdl パッケージの置き場、ref、端末を使わないリポジトリの操作。

## Agent skills

- [fdl](https://fdl.flo8s.com/.well-known/agent-skills/fdl/SKILL.md): Read the FDL docs instead of answering from memory. Versions of a DuckLake catalog on object storage, kept the way Git keeps commits Use when a task involves FDL: how it works, how to set it up or configure it, or its API.

## Agent resources

- [llms-full.txt](https://fdl.flo8s.com/llms-full.txt): The full Markdown of every page in one file.
- [Page Markdown](https://fdl.flo8s.com/index.md): Append `.md` to any page URL to fetch that page as raw Markdown.
- [JSON API](https://fdl.flo8s.com/api/docs/pages.json): Page index of the JSON docs API; each entry links the page's JSON and Markdown forms. Described by the OpenAPI document at https://fdl.flo8s.com/openapi.json.
- [Agent skills](https://fdl.flo8s.com/.well-known/agent-skills/index.json): Agent Skills discovery index of the skills this site publishes.
- [API catalog](https://fdl.flo8s.com/.well-known/api-catalog): RFC 9727 linkset of the APIs documented here.
- [AI catalog](https://fdl.flo8s.com/.well-known/ai-catalog.json): ARD manifest of the agent-facing resources on this site (MCP server, skills, APIs).
- [agent-readability.json](https://fdl.flo8s.com/agent-readability.json): Manifest of every agent-facing artifact on this site.
- [Sitemap](https://fdl.flo8s.com/sitemap.xml): Every indexable page URL with its last-modified date.