Pushing over HTTP

The requests a host answers so that writers can push without keys to its storage.

A host can take pushes over HTTP, so a writer needs no keys to its storage. The repository’s root is a URL such as https://data.example.org/towns/, and every key in the layout is a path under it.

Requests

• Writes carry Authorization: Bearer <token>. 401 without a usable token, 403 for one that may not write here
• Parts other than the last are the same size. A writer sends a file in parts when it is larger than one request may be (64 MB by default)
• A host decides which keys it takes. It may refuse writes to HEAD, fdl.json and anything outside the layout (but must answer HEAD requests, which a writer needs for a ref’s ETag), and take versions/ and data files only with If-None-Match: *
• A host that moves refs itself after its checks keeps refs.json too, and may refuse a writer’s write to it with 403. A writer treats that as done
• Moving a ref is where a host checks a push: the version is there, it is a fast-forward, the catalog matches version.json, and every data file it names is in the repository. A refusal is 409 with the reason in the body
• There is no DELETE. Retention and deletion are the host’s

Uploading straight to storage

A host may hand out upload URLs so that files go straight to its storage rather than through it, as Git LFS’s batch API does. Only files that are written once (data files and the files of a version) go this way. Refs are always written through the host.

POST <root>/?batch with the files a writer is about to upload:

{ "operation": "upload", "objects": [{ "key": "data/main/towns/a.parquet", "size": 1048576 }] }

The answer has one entry per object, in any order:

{
  "objects": [
    {
      "key": "data/main/towns/a.parquet",
      "size": 1048576,
      "actions": {
        "upload": { "href": "https://storage.example.org/…", "header": { "If-None-Match": "*" }, "expires_in": 3600 }
      }
    },
    { "key": "data/main/towns/b.parquet", "size": 2048 },
    { "key": "data/main/towns/c.parquet", "size": 9, "error": { "code": 413, "message": "Too large for one request" } }
  ]
}

• actions.upload: the writer sends PUT <href> with the file as the body, its Content-Type and the headers in header, and nothing from the repository’s own Authorization. 412 there means the object is already there. A writer that is about to use a URL past expires_in asks for that object again
• No actions and no error: the object is already there
• error with code 413: the host will not sign this one. The writer uploads it through the host (PUT, or in parts)
• Any other error: the upload is refused, with the code and the reason
• A writer asks about at most 100 objects at a time
• A host without batches answers 404, 405 or 501, and writers upload through the host
• Uploading does not make a version. Moving a ref is still where the host checks the push

Writing to it

DuckDB cannot write data files to an HTTP URL, so a working catalog of such a repository keeps two places:

• Files already in the repository stay where they are. Checking out turns their references into absolute URLs under the repository’s data_dir, so they are read over HTTP
• New files are written to a folder on the writer’s machine, the working catalog’s data path

Committing turns references under the repository’s data_dir back into relative ones, and the new files are relative already. Pushing uploads the new files the repository lacks before the catalog.