mirror of
https://github.com/marcoallegretti/karapace.git
synced 2026-03-26 21:43:09 +00:00
docs: rewrite all documentation from source code
Delete 14 old docs files (AI-generated, riddled with Phase/M1/1.0 jargon, references to non-existent commands, stale CI snippets). New documentation (6 files), written from repository source analysis: - docs/architecture.md — crate graph, engine lifecycle, identity computation, runtime backends, store design, WAL, GC, unsafe blocks - docs/cli-reference.md — all 23 commands with syntax, args, flags, exit codes, env vars, verified against crates/karapace-cli/src/main.rs - docs/storage-format.md — directory layout, objects, layers, metadata, manifest format, lock file, WAL, atomic write contract - docs/security-model.md — mount/device/env var policies with exact defaults from security.rs, trust assumptions, what is NOT protected - docs/build-and-reproducibility.md — CI env vars, RUSTFLAGS, cargo profile, reproducibility verification, toolchain pinning - docs/contributing.md — setup, verification, project layout, code standards, testing, CI workflows README.md rewritten: concise, no marketing language, prerequisites first, usage example, command table, limitations section. CONTRIBUTING.md now points to docs/contributing.md. CHANGELOG.md cleaned: removed M1-M8 labels, Phase refs, stale counts.
This commit is contained in:
parent
864d5c45f6
commit
e6e0f3dd6d
21 changed files with 1047 additions and 1898 deletions
26
CHANGELOG.md
26
CHANGELOG.md
|
|
@ -4,7 +4,7 @@ All notable changes to Karapace will be documented in this file.
|
||||||
|
|
||||||
The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/).
|
The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/).
|
||||||
|
|
||||||
## [Unreleased] — 2.0 Hardening
|
## [Unreleased]
|
||||||
|
|
||||||
### Breaking Changes
|
### Breaking Changes
|
||||||
|
|
||||||
|
|
@ -14,19 +14,15 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/).
|
||||||
- **`Engine::gc()` requires `&StoreLock`** — compile-time enforcement that callers hold the store lock before garbage collection.
|
- **`Engine::gc()` requires `&StoreLock`** — compile-time enforcement that callers hold the store lock before garbage collection.
|
||||||
- **MetadataStore checksum** — `EnvMetadata` now includes an optional `checksum` field (blake3). Written on every `put()`, verified on every `get()`. Backward-compatible via `serde(default)`.
|
- **MetadataStore checksum** — `EnvMetadata` now includes an optional `checksum` field (blake3). Written on every `put()`, verified on every `get()`. Backward-compatible via `serde(default)`.
|
||||||
|
|
||||||
### Added — 2.0 Hardening (M1–M8)
|
### Added
|
||||||
|
|
||||||
- **M1: WAL Crash Safety** — Fixed race windows in `build()` and `restore()` (rollback registered before side-effects). Added WAL protection to `destroy()`, `commit()` (layer manifest rollback), and `gc()` (WAL marker). 8 new WAL crash-safety tests.
|
|
||||||
- **M2: Integrity Hardening** — `LayerStore::get()` verifies blake3 hash on every read. `MetadataStore` embeds and verifies blake3 checksum. `verify_store_integrity()` expanded to check objects, layers, and metadata. 4 new integrity tests.
|
|
||||||
- **M3: GC Safety** — `Engine::gc()` now requires `&StoreLock` parameter (type-enforced). Snapshot layers whose parent is a live base layer are preserved during GC.
|
|
||||||
- **M4: Remote Protocol** — `X-Karapace-Protocol: 1` header sent on all HTTP backend requests (PUT, GET, HEAD). `PROTOCOL_VERSION` constant exported from `karapace-remote`. 4 new header/auth verification tests via header-capturing mock server.
|
|
||||||
- **M5: unwrap() Audit** — 0 `unwrap()` in production `src/` code. 4 `Mutex::lock().unwrap()` calls in `MockBackend` replaced with proper `RuntimeError` propagation.
|
|
||||||
- **M6: Failure Mode Testing** — 11 new tests: WAL write failure on read-only dir, build failure when WAL dir is read-only (disk-full simulation), stop() SIGTERM with real process (ESRCH path), stop() with non-existent PID, permission denied on object read, read-only metadata dir, concurrent GC lock contention, layer corruption detection, metadata corruption detection, destroy nonexistent env, invalid manifest.
|
|
||||||
- **M7: Coverage Expansion** — `verify_store_integrity()` now checks objects + layers + metadata (was objects-only). `IntegrityReport` expanded with `layers_checked/passed` and `metadata_checked/passed` fields. New tests: freeze/archive state transitions, rename environment, verify-store after fresh build, rebuild preserves new and cleans old, HTTP list_blobs, large (1MB) blob roundtrip.
|
|
||||||
- **Total: 417 tests** (24 ignored, require privileged operations). Clippy `-D warnings` clean. `cargo fmt` clean. Release build OK.
|
|
||||||
|
|
||||||
### Added — 1.0 Preparation
|
|
||||||
|
|
||||||
|
- **WAL crash safety** — Fixed race windows in `build()` and `restore()` (rollback registered before side-effects). Added WAL protection to `destroy()`, `commit()` (layer manifest rollback), and `gc()` (WAL marker).
|
||||||
|
- **Integrity hardening** — `LayerStore::get()` verifies blake3 hash on every read. `MetadataStore` embeds and verifies blake3 checksum. `verify_store_integrity()` expanded to check objects, layers, and metadata.
|
||||||
|
- **GC safety** — `Engine::gc()` now requires `&StoreLock` parameter (type-enforced). Snapshot layers whose parent is a live base layer are preserved during GC.
|
||||||
|
- **Remote protocol headers** — `X-Karapace-Protocol: 1` header sent on all HTTP backend requests (PUT, GET, HEAD). `PROTOCOL_VERSION` constant exported from `karapace-remote`.
|
||||||
|
- **unwrap() audit** — 0 `unwrap()` in production code. `Mutex::lock().unwrap()` calls in `MockBackend` replaced with proper `RuntimeError` propagation.
|
||||||
|
- **Failure mode tests** — WAL write failure, build on read-only WAL dir, stop() with SIGTERM/non-existent PID, permission denied on object read, read-only metadata dir, concurrent GC lock contention, layer/metadata corruption detection.
|
||||||
|
- **Coverage expansion** — `verify_store_integrity()` now checks objects + layers + metadata. `IntegrityReport` expanded with `layers_checked/passed` and `metadata_checked/passed` fields.
|
||||||
- **Real tar layers** — `pack_layer()`/`unpack_layer()` in karapace-store: deterministic tar creation (sorted entries, zero timestamps, owner 0:0) for regular files, directories, and symlinks. Content-addressed via blake3.
|
- **Real tar layers** — `pack_layer()`/`unpack_layer()` in karapace-store: deterministic tar creation (sorted entries, zero timestamps, owner 0:0) for regular files, directories, and symlinks. Content-addressed via blake3.
|
||||||
- **Snapshot system** — `Engine::commit()` captures overlay upper as a tar snapshot; `Engine::restore()` atomically unpacks a snapshot via staging directory swap; `Engine::list_snapshots()` lists snapshots for an environment.
|
- **Snapshot system** — `Engine::commit()` captures overlay upper as a tar snapshot; `Engine::restore()` atomically unpacks a snapshot via staging directory swap; `Engine::list_snapshots()` lists snapshots for an environment.
|
||||||
- **CLI: `snapshots` and `restore`** — new commands for snapshot management.
|
- **CLI: `snapshots` and `restore`** — new commands for snapshot management.
|
||||||
|
|
@ -34,9 +30,9 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/).
|
||||||
- **Newtype wrappers threaded through all structs** — `EnvId`, `ShortId`, `ObjectHash`, `LayerHash` now used in `EnvMetadata` across all 8 crates. Transparent serde for backward compatibility.
|
- **Newtype wrappers threaded through all structs** — `EnvId`, `ShortId`, `ObjectHash`, `LayerHash` now used in `EnvMetadata` across all 8 crates. Transparent serde for backward compatibility.
|
||||||
- **Engine::push/pull** — transfer logic moved from `karapace-remote` to `Engine` methods. `karapace-remote` is now pure I/O.
|
- **Engine::push/pull** — transfer logic moved from `karapace-remote` to `Engine` methods. `karapace-remote` is now pure I/O.
|
||||||
- **CoreError::Remote** — new error variant for remote operation failures.
|
- **CoreError::Remote** — new error variant for remote operation failures.
|
||||||
- **CLI stability contract** — `docs/cli-stability.md` defines stable command signatures for 1.x.
|
- **CLI stability contract** — `docs/cli-stability.md` defines CLI stability expectations.
|
||||||
- **Remote protocol spec** — `docs/protocol-v1.md` (v1-draft) documents blob store routes, push/pull protocol, registry format.
|
- **Remote protocol spec** — `docs/protocol-v1.md` (v1-draft) documents blob store routes, push/pull protocol, registry format.
|
||||||
- **Layer limitations doc** — `docs/layer-limitations-v1.md` documents Phase 1 limits (no xattrs, device nodes, hardlinks).
|
- **Layer limitations doc** — `docs/layer-limitations.md` documents current limits (no xattrs, device nodes, hardlinks).
|
||||||
|
|
||||||
### Changed
|
### Changed
|
||||||
|
|
||||||
|
|
|
||||||
|
|
@ -1,97 +1,5 @@
|
||||||
# Contributing to Karapace
|
# Contributing
|
||||||
|
|
||||||
## Development Setup
|
See [docs/contributing.md](docs/contributing.md) for setup, code standards, testing, and CI details.
|
||||||
|
|
||||||
```bash
|
By contributing, you agree that your contributions will be licensed under EUPL-1.2.
|
||||||
# Clone and build
|
|
||||||
git clone https://github.com/marcoallegretti/karapace.git
|
|
||||||
cd karapace
|
|
||||||
cargo build
|
|
||||||
|
|
||||||
# Run tests
|
|
||||||
cargo test --workspace
|
|
||||||
|
|
||||||
# Full verification (must pass before submitting)
|
|
||||||
cargo fmt --all --check
|
|
||||||
cargo clippy --workspace --all-targets -- -D warnings
|
|
||||||
cargo test --workspace
|
|
||||||
cargo build --release --workspace
|
|
||||||
```
|
|
||||||
|
|
||||||
## Project Structure
|
|
||||||
|
|
||||||
```
|
|
||||||
crates/
|
|
||||||
karapace-schema/ # Manifest parsing, normalization, lock file, identity hashing
|
|
||||||
karapace-store/ # Content-addressable store, metadata, layers, GC, integrity
|
|
||||||
karapace-runtime/ # Container runtime: images, sandbox, host integration, security
|
|
||||||
karapace-core/ # Build engine, lifecycle state machine, drift control, concurrency
|
|
||||||
karapace-remote/ # Remote store client, push/pull, registry
|
|
||||||
karapace-server/ # Reference remote server (tiny_http)
|
|
||||||
karapace-tui/ # Terminal UI (ratatui)
|
|
||||||
karapace-cli/ # CLI interface (23 commands)
|
|
||||||
karapace-dbus/ # D-Bus desktop integration (optional)
|
|
||||||
docs/ # Public documentation and specifications
|
|
||||||
examples/ # Ready-to-use manifest examples
|
|
||||||
data/ # systemd and D-Bus service files
|
|
||||||
```
|
|
||||||
|
|
||||||
## Architecture Principles
|
|
||||||
|
|
||||||
Before implementing any feature, verify it aligns with these principles:
|
|
||||||
|
|
||||||
1. **Determinism first.** Same manifest + lock = identical environment, always.
|
|
||||||
2. **No hidden mutable state.** All state changes are explicit and tracked.
|
|
||||||
3. **No silent drift.** Overlay changes are visible via `diff` and must be committed explicitly.
|
|
||||||
4. **No privilege escalation.** Everything runs as the unprivileged user.
|
|
||||||
5. **Convenience must not break reproducibility.** If there's a conflict, determinism wins.
|
|
||||||
|
|
||||||
See [Architecture Overview](docs/architecture.md) for the full design.
|
|
||||||
|
|
||||||
## Code Standards
|
|
||||||
|
|
||||||
- **Zero warnings**: `cargo clippy --workspace --all-targets -- -D warnings` must pass.
|
|
||||||
- **Formatted**: `cargo fmt --all --check` must pass.
|
|
||||||
- **No `unwrap()` in production code** (test code is fine).
|
|
||||||
- **No `TODO`/`FIXME`/`HACK`** in committed code.
|
|
||||||
- **All values interpolated into shell scripts must use `shell_quote()`.**
|
|
||||||
- **All mutating operations must hold a `StoreLock`.**
|
|
||||||
- **All file writes must be atomic** (use `NamedTempFile` + `persist()`).
|
|
||||||
|
|
||||||
## Testing
|
|
||||||
|
|
||||||
- All new features must include tests.
|
|
||||||
- Run the full test suite: `cargo test --workspace`
|
|
||||||
- Integration tests go in `crates/karapace-core/tests/`.
|
|
||||||
- Unit tests go in the relevant module as `#[cfg(test)] mod tests`.
|
|
||||||
|
|
||||||
## Submitting Changes
|
|
||||||
|
|
||||||
1. Fork the repository.
|
|
||||||
2. Create a feature branch from `main`.
|
|
||||||
3. Make your changes, ensuring all verification checks pass.
|
|
||||||
4. Submit a pull request with a clear description.
|
|
||||||
|
|
||||||
## Building the D-Bus Service
|
|
||||||
|
|
||||||
The D-Bus service is not compiled by default:
|
|
||||||
|
|
||||||
```bash
|
|
||||||
# Core CLI only (default)
|
|
||||||
cargo build --release
|
|
||||||
|
|
||||||
# Include D-Bus service
|
|
||||||
cargo build --release --workspace
|
|
||||||
```
|
|
||||||
|
|
||||||
## Generating Shell Completions
|
|
||||||
|
|
||||||
```bash
|
|
||||||
karapace completions bash > /etc/bash_completion.d/karapace
|
|
||||||
karapace completions zsh > /usr/share/zsh/site-functions/_karapace
|
|
||||||
karapace completions fish > ~/.config/fish/completions/karapace.fish
|
|
||||||
```
|
|
||||||
|
|
||||||
## License
|
|
||||||
|
|
||||||
By contributing, you agree that your contributions will be licensed under the European Union Public Licence v1.2 (EUPL-1.2).
|
|
||||||
|
|
|
||||||
290
README.md
290
README.md
|
|
@ -3,165 +3,20 @@
|
||||||
[](https://github.com/marcoallegretti/karapace/actions/workflows/ci.yml)
|
[](https://github.com/marcoallegretti/karapace/actions/workflows/ci.yml)
|
||||||
[](LICENSE)
|
[](LICENSE)
|
||||||
|
|
||||||
A deterministic container environment engine for immutable Linux systems.
|
Deterministic, content-addressed container environments for Linux. No root. No daemon.
|
||||||
|
|
||||||
Karapace creates isolated, reproducible development environments using Linux namespaces and overlay filesystems — no root, no daemon, no Docker. Environments are content-addressed artifacts derived from declarative TOML manifests, with full host integration for GPU, audio, Wayland, and desktop applications.
|
Karapace builds isolated development environments from declarative TOML manifests using Linux user namespaces and overlay filesystems. Each environment gets a unique identity (blake3 hash) derived from its fully resolved dependencies.
|
||||||
|
|
||||||
## What Karapace Is (and Isn't)
|
## Prerequisites
|
||||||
|
|
||||||
Karapace is **the identity and certainty layer** for reproducible environments. It is not a general-purpose container runtime.
|
- Linux with user namespaces (`CONFIG_USER_NS=y`)
|
||||||
|
- `fuse-overlayfs`
|
||||||
|
- `curl`
|
||||||
|
- Optional: `crun`/`runc`/`youki` (OCI backend)
|
||||||
|
|
||||||
| Need | Tool |
|
Run `karapace doctor` to check.
|
||||||
|---|---|
|
|
||||||
| Full system container lifecycle, advanced networking, snapshots | Incus, LXD, Podman |
|
|
||||||
| Deterministic, content-addressed, reproducible environments | **Karapace** |
|
|
||||||
| Quick disposable containers with zero config | Distrobox |
|
|
||||||
| Determinism + simplicity for casual use | **Karapace** `quick` command |
|
|
||||||
|
|
||||||
Karapace is **complementary** — it can sit on top of or alongside any container runtime. Users needing full container features use a runtime; users needing reproducibility and traceability use Karapace. The `quick` command bridges the gap for users who want both simplicity and determinism.
|
## Install
|
||||||
|
|
||||||
## Features
|
|
||||||
|
|
||||||
- **Real container isolation** — Linux user namespaces (`unshare`), `fuse-overlayfs`, `chroot`
|
|
||||||
- **No root required** — runs entirely as an unprivileged user
|
|
||||||
- **No daemon** — direct CLI, no background service needed
|
|
||||||
- **Multi-distro images** — openSUSE, Ubuntu, Debian, Fedora, Arch from LXC image servers
|
|
||||||
- **Package installation** — `zypper`, `apt`, `dnf`, `pacman` inside the container
|
|
||||||
- **Host integration** — home directory, Wayland, PipeWire, D-Bus, GPU (`/dev/dri`), audio (`/dev/snd`), SSH agent, fonts, themes
|
|
||||||
- **Desktop app export** — export GUI apps as `.desktop` files on the host
|
|
||||||
- **OCI runtime support** — optional `crun`/`runc`/`youki` backend
|
|
||||||
- **Content-addressable store** — deterministic hashing, deduplication, integrity verification
|
|
||||||
- **Overlay drift control** — diff, freeze, commit, export writable layer changes
|
|
||||||
- **OSC 777 terminal markers** — container-aware terminal integration (Konsole, etc.)
|
|
||||||
|
|
||||||
## Crate Layout
|
|
||||||
|
|
||||||
| Crate | Purpose |
|
|
||||||
|---|---|
|
|
||||||
| `karapace-schema` | Manifest v1 parsing, normalization, identity hashing, lock file |
|
|
||||||
| `karapace-store` | Content-addressable store, layers, metadata, GC, integrity |
|
|
||||||
| `karapace-runtime` | Container runtime: image download, sandbox, host integration, app export |
|
|
||||||
| `karapace-core` | Build engine, lifecycle state machine, drift control, concurrency |
|
|
||||||
| `karapace-cli` | Full CLI interface (23 commands) |
|
|
||||||
| `karapace-dbus` | Socket-activated D-Bus desktop integration (**optional**, feature-gated) |
|
|
||||||
| `karapace-remote` | Remote content-addressable store, push/pull, registry |
|
|
||||||
| `karapace-server` | Reference remote server implementing protocol v1 (tiny_http) |
|
|
||||||
| `karapace-tui` | Terminal UI for environment management (ratatui) |
|
|
||||||
|
|
||||||
## CLI Commands (23)
|
|
||||||
|
|
||||||
```
|
|
||||||
karapace build [manifest] # Build environment from manifest
|
|
||||||
karapace rebuild [manifest] # Destroy + rebuild
|
|
||||||
karapace enter <env_id> [-- cmd...] # Enter environment (or run a command)
|
|
||||||
karapace exec <env_id> -- <cmd...> # Execute command inside environment
|
|
||||||
karapace destroy <env_id> # Destroy environment
|
|
||||||
karapace stop <env_id> # Stop a running environment
|
|
||||||
karapace freeze <env_id> # Freeze environment
|
|
||||||
karapace archive <env_id> # Archive environment (preserve, prevent entry)
|
|
||||||
karapace list # List all environments
|
|
||||||
karapace inspect <env_id> # Show environment metadata
|
|
||||||
karapace diff <env_id> # Show overlay drift
|
|
||||||
karapace snapshots <env_id> # List snapshots for an environment
|
|
||||||
karapace commit <env_id> # Commit overlay drift as snapshot
|
|
||||||
karapace restore <env_id> <snapshot> # Restore overlay from snapshot
|
|
||||||
karapace gc [--dry-run] # Garbage collect store
|
|
||||||
karapace verify-store # Check store integrity
|
|
||||||
karapace push <env_id> [--tag name@tag] # Push environment to remote store
|
|
||||||
karapace pull <reference> [--remote url] # Pull environment from remote store
|
|
||||||
karapace rename <env_id> <name> # Rename environment
|
|
||||||
karapace doctor # Run diagnostic checks on system and store
|
|
||||||
karapace migrate # Check store version and migration guidance
|
|
||||||
karapace completions <shell> # Generate shell completions
|
|
||||||
karapace man-pages [dir] # Generate man pages
|
|
||||||
```
|
|
||||||
|
|
||||||
All commands support `--json` for structured output, `--store <path>` for custom store location, `--verbose` / `-v` for debug logging, and `--trace` for trace-level output.
|
|
||||||
|
|
||||||
Set `KARAPACE_LOG=debug` (or `info`, `warn`, `error`, `trace`) for fine-grained log control.
|
|
||||||
|
|
||||||
## Quick Start
|
|
||||||
|
|
||||||
```bash
|
|
||||||
# Build (core CLI only — D-Bus service is opt-in)
|
|
||||||
cargo build --release
|
|
||||||
|
|
||||||
# Build with D-Bus desktop integration
|
|
||||||
cargo build --release --workspace
|
|
||||||
```
|
|
||||||
|
|
||||||
```bash
|
|
||||||
# Write a manifest
|
|
||||||
cat > karapace.toml << 'EOF'
|
|
||||||
manifest_version = 1
|
|
||||||
|
|
||||||
[base]
|
|
||||||
image = "rolling" # openSUSE Tumbleweed (or "ubuntu/24.04", "fedora/41", "arch", etc.)
|
|
||||||
|
|
||||||
[system]
|
|
||||||
packages = ["git", "curl", "vim"]
|
|
||||||
|
|
||||||
[hardware]
|
|
||||||
gpu = true
|
|
||||||
audio = true
|
|
||||||
|
|
||||||
[runtime]
|
|
||||||
backend = "namespace"
|
|
||||||
EOF
|
|
||||||
|
|
||||||
karapace build
|
|
||||||
karapace enter <env_id>
|
|
||||||
|
|
||||||
# Run a command inside without interactive shell
|
|
||||||
karapace exec <env_id> -- git --version
|
|
||||||
|
|
||||||
# Snapshot and restore
|
|
||||||
karapace commit <env_id>
|
|
||||||
karapace snapshots <env_id>
|
|
||||||
karapace restore <env_id> <snapshot_hash>
|
|
||||||
|
|
||||||
# Push/pull to remote
|
|
||||||
karapace push <env_id> --tag my-env@latest
|
|
||||||
karapace pull my-env@latest
|
|
||||||
|
|
||||||
# List environments
|
|
||||||
karapace list
|
|
||||||
```
|
|
||||||
|
|
||||||
## Example Manifests
|
|
||||||
|
|
||||||
Ready-to-use manifests in `examples/`:
|
|
||||||
|
|
||||||
| File | Description |
|
|
||||||
|---|---|
|
|
||||||
| `examples/minimal.toml` | Bare openSUSE system, no extras |
|
|
||||||
| `examples/dev.toml` | Developer tools (git, vim, tmux, gcc, clang) |
|
|
||||||
| `examples/gui-dev.toml` | GUI development with GPU + audio passthrough |
|
|
||||||
| `examples/ubuntu-dev.toml` | Ubuntu-based with Node.js, Python, build-essential |
|
|
||||||
| `examples/rust-dev.toml` | Rust development environment |
|
|
||||||
|
|
||||||
## Shell Completions
|
|
||||||
|
|
||||||
```bash
|
|
||||||
# Bash
|
|
||||||
karapace completions bash > /etc/bash_completion.d/karapace
|
|
||||||
|
|
||||||
# Zsh
|
|
||||||
karapace completions zsh > /usr/share/zsh/site-functions/_karapace
|
|
||||||
|
|
||||||
# Fish
|
|
||||||
karapace completions fish > ~/.config/fish/completions/karapace.fish
|
|
||||||
```
|
|
||||||
|
|
||||||
## Man Pages
|
|
||||||
|
|
||||||
```bash
|
|
||||||
karapace man-pages /usr/share/man/man1
|
|
||||||
```
|
|
||||||
|
|
||||||
## Installation
|
|
||||||
|
|
||||||
### From Source (recommended)
|
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
git clone https://github.com/marcoallegretti/karapace.git
|
git clone https://github.com/marcoallegretti/karapace.git
|
||||||
|
|
@ -170,53 +25,106 @@ cargo build --release
|
||||||
sudo install -Dm755 target/release/karapace /usr/local/bin/karapace
|
sudo install -Dm755 target/release/karapace /usr/local/bin/karapace
|
||||||
```
|
```
|
||||||
|
|
||||||
### With D-Bus Service
|
Or via cargo:
|
||||||
|
|
||||||
```bash
|
|
||||||
cargo build --release --workspace
|
|
||||||
sudo install -Dm755 target/release/karapace /usr/local/bin/karapace
|
|
||||||
sudo install -Dm755 target/release/karapace-dbus /usr/local/bin/karapace-dbus
|
|
||||||
sudo install -Dm644 data/dbus/org.karapace.Manager1.service /usr/share/dbus-1/services/
|
|
||||||
sudo install -Dm644 data/systemd/karapace-dbus.service /usr/lib/systemd/user/
|
|
||||||
```
|
|
||||||
|
|
||||||
### Via Cargo
|
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
cargo install --git https://github.com/marcoallegretti/karapace.git karapace-cli
|
cargo install --git https://github.com/marcoallegretti/karapace.git karapace-cli
|
||||||
```
|
```
|
||||||
|
|
||||||
## Prerequisites
|
## Usage
|
||||||
|
|
||||||
- Linux with user namespace support (`CONFIG_USER_NS=y`)
|
```bash
|
||||||
- `fuse-overlayfs` (for overlay filesystem)
|
# Create a manifest
|
||||||
- `curl` (for image downloads)
|
cat > karapace.toml << 'EOF'
|
||||||
- Optional: `crun`/`runc`/`youki` (for OCI backend)
|
manifest_version = 1
|
||||||
|
|
||||||
Karapace checks for missing prerequisites at startup and provides distro-specific install instructions.
|
[base]
|
||||||
|
image = "rolling"
|
||||||
|
|
||||||
|
[system]
|
||||||
|
packages = ["git", "curl"]
|
||||||
|
|
||||||
|
[runtime]
|
||||||
|
backend = "namespace"
|
||||||
|
EOF
|
||||||
|
|
||||||
|
# Build and enter
|
||||||
|
karapace build
|
||||||
|
karapace enter <env_id>
|
||||||
|
|
||||||
|
# Run a command
|
||||||
|
karapace exec <env_id> -- git --version
|
||||||
|
|
||||||
|
# Snapshot
|
||||||
|
karapace commit <env_id>
|
||||||
|
karapace restore <env_id> <snapshot_hash>
|
||||||
|
```
|
||||||
|
|
||||||
|
See `examples/` for more manifests: `minimal.toml`, `dev.toml`, `gui-dev.toml`, `ubuntu-dev.toml`, `rust-dev.toml`.
|
||||||
|
|
||||||
|
## Commands
|
||||||
|
|
||||||
|
23 commands. All accept `--json`, `--store <path>`, `--verbose`, `--trace`.
|
||||||
|
|
||||||
|
```
|
||||||
|
build [manifest] [--name] Build environment
|
||||||
|
rebuild [manifest] [--name] Destroy + rebuild
|
||||||
|
enter <id> [-- cmd...] Enter environment
|
||||||
|
exec <id> -- <cmd...> Run command in environment
|
||||||
|
destroy <id> Destroy environment
|
||||||
|
stop <id> Stop running environment
|
||||||
|
freeze <id> Freeze (prevent writes)
|
||||||
|
archive <id> Archive (prevent entry)
|
||||||
|
list List environments
|
||||||
|
inspect <id> Show metadata
|
||||||
|
diff <id> Show overlay changes
|
||||||
|
snapshots <id> List snapshots
|
||||||
|
commit <id> Snapshot overlay
|
||||||
|
restore <id> <hash> Restore snapshot
|
||||||
|
gc [--dry-run] Garbage collect
|
||||||
|
verify-store Check store integrity
|
||||||
|
push <id> [--tag] [--remote] Push to remote
|
||||||
|
pull <ref> [--remote] Pull from remote
|
||||||
|
rename <id> <name> Rename environment
|
||||||
|
doctor Check prerequisites
|
||||||
|
migrate Check store version
|
||||||
|
completions <shell> Shell completions
|
||||||
|
man-pages [dir] Generate man pages
|
||||||
|
```
|
||||||
|
|
||||||
|
## Workspace
|
||||||
|
|
||||||
|
9 crates:
|
||||||
|
|
||||||
|
```
|
||||||
|
karapace-schema Manifest, normalization, lock file, identity
|
||||||
|
karapace-store Objects, layers, metadata, WAL, GC, integrity
|
||||||
|
karapace-runtime Backends (namespace/oci/mock), images, security
|
||||||
|
karapace-core Engine: lifecycle orchestration
|
||||||
|
karapace-cli CLI (23 commands)
|
||||||
|
karapace-dbus D-Bus service (optional)
|
||||||
|
karapace-tui Terminal UI (optional)
|
||||||
|
karapace-remote Remote store client, push/pull
|
||||||
|
karapace-server Reference HTTP server
|
||||||
|
```
|
||||||
|
|
||||||
|
## Limitations
|
||||||
|
|
||||||
|
- Linux only.
|
||||||
|
- Layer packing drops: extended attributes, device nodes, hardlinks, SELinux labels, ACLs.
|
||||||
|
- Base images are content-hashed but not GPG-verified.
|
||||||
|
- No MAC enforcement (SELinux/AppArmor) inside containers.
|
||||||
|
- Remote protocol has no authentication yet.
|
||||||
|
|
||||||
## Documentation
|
## Documentation
|
||||||
|
|
||||||
- **[Getting Started Guide](docs/getting-started.md)** — installation, first use, common workflows
|
- [Architecture](docs/architecture.md)
|
||||||
- [Architecture Overview](docs/architecture.md)
|
- [CLI Reference](docs/cli-reference.md)
|
||||||
- [Manifest v1 Specification](docs/manifest-spec.md)
|
- [Storage Format](docs/storage-format.md)
|
||||||
- [Lock File v2 Specification](docs/lock-spec.md)
|
|
||||||
- [Store Format v2 Specification](docs/store-spec.md)
|
|
||||||
- [Hash Contract](docs/hash-contract.md)
|
|
||||||
- [Security Model](docs/security-model.md)
|
- [Security Model](docs/security-model.md)
|
||||||
- [CLI Stability Contract](docs/cli-stability.md)
|
- [Build and Reproducibility](docs/build-and-reproducibility.md)
|
||||||
- [Remote Protocol v1 (Draft)](docs/protocol-v1.md)
|
- [Contributing](docs/contributing.md)
|
||||||
- [Layer Limitations (Phase 1)](docs/layer-limitations-v1.md)
|
|
||||||
- [Public API Reference](docs/api-reference.md)
|
|
||||||
- [Versioning Policy](docs/versioning-policy.md)
|
|
||||||
- [Verification & Supply Chain](docs/verification.md)
|
|
||||||
- [E2E Testing](docs/e2e-testing.md)
|
|
||||||
|
|
||||||
## Verification
|
## License
|
||||||
|
|
||||||
```bash
|
[EUPL-1.2](LICENSE)
|
||||||
cargo fmt --all --check
|
|
||||||
cargo clippy --workspace --all-targets -- -D warnings
|
|
||||||
cargo test --workspace
|
|
||||||
cargo build --release --workspace
|
|
||||||
```
|
|
||||||
|
|
|
||||||
|
|
@ -1,213 +0,0 @@
|
||||||
# Karapace Public API Reference
|
|
||||||
|
|
||||||
## CLI Commands
|
|
||||||
|
|
||||||
### Environment Lifecycle
|
|
||||||
|
|
||||||
#### `karapace init [manifest]`
|
|
||||||
Initialize an environment from a manifest without building. Creates metadata and a preliminary lock file.
|
|
||||||
- **Default manifest**: `karapace.toml`
|
|
||||||
|
|
||||||
#### `karapace build [manifest]`
|
|
||||||
Build an environment. Resolves dependencies, computes canonical identity, creates store objects/layers, and writes the lock file.
|
|
||||||
- **Default manifest**: `karapace.toml`
|
|
||||||
|
|
||||||
#### `karapace enter <env_id> [-- cmd...]`
|
|
||||||
Enter a built environment interactively, or run a command if `-- cmd` is provided. Transitions state to Running, then back to Built on exit.
|
|
||||||
- Accepts full env_id or short_id prefix.
|
|
||||||
|
|
||||||
#### `karapace exec <env_id> -- <cmd...>`
|
|
||||||
Execute a command inside a built environment (non-interactive). Prints stdout/stderr.
|
|
||||||
|
|
||||||
#### `karapace rebuild [manifest]`
|
|
||||||
Atomically rebuild an environment. Builds the new environment first; the old one is only destroyed after a successful build.
|
|
||||||
- Produces the same env_id for the same resolved manifest.
|
|
||||||
- If the build fails, the existing environment is preserved.
|
|
||||||
|
|
||||||
#### `karapace stop <env_id>`
|
|
||||||
Stop a running environment by sending SIGTERM/SIGKILL to its process.
|
|
||||||
|
|
||||||
#### `karapace freeze <env_id>`
|
|
||||||
Freeze an environment, preventing further entry. Transitions to Frozen state.
|
|
||||||
|
|
||||||
#### `karapace archive <env_id>`
|
|
||||||
Archive an environment. Preserves it in the store but prevents entry. Can be rebuilt later.
|
|
||||||
|
|
||||||
#### `karapace destroy <env_id>`
|
|
||||||
Destroy an environment's overlay and decrement its reference count.
|
|
||||||
- **Cannot destroy a running environment** — stop it first.
|
|
||||||
|
|
||||||
### Drift Control
|
|
||||||
|
|
||||||
#### `karapace diff <env_id>`
|
|
||||||
Show drift in the writable overlay. Lists added, modified, and removed files.
|
|
||||||
|
|
||||||
#### `karapace commit <env_id>`
|
|
||||||
Commit overlay drift into the content store as a snapshot layer.
|
|
||||||
- Only works on Built or Frozen environments.
|
|
||||||
|
|
||||||
#### `karapace export <env_id> <dest>`
|
|
||||||
Copy the writable overlay contents to a destination directory.
|
|
||||||
|
|
||||||
### Store Management
|
|
||||||
|
|
||||||
#### `karapace gc [--dry-run]`
|
|
||||||
Run garbage collection. Removes orphaned environments, layers, and objects.
|
|
||||||
- `--dry-run`: report only, do not delete.
|
|
||||||
|
|
||||||
#### `karapace verify-store`
|
|
||||||
Verify integrity of all objects in the store (blake3 content hash check).
|
|
||||||
|
|
||||||
#### `karapace verify-lock [manifest]`
|
|
||||||
Verify lock file integrity (recomputed env_id matches) and manifest consistency (no drift between manifest and lock).
|
|
||||||
|
|
||||||
### Inspection
|
|
||||||
|
|
||||||
#### `karapace inspect <env_id>`
|
|
||||||
Show environment metadata: state, layers, ref count, timestamps.
|
|
||||||
|
|
||||||
#### `karapace list`
|
|
||||||
List all known environments with short_id, state, and env_id.
|
|
||||||
|
|
||||||
#### `karapace validate [manifest]`
|
|
||||||
Validate a manifest file and print its preliminary env_id.
|
|
||||||
|
|
||||||
### Desktop Integration
|
|
||||||
|
|
||||||
#### `karapace export-app <env_id> <name> <binary>`
|
|
||||||
Export a GUI application from an environment as a `.desktop` file on the host.
|
|
||||||
|
|
||||||
#### `karapace unexport-app <env_id> <name>`
|
|
||||||
Remove an exported application's `.desktop` file from the host.
|
|
||||||
|
|
||||||
### Image Management
|
|
||||||
|
|
||||||
#### `karapace list-images`
|
|
||||||
List cached container images with status and size.
|
|
||||||
|
|
||||||
#### `karapace remove-image <name>`
|
|
||||||
Remove a cached container image from the store.
|
|
||||||
|
|
||||||
### Quick Start
|
|
||||||
|
|
||||||
#### `karapace quick [image] [-p packages] [--gpu] [--audio] [--enter]`
|
|
||||||
One-step environment creation for casual users. Generates a manifest from CLI flags, builds, and optionally enters.
|
|
||||||
- **Default image**: `rolling` (openSUSE Tumbleweed)
|
|
||||||
- `-p` / `--packages`: Comma-separated list of packages to install.
|
|
||||||
- `--gpu`: Enable GPU passthrough.
|
|
||||||
- `--audio`: Enable audio passthrough.
|
|
||||||
- `-e` / `--enter`: Enter the environment immediately after building.
|
|
||||||
- A real manifest and lock file are still generated (determinism is preserved).
|
|
||||||
|
|
||||||
Examples:
|
|
||||||
```bash
|
|
||||||
karapace quick rolling -p git,curl --enter
|
|
||||||
karapace quick ubuntu/24.04 -p build-essential,cmake --gpu
|
|
||||||
karapace quick fedora/41 --enter
|
|
||||||
```
|
|
||||||
|
|
||||||
### Tooling
|
|
||||||
|
|
||||||
#### `karapace completions <shell>`
|
|
||||||
Generate shell completions for the specified shell. Supported: `bash`, `zsh`, `fish`, `elvish`, `powershell`.
|
|
||||||
|
|
||||||
```bash
|
|
||||||
karapace completions bash > /etc/bash_completion.d/karapace
|
|
||||||
karapace completions zsh > /usr/share/zsh/site-functions/_karapace
|
|
||||||
karapace completions fish > ~/.config/fish/completions/karapace.fish
|
|
||||||
```
|
|
||||||
|
|
||||||
#### `karapace man-pages [dir]`
|
|
||||||
Generate man pages for all commands in the specified directory.
|
|
||||||
- **Default directory**: `man`
|
|
||||||
|
|
||||||
#### `karapace push <env_id> [--tag <name@tag>] [--remote <url>]`
|
|
||||||
Push an environment (metadata + layers + objects) to a remote store. Skips blobs that already exist remotely.
|
|
||||||
- **`--tag`**: Publish under a registry key (e.g. `my-env@latest`).
|
|
||||||
- **`--remote`**: Remote store URL (overrides `~/.config/karapace/remote.json`).
|
|
||||||
|
|
||||||
#### `karapace pull <reference> [--remote <url>]`
|
|
||||||
Pull an environment from a remote store. Reference can be a registry key (e.g. `my-env@latest`) or a raw env_id.
|
|
||||||
- **`--remote`**: Remote store URL (overrides config).
|
|
||||||
|
|
||||||
#### `karapace remote-list [--remote <url>]`
|
|
||||||
List environments in the remote registry.
|
|
||||||
|
|
||||||
## Global Flags
|
|
||||||
|
|
||||||
| Flag | Description |
|
|
||||||
|---|---|
|
|
||||||
| `--store <path>` | Custom store location (default: `~/.local/share/karapace`). |
|
|
||||||
| `--json` | Structured JSON output for all applicable commands. |
|
|
||||||
| `--verbose` / `-v` | Enable debug-level logging output. |
|
|
||||||
|
|
||||||
### Environment Variables
|
|
||||||
|
|
||||||
| Variable | Description |
|
|
||||||
|---|---|
|
|
||||||
| `KARAPACE_LOG` | Log level filter: `error`, `warn` (default), `info`, `debug`, `trace`. |
|
|
||||||
| `KARAPACE_STORE` | Override default store path (used by the D-Bus service). |
|
|
||||||
|
|
||||||
## Exit Codes
|
|
||||||
|
|
||||||
| Code | Meaning |
|
|
||||||
|---|---|
|
|
||||||
| `0` | Success. |
|
|
||||||
| `1` | General error. |
|
|
||||||
| `2` | Manifest validation error. |
|
|
||||||
| `3` | Store integrity error. |
|
|
||||||
|
|
||||||
## D-Bus API (Optional)
|
|
||||||
|
|
||||||
Interface: `org.karapace.Manager1`
|
|
||||||
Path: `/org/karapace/Manager1`
|
|
||||||
|
|
||||||
The D-Bus service exits after 30 seconds of idle (socket activation). Build with `cargo build -p karapace-dbus`.
|
|
||||||
|
|
||||||
All methods return proper D-Bus errors (`org.freedesktop.DBus.Error.Failed`) on failure. Mutating methods acquire the store lock automatically. Methods accepting `id_or_name` resolve by env_id, short_id, name, or prefix.
|
|
||||||
|
|
||||||
Desktop notifications are sent on build success/failure via `org.freedesktop.Notifications`.
|
|
||||||
|
|
||||||
### Properties
|
|
||||||
|
|
||||||
| Property | Type | Description |
|
|
||||||
|---|---|---|
|
|
||||||
| `ApiVersion` | `u32` | API version (currently `1`). |
|
|
||||||
| `StoreRoot` | `String` | Path to the store directory. |
|
|
||||||
|
|
||||||
### Methods
|
|
||||||
|
|
||||||
| Method | Signature | Description |
|
|
||||||
|---|---|---|
|
|
||||||
| `ListEnvironments` | `() → String` | JSON array of `{env_id, short_id, name?, state}`. |
|
|
||||||
| `GetEnvironmentStatus` | `(id_or_name) → String` | JSON status. Resolves by name. |
|
|
||||||
| `GetEnvironmentHash` | `(id_or_name) → String` | Returns env_id hash. Resolves by name. |
|
|
||||||
| `BuildEnvironment` | `(manifest_path) → String` | Build from manifest path. Sends notification. |
|
|
||||||
| `BuildNamedEnvironment` | `(manifest_path, name) → String` | Build and assign a name. Sends notification. |
|
|
||||||
| `DestroyEnvironment` | `(id_or_name) → String` | Destroy environment. Resolves by name. |
|
|
||||||
| `RunEnvironment` | `(id_or_name) → String` | Enter environment. Resolves by name. |
|
|
||||||
| `RenameEnvironment` | `(id_or_name, new_name) → String` | Rename an environment. |
|
|
||||||
| `ListPresets` | `() → String` | JSON array of `{name, description}` for built-in presets. |
|
|
||||||
| `GarbageCollect` | `(dry_run) → String` | Run GC. Acquires store lock. |
|
|
||||||
| `VerifyStore` | `() → String` | Verify store integrity. Returns `{checked, passed, failed}`. |
|
|
||||||
|
|
||||||
## Rust Crate API
|
|
||||||
|
|
||||||
### `karapace-core::Engine`
|
|
||||||
|
|
||||||
- `Engine::new(store_root)` — Create engine instance
|
|
||||||
- `Engine::init(manifest_path)` — Initialize environment (Defined state)
|
|
||||||
- `Engine::build(manifest_path)` — Full resolve → lock → build pipeline
|
|
||||||
- `Engine::enter(env_id)` — Enter environment interactively
|
|
||||||
- `Engine::exec(env_id, command)` — Execute command in environment
|
|
||||||
- `Engine::rebuild(manifest_path)` — Destroy + rebuild
|
|
||||||
- `Engine::stop(env_id)` — Stop running environment
|
|
||||||
- `Engine::freeze(env_id)` — Freeze environment
|
|
||||||
- `Engine::archive(env_id)` — Archive environment (preserve, prevent entry)
|
|
||||||
- `Engine::commit(env_id)` — Commit overlay drift
|
|
||||||
- `Engine::destroy(env_id)` — Destroy environment
|
|
||||||
- `Engine::inspect(env_id)` — Get environment metadata
|
|
||||||
- `Engine::list()` — List all environments
|
|
||||||
- `Engine::gc(dry_run)` — Garbage collection (caller must hold store lock)
|
|
||||||
- `Engine::set_name(env_id, name)` — Set or clear environment name
|
|
||||||
- `Engine::rename(env_id, new_name)` — Rename environment
|
|
||||||
|
|
@ -1,90 +1,183 @@
|
||||||
# Karapace Architecture
|
# Architecture
|
||||||
|
|
||||||
## Overview
|
## Workspace
|
||||||
|
|
||||||
Karapace is a deterministic container environment engine organized as a Cargo workspace of 9 crates. Each crate has a single responsibility and clean dependency boundaries.
|
Karapace is a Cargo workspace of 9 crates.
|
||||||
|
|
||||||
## Crate Dependency Graph
|
|
||||||
|
|
||||||
```
|
```
|
||||||
karapace-cli ─────┬──▶ karapace-core ──┬──▶ karapace-schema
|
karapace-schema Manifest parsing, normalization, lock file, identity hashing
|
||||||
│ ├──▶ karapace-store
|
karapace-store Content-addressable object store, layers, metadata, WAL, GC
|
||||||
│ ├──▶ karapace-runtime
|
karapace-runtime Container backends, image cache, security policy, prerequisites
|
||||||
│ └──▶ karapace-remote
|
karapace-core Engine: orchestrates the full environment lifecycle
|
||||||
├──▶ karapace-remote ──▶ karapace-store
|
karapace-cli CLI binary (23 commands, clap)
|
||||||
└──▶ karapace-tui ────▶ karapace-core
|
karapace-dbus D-Bus service (org.karapace.Manager1, zbus)
|
||||||
|
karapace-tui Terminal UI (ratatui, crossterm)
|
||||||
karapace-dbus ────────▶ karapace-core
|
karapace-remote Remote store client: HTTP backend, registry, push/pull
|
||||||
karapace-server ──────▶ karapace-store (standalone HTTP server)
|
karapace-server Reference HTTP server for remote store (tiny_http)
|
||||||
```
|
```
|
||||||
|
|
||||||
## Crate Responsibilities
|
## Dependency graph
|
||||||
|
|
||||||
### `karapace-schema`
|
|
||||||
Manifest v1 parsing (TOML), normalization, canonical JSON serialization, environment identity hashing, lock file v2 (resolved packages, integrity/intent verification), and built-in presets.
|
|
||||||
|
|
||||||
### `karapace-store`
|
|
||||||
Content-addressable object store (blake3), layer manifests, environment metadata (with naming, ref-counting, state machine), garbage collection with signal cancellation, and store integrity verification. All writes are atomic via `NamedTempFile` + `persist`.
|
|
||||||
|
|
||||||
### `karapace-runtime`
|
|
||||||
Container runtime abstraction (`RuntimeBackend` trait) with three backends:
|
|
||||||
- **Namespace** — `unshare` + `fuse-overlayfs` + `chroot` (unprivileged)
|
|
||||||
- **OCI** — `crun`/`runc`/`youki`
|
|
||||||
- **Mock** — deterministic test backend
|
|
||||||
|
|
||||||
Also handles image downloading, sandbox scripting, host integration (Wayland, GPU, audio, D-Bus), security policy enforcement, and desktop app export.
|
|
||||||
|
|
||||||
### `karapace-core`
|
|
||||||
The `Engine` struct orchestrates the full lifecycle: init → resolve → lock → build → enter/exec → freeze → archive → destroy. Caches `MetadataStore`, `ObjectStore`, and `LayerStore` as fields. Handles drift detection (diff/commit/export via overlay upper_dir scanning) and garbage collection delegation.
|
|
||||||
|
|
||||||
### `karapace-cli`
|
|
||||||
23 CLI commands, each in its own file under `commands/`. Shared helpers in `commands/mod.rs` (spinners, colored output, environment resolution, JSON formatting). `main.rs` is a thin dispatcher. Exit codes: 0 (success), 1 (failure), 2 (manifest error), 3 (store error).
|
|
||||||
|
|
||||||
### `karapace-dbus`
|
|
||||||
Socket-activated D-Bus service (`org.karapace.Manager1`) with 11 methods. Typed serde response structs. Desktop notifications via `notify-rust`. 30-second idle timeout for socket activation. Hardened systemd unit file.
|
|
||||||
|
|
||||||
### `karapace-remote`
|
|
||||||
Remote content-addressable store with `RemoteBackend` trait, HTTP backend (ureq), push/pull transfer with blake3 integrity verification on pull, and a JSON registry for name@tag references.
|
|
||||||
|
|
||||||
### `karapace-tui`
|
|
||||||
Interactive terminal UI (ratatui + crossterm) with list/detail/help views, vim-style keybindings, search/filter, sort cycling, freeze/archive/rename actions, and confirmation dialogs.
|
|
||||||
|
|
||||||
## Key Design Decisions
|
|
||||||
|
|
||||||
1. **Content-addressed identity** — `env_id` is computed from the *resolved* lock file (pinned versions + base image content digest), not from unresolved manifest data.
|
|
||||||
|
|
||||||
2. **Atomic operations** — All store writes use `NamedTempFile` + `persist` for crash safety. Rebuild builds the new environment before destroying the old one.
|
|
||||||
|
|
||||||
3. **No `unwrap()` in production** — All error paths are handled with proper error types (`StoreError`, `CoreError`, `RemoteError`, `RuntimeError`).
|
|
||||||
|
|
||||||
4. **Store locking** — `StoreLock` file lock on all mutating operations (CLI + D-Bus). GC respects active/archived environments.
|
|
||||||
|
|
||||||
5. **Layered security** — Mount whitelist, device policy, env var allow/deny, resource limits. No privilege escalation.
|
|
||||||
|
|
||||||
## Data Flow
|
|
||||||
|
|
||||||
```
|
```
|
||||||
Manifest (TOML)
|
karapace-cli ──┬──> karapace-core ──┬──> karapace-schema
|
||||||
│
|
│ ├──> karapace-store
|
||||||
▼
|
│ ├──> karapace-runtime
|
||||||
NormalizedManifest (canonical JSON)
|
│ └──> karapace-remote
|
||||||
│
|
├──> karapace-runtime
|
||||||
▼ resolve (RuntimeBackend)
|
└──> karapace-store
|
||||||
ResolutionResult (base_image_digest + resolved_packages)
|
|
||||||
│
|
karapace-dbus ────> karapace-core
|
||||||
▼
|
karapace-tui ─────> karapace-core
|
||||||
LockFile v2 (pinned, verifiable)
|
karapace-remote ──> karapace-store
|
||||||
│
|
karapace-server ──> karapace-remote, karapace-store
|
||||||
▼ compute_identity()
|
|
||||||
EnvIdentity (env_id = blake3 of canonical lock)
|
|
||||||
│
|
|
||||||
▼ build (store objects + layers + metadata)
|
|
||||||
Built Environment (overlay filesystem)
|
|
||||||
```
|
```
|
||||||
|
|
||||||
### `karapace-server`
|
## Engine lifecycle
|
||||||
Reference remote server implementing protocol v1 over HTTP (tiny_http). Provides blob storage, registry, and list endpoints. Used for testing push/pull workflows.
|
|
||||||
|
|
||||||
## Test Coverage
|
`karapace-core::Engine` is the central orchestrator. All state transitions go through it.
|
||||||
|
|
||||||
417 tests across all crates. 24 ignored tests require privileged operations (real `unshare`, `fuse-overlayfs`, ENOSPC simulation, namespace access).
|
```
|
||||||
|
┌─────────┐
|
||||||
|
build() ───> │ Defined │
|
||||||
|
└────┬────┘
|
||||||
|
│ resolve → lock → build
|
||||||
|
v
|
||||||
|
┌─────────┐
|
||||||
|
│ Built │ <── rebuild()
|
||||||
|
└──┬──┬───┘
|
||||||
|
enter() │ │ │ freeze()
|
||||||
|
v │ v
|
||||||
|
┌─────────┐ ┌─────────┐
|
||||||
|
│ Running │ │ Frozen │
|
||||||
|
└─────────┘ └────┬────┘
|
||||||
|
│ archive()
|
||||||
|
v
|
||||||
|
┌──────────┐
|
||||||
|
│ Archived │
|
||||||
|
└──────────┘
|
||||||
|
```
|
||||||
|
|
||||||
|
State transitions are validated in `karapace-core/src/lifecycle.rs`. Invalid transitions return `CoreError`.
|
||||||
|
|
||||||
|
### Build pipeline
|
||||||
|
|
||||||
|
`Engine::build(manifest_path)` executes:
|
||||||
|
|
||||||
|
1. Parse manifest (`karapace-schema::parse_manifest_file`)
|
||||||
|
2. Normalize (`ManifestV1::normalize`) — sort packages, deduplicate, lowercase backend
|
||||||
|
3. Select runtime backend (`karapace-runtime::select_backend`)
|
||||||
|
4. Resolve — backend downloads base image, computes content digest, queries package manager for exact versions → `ResolutionResult`
|
||||||
|
5. Create lock file (`LockFile::from_resolved`) with pinned versions and content digest
|
||||||
|
6. Compute identity (`LockFile::compute_identity`) → `env_id` (blake3)
|
||||||
|
7. Store manifest as object, create layers, write metadata
|
||||||
|
8. Backend builds the environment filesystem
|
||||||
|
9. Write lock file to disk
|
||||||
|
|
||||||
|
### Identity computation
|
||||||
|
|
||||||
|
Defined in `karapace-schema/src/lock.rs::LockFile::compute_identity()`.
|
||||||
|
|
||||||
|
Input fed to blake3 in order:
|
||||||
|
- `base_digest:<content_hash>`
|
||||||
|
- `pkg:<name>@<version>` for each resolved package (sorted)
|
||||||
|
- `app:<name>` for each app (sorted)
|
||||||
|
- `hw:gpu` / `hw:audio` if enabled
|
||||||
|
- `mount:<label>:<host>:<container>` for each mount (sorted)
|
||||||
|
- `backend:<name>`
|
||||||
|
- `net:isolated` if enabled
|
||||||
|
- `cpu:<value>` / `mem:<value>` if set
|
||||||
|
|
||||||
|
Output: 64-character hex blake3 digest. First 12 characters = `short_id`.
|
||||||
|
|
||||||
|
## Container runtime
|
||||||
|
|
||||||
|
`karapace-runtime/src/backend.rs` defines `RuntimeBackend` trait:
|
||||||
|
|
||||||
|
```rust
|
||||||
|
pub trait RuntimeBackend: Send + Sync {
|
||||||
|
fn name(&self) -> &str;
|
||||||
|
fn available(&self) -> bool;
|
||||||
|
fn resolve(&self, spec: &RuntimeSpec) -> Result<ResolutionResult, RuntimeError>;
|
||||||
|
fn build(&self, spec: &RuntimeSpec) -> Result<(), RuntimeError>;
|
||||||
|
fn enter(&self, spec: &RuntimeSpec) -> Result<(), RuntimeError>;
|
||||||
|
fn exec(&self, spec: &RuntimeSpec, command: &[String]) -> Result<Output, RuntimeError>;
|
||||||
|
fn destroy(&self, spec: &RuntimeSpec) -> Result<(), RuntimeError>;
|
||||||
|
fn status(&self, env_id: &str) -> Result<RuntimeStatus, RuntimeError>;
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
Three backends (`karapace-runtime/src/backend.rs::select_backend`):
|
||||||
|
|
||||||
|
| Backend | Implementation | Use |
|
||||||
|
|---------|---------------|-----|
|
||||||
|
| `namespace` | `unshare` + `fuse-overlayfs` + `chroot` | Default. Unprivileged. |
|
||||||
|
| `oci` | `crun` / `runc` / `youki` | OCI-compatible runtimes. |
|
||||||
|
| `mock` | Deterministic stubs | Testing only. |
|
||||||
|
|
||||||
|
## Image cache
|
||||||
|
|
||||||
|
`karapace-runtime/src/image.rs::ImageCache` stores downloaded base images under `<store_root>/images/<cache_key>/rootfs/`.
|
||||||
|
|
||||||
|
Images are fetched from `images.linuxcontainers.org`. The content digest is a blake3 hash of the rootfs directory tree (`compute_image_digest`). Package manager is auto-detected from rootfs contents (`detect_package_manager`).
|
||||||
|
|
||||||
|
## Content-addressable store
|
||||||
|
|
||||||
|
All persistent data lives under `<store_root>/store/`. See [storage-format.md](storage-format.md) for the full layout.
|
||||||
|
|
||||||
|
- **Objects**: keyed by blake3 hash of content. Written atomically (tempfile + rename). Verified on every read.
|
||||||
|
- **Layers**: JSON manifests describing tar archives. Kinds: `Base`, `Dependency`, `Policy`, `Snapshot`.
|
||||||
|
- **Metadata**: JSON per environment. Includes state, layers, ref count, checksum.
|
||||||
|
|
||||||
|
## Snapshot mechanism
|
||||||
|
|
||||||
|
`Engine::commit(env_id)`:
|
||||||
|
1. Pack the overlay upper directory into a deterministic tar (`pack_layer`)
|
||||||
|
2. Store tar as object
|
||||||
|
3. Create a `Snapshot` layer manifest with composite hash: `blake3("snapshot:{env_id}:{base_layer}:{tar_hash}")`
|
||||||
|
|
||||||
|
`Engine::restore(env_id, snapshot_hash)`:
|
||||||
|
1. Retrieve snapshot layer and its tar object
|
||||||
|
2. Unpack to `store/staging/restore-{env_id}`
|
||||||
|
3. Atomic rename-swap with the environment's upper directory
|
||||||
|
|
||||||
|
Deterministic packing: entries sorted, timestamps zeroed, owner `0:0`, permissions preserved. Symlinks preserved. Extended attributes, device nodes, hardlinks, ACLs, SELinux labels are dropped.
|
||||||
|
|
||||||
|
## Garbage collection
|
||||||
|
|
||||||
|
`Engine::gc(store_lock, dry_run)` in `karapace-core/src/engine.rs`. Caller must hold `StoreLock`.
|
||||||
|
|
||||||
|
Protected from collection:
|
||||||
|
- Environments with state `Running` or `Archived`
|
||||||
|
- Layers referenced by any live environment
|
||||||
|
- Snapshot layers whose parent is a live base layer
|
||||||
|
- Objects referenced by any live layer or live metadata `manifest_hash`
|
||||||
|
|
||||||
|
Everything else is orphaned and removed. GC supports `SIGINT`/`SIGTERM` cancellation.
|
||||||
|
|
||||||
|
## Write-ahead log
|
||||||
|
|
||||||
|
`karapace-store/src/wal.rs`. JSON entries in `store/wal/`.
|
||||||
|
|
||||||
|
Operations tracked: `Build`, `Rebuild`, `Commit`, `Restore`, `Destroy`, `Gc`.
|
||||||
|
|
||||||
|
Each entry records rollback steps (`RemoveDir`, `RemoveFile`). On `Engine::new()`, incomplete WAL entries are replayed in reverse order, then deleted. Corrupt entries are silently removed.
|
||||||
|
|
||||||
|
## Concurrency
|
||||||
|
|
||||||
|
`karapace-core/src/concurrency.rs::StoreLock` uses `flock(2)` on `store/.lock`. All mutating CLI commands and D-Bus methods acquire this lock.
|
||||||
|
|
||||||
|
## Signal handling
|
||||||
|
|
||||||
|
`karapace-core/src/concurrency.rs::install_signal_handler()` registers `SIGINT`/`SIGTERM` via `ctrlc` crate. Sets an atomic flag checked by GC and long-running operations.
|
||||||
|
|
||||||
|
## Unsafe code
|
||||||
|
|
||||||
|
Five `unsafe` blocks in the codebase:
|
||||||
|
|
||||||
|
| Location | Call | Purpose |
|
||||||
|
|----------|------|---------|
|
||||||
|
| `karapace-core/src/engine.rs:455` | `libc::kill(SIGTERM)` | Stop running environment |
|
||||||
|
| `karapace-core/src/engine.rs:475` | `libc::kill(SIGKILL)` | Force-kill after timeout |
|
||||||
|
| `karapace-runtime/src/sandbox.rs:46` | `libc::getuid()` | Get current UID for namespace setup |
|
||||||
|
| `karapace-runtime/src/sandbox.rs:53` | `libc::getgid()` | Get current GID for namespace setup |
|
||||||
|
| `karapace-runtime/src/terminal.rs:41` | `libc::isatty()` | Detect terminal for interactive mode |
|
||||||
|
|
|
||||||
83
docs/build-and-reproducibility.md
Normal file
83
docs/build-and-reproducibility.md
Normal file
|
|
@ -0,0 +1,83 @@
|
||||||
|
# Build and Reproducibility
|
||||||
|
|
||||||
|
## Deterministic identity
|
||||||
|
|
||||||
|
The environment identity (`env_id`) is a blake3 hash computed from the fully resolved lock file — not from the unresolved manifest. Two identical lock files on any machine produce the same `env_id`.
|
||||||
|
|
||||||
|
The identity is computed in `karapace-schema/src/lock.rs::LockFile::compute_identity()`. See [architecture.md](architecture.md) for the full list of hash inputs.
|
||||||
|
|
||||||
|
## Lock file
|
||||||
|
|
||||||
|
`karapace build` writes `karapace.lock` next to the manifest. This file pins:
|
||||||
|
|
||||||
|
- Base image content digest (blake3 of rootfs)
|
||||||
|
- Exact package versions (queried from the package manager inside the image)
|
||||||
|
- All manifest-declared settings (hardware, mounts, backend, resource limits)
|
||||||
|
|
||||||
|
The lock file should be committed to version control.
|
||||||
|
|
||||||
|
## Reproducibility constraints
|
||||||
|
|
||||||
|
Given the same lock file and the same base image content, builds produce the same `env_id`. The overlay filesystem content depends on the package manager's behavior, which Karapace does not control.
|
||||||
|
|
||||||
|
Karapace guarantees identity reproducibility (same inputs → same `env_id`). It does not guarantee bit-for-bit filesystem reproducibility across different package repository states.
|
||||||
|
|
||||||
|
## CI release builds
|
||||||
|
|
||||||
|
Configured in `.github/workflows/ci.yml` and `.github/workflows/release.yml`.
|
||||||
|
|
||||||
|
### Environment variables
|
||||||
|
|
||||||
|
| Variable | Value | Purpose |
|
||||||
|
|----------|-------|---------|
|
||||||
|
| `CARGO_INCREMENTAL` | `0` | Disable incremental compilation |
|
||||||
|
| `SOURCE_DATE_EPOCH` | `0` | Deterministic timestamps in build artifacts |
|
||||||
|
|
||||||
|
### RUSTFLAGS
|
||||||
|
|
||||||
|
```
|
||||||
|
-D warnings
|
||||||
|
--remap-path-prefix /home/runner/work=src
|
||||||
|
--remap-path-prefix /home/runner/.cargo/registry/src=crate
|
||||||
|
--remap-path-prefix /home/runner/.rustup=rustup
|
||||||
|
```
|
||||||
|
|
||||||
|
Path remapping eliminates runner-specific filesystem paths from the binary.
|
||||||
|
|
||||||
|
### Cargo profile (release)
|
||||||
|
|
||||||
|
Defined in workspace `Cargo.toml`:
|
||||||
|
|
||||||
|
```toml
|
||||||
|
[profile.release]
|
||||||
|
strip = true
|
||||||
|
lto = "thin"
|
||||||
|
```
|
||||||
|
|
||||||
|
### Build procedure
|
||||||
|
|
||||||
|
1. `cargo clean` — eliminates stale intermediate artifacts
|
||||||
|
2. `cargo build --release --target <target> -p karapace-cli -p karapace-dbus`
|
||||||
|
|
||||||
|
Both glibc (`x86_64-unknown-linux-gnu`) and musl (`x86_64-unknown-linux-musl`) targets are built. Musl binaries are fully statically linked.
|
||||||
|
|
||||||
|
### Reproducibility verification in CI
|
||||||
|
|
||||||
|
CI runs two types of reproducibility checks:
|
||||||
|
|
||||||
|
- **Same-run:** two sequential builds on the same runner, output hashes compared
|
||||||
|
- **Cross-run:** builds on `ubuntu-latest` and `ubuntu-22.04`, hashes compared (warning-only — different system libraries may cause differences in glibc builds)
|
||||||
|
|
||||||
|
Musl builds are expected to be runner-independent.
|
||||||
|
|
||||||
|
**Constraint:** build invocations must use identical `-p` flags. Building `-p karapace-cli` alone may produce a different binary than `-p karapace-cli -p karapace-dbus` due to codegen unit ordering.
|
||||||
|
|
||||||
|
## Local development builds
|
||||||
|
|
||||||
|
`.cargo/config.toml` configures path remapping for the project maintainer. Other developers should update the paths or set `RUSTFLAGS` directly.
|
||||||
|
|
||||||
|
Local builds are for development. CI builds are the authoritative release artifacts.
|
||||||
|
|
||||||
|
## Pinned toolchain
|
||||||
|
|
||||||
|
CI pins Rust toolchain version via `RUST_TOOLCHAIN` environment variable (currently `1.93`). This is set in the workflow `env` block and used with `dtolnay/rust-toolchain@stable`.
|
||||||
286
docs/cli-reference.md
Normal file
286
docs/cli-reference.md
Normal file
|
|
@ -0,0 +1,286 @@
|
||||||
|
# CLI Reference
|
||||||
|
|
||||||
|
Binary: `karapace`. Defined in `crates/karapace-cli/src/main.rs`.
|
||||||
|
|
||||||
|
## Global flags
|
||||||
|
|
||||||
|
| Flag | Default | Description |
|
||||||
|
|------|---------|-------------|
|
||||||
|
| `--store <path>` | `~/.local/share/karapace` | Store directory path |
|
||||||
|
| `--json` | `false` | JSON output |
|
||||||
|
| `--verbose` / `-v` | `false` | Debug-level logging |
|
||||||
|
| `--trace` | `false` | Trace-level logging (implies debug) |
|
||||||
|
|
||||||
|
## Environment variables
|
||||||
|
|
||||||
|
| Variable | Used by | Description |
|
||||||
|
|----------|---------|-------------|
|
||||||
|
| `KARAPACE_LOG` | cli, dbus | Log level filter: `error`, `warn`, `info`, `debug`, `trace`. Overrides `--verbose`/`--trace`. |
|
||||||
|
| `KARAPACE_STORE` | dbus | Override default store path. |
|
||||||
|
| `KARAPACE_SKIP_PREREQS` | cli | Set to `1` to skip runtime prerequisite checks. |
|
||||||
|
|
||||||
|
## Exit codes
|
||||||
|
|
||||||
|
| Code | Constant | Condition |
|
||||||
|
|------|----------|-----------|
|
||||||
|
| 0 | `EXIT_SUCCESS` | Success |
|
||||||
|
| 1 | `EXIT_FAILURE` | General error |
|
||||||
|
| 2 | `EXIT_MANIFEST_ERROR` | Manifest parse or validation error |
|
||||||
|
| 3 | `EXIT_STORE_ERROR` | Store integrity or lock error |
|
||||||
|
|
||||||
|
Defined in `crates/karapace-cli/src/commands/mod.rs`.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Commands
|
||||||
|
|
||||||
|
### `build`
|
||||||
|
|
||||||
|
Build an environment from a manifest.
|
||||||
|
|
||||||
|
```
|
||||||
|
karapace build [manifest] [--name <name>]
|
||||||
|
```
|
||||||
|
|
||||||
|
| Argument | Default | Description |
|
||||||
|
|----------|---------|-------------|
|
||||||
|
| `manifest` | `karapace.toml` | Path to manifest file |
|
||||||
|
| `--name` | — | Assign a human-readable name |
|
||||||
|
|
||||||
|
Executes: parse → normalize → resolve → lock → build. Writes `karapace.lock` next to the manifest. Requires runtime prerequisites (user namespaces, fuse-overlayfs).
|
||||||
|
|
||||||
|
### `rebuild`
|
||||||
|
|
||||||
|
Destroy the existing environment and build a new one from the manifest.
|
||||||
|
|
||||||
|
```
|
||||||
|
karapace rebuild [manifest] [--name <name>]
|
||||||
|
```
|
||||||
|
|
||||||
|
Same arguments as `build`. The old environment is destroyed only after the new one builds successfully.
|
||||||
|
|
||||||
|
### `enter`
|
||||||
|
|
||||||
|
Enter an environment interactively, or run a command.
|
||||||
|
|
||||||
|
```
|
||||||
|
karapace enter <env_id> [-- cmd...]
|
||||||
|
```
|
||||||
|
|
||||||
|
| Argument | Description |
|
||||||
|
|----------|-------------|
|
||||||
|
| `env_id` | Full env_id, short_id, or name |
|
||||||
|
| `-- cmd...` | Optional command to run instead of interactive shell |
|
||||||
|
|
||||||
|
Sets state to `Running` on entry, back to `Built` on exit.
|
||||||
|
|
||||||
|
### `exec`
|
||||||
|
|
||||||
|
Run a command inside an environment (non-interactive).
|
||||||
|
|
||||||
|
```
|
||||||
|
karapace exec <env_id> -- <cmd...>
|
||||||
|
```
|
||||||
|
|
||||||
|
| Argument | Description |
|
||||||
|
|----------|-------------|
|
||||||
|
| `env_id` | Full env_id, short_id, or name |
|
||||||
|
| `cmd...` | Required. Command and arguments. |
|
||||||
|
|
||||||
|
### `destroy`
|
||||||
|
|
||||||
|
Destroy an environment and its overlay.
|
||||||
|
|
||||||
|
```
|
||||||
|
karapace destroy <env_id>
|
||||||
|
```
|
||||||
|
|
||||||
|
Cannot destroy a `Running` environment. Stop it first.
|
||||||
|
|
||||||
|
### `stop`
|
||||||
|
|
||||||
|
Stop a running environment.
|
||||||
|
|
||||||
|
```
|
||||||
|
karapace stop <env_id>
|
||||||
|
```
|
||||||
|
|
||||||
|
Sends `SIGTERM`. If the process does not exit within the timeout, sends `SIGKILL`.
|
||||||
|
|
||||||
|
### `freeze`
|
||||||
|
|
||||||
|
Freeze an environment. Prevents further writes.
|
||||||
|
|
||||||
|
```
|
||||||
|
karapace freeze <env_id>
|
||||||
|
```
|
||||||
|
|
||||||
|
### `archive`
|
||||||
|
|
||||||
|
Archive an environment. Prevents entry but preserves store data.
|
||||||
|
|
||||||
|
```
|
||||||
|
karapace archive <env_id>
|
||||||
|
```
|
||||||
|
|
||||||
|
Archived environments are protected from garbage collection.
|
||||||
|
|
||||||
|
### `list`
|
||||||
|
|
||||||
|
List all environments.
|
||||||
|
|
||||||
|
```
|
||||||
|
karapace list
|
||||||
|
```
|
||||||
|
|
||||||
|
Output columns: `SHORT_ID`, `NAME`, `STATE`, `ENV_ID`.
|
||||||
|
|
||||||
|
### `inspect`
|
||||||
|
|
||||||
|
Show environment metadata.
|
||||||
|
|
||||||
|
```
|
||||||
|
karapace inspect <env_id>
|
||||||
|
```
|
||||||
|
|
||||||
|
### `diff`
|
||||||
|
|
||||||
|
Show changes in the writable overlay.
|
||||||
|
|
||||||
|
```
|
||||||
|
karapace diff <env_id>
|
||||||
|
```
|
||||||
|
|
||||||
|
Lists added, modified, and removed files relative to the base layer.
|
||||||
|
|
||||||
|
### `snapshots`
|
||||||
|
|
||||||
|
List snapshots for an environment.
|
||||||
|
|
||||||
|
```
|
||||||
|
karapace snapshots <env_id>
|
||||||
|
```
|
||||||
|
|
||||||
|
### `commit`
|
||||||
|
|
||||||
|
Save overlay changes as a snapshot layer.
|
||||||
|
|
||||||
|
```
|
||||||
|
karapace commit <env_id>
|
||||||
|
```
|
||||||
|
|
||||||
|
Only valid for `Built` or `Frozen` environments.
|
||||||
|
|
||||||
|
### `restore`
|
||||||
|
|
||||||
|
Restore an environment's overlay from a snapshot.
|
||||||
|
|
||||||
|
```
|
||||||
|
karapace restore <env_id> <snapshot_hash>
|
||||||
|
```
|
||||||
|
|
||||||
|
| Argument | Description |
|
||||||
|
|----------|-------------|
|
||||||
|
| `env_id` | Environment to restore |
|
||||||
|
| `snapshot_hash` | Layer hash from `snapshots` output |
|
||||||
|
|
||||||
|
### `gc`
|
||||||
|
|
||||||
|
Garbage collect orphaned store data.
|
||||||
|
|
||||||
|
```
|
||||||
|
karapace gc [--dry-run]
|
||||||
|
```
|
||||||
|
|
||||||
|
| Flag | Description |
|
||||||
|
|------|-------------|
|
||||||
|
| `--dry-run` | Report what would be removed without deleting |
|
||||||
|
|
||||||
|
### `verify-store`
|
||||||
|
|
||||||
|
Verify integrity of all objects in the store.
|
||||||
|
|
||||||
|
```
|
||||||
|
karapace verify-store
|
||||||
|
```
|
||||||
|
|
||||||
|
Re-hashes every object, layer, and metadata entry against its stored key or checksum.
|
||||||
|
|
||||||
|
### `push`
|
||||||
|
|
||||||
|
Push an environment to a remote store.
|
||||||
|
|
||||||
|
```
|
||||||
|
karapace push <env_id> [--tag <name@tag>] [--remote <url>]
|
||||||
|
```
|
||||||
|
|
||||||
|
| Flag | Description |
|
||||||
|
|------|-------------|
|
||||||
|
| `--tag` | Registry key, e.g. `my-env@latest` |
|
||||||
|
| `--remote` | Remote URL. Overrides `~/.config/karapace/remote.json`. |
|
||||||
|
|
||||||
|
Skips blobs that already exist on the remote.
|
||||||
|
|
||||||
|
### `pull`
|
||||||
|
|
||||||
|
Pull an environment from a remote store.
|
||||||
|
|
||||||
|
```
|
||||||
|
karapace pull <reference> [--remote <url>]
|
||||||
|
```
|
||||||
|
|
||||||
|
| Argument | Description |
|
||||||
|
|----------|-------------|
|
||||||
|
| `reference` | Registry key (`name@tag`) or raw `env_id` |
|
||||||
|
|
||||||
|
Downloaded objects are verified with blake3 before storage.
|
||||||
|
|
||||||
|
### `rename`
|
||||||
|
|
||||||
|
Rename an environment.
|
||||||
|
|
||||||
|
```
|
||||||
|
karapace rename <env_id> <new_name>
|
||||||
|
```
|
||||||
|
|
||||||
|
Names must match `[a-zA-Z0-9_-]`, 1–64 characters. Validated in `karapace-store/src/metadata.rs::validate_env_name`.
|
||||||
|
|
||||||
|
### `completions`
|
||||||
|
|
||||||
|
Generate shell completions.
|
||||||
|
|
||||||
|
```
|
||||||
|
karapace completions <shell>
|
||||||
|
```
|
||||||
|
|
||||||
|
Supported shells: `bash`, `zsh`, `fish`, `elvish`, `powershell`.
|
||||||
|
|
||||||
|
### `man-pages`
|
||||||
|
|
||||||
|
Generate man pages.
|
||||||
|
|
||||||
|
```
|
||||||
|
karapace man-pages [dir]
|
||||||
|
```
|
||||||
|
|
||||||
|
| Argument | Default | Description |
|
||||||
|
|----------|---------|-------------|
|
||||||
|
| `dir` | `man` | Output directory |
|
||||||
|
|
||||||
|
### `doctor`
|
||||||
|
|
||||||
|
Check system prerequisites and store health.
|
||||||
|
|
||||||
|
```
|
||||||
|
karapace doctor
|
||||||
|
```
|
||||||
|
|
||||||
|
Checks: user namespace support, `fuse-overlayfs` availability, `curl` availability. Exits non-zero if any check fails.
|
||||||
|
|
||||||
|
### `migrate`
|
||||||
|
|
||||||
|
Check store format version and show migration guidance.
|
||||||
|
|
||||||
|
```
|
||||||
|
karapace migrate
|
||||||
|
```
|
||||||
|
|
@ -1,84 +0,0 @@
|
||||||
# Karapace CLI Stability Contract
|
|
||||||
|
|
||||||
## Scope
|
|
||||||
|
|
||||||
This document defines the stability guarantee for the Karapace CLI between 1.x releases.
|
|
||||||
|
|
||||||
## Stable Commands (21)
|
|
||||||
|
|
||||||
The following commands have **stable signatures** — no breaking changes to arguments, flags, or output format between 1.x releases:
|
|
||||||
|
|
||||||
| Command | Description |
|
|
||||||
|---------|-------------|
|
|
||||||
| `build` | Build environment from manifest |
|
|
||||||
| `rebuild` | Destroy + rebuild environment |
|
|
||||||
| `enter` | Enter environment interactively |
|
|
||||||
| `exec` | Execute command inside environment |
|
|
||||||
| `destroy` | Destroy environment |
|
|
||||||
| `stop` | Stop a running environment |
|
|
||||||
| `freeze` | Freeze environment (read-only overlay) |
|
|
||||||
| `archive` | Archive environment (preserve, prevent entry) |
|
|
||||||
| `list` | List all environments |
|
|
||||||
| `inspect` | Show environment metadata |
|
|
||||||
| `diff` | Show overlay drift |
|
|
||||||
| `snapshots` | List snapshots for an environment |
|
|
||||||
| `commit` | Commit overlay drift as snapshot |
|
|
||||||
| `restore` | Restore overlay from snapshot |
|
|
||||||
| `gc` | Garbage collect orphaned store data |
|
|
||||||
| `verify-store` | Check store integrity |
|
|
||||||
| `push` | Push environment to remote store |
|
|
||||||
| `pull` | Pull environment from remote store |
|
|
||||||
| `rename` | Rename environment |
|
|
||||||
| `doctor` | Run diagnostic checks on system and store |
|
|
||||||
| `migrate` | Check store version and show migration guidance |
|
|
||||||
|
|
||||||
## Zero-Maintenance Commands (2)
|
|
||||||
|
|
||||||
These commands are auto-generated and have no hand-maintained logic:
|
|
||||||
|
|
||||||
| Command | Description |
|
|
||||||
|---------|-------------|
|
|
||||||
| `completions` | Generate shell completions (bash/zsh/fish/elvish/powershell) |
|
|
||||||
| `man-pages` | Generate man pages |
|
|
||||||
|
|
||||||
**Total: 23 commands.**
|
|
||||||
|
|
||||||
## Global Flags
|
|
||||||
|
|
||||||
All commands accept these flags (stable):
|
|
||||||
|
|
||||||
- `--store <path>` — custom store location (default: `~/.local/share/karapace`)
|
|
||||||
- `--json` — structured JSON output on all query and store commands
|
|
||||||
- `--verbose` / `-v` — enable debug logging
|
|
||||||
- `--trace` — enable trace-level logging (more detailed than `--verbose`)
|
|
||||||
|
|
||||||
## What "Stable" Means
|
|
||||||
|
|
||||||
- **No removed flags** — existing flags continue to work.
|
|
||||||
- **No changed flag semantics** — same flag produces same behavior.
|
|
||||||
- **No changed exit codes** — exit code meanings are fixed.
|
|
||||||
- **No changed JSON output keys** — new keys may be added, existing keys are never removed or renamed.
|
|
||||||
- **New flags may be added** — additive changes are allowed.
|
|
||||||
|
|
||||||
## What May Change
|
|
||||||
|
|
||||||
- Human-readable (non-JSON) output formatting.
|
|
||||||
- Spinner and progress indicator appearance.
|
|
||||||
- Error message wording (not error codes).
|
|
||||||
- Addition of new commands.
|
|
||||||
- Addition of new optional flags to existing commands.
|
|
||||||
|
|
||||||
## Removed Commands
|
|
||||||
|
|
||||||
The following commands were removed before 1.0 and will not return:
|
|
||||||
|
|
||||||
`init`, `preset`, `list-presets`, `export-app`, `unexport-app`, `quick`, `validate`, `verify-lock`, `export`, `list-images`, `remove-image`, `remote-list`, `tui`
|
|
||||||
|
|
||||||
## Exit Codes
|
|
||||||
|
|
||||||
| Code | Meaning |
|
|
||||||
|------|---------|
|
|
||||||
| 0 | Success |
|
|
||||||
| 1 | General failure |
|
|
||||||
| 2 | Manifest error (parse, validation) |
|
|
||||||
| 3 | Store error (integrity, version mismatch) |
|
|
||||||
98
docs/contributing.md
Normal file
98
docs/contributing.md
Normal file
|
|
@ -0,0 +1,98 @@
|
||||||
|
# Contributing
|
||||||
|
|
||||||
|
## Setup
|
||||||
|
|
||||||
|
```bash
|
||||||
|
git clone https://github.com/marcoallegretti/karapace.git
|
||||||
|
cd karapace
|
||||||
|
cargo build
|
||||||
|
cargo test --workspace
|
||||||
|
```
|
||||||
|
|
||||||
|
## Verification
|
||||||
|
|
||||||
|
All of these must pass before submitting changes:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
cargo fmt --all --check
|
||||||
|
cargo clippy --workspace --all-targets -- -D warnings
|
||||||
|
cargo test --workspace
|
||||||
|
cargo build --release --workspace
|
||||||
|
```
|
||||||
|
|
||||||
|
## Project layout
|
||||||
|
|
||||||
|
```
|
||||||
|
crates/
|
||||||
|
karapace-schema/ Manifest parsing, normalization, lock file, identity
|
||||||
|
karapace-store/ Content-addressable store, metadata, layers, WAL, GC
|
||||||
|
karapace-runtime/ Container backends, images, sandbox, security policy
|
||||||
|
karapace-core/ Engine: lifecycle orchestration, drift, concurrency
|
||||||
|
karapace-cli/ CLI binary (23 commands)
|
||||||
|
karapace-dbus/ D-Bus service (optional, not in default-members)
|
||||||
|
karapace-tui/ Terminal UI (optional, not in default-members)
|
||||||
|
karapace-remote/ Remote store client, push/pull, registry
|
||||||
|
karapace-server/ Reference HTTP server for remote store
|
||||||
|
docs/ Public documentation
|
||||||
|
docu_dev/ Internal development notes (not shipped)
|
||||||
|
data/ systemd and D-Bus service files
|
||||||
|
```
|
||||||
|
|
||||||
|
`default-members` in `Cargo.toml`: schema, store, runtime, core, cli, server. The D-Bus service and TUI are opt-in.
|
||||||
|
|
||||||
|
## Code standards
|
||||||
|
|
||||||
|
- `cargo clippy -- -D warnings` — zero warnings.
|
||||||
|
- `cargo fmt` — enforced.
|
||||||
|
- No `unwrap()` in production `src/` code. Tests are fine.
|
||||||
|
- All values interpolated into shell commands must use `shell_quote()`.
|
||||||
|
- All mutating operations must hold a `StoreLock`.
|
||||||
|
- All file writes must be atomic (`NamedTempFile` + `persist()`).
|
||||||
|
|
||||||
|
## Testing
|
||||||
|
|
||||||
|
- Unit tests: `#[cfg(test)] mod tests` in the relevant module.
|
||||||
|
- Integration tests: `crates/karapace-core/tests/`.
|
||||||
|
- E2E tests: `crates/karapace-core/tests/e2e.rs` — `#[ignore]`, require user namespaces and `fuse-overlayfs`.
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# Unit + integration tests
|
||||||
|
cargo test --workspace
|
||||||
|
|
||||||
|
# E2E tests (requires Linux with user namespaces)
|
||||||
|
cargo test --test e2e -- --ignored --test-threads=1
|
||||||
|
```
|
||||||
|
|
||||||
|
## CI
|
||||||
|
|
||||||
|
Three workflows in `.github/workflows/`:
|
||||||
|
|
||||||
|
| Workflow | File | What it checks |
|
||||||
|
|----------|------|----------------|
|
||||||
|
| CI | `ci.yml` | fmt, clippy, tests (Ubuntu + Fedora), E2E, ENOSPC, release builds, reproducibility, smoke tests, lockfile, cargo-deny |
|
||||||
|
| Release | `release.yml` | Builds release binaries, signs with cosign, generates SBOM and provenance |
|
||||||
|
| Supply Chain | `supply-chain-test.yml` | Tamper detection, signature verification, adversarial injection tests |
|
||||||
|
|
||||||
|
The `ci-contract` job in `ci.yml` verifies that all required jobs are present. See `CI_CONTRACT.md`.
|
||||||
|
|
||||||
|
## Building the D-Bus service
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# CLI only (default)
|
||||||
|
cargo build --release
|
||||||
|
|
||||||
|
# CLI + D-Bus service
|
||||||
|
cargo build --release --workspace
|
||||||
|
```
|
||||||
|
|
||||||
|
## Shell completions
|
||||||
|
|
||||||
|
```bash
|
||||||
|
karapace completions bash > ~/.local/share/bash-completion/completions/karapace
|
||||||
|
karapace completions zsh > ~/.local/share/zsh/site-functions/_karapace
|
||||||
|
karapace completions fish > ~/.config/fish/completions/karapace.fish
|
||||||
|
```
|
||||||
|
|
||||||
|
## License
|
||||||
|
|
||||||
|
Contributions are licensed under EUPL-1.2.
|
||||||
|
|
@ -1,77 +0,0 @@
|
||||||
# End-to-End Testing
|
|
||||||
|
|
||||||
Karapace includes end-to-end tests that exercise the real namespace backend with `unshare`, `fuse-overlayfs`, and actual container images.
|
|
||||||
|
|
||||||
## Prerequisites
|
|
||||||
|
|
||||||
- Linux with user namespace support (`CONFIG_USER_NS=y`)
|
|
||||||
- `fuse-overlayfs` installed
|
|
||||||
- `curl` installed
|
|
||||||
- Network access (images are downloaded from `images.linuxcontainers.org`)
|
|
||||||
|
|
||||||
### Install on openSUSE Tumbleweed
|
|
||||||
|
|
||||||
```bash
|
|
||||||
sudo zypper install fuse-overlayfs curl
|
|
||||||
```
|
|
||||||
|
|
||||||
### Install on Ubuntu/Debian
|
|
||||||
|
|
||||||
```bash
|
|
||||||
sudo apt-get install fuse-overlayfs curl
|
|
||||||
```
|
|
||||||
|
|
||||||
### Install on Fedora
|
|
||||||
|
|
||||||
```bash
|
|
||||||
sudo dnf install fuse-overlayfs curl
|
|
||||||
```
|
|
||||||
|
|
||||||
## Running E2E Tests
|
|
||||||
|
|
||||||
E2E tests are `#[ignore]` by default. Run them explicitly:
|
|
||||||
|
|
||||||
```bash
|
|
||||||
cargo test --test e2e -- --ignored --test-threads=1
|
|
||||||
```
|
|
||||||
|
|
||||||
The `--test-threads=1` flag is important: E2E tests mount overlays and download images, so parallel execution can cause resource conflicts.
|
|
||||||
|
|
||||||
## Test Descriptions
|
|
||||||
|
|
||||||
| Test | What it does |
|
|
||||||
|---|---|
|
|
||||||
| `e2e_build_minimal_namespace` | Build a minimal openSUSE Tumbleweed environment with no packages |
|
|
||||||
| `e2e_exec_in_namespace` | Build + exec `echo hello` inside the container |
|
|
||||||
| `e2e_destroy_cleans_up` | Build + destroy, verify env_dir is removed |
|
|
||||||
| `e2e_rebuild_determinism` | Build + rebuild, verify env_id is identical |
|
|
||||||
| `e2e_build_with_packages` | Build with `which` package, verify resolved versions in lock file |
|
|
||||||
|
|
||||||
## CI
|
|
||||||
|
|
||||||
The GitHub Actions CI workflow includes an E2E job that runs on `ubuntu-latest`:
|
|
||||||
|
|
||||||
```yaml
|
|
||||||
e2e:
|
|
||||||
name: E2E Tests
|
|
||||||
runs-on: ubuntu-latest
|
|
||||||
needs: [test]
|
|
||||||
steps:
|
|
||||||
- uses: actions/checkout@v4
|
|
||||||
- uses: dtolnay/rust-toolchain@stable
|
|
||||||
- uses: Swatinem/rust-cache@v2
|
|
||||||
- name: Install prerequisites
|
|
||||||
run: |
|
|
||||||
sudo apt-get update -qq
|
|
||||||
sudo apt-get install -y -qq fuse-overlayfs curl
|
|
||||||
sudo sysctl -w kernel.unprivileged_userns_clone=1 || true
|
|
||||||
- name: Run E2E tests
|
|
||||||
run: cargo test --test e2e -- --ignored --test-threads=1
|
|
||||||
```
|
|
||||||
|
|
||||||
## Troubleshooting
|
|
||||||
|
|
||||||
- **"unshare: user namespaces not available"** — Enable with `sysctl kernel.unprivileged_userns_clone=1`
|
|
||||||
- **"fuse-overlayfs not found"** — Install the `fuse-overlayfs` package
|
|
||||||
- **"failed to download image"** — Check network connectivity and DNS
|
|
||||||
- **Stale mounts after failed test** — Run `fusermount3 -u /path/to/merged` or reboot
|
|
||||||
|
|
@ -1,282 +0,0 @@
|
||||||
# Getting Started with Karapace
|
|
||||||
|
|
||||||
Karapace creates isolated, reproducible development environments on Linux using
|
|
||||||
namespaces and overlay filesystems. No root, no daemon, no Docker.
|
|
||||||
|
|
||||||
This guide walks you through installation, first use, and common workflows.
|
|
||||||
|
|
||||||
## Prerequisites
|
|
||||||
|
|
||||||
Karapace requires a Linux system with:
|
|
||||||
|
|
||||||
- **User namespace support** (`CONFIG_USER_NS=y` — enabled on all major distros)
|
|
||||||
- **fuse-overlayfs** (overlay filesystem in userspace)
|
|
||||||
- **curl** (for downloading base images)
|
|
||||||
|
|
||||||
Optional:
|
|
||||||
|
|
||||||
- **crun**, **runc**, or **youki** (only if using the OCI backend)
|
|
||||||
|
|
||||||
### Install prerequisites by distro
|
|
||||||
|
|
||||||
**openSUSE Tumbleweed / Leap:**
|
|
||||||
```bash
|
|
||||||
sudo zypper install fuse-overlayfs curl
|
|
||||||
```
|
|
||||||
|
|
||||||
**Ubuntu / Debian:**
|
|
||||||
```bash
|
|
||||||
sudo apt install fuse-overlayfs curl
|
|
||||||
```
|
|
||||||
|
|
||||||
**Fedora:**
|
|
||||||
```bash
|
|
||||||
sudo dnf install fuse-overlayfs curl
|
|
||||||
```
|
|
||||||
|
|
||||||
**Arch Linux:**
|
|
||||||
```bash
|
|
||||||
sudo pacman -S fuse-overlayfs curl
|
|
||||||
```
|
|
||||||
|
|
||||||
Run `karapace doctor` at any time to check that all prerequisites are met.
|
|
||||||
|
|
||||||
## Installation
|
|
||||||
|
|
||||||
### From source (recommended)
|
|
||||||
|
|
||||||
```bash
|
|
||||||
git clone https://github.com/marcoallegretti/karapace.git
|
|
||||||
cd karapace
|
|
||||||
cargo build --release
|
|
||||||
sudo install -Dm755 target/release/karapace /usr/local/bin/karapace
|
|
||||||
```
|
|
||||||
|
|
||||||
### Via cargo install
|
|
||||||
|
|
||||||
```bash
|
|
||||||
cargo install --git https://github.com/marcoallegretti/karapace.git karapace-cli
|
|
||||||
```
|
|
||||||
|
|
||||||
### Shell completions
|
|
||||||
|
|
||||||
```bash
|
|
||||||
# Bash
|
|
||||||
karapace completions bash > ~/.local/share/bash-completion/completions/karapace
|
|
||||||
|
|
||||||
# Zsh
|
|
||||||
karapace completions zsh > ~/.local/share/zsh/site-functions/_karapace
|
|
||||||
|
|
||||||
# Fish
|
|
||||||
karapace completions fish > ~/.config/fish/completions/karapace.fish
|
|
||||||
```
|
|
||||||
|
|
||||||
## Your first environment
|
|
||||||
|
|
||||||
### 1. Write a manifest
|
|
||||||
|
|
||||||
Create a file called `karapace.toml`:
|
|
||||||
|
|
||||||
```toml
|
|
||||||
manifest_version = 1
|
|
||||||
|
|
||||||
[base]
|
|
||||||
image = "rolling" # openSUSE Tumbleweed
|
|
||||||
|
|
||||||
[system]
|
|
||||||
packages = ["git", "curl"]
|
|
||||||
|
|
||||||
[runtime]
|
|
||||||
backend = "namespace"
|
|
||||||
```
|
|
||||||
|
|
||||||
Available base images: `"rolling"` (openSUSE Tumbleweed), `"ubuntu/24.04"`,
|
|
||||||
`"debian/12"`, `"fedora/41"`, `"arch"`.
|
|
||||||
|
|
||||||
### 2. Build the environment
|
|
||||||
|
|
||||||
```bash
|
|
||||||
karapace build
|
|
||||||
```
|
|
||||||
|
|
||||||
This downloads the base image, installs the requested packages, and produces
|
|
||||||
a content-addressed environment. The output shows the environment ID (`env_id`)
|
|
||||||
and a short ID for convenience.
|
|
||||||
|
|
||||||
### 3. Enter the environment
|
|
||||||
|
|
||||||
```bash
|
|
||||||
karapace enter <env_id>
|
|
||||||
```
|
|
||||||
|
|
||||||
You can use the short ID (first 8 characters) or a name instead of the full ID.
|
|
||||||
Inside the environment you have a full Linux userspace with the packages you
|
|
||||||
requested.
|
|
||||||
|
|
||||||
### 4. Run a single command
|
|
||||||
|
|
||||||
```bash
|
|
||||||
karapace exec <env_id> -- git --version
|
|
||||||
```
|
|
||||||
|
|
||||||
## Naming environments
|
|
||||||
|
|
||||||
By default, environments are identified by their content hash. You can assign
|
|
||||||
a human-readable name:
|
|
||||||
|
|
||||||
```bash
|
|
||||||
karapace build --name mydev
|
|
||||||
karapace enter mydev
|
|
||||||
```
|
|
||||||
|
|
||||||
Or rename an existing environment:
|
|
||||||
|
|
||||||
```bash
|
|
||||||
karapace rename <env_id> mydev
|
|
||||||
```
|
|
||||||
|
|
||||||
## Common workflows
|
|
||||||
|
|
||||||
### Snapshot and restore
|
|
||||||
|
|
||||||
After making changes inside an environment (installing extra packages, editing
|
|
||||||
config files), you can snapshot and later restore:
|
|
||||||
|
|
||||||
```bash
|
|
||||||
# See what changed
|
|
||||||
karapace diff mydev
|
|
||||||
|
|
||||||
# Save a snapshot
|
|
||||||
karapace commit mydev
|
|
||||||
|
|
||||||
# List snapshots
|
|
||||||
karapace snapshots mydev
|
|
||||||
|
|
||||||
# Restore a previous snapshot
|
|
||||||
karapace restore mydev <snapshot_hash>
|
|
||||||
```
|
|
||||||
|
|
||||||
### Freeze and archive
|
|
||||||
|
|
||||||
```bash
|
|
||||||
# Freeze: prevent further changes (still enterable in read-only mode)
|
|
||||||
karapace freeze mydev
|
|
||||||
|
|
||||||
# Archive: preserve metadata but prevent entry
|
|
||||||
karapace archive mydev
|
|
||||||
```
|
|
||||||
|
|
||||||
### Rebuild from scratch
|
|
||||||
|
|
||||||
If you change your manifest, rebuild destroys the old environment and builds
|
|
||||||
a new one:
|
|
||||||
|
|
||||||
```bash
|
|
||||||
karapace rebuild
|
|
||||||
```
|
|
||||||
|
|
||||||
### Push and pull (remote sharing)
|
|
||||||
|
|
||||||
```bash
|
|
||||||
# Push to a remote store
|
|
||||||
karapace push mydev --tag my-env@latest
|
|
||||||
|
|
||||||
# Pull on another machine
|
|
||||||
karapace pull my-env@latest --remote https://your-server.example.com
|
|
||||||
```
|
|
||||||
|
|
||||||
### GUI application export
|
|
||||||
|
|
||||||
Export a GUI application from inside the container to your host desktop:
|
|
||||||
|
|
||||||
```bash
|
|
||||||
karapace exec mydev -- karapace-export-app firefox
|
|
||||||
```
|
|
||||||
|
|
||||||
This creates a `.desktop` file on the host that launches the app inside the
|
|
||||||
container with GPU and audio passthrough.
|
|
||||||
|
|
||||||
## Hardware passthrough
|
|
||||||
|
|
||||||
Enable GPU and audio in your manifest:
|
|
||||||
|
|
||||||
```toml
|
|
||||||
[hardware]
|
|
||||||
gpu = true # Passes /dev/dri into the container
|
|
||||||
audio = true # Passes PipeWire/PulseAudio socket
|
|
||||||
```
|
|
||||||
|
|
||||||
Karapace also forwards Wayland, X11, D-Bus session bus, SSH agent, fonts,
|
|
||||||
and GTK/icon themes automatically when available.
|
|
||||||
|
|
||||||
## Custom bind mounts
|
|
||||||
|
|
||||||
Mount host directories into the container:
|
|
||||||
|
|
||||||
```toml
|
|
||||||
[mounts]
|
|
||||||
workspace = "~/projects:/workspace"
|
|
||||||
data = "/data/datasets:/datasets"
|
|
||||||
```
|
|
||||||
|
|
||||||
## Built-in presets
|
|
||||||
|
|
||||||
For quick setup without writing a manifest:
|
|
||||||
|
|
||||||
```bash
|
|
||||||
# List available presets
|
|
||||||
karapace list-presets
|
|
||||||
|
|
||||||
# Build from a preset
|
|
||||||
karapace preset dev-rust
|
|
||||||
```
|
|
||||||
|
|
||||||
Available presets: `dev`, `dev-rust`, `dev-python`, `gui-app`, `gaming`, `minimal`.
|
|
||||||
|
|
||||||
## Quick one-liner
|
|
||||||
|
|
||||||
The `quick` command combines build + enter in a single step:
|
|
||||||
|
|
||||||
```bash
|
|
||||||
karapace quick
|
|
||||||
```
|
|
||||||
|
|
||||||
This uses the `karapace.toml` in the current directory (or creates a minimal one).
|
|
||||||
|
|
||||||
## Diagnostics
|
|
||||||
|
|
||||||
```bash
|
|
||||||
# Check system prerequisites
|
|
||||||
karapace doctor
|
|
||||||
|
|
||||||
# Verify store integrity
|
|
||||||
karapace verify-store
|
|
||||||
|
|
||||||
# List all environments
|
|
||||||
karapace list
|
|
||||||
|
|
||||||
# Inspect an environment
|
|
||||||
karapace inspect mydev
|
|
||||||
|
|
||||||
# Garbage collect unused objects
|
|
||||||
karapace gc --dry-run
|
|
||||||
karapace gc
|
|
||||||
```
|
|
||||||
|
|
||||||
## Environment variables
|
|
||||||
|
|
||||||
| Variable | Effect |
|
|
||||||
|----------|--------|
|
|
||||||
| `KARAPACE_LOG` | Log level: `error`, `warn`, `info`, `debug`, `trace` |
|
|
||||||
| `KARAPACE_STORE` | Custom store directory (default: `~/.local/share/karapace`) |
|
|
||||||
|
|
||||||
Or use CLI flags: `--verbose` / `-v` for debug, `--trace` for trace,
|
|
||||||
`--store <path>` for a custom store, `--json` for machine-readable output.
|
|
||||||
|
|
||||||
## Next steps
|
|
||||||
|
|
||||||
- [Manifest v1 Specification](manifest-spec.md) — full manifest reference
|
|
||||||
- [Architecture Overview](architecture.md) — how Karapace works internally
|
|
||||||
- [CLI Stability Contract](cli-stability.md) — which commands are stable
|
|
||||||
- [Security Model](security-model.md) — isolation guarantees and threat model
|
|
||||||
- [Verification](verification.md) — verifying release artifact integrity
|
|
||||||
|
|
@ -1,60 +0,0 @@
|
||||||
# Karapace Hash Contract
|
|
||||||
|
|
||||||
## Overview
|
|
||||||
|
|
||||||
The environment identity (`env_id`) is a deterministic blake3 hash that uniquely identifies an environment's fully resolved state. Two identical lock files on any machine must produce the same `env_id`.
|
|
||||||
|
|
||||||
## Algorithm
|
|
||||||
|
|
||||||
Blake3 (256-bit output, hex-encoded, 64 characters).
|
|
||||||
|
|
||||||
## Two-Phase Identity
|
|
||||||
|
|
||||||
Karapace computes identity in two phases:
|
|
||||||
|
|
||||||
### Preliminary Identity (`compute_env_id`)
|
|
||||||
|
|
||||||
Used only during `init` (before resolution) and for internal lookup. Computed from unresolved manifest data. **Not the canonical identity.**
|
|
||||||
|
|
||||||
### Canonical Identity (`LockFile::compute_identity`)
|
|
||||||
|
|
||||||
The authoritative identity used after `build`. Computed from the fully resolved lock file state. This is what gets stored in metadata and the lock file.
|
|
||||||
|
|
||||||
## Canonical Hash Input
|
|
||||||
|
|
||||||
The canonical hash includes the following inputs, fed in order:
|
|
||||||
|
|
||||||
1. **Base image content digest**: `base_digest:<blake3_of_rootfs>` — real content hash, not a tag name hash.
|
|
||||||
2. **Resolved packages**: each as `pkg:<name>@<version>` (sorted by name).
|
|
||||||
3. **Resolved apps**: each as `app:<name>` (sorted).
|
|
||||||
4. **Hardware policy**: `hw:gpu` if GPU enabled, `hw:audio` if audio enabled.
|
|
||||||
5. **Mount policy**: each as `mount:<label>:<host_path>:<container_path>` (sorted by label).
|
|
||||||
6. **Runtime backend**: `backend:<name>` (lowercased).
|
|
||||||
7. **Network isolation**: `net:isolated` if enabled.
|
|
||||||
8. **CPU shares**: `cpu:<value>` if set.
|
|
||||||
9. **Memory limit**: `mem:<value>` if set.
|
|
||||||
|
|
||||||
## Hash MUST NOT Include
|
|
||||||
|
|
||||||
- Writable overlay state (mutable drift).
|
|
||||||
- Timestamps (creation, modification).
|
|
||||||
- Host-specific non-declared paths.
|
|
||||||
- Machine identifiers (hostname, MAC, etc.).
|
|
||||||
- Store location.
|
|
||||||
- Unresolved package names without versions.
|
|
||||||
|
|
||||||
## Properties
|
|
||||||
|
|
||||||
- **Deterministic**: same resolved inputs → same hash, always.
|
|
||||||
- **Stable**: consistent across identical systems with same resolved packages.
|
|
||||||
- **Immutable**: once built, the env_id never changes for that lock state.
|
|
||||||
- **Version-sensitive**: different package versions produce different identities.
|
|
||||||
|
|
||||||
## Short ID
|
|
||||||
|
|
||||||
The `short_id` is the first 12 hex characters of `env_id`. Used for display and prefix-matching in the CLI.
|
|
||||||
|
|
||||||
## Implementation
|
|
||||||
|
|
||||||
- **Canonical**: `karapace-schema/src/lock.rs::LockFile::compute_identity()`
|
|
||||||
- **Preliminary**: `karapace-schema/src/identity.rs::compute_env_id()`
|
|
||||||
|
|
@ -1,77 +0,0 @@
|
||||||
# Karapace Layer Limitations — Phase 1 (v1.0)
|
|
||||||
|
|
||||||
## Overview
|
|
||||||
|
|
||||||
Karapace 1.0 ships with Phase 1 layer support: deterministic tar-based content-addressed layers. This document describes what is supported, what is not, and what is planned for future phases.
|
|
||||||
|
|
||||||
## Supported (Phase 1)
|
|
||||||
|
|
||||||
- **Regular files** — full content preservation, deterministic packing.
|
|
||||||
- **Directories** — including empty directories.
|
|
||||||
- **Symbolic links** — target path preserved exactly.
|
|
||||||
- **Deterministic packing** — sorted entries, zero timestamps (`mtime = 0`), owner `0:0`, consistent permissions.
|
|
||||||
- **Content addressing** — blake3 hash of the tar archive.
|
|
||||||
- **Snapshot layers** — `commit` captures overlay upper directory as a tar, `restore` unpacks it atomically.
|
|
||||||
- **Composite snapshot identity** — snapshot layer hash is `blake3("snapshot:{env_id}:{base_layer}:{tar_hash}")` to prevent collision with base layers.
|
|
||||||
- **Atomic restore** — unpack to staging directory, then rename-swap with the upper directory.
|
|
||||||
|
|
||||||
## Not Supported (Phase 1)
|
|
||||||
|
|
||||||
The following filesystem features are **silently dropped** during `pack_layer`:
|
|
||||||
|
|
||||||
| Feature | Status | Planned |
|
|
||||||
|---------|--------|---------|
|
|
||||||
| Extended attributes (xattrs) | Dropped | Phase 2 (1.1) |
|
|
||||||
| Device nodes | Dropped | Phase 2 (1.1) |
|
|
||||||
| Hardlinks | Stored as regular files (deduplicated content) | Phase 2 (1.1) |
|
|
||||||
| SELinux labels | Dropped | Phase 2 (1.1) |
|
|
||||||
| ACLs | Dropped | Phase 2 (1.1) |
|
|
||||||
| Sparse files | Stored as full files | Phase 2 (1.1) |
|
|
||||||
| UID/GID remapping | Not supported | Phase 3 (2.0) |
|
|
||||||
| Per-file dedup | Not supported | Phase 3 (2.0) |
|
|
||||||
| Whiteout files (overlay deletion markers) | Included as regular files | Phase 2 (1.1) |
|
|
||||||
|
|
||||||
## Implications
|
|
||||||
|
|
||||||
### Security-Sensitive Workloads
|
|
||||||
|
|
||||||
Environments relying on SELinux labels, xattrs for capabilities (`security.capability`), or ACLs will not have those attributes preserved across `commit`/`restore` cycles. This is acceptable for development environments but not for production container images.
|
|
||||||
|
|
||||||
### Hardlinks
|
|
||||||
|
|
||||||
If two files in the upper directory are hardlinked, they will be stored as separate regular files in the tar. This means:
|
|
||||||
- Restoring from a snapshot may increase disk usage.
|
|
||||||
- File identity (inode sharing) is not preserved.
|
|
||||||
|
|
||||||
### Device Nodes
|
|
||||||
|
|
||||||
Device nodes (`/dev/*`) created inside the environment are dropped during packing. This is intentional — device nodes are host-specific and should not be stored in content-addressed layers.
|
|
||||||
|
|
||||||
### Whiteout Handling
|
|
||||||
|
|
||||||
Overlay whiteout files (`.wh.*` and `.wh..wh..opq`) are currently stored as-is in the tar. They are only meaningful when applied on top of the correct base layer. Restoring a snapshot to a different base layer may produce incorrect results.
|
|
||||||
|
|
||||||
## Determinism Guarantees
|
|
||||||
|
|
||||||
| Property | Guaranteed |
|
|
||||||
|----------|-----------|
|
|
||||||
| Same directory content → same tar bytes | Yes |
|
|
||||||
| Same tar bytes → same blake3 hash | Yes |
|
|
||||||
| Roundtrip fidelity (regular files, dirs, symlinks) | Yes |
|
|
||||||
| Timestamp preservation | No (zeroed for determinism) |
|
|
||||||
| Owner/group preservation | No (set to 0:0) |
|
|
||||||
| Permission preservation | Yes (mode bits preserved) |
|
|
||||||
|
|
||||||
## Phase 2 Roadmap (1.1)
|
|
||||||
|
|
||||||
- Extended attribute support via `tar` crate's xattr feature.
|
|
||||||
- Hardlink detection and deduplication within a single layer.
|
|
||||||
- Overlay whiteout awareness (proper deletion semantics).
|
|
||||||
- Device node opt-in for privileged builds.
|
|
||||||
- SELinux label preservation.
|
|
||||||
|
|
||||||
## Phase 3 Roadmap (2.0)
|
|
||||||
|
|
||||||
- UID/GID remapping for rootless environments.
|
|
||||||
- Per-file content deduplication across layers.
|
|
||||||
- Layer diffing and incremental snapshots.
|
|
||||||
|
|
@ -1,94 +0,0 @@
|
||||||
# Karapace Lock File Specification (v2)
|
|
||||||
|
|
||||||
## Overview
|
|
||||||
|
|
||||||
The lock file (`karapace.lock`) captures the fully resolved state of an environment at build time. It ensures reproducible builds by pinning all dependency versions and the base image content digest.
|
|
||||||
|
|
||||||
The environment identity (`env_id`) is computed from the **locked** state — resolved package versions and content digest — not from unresolved manifest data. This guarantees: same lockfile → same env_id → same environment.
|
|
||||||
|
|
||||||
## Format
|
|
||||||
|
|
||||||
TOML with the following fields:
|
|
||||||
|
|
||||||
| Field | Type | Description |
|
|
||||||
|---|---|---|
|
|
||||||
| `lock_version` | `u32` | Lock format version. Must be `2`. |
|
|
||||||
| `env_id` | `string` | Full 64-character blake3 hex digest computed from locked state. |
|
|
||||||
| `short_id` | `string` | First 12 characters of `env_id`. |
|
|
||||||
| `base_image` | `string` | Base image identifier from manifest (e.g. `"rolling"`). |
|
|
||||||
| `base_image_digest` | `string` | Blake3 content digest of the actual base image rootfs. |
|
|
||||||
| `resolved_packages` | `array` | Sorted list of `{ name, version }` tables with pinned versions. |
|
|
||||||
| `resolved_apps` | `string[]` | Sorted, deduplicated list of resolved GUI apps. |
|
|
||||||
| `runtime_backend` | `string` | Normalized runtime backend name (lowercased). |
|
|
||||||
| `hardware_gpu` | `bool` | Whether GPU passthrough is requested. |
|
|
||||||
| `hardware_audio` | `bool` | Whether audio passthrough is requested. |
|
|
||||||
| `network_isolation` | `bool` | Whether network is isolated. |
|
|
||||||
| `mounts` | `array` | Sorted list of `{ label, host_path, container_path }` tables. |
|
|
||||||
| `cpu_shares` | `u64?` | CPU shares limit (optional). |
|
|
||||||
| `memory_limit_mb` | `u64?` | Memory limit in MB (optional). |
|
|
||||||
|
|
||||||
## Resolved Packages
|
|
||||||
|
|
||||||
Each entry in `resolved_packages` is a table:
|
|
||||||
|
|
||||||
```toml
|
|
||||||
[[resolved_packages]]
|
|
||||||
name = "git"
|
|
||||||
version = "2.44.0-1"
|
|
||||||
|
|
||||||
[[resolved_packages]]
|
|
||||||
name = "clang"
|
|
||||||
version = "17.0.6-1"
|
|
||||||
```
|
|
||||||
|
|
||||||
Versions are queried from the actual package manager inside the base image during the resolve phase. This ensures identity is based on real installed versions, not just package names.
|
|
||||||
|
|
||||||
## Invariants
|
|
||||||
|
|
||||||
- The lock file contains **content digests**, not tags or mutable references.
|
|
||||||
- `base_image_digest` is a blake3 hash of the actual rootfs content, not a hash of the image tag name.
|
|
||||||
- `env_id` is computed from all locked fields via `LockFile::compute_identity()`.
|
|
||||||
- Verification is split into two checks:
|
|
||||||
- **Integrity**: `verify_integrity()` — recomputes `env_id` and compares to the stored value.
|
|
||||||
- **Manifest intent**: `verify_manifest_intent()` — checks that the manifest hasn't drifted from what was locked.
|
|
||||||
|
|
||||||
## Generation
|
|
||||||
|
|
||||||
- Generated automatically by `karapace init` (with preliminary unresolved versions) and `karapace build` (with fully resolved versions).
|
|
||||||
- Written atomically (tempfile + rename) to the same directory as the manifest.
|
|
||||||
- Should be committed to version control for reproducible builds.
|
|
||||||
|
|
||||||
## Manual Override
|
|
||||||
|
|
||||||
- The lock file can only be regenerated by re-running `karapace build`.
|
|
||||||
- There is no `karapace lock --update` command; rebuilding is the only path.
|
|
||||||
- `karapace verify-lock` checks both integrity and manifest consistency.
|
|
||||||
|
|
||||||
## Example
|
|
||||||
|
|
||||||
```toml
|
|
||||||
lock_version = 2
|
|
||||||
env_id = "46e1d96fdd6fd988092fbcd19b1d89f2b080f3e74d0f4984b4ba45ca5b95e594"
|
|
||||||
short_id = "46e1d96fdd6f"
|
|
||||||
base_image = "rolling"
|
|
||||||
base_image_digest = "a1b2c3d4e5f6..."
|
|
||||||
runtime_backend = "namespace"
|
|
||||||
hardware_gpu = true
|
|
||||||
hardware_audio = false
|
|
||||||
network_isolation = false
|
|
||||||
|
|
||||||
[[resolved_packages]]
|
|
||||||
name = "clang"
|
|
||||||
version = "17.0.6-1"
|
|
||||||
|
|
||||||
[[resolved_packages]]
|
|
||||||
name = "git"
|
|
||||||
version = "2.44.0-1"
|
|
||||||
|
|
||||||
resolved_apps = ["debugger", "ide"]
|
|
||||||
|
|
||||||
[[mounts]]
|
|
||||||
label = "workspace"
|
|
||||||
host_path = "./"
|
|
||||||
container_path = "/workspace"
|
|
||||||
```
|
|
||||||
|
|
@ -1,113 +0,0 @@
|
||||||
# Karapace Manifest v0.1 Specification
|
|
||||||
|
|
||||||
## Overview
|
|
||||||
|
|
||||||
The Karapace manifest is a TOML file (typically `karapace.toml`) that declaratively defines an environment. It is the single source of truth for environment identity.
|
|
||||||
|
|
||||||
## Schema
|
|
||||||
|
|
||||||
### Required Fields
|
|
||||||
|
|
||||||
| Field | Type | Description |
|
|
||||||
|---|---|---|
|
|
||||||
| `manifest_version` | `u32` | Must be `1`. |
|
|
||||||
| `base.image` | `string` | Base image identifier. Must not be empty. |
|
|
||||||
|
|
||||||
### Optional Sections
|
|
||||||
|
|
||||||
#### `[system]`
|
|
||||||
|
|
||||||
| Field | Type | Default | Description |
|
|
||||||
|---|---|---|---|
|
|
||||||
| `packages` | `string[]` | `[]` | System packages to install. Duplicates are deduplicated during normalization. |
|
|
||||||
|
|
||||||
#### `[gui]`
|
|
||||||
|
|
||||||
| Field | Type | Default | Description |
|
|
||||||
|---|---|---|---|
|
|
||||||
| `apps` | `string[]` | `[]` | GUI applications to install. |
|
|
||||||
|
|
||||||
#### `[hardware]`
|
|
||||||
|
|
||||||
| Field | Type | Default | Description |
|
|
||||||
|---|---|---|---|
|
|
||||||
| `gpu` | `bool` | `false` | Request GPU passthrough (`/dev/dri`). |
|
|
||||||
| `audio` | `bool` | `false` | Request audio device passthrough (`/dev/snd`). |
|
|
||||||
|
|
||||||
#### `[mounts]`
|
|
||||||
|
|
||||||
Flat key-value pairs. Each key is a label; each value is `<host_path>:<container_path>`.
|
|
||||||
|
|
||||||
- Labels must not be empty.
|
|
||||||
- The `:` separator is required.
|
|
||||||
- Absolute host paths are validated against the mount whitelist.
|
|
||||||
- Relative host paths (e.g. `./`) are always permitted.
|
|
||||||
|
|
||||||
#### `[runtime]`
|
|
||||||
|
|
||||||
| Field | Type | Default | Description |
|
|
||||||
|---|---|---|---|
|
|
||||||
| `backend` | `string` | `"namespace"` | Runtime backend: `namespace`, `oci`, or `mock`. |
|
|
||||||
| `network_isolation` | `bool` | `false` | Isolate network from host. |
|
|
||||||
|
|
||||||
#### `[runtime.resource_limits]`
|
|
||||||
|
|
||||||
| Field | Type | Default | Description |
|
|
||||||
|---|---|---|---|
|
|
||||||
| `cpu_shares` | `u64?` | `null` | CPU shares limit. |
|
|
||||||
| `memory_limit_mb` | `u64?` | `null` | Memory limit in MB. |
|
|
||||||
|
|
||||||
## Validation Rules
|
|
||||||
|
|
||||||
1. `manifest_version` must equal `1`.
|
|
||||||
2. Unknown fields at any level cause a parse error (`deny_unknown_fields`).
|
|
||||||
3. `base.image` must not be empty or whitespace-only.
|
|
||||||
4. Mount specs must contain exactly one `:` with non-empty sides.
|
|
||||||
|
|
||||||
## Normalization
|
|
||||||
|
|
||||||
During normalization:
|
|
||||||
|
|
||||||
- All string values are trimmed.
|
|
||||||
- `system.packages` and `gui.apps` are sorted, deduplicated.
|
|
||||||
- Mounts are sorted by label.
|
|
||||||
- `runtime.backend` is lowercased.
|
|
||||||
|
|
||||||
The normalized form is serialized to canonical JSON for hashing.
|
|
||||||
|
|
||||||
## Example
|
|
||||||
|
|
||||||
```toml
|
|
||||||
manifest_version = 1
|
|
||||||
|
|
||||||
[base]
|
|
||||||
image = "rolling"
|
|
||||||
|
|
||||||
[system]
|
|
||||||
packages = ["clang", "cmake", "git"]
|
|
||||||
|
|
||||||
[gui]
|
|
||||||
apps = ["ide", "debugger"]
|
|
||||||
|
|
||||||
[hardware]
|
|
||||||
gpu = true
|
|
||||||
audio = true
|
|
||||||
|
|
||||||
[mounts]
|
|
||||||
workspace = "./:/workspace"
|
|
||||||
|
|
||||||
[runtime]
|
|
||||||
backend = "namespace"
|
|
||||||
network_isolation = false
|
|
||||||
|
|
||||||
[runtime.resource_limits]
|
|
||||||
cpu_shares = 1024
|
|
||||||
memory_limit_mb = 4096
|
|
||||||
```
|
|
||||||
|
|
||||||
## Versioning
|
|
||||||
|
|
||||||
- The manifest format is versioned via `manifest_version`.
|
|
||||||
- Only version `1` is supported in Karapace 0.1.
|
|
||||||
- Future versions will increment this field.
|
|
||||||
- Backward-incompatible changes require a version bump.
|
|
||||||
|
|
@ -1,109 +0,0 @@
|
||||||
# Karapace Remote Protocol v1 (Draft)
|
|
||||||
|
|
||||||
> **Status: v1-draft** — This protocol is subject to change before 1.1.
|
|
||||||
|
|
||||||
## Overview
|
|
||||||
|
|
||||||
The Karapace remote protocol defines how environments are transferred between local stores and a remote HTTP backend. It is a content-addressable blob store with an optional registry index for named references.
|
|
||||||
|
|
||||||
## Blob Types
|
|
||||||
|
|
||||||
| Kind | Key | Content |
|
|
||||||
|------|-----|---------|
|
|
||||||
| `Object` | blake3 hex hash | Raw object data (tar layers, manifests) |
|
|
||||||
| `Layer` | layer hash | JSON `LayerManifest` |
|
|
||||||
| `Metadata` | env_id | JSON `EnvMetadata` |
|
|
||||||
|
|
||||||
## HTTP Routes
|
|
||||||
|
|
||||||
All routes are relative to the remote base URL.
|
|
||||||
|
|
||||||
### Blobs
|
|
||||||
|
|
||||||
| Method | Route | Description |
|
|
||||||
|--------|-------|-------------|
|
|
||||||
| `PUT` | `/blobs/{kind}/{key}` | Upload a blob |
|
|
||||||
| `GET` | `/blobs/{kind}/{key}` | Download a blob |
|
|
||||||
| `HEAD` | `/blobs/{kind}/{key}` | Check if blob exists |
|
|
||||||
| `GET` | `/blobs/{kind}` | List blob keys |
|
|
||||||
|
|
||||||
### Registry
|
|
||||||
|
|
||||||
| Method | Route | Description |
|
|
||||||
|--------|-------|-------------|
|
|
||||||
| `PUT` | `/registry` | Upload registry index |
|
|
||||||
| `GET` | `/registry` | Download registry index |
|
|
||||||
|
|
||||||
## Push Protocol
|
|
||||||
|
|
||||||
1. Read local `EnvMetadata` for the target env_id.
|
|
||||||
2. Collect layer hashes: `base_layer` + `dependency_layers`.
|
|
||||||
3. For each layer, collect `object_refs`.
|
|
||||||
4. Upload objects (skip if `HEAD` returns 200).
|
|
||||||
5. Upload layer manifests (skip if `HEAD` returns 200).
|
|
||||||
6. Upload metadata blob.
|
|
||||||
7. If a registry tag is provided, download current registry, merge entry, upload updated registry.
|
|
||||||
|
|
||||||
## Pull Protocol
|
|
||||||
|
|
||||||
1. Download `Metadata` blob for the target env_id.
|
|
||||||
2. Collect layer hashes from metadata.
|
|
||||||
3. Download layer manifests (skip if local store has them).
|
|
||||||
4. Collect `object_refs` from all layers.
|
|
||||||
5. Download objects (skip if local store has them).
|
|
||||||
6. **Verify integrity**: compute blake3 hash of each downloaded object; reject if hash does not match key.
|
|
||||||
7. Store metadata locally.
|
|
||||||
|
|
||||||
## Registry Format
|
|
||||||
|
|
||||||
```json
|
|
||||||
{
|
|
||||||
"entries": {
|
|
||||||
"my-env@latest": {
|
|
||||||
"env_id": "abc123...",
|
|
||||||
"short_id": "abc123def456",
|
|
||||||
"name": "my-env",
|
|
||||||
"pushed_at": "2026-01-15T12:00:00Z"
|
|
||||||
}
|
|
||||||
}
|
|
||||||
}
|
|
||||||
```
|
|
||||||
|
|
||||||
### Reference Resolution
|
|
||||||
|
|
||||||
References follow the format `name@tag` (default tag: `latest`). Resolution:
|
|
||||||
1. Parse reference into `(name, tag)`.
|
|
||||||
2. Look up `{name}@{tag}` in the registry.
|
|
||||||
3. Return the associated `env_id`.
|
|
||||||
|
|
||||||
## Integrity
|
|
||||||
|
|
||||||
- All objects are keyed by their blake3 hash.
|
|
||||||
- On pull, every downloaded object is re-hashed and compared to its key.
|
|
||||||
- Mismatches produce `RemoteError::IntegrityFailure`.
|
|
||||||
- Layer manifests and metadata are JSON — no hash verification (they are keyed by logical identifiers, not content hashes).
|
|
||||||
|
|
||||||
## Headers
|
|
||||||
|
|
||||||
| Header | Value | When |
|
|
||||||
|--------|-------|------|
|
|
||||||
| `Content-Type` | `application/octet-stream` | Blob uploads/downloads |
|
|
||||||
| `Content-Type` | `application/json` | Registry uploads/downloads |
|
|
||||||
|
|
||||||
## Version Negotiation
|
|
||||||
|
|
||||||
Not yet implemented. The protocol version will be negotiated via a `X-Karapace-Protocol` header in 1.1.
|
|
||||||
|
|
||||||
## Error Responses
|
|
||||||
|
|
||||||
| Status | Meaning |
|
|
||||||
|--------|---------|
|
|
||||||
| 200 | Success |
|
|
||||||
| 404 | Blob or registry not found |
|
|
||||||
| 500 | Server error |
|
|
||||||
|
|
||||||
## Security Considerations
|
|
||||||
|
|
||||||
- No authentication in v1-draft. Authentication headers will be added in 1.1.
|
|
||||||
- HTTPS is strongly recommended for all remote URLs.
|
|
||||||
- Object integrity is verified client-side via blake3.
|
|
||||||
|
|
@ -1,83 +1,105 @@
|
||||||
# Karapace Security Model
|
# Security Model
|
||||||
|
|
||||||
## Core Invariants
|
Defined in `karapace-runtime/src/security.rs::SecurityPolicy`.
|
||||||
|
|
||||||
1. **Host filesystem protected by default.** No writes outside the store and overlay.
|
## Privilege model
|
||||||
2. **No implicit `/dev` access.** Device passthrough is explicit and opt-in.
|
|
||||||
3. **GPU passthrough requires `hardware.gpu = true`** in the manifest.
|
|
||||||
4. **Audio passthrough requires `hardware.audio = true`** in the manifest.
|
|
||||||
5. **Network isolation is configurable** via `runtime.network_isolation`.
|
|
||||||
6. **No permanent root daemon required.** All operations are rootless.
|
|
||||||
7. **No SUID binaries required.**
|
|
||||||
|
|
||||||
## Mount Policy
|
Karapace runs entirely as an unprivileged user. No SUID binaries. No root daemon. Isolation is provided by Linux user namespaces (`unshare`).
|
||||||
|
|
||||||
- Absolute host paths are checked against a whitelist.
|
The OCI backend delegates to rootless runtimes (`crun`, `runc`, `youki`).
|
||||||
- Default allowed prefixes: `/home`, `/tmp`.
|
|
||||||
- Relative paths (e.g. `./`) are always allowed.
|
|
||||||
- Mounts outside the whitelist are rejected at build time.
|
|
||||||
|
|
||||||
## Device Policy
|
## Mount policy
|
||||||
|
|
||||||
- Default policy denies all device access.
|
Absolute host paths in manifest `[mounts]` are validated against an allowlist.
|
||||||
- `hardware.gpu = true` adds `/dev/dri` to allowed devices.
|
|
||||||
- `hardware.audio = true` adds `/dev/snd` to allowed devices.
|
|
||||||
- No implicit device passthrough.
|
|
||||||
|
|
||||||
## Environment Variable Control
|
**Default allowed prefixes:** `/home`, `/tmp`.
|
||||||
|
|
||||||
- A whitelist of safe environment variables is passed to the environment:
|
Relative paths (e.g. `./`) are always permitted. Mounts outside the allowlist are rejected at build time with `RuntimeError::MountDenied`.
|
||||||
`TERM`, `LANG`, `HOME`, `USER`, `PATH`, `SHELL`, `XDG_RUNTIME_DIR`.
|
|
||||||
- A denylist prevents sensitive variables from leaking:
|
|
||||||
`SSH_AUTH_SOCK`, `GPG_AGENT_INFO`, `AWS_SECRET_ACCESS_KEY`, `DOCKER_HOST`.
|
|
||||||
- Only whitelisted, non-denied variables are propagated.
|
|
||||||
|
|
||||||
## Resource Limits
|
Path traversal is prevented by `canonicalize_logical()` in `security.rs`, which resolves `..` components before checking the prefix.
|
||||||
|
|
||||||
- CPU shares and memory limits are declared in the manifest.
|
Defined in `SecurityPolicy::validate_mounts`.
|
||||||
- The security policy enforces upper bounds.
|
|
||||||
- Exceeding policy limits causes a build failure, not a silent cap.
|
|
||||||
|
|
||||||
## Privilege Model
|
## Device policy
|
||||||
|
|
||||||
- No privileged escalation at any point.
|
Default policy denies all device access.
|
||||||
- User namespaces provide isolation without root.
|
|
||||||
- The OCI backend uses rootless runtimes (crun, runc, youki).
|
|
||||||
|
|
||||||
## Threat Model
|
- `hardware.gpu = true` → allows `/dev/dri`
|
||||||
|
- `hardware.audio = true` → allows `/dev/snd`
|
||||||
|
|
||||||
### Privilege Boundary
|
No implicit device passthrough. Defined in `SecurityPolicy::validate_devices`.
|
||||||
|
|
||||||
- Karapace operates entirely within the user's privilege level.
|
## Environment variable control
|
||||||
- The store is owned by the user and protected by filesystem permissions.
|
|
||||||
- No daemon runs with elevated privileges.
|
|
||||||
|
|
||||||
### Attack Surface
|
**Allowed** (propagated into the container):
|
||||||
|
|
||||||
- **Manifest parsing**: strict TOML parser with `deny_unknown_fields`.
|
```
|
||||||
- **Store integrity**: blake3 verification on every object read.
|
TERM, LANG, HOME, USER, PATH, SHELL, XDG_RUNTIME_DIR
|
||||||
- **Image integrity**: blake3 digest stored on download, verified on cache hit.
|
```
|
||||||
- **Shell injection**: all paths and values in sandbox scripts use POSIX single-quote escaping (`shell_quote`). Environment variable keys are validated against `[a-zA-Z0-9_]`.
|
|
||||||
- **Mount injection**: prevented by whitelist enforcement.
|
|
||||||
- **Environment variable leakage**: prevented by deny/allow lists.
|
|
||||||
- **Concurrent access**: file locking on all mutating CLI and D-Bus operations.
|
|
||||||
- **Process safety**: cannot destroy a running environment (must stop first).
|
|
||||||
- **Rebuild atomicity**: new environment is built before the old one is destroyed.
|
|
||||||
|
|
||||||
### Isolation Assumptions
|
**Denied** (never propagated):
|
||||||
|
|
||||||
- The host kernel provides functioning user namespaces.
|
```
|
||||||
- The filesystem permissions on the store directory are correct.
|
SSH_AUTH_SOCK, GPG_AGENT_INFO, AWS_SECRET_ACCESS_KEY, DOCKER_HOST
|
||||||
- The OCI runtime (if used) is trusted and correctly installed.
|
```
|
||||||
|
|
||||||
### Security Review Checklist
|
Only variables present in the allowed list and absent from the denied list are passed through. Defined in `SecurityPolicy::filter_env_vars`.
|
||||||
|
|
||||||
- [x] No SUID binaries.
|
## Resource limits
|
||||||
- [x] No root daemon.
|
|
||||||
- [x] Mount whitelist enforced.
|
Declared in manifest `[runtime.resource_limits]`:
|
||||||
- [x] Device passthrough explicit.
|
|
||||||
- [x] Environment variables controlled.
|
- `cpu_shares`: CPU shares limit
|
||||||
- [x] Resource limits supported.
|
- `memory_limit_mb`: memory limit in MB
|
||||||
- [x] Store integrity verified.
|
|
||||||
- [x] Concurrent access safe.
|
If the policy defines upper bounds, requesting values above them causes a build-time error (`RuntimeError::ResourceLimitExceeded`). Defined in `SecurityPolicy::validate_resource_limits`.
|
||||||
- [x] Signal handling (SIGINT/SIGTERM) clean.
|
|
||||||
|
## Store integrity
|
||||||
|
|
||||||
|
- **Objects:** blake3 hash verified on every read. Key = hash of content.
|
||||||
|
- **Metadata:** blake3 checksum embedded in each metadata file, verified on every `get()`.
|
||||||
|
- **Layers:** file content re-hashed against filename on read.
|
||||||
|
- **Images:** content digest stored on download, re-verified on cache hits.
|
||||||
|
|
||||||
|
## Concurrency
|
||||||
|
|
||||||
|
File locking (`flock(2)`) on `store/.lock` for all mutating operations. Both CLI and D-Bus service acquire the lock. Defined in `karapace-core/src/concurrency.rs::StoreLock`.
|
||||||
|
|
||||||
|
Running environments cannot be destroyed (must be stopped first).
|
||||||
|
|
||||||
|
## Shell injection prevention
|
||||||
|
|
||||||
|
Sandbox scripts use POSIX single-quote escaping for all paths and values. Environment variable keys are validated against `[a-zA-Z0-9_]`. Defined in `karapace-runtime/src/sandbox.rs`.
|
||||||
|
|
||||||
|
## Manifest parsing
|
||||||
|
|
||||||
|
Strict TOML parser with `deny_unknown_fields` on all sections. Unknown keys cause a parse error.
|
||||||
|
|
||||||
|
## What is NOT protected
|
||||||
|
|
||||||
|
- Karapace does not protect against a compromised host kernel.
|
||||||
|
- Karapace does not verify the authenticity of upstream base images beyond content hashing.
|
||||||
|
- The OCI runtime (if used) is trusted.
|
||||||
|
- Filesystem permissions on the store directory are the user's responsibility.
|
||||||
|
- Network isolation (`runtime.network_isolation`) depends on the backend implementation.
|
||||||
|
- No MAC (SELinux/AppArmor) enforcement within the container.
|
||||||
|
|
||||||
|
## Trust assumptions
|
||||||
|
|
||||||
|
1. The host kernel provides functioning, secure user namespaces.
|
||||||
|
2. The store directory has correct ownership and permissions.
|
||||||
|
3. `fuse-overlayfs` is correctly installed and not compromised.
|
||||||
|
4. The OCI runtime (if used) is a trusted binary.
|
||||||
|
5. Base images from `images.linuxcontainers.org` are fetched over HTTPS but not GPG-verified.
|
||||||
|
|
||||||
|
## Unsafe code
|
||||||
|
|
||||||
|
Five `unsafe` blocks, all FFI calls to libc:
|
||||||
|
|
||||||
|
| File | Call | Purpose |
|
||||||
|
|------|------|---------|
|
||||||
|
| `karapace-core/src/engine.rs` | `libc::kill(SIGTERM)` | Stop running environment |
|
||||||
|
| `karapace-core/src/engine.rs` | `libc::kill(SIGKILL)` | Force-kill after timeout |
|
||||||
|
| `karapace-runtime/src/sandbox.rs` | `libc::getuid()` | Get UID for namespace mapping |
|
||||||
|
| `karapace-runtime/src/sandbox.rs` | `libc::getgid()` | Get GID for namespace mapping |
|
||||||
|
| `karapace-runtime/src/terminal.rs` | `libc::isatty()` | Detect terminal for interactive mode |
|
||||||
|
|
|
||||||
212
docs/storage-format.md
Normal file
212
docs/storage-format.md
Normal file
|
|
@ -0,0 +1,212 @@
|
||||||
|
# Storage Format
|
||||||
|
|
||||||
|
Store format version: **2**. Defined in `karapace-store/src/layout.rs::STORE_FORMAT_VERSION`.
|
||||||
|
|
||||||
|
## Directory layout
|
||||||
|
|
||||||
|
Default root: `~/.local/share/karapace`.
|
||||||
|
|
||||||
|
```
|
||||||
|
<root>/
|
||||||
|
store/
|
||||||
|
version # { "format_version": 2 }
|
||||||
|
.lock # flock(2) exclusive lock
|
||||||
|
objects/<blake3_hex> # content-addressable blobs
|
||||||
|
layers/<blake3_hex> # layer manifests (JSON)
|
||||||
|
metadata/<env_id> # environment metadata (JSON)
|
||||||
|
staging/ # temp workspace for atomic operations
|
||||||
|
wal/<op_id>.json # write-ahead log entries
|
||||||
|
env/
|
||||||
|
<env_id>/
|
||||||
|
upper/ # overlay writable layer
|
||||||
|
overlay/ # overlay mount point
|
||||||
|
images/
|
||||||
|
<cache_key>/
|
||||||
|
rootfs/ # extracted base image filesystem
|
||||||
|
```
|
||||||
|
|
||||||
|
Paths defined in `karapace-store/src/layout.rs::StoreLayout`.
|
||||||
|
|
||||||
|
## Version file
|
||||||
|
|
||||||
|
```json
|
||||||
|
{ "format_version": 2 }
|
||||||
|
```
|
||||||
|
|
||||||
|
Checked on every store access. Mismatched versions are rejected with `StoreError::VersionMismatch`.
|
||||||
|
|
||||||
|
## Objects
|
||||||
|
|
||||||
|
Content-addressable blobs keyed by blake3 hex digest of their content.
|
||||||
|
|
||||||
|
- Write: `NamedTempFile` in objects dir → write content → `sync_all()` → `persist()` (atomic rename)
|
||||||
|
- Read: read file → recompute blake3 → compare to filename → reject on mismatch
|
||||||
|
- Idempotent: writing identical content is a no-op
|
||||||
|
|
||||||
|
Defined in `karapace-store/src/objects.rs::ObjectStore`.
|
||||||
|
|
||||||
|
## Layers
|
||||||
|
|
||||||
|
JSON files in `store/layers/`. Each describes a tar archive stored in the object store.
|
||||||
|
|
||||||
|
```json
|
||||||
|
{
|
||||||
|
"hash": "<layer_hash>",
|
||||||
|
"kind": "Base | Dependency | Policy | Snapshot",
|
||||||
|
"parent": "<parent_hash> | null",
|
||||||
|
"object_refs": ["<hash>", ...],
|
||||||
|
"read_only": true,
|
||||||
|
"tar_hash": "<blake3_of_tar>"
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
Defined in `karapace-store/src/layers.rs::LayerManifest`.
|
||||||
|
|
||||||
|
**Layer kinds:**
|
||||||
|
|
||||||
|
| Kind | Hash computation | Parent |
|
||||||
|
|------|-----------------|--------|
|
||||||
|
| `Base` | `tar_hash` | None |
|
||||||
|
| `Dependency` | `tar_hash` | Base layer |
|
||||||
|
| `Policy` | `tar_hash` | — |
|
||||||
|
| `Snapshot` | `blake3("snapshot:{env_id}:{base_layer}:{tar_hash}")` | Base layer |
|
||||||
|
|
||||||
|
Layer integrity is verified on read: the file content is re-hashed and compared to the filename.
|
||||||
|
|
||||||
|
### Deterministic tar packing
|
||||||
|
|
||||||
|
`karapace-store/src/layers.rs::pack_layer(source_dir)`:
|
||||||
|
|
||||||
|
- Entries sorted by path
|
||||||
|
- Timestamps set to 0
|
||||||
|
- Owner set to `0:0`
|
||||||
|
- Permissions preserved
|
||||||
|
- Symlink targets preserved
|
||||||
|
|
||||||
|
**Dropped during packing:** extended attributes, device nodes, hardlinks (stored as regular files), SELinux labels, ACLs, sparse file holes.
|
||||||
|
|
||||||
|
`unpack_layer(tar_data, target_dir)` reverses the process.
|
||||||
|
|
||||||
|
## Metadata
|
||||||
|
|
||||||
|
JSON files in `store/metadata/`, one per environment. Filename is the `env_id`.
|
||||||
|
|
||||||
|
```json
|
||||||
|
{
|
||||||
|
"env_id": "...",
|
||||||
|
"short_id": "...",
|
||||||
|
"name": null,
|
||||||
|
"state": "Built",
|
||||||
|
"manifest_hash": "<object_hash>",
|
||||||
|
"base_layer": "<layer_hash>",
|
||||||
|
"dependency_layers": [],
|
||||||
|
"policy_layer": null,
|
||||||
|
"created_at": "RFC3339",
|
||||||
|
"updated_at": "RFC3339",
|
||||||
|
"ref_count": 1,
|
||||||
|
"checksum": "<blake3_of_json>"
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
Defined in `karapace-store/src/metadata.rs::EnvMetadata`.
|
||||||
|
|
||||||
|
**States:** `Defined`, `Built`, `Running`, `Frozen`, `Archived`.
|
||||||
|
|
||||||
|
**Checksum:** blake3 of the JSON content (excluding the checksum field itself). Computed on every `put()`, verified on every `get()`. Absent in legacy metadata (`#[serde(default)]`).
|
||||||
|
|
||||||
|
**Names:** optional, validated by `validate_env_name`: pattern `[a-zA-Z0-9_-]`, 1–64 characters. Unique across all environments.
|
||||||
|
|
||||||
|
## Manifest format
|
||||||
|
|
||||||
|
File: `karapace.toml`. Parsed by `karapace-schema/src/manifest.rs`.
|
||||||
|
|
||||||
|
```toml
|
||||||
|
manifest_version = 1
|
||||||
|
|
||||||
|
[base]
|
||||||
|
image = "rolling"
|
||||||
|
|
||||||
|
[system]
|
||||||
|
packages = ["git", "curl"]
|
||||||
|
|
||||||
|
[gui]
|
||||||
|
apps = []
|
||||||
|
|
||||||
|
[hardware]
|
||||||
|
gpu = false
|
||||||
|
audio = false
|
||||||
|
|
||||||
|
[mounts]
|
||||||
|
workspace = "./:/workspace"
|
||||||
|
|
||||||
|
[runtime]
|
||||||
|
backend = "namespace"
|
||||||
|
network_isolation = false
|
||||||
|
|
||||||
|
[runtime.resource_limits]
|
||||||
|
cpu_shares = 1024
|
||||||
|
memory_limit_mb = 4096
|
||||||
|
```
|
||||||
|
|
||||||
|
**Required:** `manifest_version` (must be `1`), `base.image` (non-empty).
|
||||||
|
|
||||||
|
**Optional:** all other sections. Unknown fields cause a parse error (`deny_unknown_fields`).
|
||||||
|
|
||||||
|
**Normalization** (`ManifestV1::normalize`): trim strings, sort and deduplicate packages/apps, sort mounts by label, lowercase backend name. Produces `NormalizedManifest` with a `canonical_json()` method.
|
||||||
|
|
||||||
|
## Lock file
|
||||||
|
|
||||||
|
File: `karapace.lock`. Written next to the manifest. TOML format.
|
||||||
|
|
||||||
|
```toml
|
||||||
|
lock_version = 2
|
||||||
|
env_id = "46e1d96f..."
|
||||||
|
short_id = "46e1d96fdd6f"
|
||||||
|
base_image = "rolling"
|
||||||
|
base_image_digest = "a1b2c3d4..."
|
||||||
|
runtime_backend = "namespace"
|
||||||
|
hardware_gpu = false
|
||||||
|
hardware_audio = false
|
||||||
|
network_isolation = false
|
||||||
|
|
||||||
|
[[resolved_packages]]
|
||||||
|
name = "git"
|
||||||
|
version = "2.44.0-1"
|
||||||
|
```
|
||||||
|
|
||||||
|
Defined in `karapace-schema/src/lock.rs::LockFile`.
|
||||||
|
|
||||||
|
**Verification:**
|
||||||
|
- `verify_integrity()`: recomputes `env_id` from locked fields, compares to stored value
|
||||||
|
- `verify_manifest_intent()`: checks manifest hasn't drifted from what was locked
|
||||||
|
|
||||||
|
## Hashing
|
||||||
|
|
||||||
|
All hashing uses **blake3**, 256-bit output, hex-encoded (64 characters).
|
||||||
|
|
||||||
|
Used for: object keys, layer hashes, env_id computation, metadata checksums, image content digests.
|
||||||
|
|
||||||
|
## Write-ahead log
|
||||||
|
|
||||||
|
`store/wal/<op_id>.json`. Defined in `karapace-store/src/wal.rs`.
|
||||||
|
|
||||||
|
```json
|
||||||
|
{
|
||||||
|
"op_id": "20260215120000123-a1b2c3d4",
|
||||||
|
"kind": "Build",
|
||||||
|
"env_id": "...",
|
||||||
|
"timestamp": "RFC3339",
|
||||||
|
"rollback_steps": [
|
||||||
|
{ "RemoveDir": "/path" },
|
||||||
|
{ "RemoveFile": "/path" }
|
||||||
|
]
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
**Operations:** `Build`, `Rebuild`, `Commit`, `Restore`, `Destroy`, `Gc`.
|
||||||
|
|
||||||
|
**Recovery:** on `Engine::new()`, all WAL entries are scanned. Each entry's rollback steps execute in reverse order. The entry is then deleted. Corrupt entries are silently removed.
|
||||||
|
|
||||||
|
## Atomic write contract
|
||||||
|
|
||||||
|
All store writes follow: `NamedTempFile::new_in(dir)` → write → `flush()` → `persist()` (atomic rename). No partial files are visible. Defined throughout `karapace-store`.
|
||||||
|
|
@ -1,150 +0,0 @@
|
||||||
# Karapace Store Format Specification (v2)
|
|
||||||
|
|
||||||
## Overview
|
|
||||||
|
|
||||||
The Karapace store is a content-addressable filesystem structure that holds all environment data: objects, layers, metadata, environment directories, and crash recovery state.
|
|
||||||
|
|
||||||
## Directory Layout
|
|
||||||
|
|
||||||
```
|
|
||||||
<store_root>/
|
|
||||||
store/
|
|
||||||
version # JSON: { "format_version": 2 }
|
|
||||||
.lock # flock(2) file for exclusive access
|
|
||||||
objects/<hash> # Content-addressable blobs (blake3)
|
|
||||||
layers/<hash> # Layer manifests (JSON)
|
|
||||||
metadata/<env_id> # Environment metadata (JSON)
|
|
||||||
staging/ # Temporary workspace for atomic operations
|
|
||||||
wal/ # Write-ahead log entries (JSON)
|
|
||||||
env/
|
|
||||||
<env_id>/
|
|
||||||
upper/ # Writable overlay layer (fuse-overlayfs upperdir)
|
|
||||||
lower -> ... # Symlink to base image rootfs
|
|
||||||
work/ # Overlay workdir (ephemeral)
|
|
||||||
merged/ # Overlay mount point
|
|
||||||
images/
|
|
||||||
<cache_key>/
|
|
||||||
rootfs/ # Extracted base image rootfs
|
|
||||||
```
|
|
||||||
|
|
||||||
## Format Version
|
|
||||||
|
|
||||||
- Current version: **2**
|
|
||||||
- Stored in `store/version` as JSON.
|
|
||||||
- Checked on every store access; mismatches are rejected.
|
|
||||||
- Version 1 stores are not auto-migrated; a clean rebuild is required.
|
|
||||||
|
|
||||||
## Objects
|
|
||||||
|
|
||||||
- Keyed by blake3 hex digest of their content.
|
|
||||||
- Written atomically: write to tempfile, then rename.
|
|
||||||
- Integrity verified on every read: content re-hashed and compared to filename.
|
|
||||||
- Idempotent: writing the same content twice is a no-op.
|
|
||||||
|
|
||||||
## Layers
|
|
||||||
|
|
||||||
Each layer is a JSON manifest:
|
|
||||||
|
|
||||||
```json
|
|
||||||
{
|
|
||||||
"hash": "<layer_hash>",
|
|
||||||
"kind": "Base" | "Dependency" | "Policy" | "Snapshot",
|
|
||||||
"parent": "<parent_hash>" | null,
|
|
||||||
"object_refs": ["<hash>", ...],
|
|
||||||
"read_only": true,
|
|
||||||
"tar_hash": "<blake3_hash>"
|
|
||||||
}
|
|
||||||
```
|
|
||||||
|
|
||||||
- `tar_hash` (v2): blake3 hash of the deterministic tar archive stored in the object store.
|
|
||||||
- Base layers have no parent. Their `hash` equals their `tar_hash`.
|
|
||||||
- Dependency layers reference a base parent.
|
|
||||||
- Snapshot layers are created by `commit`. Their `hash` is a composite identity: `blake3("snapshot:{env_id}:{base_layer}:{tar_hash}")` to prevent collision with base layers.
|
|
||||||
|
|
||||||
## Metadata
|
|
||||||
|
|
||||||
Each environment has a JSON metadata file:
|
|
||||||
|
|
||||||
```json
|
|
||||||
{
|
|
||||||
"env_id": "...",
|
|
||||||
"short_id": "...",
|
|
||||||
"name": "my-env",
|
|
||||||
"state": "Defined" | "Built" | "Running" | "Frozen" | "Archived",
|
|
||||||
"manifest_hash": "<object_hash>",
|
|
||||||
"base_layer": "<layer_hash>",
|
|
||||||
"dependency_layers": ["<hash>", ...],
|
|
||||||
"policy_layer": null | "<hash>",
|
|
||||||
"created_at": "RFC3339",
|
|
||||||
"updated_at": "RFC3339",
|
|
||||||
"ref_count": 1
|
|
||||||
}
|
|
||||||
```
|
|
||||||
|
|
||||||
- `name` is optional (`#[serde(default)]`). Old metadata without this field deserializes correctly.
|
|
||||||
|
|
||||||
## Atomic Write Contract
|
|
||||||
|
|
||||||
All writes follow the pattern:
|
|
||||||
1. Create `NamedTempFile` in the target directory.
|
|
||||||
2. Write full content.
|
|
||||||
3. `flush()`.
|
|
||||||
4. `persist()` (atomic rename).
|
|
||||||
|
|
||||||
This ensures no partial files are visible.
|
|
||||||
|
|
||||||
## Garbage Collection
|
|
||||||
|
|
||||||
- Environments with `ref_count == 0` and state not in {`Running`, `Archived`} are eligible for collection.
|
|
||||||
- Layers not referenced by any live environment are orphaned.
|
|
||||||
- Objects not referenced by any live layer or live metadata (`manifest_hash`) are orphaned.
|
|
||||||
- GC never deletes running or archived environments.
|
|
||||||
- GC supports graceful cancellation via signal handler (`SIGINT`/`SIGTERM`).
|
|
||||||
- `--dry-run` reports what would be removed without acting.
|
|
||||||
- The caller must hold the store lock before running GC.
|
|
||||||
|
|
||||||
## Write-Ahead Log (WAL)
|
|
||||||
|
|
||||||
The `store/wal/` directory contains JSON entries for in-flight mutating operations. Each entry tracks:
|
|
||||||
|
|
||||||
```json
|
|
||||||
{
|
|
||||||
"op_id": "20260215120000123-a1b2c3d4",
|
|
||||||
"kind": "Build" | "Rebuild" | "Commit" | "Restore" | "Destroy",
|
|
||||||
"env_id": "...",
|
|
||||||
"timestamp": "RFC3339",
|
|
||||||
"rollback_steps": [
|
|
||||||
{ "RemoveDir": "/path/to/orphaned/dir" },
|
|
||||||
{ "RemoveFile": "/path/to/orphaned/file" }
|
|
||||||
]
|
|
||||||
}
|
|
||||||
```
|
|
||||||
|
|
||||||
### Recovery Protocol
|
|
||||||
|
|
||||||
1. On `Engine::new()`, the WAL directory is scanned for incomplete entries.
|
|
||||||
2. Each entry's rollback steps are executed in **reverse order**.
|
|
||||||
3. The WAL entry is then removed.
|
|
||||||
4. Corrupt or unreadable WAL entries are silently deleted.
|
|
||||||
|
|
||||||
### Invariants
|
|
||||||
|
|
||||||
- **INV-W1**: Kill during rebuild → next startup rolls back orphaned env_dir.
|
|
||||||
- **INV-W2**: Kill during build → orphaned env_dir cleaned.
|
|
||||||
- **INV-W3**: Successful operations leave zero WAL entries.
|
|
||||||
|
|
||||||
## Staging Directory
|
|
||||||
|
|
||||||
The `store/staging/` directory is a temporary workspace used for atomic operations:
|
|
||||||
|
|
||||||
- **Restore**: snapshot tar is unpacked into `staging/restore-{env_id}`, then renamed to replace the overlay upper directory.
|
|
||||||
- **Layer packing**: temporary files during tar creation.
|
|
||||||
|
|
||||||
The staging directory is cleaned up after each operation. Leftover staging data is safe to delete.
|
|
||||||
|
|
||||||
## Backward Compatibility
|
|
||||||
|
|
||||||
- Layout changes require a format version bump.
|
|
||||||
- Karapace 1.0 requires format version 2.
|
|
||||||
- Version 1 stores are not auto-migrated; environments must be rebuilt.
|
|
||||||
- The `name` and `tar_hash` fields use `#[serde(default)]` for forward-compatible deserialization.
|
|
||||||
|
|
@ -1,132 +0,0 @@
|
||||||
# Verifying Karapace Downloads
|
|
||||||
|
|
||||||
Every tagged release includes signed binaries for both glibc and musl (static) targets,
|
|
||||||
SHA256 checksums, and in-toto provenance attestations. All artifacts are signed with
|
|
||||||
cosign using GitHub Actions OIDC — no manual keys involved.
|
|
||||||
|
|
||||||
These instructions are mechanically verified in CI: the `verify-docs-executable` job
|
|
||||||
in `supply-chain-test.yml` executes these exact commands on every PR.
|
|
||||||
|
|
||||||
## Choosing a Binary
|
|
||||||
|
|
||||||
| Binary | Linking | Use Case |
|
|
||||||
|--------|---------|----------|
|
|
||||||
| `karapace-linux-x86_64-gnu` | Dynamic (glibc) | Standard Linux distributions |
|
|
||||||
| `karapace-linux-x86_64-musl` | Static (musl) | Minimal containers, Alpine, any distro |
|
|
||||||
| `karapace-dbus-linux-x86_64-gnu` | Dynamic (glibc) | D-Bus service on standard distros |
|
|
||||||
| `karapace-dbus-linux-x86_64-musl` | Static (musl) | D-Bus service in containers |
|
|
||||||
|
|
||||||
The musl binaries are fully statically linked and require no system libraries.
|
|
||||||
|
|
||||||
## 1. Verify SHA256 Checksums
|
|
||||||
|
|
||||||
```bash
|
|
||||||
# For glibc binaries:
|
|
||||||
sha256sum -c SHA256SUMS-gnu
|
|
||||||
|
|
||||||
# For musl binaries:
|
|
||||||
sha256sum -c SHA256SUMS-musl
|
|
||||||
```
|
|
||||||
|
|
||||||
Both `karapace` and `karapace-dbus` must show `OK`.
|
|
||||||
|
|
||||||
## 2. Verify Cosign Signatures
|
|
||||||
|
|
||||||
Install [cosign](https://docs.sigstore.dev/cosign/system_config/installation/):
|
|
||||||
|
|
||||||
```bash
|
|
||||||
# Verify karapace binary (use -gnu or -musl suffix as appropriate)
|
|
||||||
cosign verify-blob karapace-linux-x86_64-gnu \
|
|
||||||
--signature karapace-gnu.sig \
|
|
||||||
--certificate karapace-gnu.crt \
|
|
||||||
--certificate-identity-regexp 'https://github.com/marcoallegretti/karapace' \
|
|
||||||
--certificate-oidc-issuer https://token.actions.githubusercontent.com
|
|
||||||
|
|
||||||
# Verify karapace-dbus binary
|
|
||||||
cosign verify-blob karapace-dbus-linux-x86_64-gnu \
|
|
||||||
--signature karapace-dbus-gnu.sig \
|
|
||||||
--certificate karapace-dbus-gnu.crt \
|
|
||||||
--certificate-identity-regexp 'https://github.com/marcoallegretti/karapace' \
|
|
||||||
--certificate-oidc-issuer https://token.actions.githubusercontent.com
|
|
||||||
```
|
|
||||||
|
|
||||||
Both commands must print `Verified OK`. Replace `-gnu` with `-musl` for static binaries.
|
|
||||||
|
|
||||||
## 3. Verify Provenance Attestation
|
|
||||||
|
|
||||||
```bash
|
|
||||||
cosign verify-blob provenance-gnu.json \
|
|
||||||
--signature provenance-gnu.json.sig \
|
|
||||||
--certificate provenance-gnu.json.crt \
|
|
||||||
--certificate-identity-regexp 'https://github.com/marcoallegretti/karapace' \
|
|
||||||
--certificate-oidc-issuer https://token.actions.githubusercontent.com
|
|
||||||
```
|
|
||||||
|
|
||||||
Inspect the provenance to verify the build origin:
|
|
||||||
|
|
||||||
```bash
|
|
||||||
python3 -c "
|
|
||||||
import json
|
|
||||||
with open('provenance-gnu.json') as f:
|
|
||||||
prov = json.load(f)
|
|
||||||
src = prov['predicate']['invocation']['configSource']
|
|
||||||
print(f'Commit: {src[\"digest\"][\"sha1\"]}')
|
|
||||||
print(f'Repo: {src[\"uri\"]}')
|
|
||||||
print(f'Workflow: {src[\"entryPoint\"]}')
|
|
||||||
print(f'Builder: {prov[\"predicate\"][\"builder\"][\"id\"]}')
|
|
||||||
"
|
|
||||||
```
|
|
||||||
|
|
||||||
## 4. Inspect SBOM
|
|
||||||
|
|
||||||
The CycloneDX SBOM lists all Rust dependencies and their versions:
|
|
||||||
|
|
||||||
```bash
|
|
||||||
python3 -m json.tool karapace_bom.json | head -50
|
|
||||||
```
|
|
||||||
|
|
||||||
## Build Reproducibility
|
|
||||||
|
|
||||||
All CI release builds enforce:
|
|
||||||
- `CARGO_INCREMENTAL=0` — disables incremental compilation
|
|
||||||
- `cargo clean` before every release build — eliminates stale intermediate artifacts
|
|
||||||
- `SOURCE_DATE_EPOCH=0` — deterministic timestamps
|
|
||||||
- `--remap-path-prefix` — eliminates runner-specific filesystem paths
|
|
||||||
- `strip = true` + `lto = "thin"` — deterministic output
|
|
||||||
|
|
||||||
**Reproducibility requirement:** Build invocations must use identical `-p` flags.
|
|
||||||
Building `-p karapace-cli` alone may produce a different binary than
|
|
||||||
`-p karapace-cli -p karapace-dbus` due to codegen unit ordering.
|
|
||||||
|
|
||||||
Both glibc and musl builds are verified for same-run and cross-run reproducibility
|
|
||||||
(ubuntu-latest vs ubuntu-22.04). Musl static builds are expected to be fully
|
|
||||||
runner-independent since they have no dynamic library dependencies.
|
|
||||||
|
|
||||||
## Local Development Builds
|
|
||||||
|
|
||||||
Local dev builds use `.cargo/config.toml` to remap dependency paths via `--remap-path-prefix`.
|
|
||||||
This eliminates local filesystem paths from release binaries. The remapping is configured for
|
|
||||||
the project maintainer's paths; other developers should update the paths in `.cargo/config.toml`
|
|
||||||
or set `RUSTFLAGS` directly.
|
|
||||||
|
|
||||||
**Local builds are for development only. CI builds are the authoritative release artifacts.**
|
|
||||||
|
|
||||||
## Release Artifacts
|
|
||||||
|
|
||||||
Each GitHub release contains artifacts for both `x86_64-unknown-linux-gnu` (glibc) and
|
|
||||||
`x86_64-unknown-linux-musl` (static) targets:
|
|
||||||
|
|
||||||
| File | Description |
|
|
||||||
|------|-------------|
|
|
||||||
| `karapace-linux-x86_64-gnu` | CLI binary (glibc) |
|
|
||||||
| `karapace-linux-x86_64-musl` | CLI binary (static musl) |
|
|
||||||
| `karapace-dbus-linux-x86_64-gnu` | D-Bus service binary (glibc) |
|
|
||||||
| `karapace-dbus-linux-x86_64-musl` | D-Bus service binary (static musl) |
|
|
||||||
| `SHA256SUMS-gnu` | SHA256 checksums for glibc binaries |
|
|
||||||
| `SHA256SUMS-musl` | SHA256 checksums for musl binaries |
|
|
||||||
| `karapace-gnu.sig` / `.crt` | Cosign signature + certificate (glibc CLI) |
|
|
||||||
| `karapace-musl.sig` / `.crt` | Cosign signature + certificate (musl CLI) |
|
|
||||||
| `karapace-dbus-gnu.sig` / `.crt` | Cosign signature + certificate (glibc D-Bus) |
|
|
||||||
| `karapace-dbus-musl.sig` / `.crt` | Cosign signature + certificate (musl D-Bus) |
|
|
||||||
| `provenance-gnu.json` / `.sig` / `.crt` | Provenance attestation (glibc) |
|
|
||||||
| `provenance-musl.json` / `.sig` / `.crt` | Provenance attestation (musl) |
|
|
||||||
|
|
@ -1,66 +0,0 @@
|
||||||
# Karapace Versioning Policy
|
|
||||||
|
|
||||||
## Versioned Artifacts
|
|
||||||
|
|
||||||
Karapace versions three independent artifacts:
|
|
||||||
|
|
||||||
| Artifact | Current Version | Location |
|
|
||||||
|---|---|---|
|
|
||||||
| Manifest format | `1` | `manifest_version` field in manifest |
|
|
||||||
| Store layout | `2` | `store/version` file |
|
|
||||||
| DBus API | `1` | `org.karapace.Manager1` interface |
|
|
||||||
| Remote protocol | `v1-draft` | `X-Karapace-Protocol` header |
|
|
||||||
|
|
||||||
## Freeze Policies
|
|
||||||
|
|
||||||
### Store Format Freeze
|
|
||||||
|
|
||||||
- **Store layout v2 is frozen as of Karapace 1.0.**
|
|
||||||
- No breaking changes to the store directory layout, object naming scheme, or metadata JSON schema within 1.x releases.
|
|
||||||
- New optional fields may be added to metadata JSON with `#[serde(default)]`.
|
|
||||||
- Store v1 is not supported; Karapace 1.0+ rejects v1 stores with a clear error.
|
|
||||||
- If a future major version changes the store format, `karapace migrate` will be provided.
|
|
||||||
|
|
||||||
### Remote Protocol Freeze
|
|
||||||
|
|
||||||
- **Remote protocol v1 is currently `v1-draft`** and may change before Karapace 1.1.
|
|
||||||
- The protocol will be frozen (declared stable) in Karapace 1.1.
|
|
||||||
- After freeze: blob routes, registry format, and integrity checking are stable.
|
|
||||||
- New optional endpoints may be added without a version bump.
|
|
||||||
- Breaking changes require a protocol version bump and `X-Karapace-Protocol` negotiation.
|
|
||||||
|
|
||||||
## Compatibility Rules
|
|
||||||
|
|
||||||
### Manifest Format
|
|
||||||
|
|
||||||
- The `manifest_version` field is required and checked on parse.
|
|
||||||
- Only version `1` is supported in Karapace 0.1.
|
|
||||||
- Adding optional fields is a backward-compatible change (no version bump).
|
|
||||||
- Removing or renaming fields requires a version bump.
|
|
||||||
- Changing normalization or hashing behavior requires a version bump.
|
|
||||||
|
|
||||||
### Store Layout
|
|
||||||
|
|
||||||
- The store format version is stored in `store/version`.
|
|
||||||
- Karapace refuses to operate on a store with a different version.
|
|
||||||
- Adding new file types to the store is backward-compatible.
|
|
||||||
- Changing the object naming scheme or layout requires a version bump.
|
|
||||||
|
|
||||||
### DBus API
|
|
||||||
|
|
||||||
- The API version is returned by `ApiVersion()`.
|
|
||||||
- Adding new methods is backward-compatible.
|
|
||||||
- Changing method signatures requires a version bump.
|
|
||||||
- The interface name includes the major version (`Manager1`).
|
|
||||||
|
|
||||||
## Release Policy
|
|
||||||
|
|
||||||
- **Patch releases** (0.1.x): bug fixes only, no format changes.
|
|
||||||
- **Minor releases** (0.x.0): may add backward-compatible features.
|
|
||||||
- **Major releases** (x.0.0): may break backward compatibility with version bumps.
|
|
||||||
|
|
||||||
## Migration
|
|
||||||
|
|
||||||
- Karapace 0.1 does not support migration from other tools.
|
|
||||||
- Future versions may include `karapace migrate` for store upgrades.
|
|
||||||
- Manifest version migration is the user's responsibility.
|
|
||||||
Loading…
Reference in a new issue