From e6e0f3dd6d2bb3cf77f5f97d929fdc426bf20f99 Mon Sep 17 00:00:00 2001 From: Marco Allegretti Date: Mon, 23 Feb 2026 01:06:42 +0100 Subject: [PATCH] docs: rewrite all documentation from source code MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Delete 14 old docs files (AI-generated, riddled with Phase/M1/1.0 jargon, references to non-existent commands, stale CI snippets). New documentation (6 files), written from repository source analysis: - docs/architecture.md — crate graph, engine lifecycle, identity computation, runtime backends, store design, WAL, GC, unsafe blocks - docs/cli-reference.md — all 23 commands with syntax, args, flags, exit codes, env vars, verified against crates/karapace-cli/src/main.rs - docs/storage-format.md — directory layout, objects, layers, metadata, manifest format, lock file, WAL, atomic write contract - docs/security-model.md — mount/device/env var policies with exact defaults from security.rs, trust assumptions, what is NOT protected - docs/build-and-reproducibility.md — CI env vars, RUSTFLAGS, cargo profile, reproducibility verification, toolchain pinning - docs/contributing.md — setup, verification, project layout, code standards, testing, CI workflows README.md rewritten: concise, no marketing language, prerequisites first, usage example, command table, limitations section. CONTRIBUTING.md now points to docs/contributing.md. CHANGELOG.md cleaned: removed M1-M8 labels, Phase refs, stale counts. --- CHANGELOG.md | 26 ++- CONTRIBUTING.md | 98 +--------- README.md | 290 ++++++++++-------------------- docs/api-reference.md | 213 ---------------------- docs/architecture.md | 249 +++++++++++++++++-------- docs/build-and-reproducibility.md | 83 +++++++++ docs/cli-reference.md | 286 +++++++++++++++++++++++++++++ docs/cli-stability.md | 84 --------- docs/contributing.md | 98 ++++++++++ docs/e2e-testing.md | 77 -------- docs/getting-started.md | 282 ----------------------------- docs/hash-contract.md | 60 ------- docs/layer-limitations-v1.md | 77 -------- docs/lock-spec.md | 94 ---------- docs/manifest-spec.md | 113 ------------ docs/protocol-v1.md | 109 ----------- docs/security-model.md | 146 ++++++++------- docs/storage-format.md | 212 ++++++++++++++++++++++ docs/store-spec.md | 150 ---------------- docs/verification.md | 132 -------------- docs/versioning-policy.md | 66 ------- 21 files changed, 1047 insertions(+), 1898 deletions(-) delete mode 100644 docs/api-reference.md create mode 100644 docs/build-and-reproducibility.md create mode 100644 docs/cli-reference.md delete mode 100644 docs/cli-stability.md create mode 100644 docs/contributing.md delete mode 100644 docs/e2e-testing.md delete mode 100644 docs/getting-started.md delete mode 100644 docs/hash-contract.md delete mode 100644 docs/layer-limitations-v1.md delete mode 100644 docs/lock-spec.md delete mode 100644 docs/manifest-spec.md delete mode 100644 docs/protocol-v1.md create mode 100644 docs/storage-format.md delete mode 100644 docs/store-spec.md delete mode 100644 docs/verification.md delete mode 100644 docs/versioning-policy.md diff --git a/CHANGELOG.md b/CHANGELOG.md index d2550a3..6f07aa4 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -4,7 +4,7 @@ All notable changes to Karapace will be documented in this file. The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/). -## [Unreleased] — 2.0 Hardening +## [Unreleased] ### Breaking Changes @@ -14,19 +14,15 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/). - **`Engine::gc()` requires `&StoreLock`** — compile-time enforcement that callers hold the store lock before garbage collection. - **MetadataStore checksum** — `EnvMetadata` now includes an optional `checksum` field (blake3). Written on every `put()`, verified on every `get()`. Backward-compatible via `serde(default)`. -### Added — 2.0 Hardening (M1–M8) - -- **M1: WAL Crash Safety** — Fixed race windows in `build()` and `restore()` (rollback registered before side-effects). Added WAL protection to `destroy()`, `commit()` (layer manifest rollback), and `gc()` (WAL marker). 8 new WAL crash-safety tests. -- **M2: Integrity Hardening** — `LayerStore::get()` verifies blake3 hash on every read. `MetadataStore` embeds and verifies blake3 checksum. `verify_store_integrity()` expanded to check objects, layers, and metadata. 4 new integrity tests. -- **M3: GC Safety** — `Engine::gc()` now requires `&StoreLock` parameter (type-enforced). Snapshot layers whose parent is a live base layer are preserved during GC. -- **M4: Remote Protocol** — `X-Karapace-Protocol: 1` header sent on all HTTP backend requests (PUT, GET, HEAD). `PROTOCOL_VERSION` constant exported from `karapace-remote`. 4 new header/auth verification tests via header-capturing mock server. -- **M5: unwrap() Audit** — 0 `unwrap()` in production `src/` code. 4 `Mutex::lock().unwrap()` calls in `MockBackend` replaced with proper `RuntimeError` propagation. -- **M6: Failure Mode Testing** — 11 new tests: WAL write failure on read-only dir, build failure when WAL dir is read-only (disk-full simulation), stop() SIGTERM with real process (ESRCH path), stop() with non-existent PID, permission denied on object read, read-only metadata dir, concurrent GC lock contention, layer corruption detection, metadata corruption detection, destroy nonexistent env, invalid manifest. -- **M7: Coverage Expansion** — `verify_store_integrity()` now checks objects + layers + metadata (was objects-only). `IntegrityReport` expanded with `layers_checked/passed` and `metadata_checked/passed` fields. New tests: freeze/archive state transitions, rename environment, verify-store after fresh build, rebuild preserves new and cleans old, HTTP list_blobs, large (1MB) blob roundtrip. -- **Total: 417 tests** (24 ignored, require privileged operations). Clippy `-D warnings` clean. `cargo fmt` clean. Release build OK. - -### Added — 1.0 Preparation +### Added +- **WAL crash safety** — Fixed race windows in `build()` and `restore()` (rollback registered before side-effects). Added WAL protection to `destroy()`, `commit()` (layer manifest rollback), and `gc()` (WAL marker). +- **Integrity hardening** — `LayerStore::get()` verifies blake3 hash on every read. `MetadataStore` embeds and verifies blake3 checksum. `verify_store_integrity()` expanded to check objects, layers, and metadata. +- **GC safety** — `Engine::gc()` now requires `&StoreLock` parameter (type-enforced). Snapshot layers whose parent is a live base layer are preserved during GC. +- **Remote protocol headers** — `X-Karapace-Protocol: 1` header sent on all HTTP backend requests (PUT, GET, HEAD). `PROTOCOL_VERSION` constant exported from `karapace-remote`. +- **unwrap() audit** — 0 `unwrap()` in production code. `Mutex::lock().unwrap()` calls in `MockBackend` replaced with proper `RuntimeError` propagation. +- **Failure mode tests** — WAL write failure, build on read-only WAL dir, stop() with SIGTERM/non-existent PID, permission denied on object read, read-only metadata dir, concurrent GC lock contention, layer/metadata corruption detection. +- **Coverage expansion** — `verify_store_integrity()` now checks objects + layers + metadata. `IntegrityReport` expanded with `layers_checked/passed` and `metadata_checked/passed` fields. - **Real tar layers** — `pack_layer()`/`unpack_layer()` in karapace-store: deterministic tar creation (sorted entries, zero timestamps, owner 0:0) for regular files, directories, and symlinks. Content-addressed via blake3. - **Snapshot system** — `Engine::commit()` captures overlay upper as a tar snapshot; `Engine::restore()` atomically unpacks a snapshot via staging directory swap; `Engine::list_snapshots()` lists snapshots for an environment. - **CLI: `snapshots` and `restore`** — new commands for snapshot management. @@ -34,9 +30,9 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/). - **Newtype wrappers threaded through all structs** — `EnvId`, `ShortId`, `ObjectHash`, `LayerHash` now used in `EnvMetadata` across all 8 crates. Transparent serde for backward compatibility. - **Engine::push/pull** — transfer logic moved from `karapace-remote` to `Engine` methods. `karapace-remote` is now pure I/O. - **CoreError::Remote** — new error variant for remote operation failures. -- **CLI stability contract** — `docs/cli-stability.md` defines stable command signatures for 1.x. +- **CLI stability contract** — `docs/cli-stability.md` defines CLI stability expectations. - **Remote protocol spec** — `docs/protocol-v1.md` (v1-draft) documents blob store routes, push/pull protocol, registry format. -- **Layer limitations doc** — `docs/layer-limitations-v1.md` documents Phase 1 limits (no xattrs, device nodes, hardlinks). +- **Layer limitations doc** — `docs/layer-limitations.md` documents current limits (no xattrs, device nodes, hardlinks). ### Changed diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index bde7cb9..4ff0a4b 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -1,97 +1,5 @@ -# Contributing to Karapace +# Contributing -## Development Setup +See [docs/contributing.md](docs/contributing.md) for setup, code standards, testing, and CI details. -```bash -# Clone and build -git clone https://github.com/marcoallegretti/karapace.git -cd karapace -cargo build - -# Run tests -cargo test --workspace - -# Full verification (must pass before submitting) -cargo fmt --all --check -cargo clippy --workspace --all-targets -- -D warnings -cargo test --workspace -cargo build --release --workspace -``` - -## Project Structure - -``` -crates/ - karapace-schema/ # Manifest parsing, normalization, lock file, identity hashing - karapace-store/ # Content-addressable store, metadata, layers, GC, integrity - karapace-runtime/ # Container runtime: images, sandbox, host integration, security - karapace-core/ # Build engine, lifecycle state machine, drift control, concurrency - karapace-remote/ # Remote store client, push/pull, registry - karapace-server/ # Reference remote server (tiny_http) - karapace-tui/ # Terminal UI (ratatui) - karapace-cli/ # CLI interface (23 commands) - karapace-dbus/ # D-Bus desktop integration (optional) -docs/ # Public documentation and specifications -examples/ # Ready-to-use manifest examples -data/ # systemd and D-Bus service files -``` - -## Architecture Principles - -Before implementing any feature, verify it aligns with these principles: - -1. **Determinism first.** Same manifest + lock = identical environment, always. -2. **No hidden mutable state.** All state changes are explicit and tracked. -3. **No silent drift.** Overlay changes are visible via `diff` and must be committed explicitly. -4. **No privilege escalation.** Everything runs as the unprivileged user. -5. **Convenience must not break reproducibility.** If there's a conflict, determinism wins. - -See [Architecture Overview](docs/architecture.md) for the full design. - -## Code Standards - -- **Zero warnings**: `cargo clippy --workspace --all-targets -- -D warnings` must pass. -- **Formatted**: `cargo fmt --all --check` must pass. -- **No `unwrap()` in production code** (test code is fine). -- **No `TODO`/`FIXME`/`HACK`** in committed code. -- **All values interpolated into shell scripts must use `shell_quote()`.** -- **All mutating operations must hold a `StoreLock`.** -- **All file writes must be atomic** (use `NamedTempFile` + `persist()`). - -## Testing - -- All new features must include tests. -- Run the full test suite: `cargo test --workspace` -- Integration tests go in `crates/karapace-core/tests/`. -- Unit tests go in the relevant module as `#[cfg(test)] mod tests`. - -## Submitting Changes - -1. Fork the repository. -2. Create a feature branch from `main`. -3. Make your changes, ensuring all verification checks pass. -4. Submit a pull request with a clear description. - -## Building the D-Bus Service - -The D-Bus service is not compiled by default: - -```bash -# Core CLI only (default) -cargo build --release - -# Include D-Bus service -cargo build --release --workspace -``` - -## Generating Shell Completions - -```bash -karapace completions bash > /etc/bash_completion.d/karapace -karapace completions zsh > /usr/share/zsh/site-functions/_karapace -karapace completions fish > ~/.config/fish/completions/karapace.fish -``` - -## License - -By contributing, you agree that your contributions will be licensed under the European Union Public Licence v1.2 (EUPL-1.2). +By contributing, you agree that your contributions will be licensed under EUPL-1.2. diff --git a/README.md b/README.md index d1cd753..17f1a1a 100644 --- a/README.md +++ b/README.md @@ -3,165 +3,20 @@ [![CI](https://github.com/marcoallegretti/karapace/actions/workflows/ci.yml/badge.svg)](https://github.com/marcoallegretti/karapace/actions/workflows/ci.yml) [![License: EUPL-1.2](https://img.shields.io/badge/License-EUPL_1.2-blue.svg)](LICENSE) -A deterministic container environment engine for immutable Linux systems. +Deterministic, content-addressed container environments for Linux. No root. No daemon. -Karapace creates isolated, reproducible development environments using Linux namespaces and overlay filesystems — no root, no daemon, no Docker. Environments are content-addressed artifacts derived from declarative TOML manifests, with full host integration for GPU, audio, Wayland, and desktop applications. +Karapace builds isolated development environments from declarative TOML manifests using Linux user namespaces and overlay filesystems. Each environment gets a unique identity (blake3 hash) derived from its fully resolved dependencies. -## What Karapace Is (and Isn't) +## Prerequisites -Karapace is **the identity and certainty layer** for reproducible environments. It is not a general-purpose container runtime. +- Linux with user namespaces (`CONFIG_USER_NS=y`) +- `fuse-overlayfs` +- `curl` +- Optional: `crun`/`runc`/`youki` (OCI backend) -| Need | Tool | -|---|---| -| Full system container lifecycle, advanced networking, snapshots | Incus, LXD, Podman | -| Deterministic, content-addressed, reproducible environments | **Karapace** | -| Quick disposable containers with zero config | Distrobox | -| Determinism + simplicity for casual use | **Karapace** `quick` command | +Run `karapace doctor` to check. -Karapace is **complementary** — it can sit on top of or alongside any container runtime. Users needing full container features use a runtime; users needing reproducibility and traceability use Karapace. The `quick` command bridges the gap for users who want both simplicity and determinism. - -## Features - -- **Real container isolation** — Linux user namespaces (`unshare`), `fuse-overlayfs`, `chroot` -- **No root required** — runs entirely as an unprivileged user -- **No daemon** — direct CLI, no background service needed -- **Multi-distro images** — openSUSE, Ubuntu, Debian, Fedora, Arch from LXC image servers -- **Package installation** — `zypper`, `apt`, `dnf`, `pacman` inside the container -- **Host integration** — home directory, Wayland, PipeWire, D-Bus, GPU (`/dev/dri`), audio (`/dev/snd`), SSH agent, fonts, themes -- **Desktop app export** — export GUI apps as `.desktop` files on the host -- **OCI runtime support** — optional `crun`/`runc`/`youki` backend -- **Content-addressable store** — deterministic hashing, deduplication, integrity verification -- **Overlay drift control** — diff, freeze, commit, export writable layer changes -- **OSC 777 terminal markers** — container-aware terminal integration (Konsole, etc.) - -## Crate Layout - -| Crate | Purpose | -|---|---| -| `karapace-schema` | Manifest v1 parsing, normalization, identity hashing, lock file | -| `karapace-store` | Content-addressable store, layers, metadata, GC, integrity | -| `karapace-runtime` | Container runtime: image download, sandbox, host integration, app export | -| `karapace-core` | Build engine, lifecycle state machine, drift control, concurrency | -| `karapace-cli` | Full CLI interface (23 commands) | -| `karapace-dbus` | Socket-activated D-Bus desktop integration (**optional**, feature-gated) | -| `karapace-remote` | Remote content-addressable store, push/pull, registry | -| `karapace-server` | Reference remote server implementing protocol v1 (tiny_http) | -| `karapace-tui` | Terminal UI for environment management (ratatui) | - -## CLI Commands (23) - -``` -karapace build [manifest] # Build environment from manifest -karapace rebuild [manifest] # Destroy + rebuild -karapace enter [-- cmd...] # Enter environment (or run a command) -karapace exec -- # Execute command inside environment -karapace destroy # Destroy environment -karapace stop # Stop a running environment -karapace freeze # Freeze environment -karapace archive # Archive environment (preserve, prevent entry) -karapace list # List all environments -karapace inspect # Show environment metadata -karapace diff # Show overlay drift -karapace snapshots # List snapshots for an environment -karapace commit # Commit overlay drift as snapshot -karapace restore # Restore overlay from snapshot -karapace gc [--dry-run] # Garbage collect store -karapace verify-store # Check store integrity -karapace push [--tag name@tag] # Push environment to remote store -karapace pull [--remote url] # Pull environment from remote store -karapace rename # Rename environment -karapace doctor # Run diagnostic checks on system and store -karapace migrate # Check store version and migration guidance -karapace completions # Generate shell completions -karapace man-pages [dir] # Generate man pages -``` - -All commands support `--json` for structured output, `--store ` for custom store location, `--verbose` / `-v` for debug logging, and `--trace` for trace-level output. - -Set `KARAPACE_LOG=debug` (or `info`, `warn`, `error`, `trace`) for fine-grained log control. - -## Quick Start - -```bash -# Build (core CLI only — D-Bus service is opt-in) -cargo build --release - -# Build with D-Bus desktop integration -cargo build --release --workspace -``` - -```bash -# Write a manifest -cat > karapace.toml << 'EOF' -manifest_version = 1 - -[base] -image = "rolling" # openSUSE Tumbleweed (or "ubuntu/24.04", "fedora/41", "arch", etc.) - -[system] -packages = ["git", "curl", "vim"] - -[hardware] -gpu = true -audio = true - -[runtime] -backend = "namespace" -EOF - -karapace build -karapace enter - -# Run a command inside without interactive shell -karapace exec -- git --version - -# Snapshot and restore -karapace commit -karapace snapshots -karapace restore - -# Push/pull to remote -karapace push --tag my-env@latest -karapace pull my-env@latest - -# List environments -karapace list -``` - -## Example Manifests - -Ready-to-use manifests in `examples/`: - -| File | Description | -|---|---| -| `examples/minimal.toml` | Bare openSUSE system, no extras | -| `examples/dev.toml` | Developer tools (git, vim, tmux, gcc, clang) | -| `examples/gui-dev.toml` | GUI development with GPU + audio passthrough | -| `examples/ubuntu-dev.toml` | Ubuntu-based with Node.js, Python, build-essential | -| `examples/rust-dev.toml` | Rust development environment | - -## Shell Completions - -```bash -# Bash -karapace completions bash > /etc/bash_completion.d/karapace - -# Zsh -karapace completions zsh > /usr/share/zsh/site-functions/_karapace - -# Fish -karapace completions fish > ~/.config/fish/completions/karapace.fish -``` - -## Man Pages - -```bash -karapace man-pages /usr/share/man/man1 -``` - -## Installation - -### From Source (recommended) +## Install ```bash git clone https://github.com/marcoallegretti/karapace.git @@ -170,53 +25,106 @@ cargo build --release sudo install -Dm755 target/release/karapace /usr/local/bin/karapace ``` -### With D-Bus Service - -```bash -cargo build --release --workspace -sudo install -Dm755 target/release/karapace /usr/local/bin/karapace -sudo install -Dm755 target/release/karapace-dbus /usr/local/bin/karapace-dbus -sudo install -Dm644 data/dbus/org.karapace.Manager1.service /usr/share/dbus-1/services/ -sudo install -Dm644 data/systemd/karapace-dbus.service /usr/lib/systemd/user/ -``` - -### Via Cargo +Or via cargo: ```bash cargo install --git https://github.com/marcoallegretti/karapace.git karapace-cli ``` -## Prerequisites +## Usage -- Linux with user namespace support (`CONFIG_USER_NS=y`) -- `fuse-overlayfs` (for overlay filesystem) -- `curl` (for image downloads) -- Optional: `crun`/`runc`/`youki` (for OCI backend) +```bash +# Create a manifest +cat > karapace.toml << 'EOF' +manifest_version = 1 -Karapace checks for missing prerequisites at startup and provides distro-specific install instructions. +[base] +image = "rolling" + +[system] +packages = ["git", "curl"] + +[runtime] +backend = "namespace" +EOF + +# Build and enter +karapace build +karapace enter + +# Run a command +karapace exec -- git --version + +# Snapshot +karapace commit +karapace restore +``` + +See `examples/` for more manifests: `minimal.toml`, `dev.toml`, `gui-dev.toml`, `ubuntu-dev.toml`, `rust-dev.toml`. + +## Commands + +23 commands. All accept `--json`, `--store `, `--verbose`, `--trace`. + +``` +build [manifest] [--name] Build environment +rebuild [manifest] [--name] Destroy + rebuild +enter [-- cmd...] Enter environment +exec -- Run command in environment +destroy Destroy environment +stop Stop running environment +freeze Freeze (prevent writes) +archive Archive (prevent entry) +list List environments +inspect Show metadata +diff Show overlay changes +snapshots List snapshots +commit Snapshot overlay +restore Restore snapshot +gc [--dry-run] Garbage collect +verify-store Check store integrity +push [--tag] [--remote] Push to remote +pull [--remote] Pull from remote +rename Rename environment +doctor Check prerequisites +migrate Check store version +completions Shell completions +man-pages [dir] Generate man pages +``` + +## Workspace + +9 crates: + +``` +karapace-schema Manifest, normalization, lock file, identity +karapace-store Objects, layers, metadata, WAL, GC, integrity +karapace-runtime Backends (namespace/oci/mock), images, security +karapace-core Engine: lifecycle orchestration +karapace-cli CLI (23 commands) +karapace-dbus D-Bus service (optional) +karapace-tui Terminal UI (optional) +karapace-remote Remote store client, push/pull +karapace-server Reference HTTP server +``` + +## Limitations + +- Linux only. +- Layer packing drops: extended attributes, device nodes, hardlinks, SELinux labels, ACLs. +- Base images are content-hashed but not GPG-verified. +- No MAC enforcement (SELinux/AppArmor) inside containers. +- Remote protocol has no authentication yet. ## Documentation -- **[Getting Started Guide](docs/getting-started.md)** — installation, first use, common workflows -- [Architecture Overview](docs/architecture.md) -- [Manifest v1 Specification](docs/manifest-spec.md) -- [Lock File v2 Specification](docs/lock-spec.md) -- [Store Format v2 Specification](docs/store-spec.md) -- [Hash Contract](docs/hash-contract.md) +- [Architecture](docs/architecture.md) +- [CLI Reference](docs/cli-reference.md) +- [Storage Format](docs/storage-format.md) - [Security Model](docs/security-model.md) -- [CLI Stability Contract](docs/cli-stability.md) -- [Remote Protocol v1 (Draft)](docs/protocol-v1.md) -- [Layer Limitations (Phase 1)](docs/layer-limitations-v1.md) -- [Public API Reference](docs/api-reference.md) -- [Versioning Policy](docs/versioning-policy.md) -- [Verification & Supply Chain](docs/verification.md) -- [E2E Testing](docs/e2e-testing.md) +- [Build and Reproducibility](docs/build-and-reproducibility.md) +- [Contributing](docs/contributing.md) -## Verification +## License -```bash -cargo fmt --all --check -cargo clippy --workspace --all-targets -- -D warnings -cargo test --workspace -cargo build --release --workspace -``` +[EUPL-1.2](LICENSE) diff --git a/docs/api-reference.md b/docs/api-reference.md deleted file mode 100644 index dbd8941..0000000 --- a/docs/api-reference.md +++ /dev/null @@ -1,213 +0,0 @@ -# Karapace Public API Reference - -## CLI Commands - -### Environment Lifecycle - -#### `karapace init [manifest]` -Initialize an environment from a manifest without building. Creates metadata and a preliminary lock file. -- **Default manifest**: `karapace.toml` - -#### `karapace build [manifest]` -Build an environment. Resolves dependencies, computes canonical identity, creates store objects/layers, and writes the lock file. -- **Default manifest**: `karapace.toml` - -#### `karapace enter [-- cmd...]` -Enter a built environment interactively, or run a command if `-- cmd` is provided. Transitions state to Running, then back to Built on exit. -- Accepts full env_id or short_id prefix. - -#### `karapace exec -- ` -Execute a command inside a built environment (non-interactive). Prints stdout/stderr. - -#### `karapace rebuild [manifest]` -Atomically rebuild an environment. Builds the new environment first; the old one is only destroyed after a successful build. -- Produces the same env_id for the same resolved manifest. -- If the build fails, the existing environment is preserved. - -#### `karapace stop ` -Stop a running environment by sending SIGTERM/SIGKILL to its process. - -#### `karapace freeze ` -Freeze an environment, preventing further entry. Transitions to Frozen state. - -#### `karapace archive ` -Archive an environment. Preserves it in the store but prevents entry. Can be rebuilt later. - -#### `karapace destroy ` -Destroy an environment's overlay and decrement its reference count. -- **Cannot destroy a running environment** — stop it first. - -### Drift Control - -#### `karapace diff ` -Show drift in the writable overlay. Lists added, modified, and removed files. - -#### `karapace commit ` -Commit overlay drift into the content store as a snapshot layer. -- Only works on Built or Frozen environments. - -#### `karapace export ` -Copy the writable overlay contents to a destination directory. - -### Store Management - -#### `karapace gc [--dry-run]` -Run garbage collection. Removes orphaned environments, layers, and objects. -- `--dry-run`: report only, do not delete. - -#### `karapace verify-store` -Verify integrity of all objects in the store (blake3 content hash check). - -#### `karapace verify-lock [manifest]` -Verify lock file integrity (recomputed env_id matches) and manifest consistency (no drift between manifest and lock). - -### Inspection - -#### `karapace inspect ` -Show environment metadata: state, layers, ref count, timestamps. - -#### `karapace list` -List all known environments with short_id, state, and env_id. - -#### `karapace validate [manifest]` -Validate a manifest file and print its preliminary env_id. - -### Desktop Integration - -#### `karapace export-app ` -Export a GUI application from an environment as a `.desktop` file on the host. - -#### `karapace unexport-app ` -Remove an exported application's `.desktop` file from the host. - -### Image Management - -#### `karapace list-images` -List cached container images with status and size. - -#### `karapace remove-image ` -Remove a cached container image from the store. - -### Quick Start - -#### `karapace quick [image] [-p packages] [--gpu] [--audio] [--enter]` -One-step environment creation for casual users. Generates a manifest from CLI flags, builds, and optionally enters. -- **Default image**: `rolling` (openSUSE Tumbleweed) -- `-p` / `--packages`: Comma-separated list of packages to install. -- `--gpu`: Enable GPU passthrough. -- `--audio`: Enable audio passthrough. -- `-e` / `--enter`: Enter the environment immediately after building. -- A real manifest and lock file are still generated (determinism is preserved). - -Examples: -```bash -karapace quick rolling -p git,curl --enter -karapace quick ubuntu/24.04 -p build-essential,cmake --gpu -karapace quick fedora/41 --enter -``` - -### Tooling - -#### `karapace completions ` -Generate shell completions for the specified shell. Supported: `bash`, `zsh`, `fish`, `elvish`, `powershell`. - -```bash -karapace completions bash > /etc/bash_completion.d/karapace -karapace completions zsh > /usr/share/zsh/site-functions/_karapace -karapace completions fish > ~/.config/fish/completions/karapace.fish -``` - -#### `karapace man-pages [dir]` -Generate man pages for all commands in the specified directory. -- **Default directory**: `man` - -#### `karapace push [--tag ] [--remote ]` -Push an environment (metadata + layers + objects) to a remote store. Skips blobs that already exist remotely. -- **`--tag`**: Publish under a registry key (e.g. `my-env@latest`). -- **`--remote`**: Remote store URL (overrides `~/.config/karapace/remote.json`). - -#### `karapace pull [--remote ]` -Pull an environment from a remote store. Reference can be a registry key (e.g. `my-env@latest`) or a raw env_id. -- **`--remote`**: Remote store URL (overrides config). - -#### `karapace remote-list [--remote ]` -List environments in the remote registry. - -## Global Flags - -| Flag | Description | -|---|---| -| `--store ` | Custom store location (default: `~/.local/share/karapace`). | -| `--json` | Structured JSON output for all applicable commands. | -| `--verbose` / `-v` | Enable debug-level logging output. | - -### Environment Variables - -| Variable | Description | -|---|---| -| `KARAPACE_LOG` | Log level filter: `error`, `warn` (default), `info`, `debug`, `trace`. | -| `KARAPACE_STORE` | Override default store path (used by the D-Bus service). | - -## Exit Codes - -| Code | Meaning | -|---|---| -| `0` | Success. | -| `1` | General error. | -| `2` | Manifest validation error. | -| `3` | Store integrity error. | - -## D-Bus API (Optional) - -Interface: `org.karapace.Manager1` -Path: `/org/karapace/Manager1` - -The D-Bus service exits after 30 seconds of idle (socket activation). Build with `cargo build -p karapace-dbus`. - -All methods return proper D-Bus errors (`org.freedesktop.DBus.Error.Failed`) on failure. Mutating methods acquire the store lock automatically. Methods accepting `id_or_name` resolve by env_id, short_id, name, or prefix. - -Desktop notifications are sent on build success/failure via `org.freedesktop.Notifications`. - -### Properties - -| Property | Type | Description | -|---|---|---| -| `ApiVersion` | `u32` | API version (currently `1`). | -| `StoreRoot` | `String` | Path to the store directory. | - -### Methods - -| Method | Signature | Description | -|---|---|---| -| `ListEnvironments` | `() → String` | JSON array of `{env_id, short_id, name?, state}`. | -| `GetEnvironmentStatus` | `(id_or_name) → String` | JSON status. Resolves by name. | -| `GetEnvironmentHash` | `(id_or_name) → String` | Returns env_id hash. Resolves by name. | -| `BuildEnvironment` | `(manifest_path) → String` | Build from manifest path. Sends notification. | -| `BuildNamedEnvironment` | `(manifest_path, name) → String` | Build and assign a name. Sends notification. | -| `DestroyEnvironment` | `(id_or_name) → String` | Destroy environment. Resolves by name. | -| `RunEnvironment` | `(id_or_name) → String` | Enter environment. Resolves by name. | -| `RenameEnvironment` | `(id_or_name, new_name) → String` | Rename an environment. | -| `ListPresets` | `() → String` | JSON array of `{name, description}` for built-in presets. | -| `GarbageCollect` | `(dry_run) → String` | Run GC. Acquires store lock. | -| `VerifyStore` | `() → String` | Verify store integrity. Returns `{checked, passed, failed}`. | - -## Rust Crate API - -### `karapace-core::Engine` - -- `Engine::new(store_root)` — Create engine instance -- `Engine::init(manifest_path)` — Initialize environment (Defined state) -- `Engine::build(manifest_path)` — Full resolve → lock → build pipeline -- `Engine::enter(env_id)` — Enter environment interactively -- `Engine::exec(env_id, command)` — Execute command in environment -- `Engine::rebuild(manifest_path)` — Destroy + rebuild -- `Engine::stop(env_id)` — Stop running environment -- `Engine::freeze(env_id)` — Freeze environment -- `Engine::archive(env_id)` — Archive environment (preserve, prevent entry) -- `Engine::commit(env_id)` — Commit overlay drift -- `Engine::destroy(env_id)` — Destroy environment -- `Engine::inspect(env_id)` — Get environment metadata -- `Engine::list()` — List all environments -- `Engine::gc(dry_run)` — Garbage collection (caller must hold store lock) -- `Engine::set_name(env_id, name)` — Set or clear environment name -- `Engine::rename(env_id, new_name)` — Rename environment diff --git a/docs/architecture.md b/docs/architecture.md index 1545d44..129d2c6 100644 --- a/docs/architecture.md +++ b/docs/architecture.md @@ -1,90 +1,183 @@ -# Karapace Architecture +# Architecture -## Overview +## Workspace -Karapace is a deterministic container environment engine organized as a Cargo workspace of 9 crates. Each crate has a single responsibility and clean dependency boundaries. - -## Crate Dependency Graph +Karapace is a Cargo workspace of 9 crates. ``` -karapace-cli ─────┬──▶ karapace-core ──┬──▶ karapace-schema - │ ├──▶ karapace-store - │ ├──▶ karapace-runtime - │ └──▶ karapace-remote - ├──▶ karapace-remote ──▶ karapace-store - └──▶ karapace-tui ────▶ karapace-core - -karapace-dbus ────────▶ karapace-core -karapace-server ──────▶ karapace-store (standalone HTTP server) +karapace-schema Manifest parsing, normalization, lock file, identity hashing +karapace-store Content-addressable object store, layers, metadata, WAL, GC +karapace-runtime Container backends, image cache, security policy, prerequisites +karapace-core Engine: orchestrates the full environment lifecycle +karapace-cli CLI binary (23 commands, clap) +karapace-dbus D-Bus service (org.karapace.Manager1, zbus) +karapace-tui Terminal UI (ratatui, crossterm) +karapace-remote Remote store client: HTTP backend, registry, push/pull +karapace-server Reference HTTP server for remote store (tiny_http) ``` -## Crate Responsibilities - -### `karapace-schema` -Manifest v1 parsing (TOML), normalization, canonical JSON serialization, environment identity hashing, lock file v2 (resolved packages, integrity/intent verification), and built-in presets. - -### `karapace-store` -Content-addressable object store (blake3), layer manifests, environment metadata (with naming, ref-counting, state machine), garbage collection with signal cancellation, and store integrity verification. All writes are atomic via `NamedTempFile` + `persist`. - -### `karapace-runtime` -Container runtime abstraction (`RuntimeBackend` trait) with three backends: -- **Namespace** — `unshare` + `fuse-overlayfs` + `chroot` (unprivileged) -- **OCI** — `crun`/`runc`/`youki` -- **Mock** — deterministic test backend - -Also handles image downloading, sandbox scripting, host integration (Wayland, GPU, audio, D-Bus), security policy enforcement, and desktop app export. - -### `karapace-core` -The `Engine` struct orchestrates the full lifecycle: init → resolve → lock → build → enter/exec → freeze → archive → destroy. Caches `MetadataStore`, `ObjectStore`, and `LayerStore` as fields. Handles drift detection (diff/commit/export via overlay upper_dir scanning) and garbage collection delegation. - -### `karapace-cli` -23 CLI commands, each in its own file under `commands/`. Shared helpers in `commands/mod.rs` (spinners, colored output, environment resolution, JSON formatting). `main.rs` is a thin dispatcher. Exit codes: 0 (success), 1 (failure), 2 (manifest error), 3 (store error). - -### `karapace-dbus` -Socket-activated D-Bus service (`org.karapace.Manager1`) with 11 methods. Typed serde response structs. Desktop notifications via `notify-rust`. 30-second idle timeout for socket activation. Hardened systemd unit file. - -### `karapace-remote` -Remote content-addressable store with `RemoteBackend` trait, HTTP backend (ureq), push/pull transfer with blake3 integrity verification on pull, and a JSON registry for name@tag references. - -### `karapace-tui` -Interactive terminal UI (ratatui + crossterm) with list/detail/help views, vim-style keybindings, search/filter, sort cycling, freeze/archive/rename actions, and confirmation dialogs. - -## Key Design Decisions - -1. **Content-addressed identity** — `env_id` is computed from the *resolved* lock file (pinned versions + base image content digest), not from unresolved manifest data. - -2. **Atomic operations** — All store writes use `NamedTempFile` + `persist` for crash safety. Rebuild builds the new environment before destroying the old one. - -3. **No `unwrap()` in production** — All error paths are handled with proper error types (`StoreError`, `CoreError`, `RemoteError`, `RuntimeError`). - -4. **Store locking** — `StoreLock` file lock on all mutating operations (CLI + D-Bus). GC respects active/archived environments. - -5. **Layered security** — Mount whitelist, device policy, env var allow/deny, resource limits. No privilege escalation. - -## Data Flow +## Dependency graph ``` -Manifest (TOML) - │ - ▼ -NormalizedManifest (canonical JSON) - │ - ▼ resolve (RuntimeBackend) -ResolutionResult (base_image_digest + resolved_packages) - │ - ▼ -LockFile v2 (pinned, verifiable) - │ - ▼ compute_identity() -EnvIdentity (env_id = blake3 of canonical lock) - │ - ▼ build (store objects + layers + metadata) -Built Environment (overlay filesystem) +karapace-cli ──┬──> karapace-core ──┬──> karapace-schema + │ ├──> karapace-store + │ ├──> karapace-runtime + │ └──> karapace-remote + ├──> karapace-runtime + └──> karapace-store + +karapace-dbus ────> karapace-core +karapace-tui ─────> karapace-core +karapace-remote ──> karapace-store +karapace-server ──> karapace-remote, karapace-store ``` -### `karapace-server` -Reference remote server implementing protocol v1 over HTTP (tiny_http). Provides blob storage, registry, and list endpoints. Used for testing push/pull workflows. +## Engine lifecycle -## Test Coverage +`karapace-core::Engine` is the central orchestrator. All state transitions go through it. -417 tests across all crates. 24 ignored tests require privileged operations (real `unshare`, `fuse-overlayfs`, ENOSPC simulation, namespace access). +``` + ┌─────────┐ + build() ───> │ Defined │ + └────┬────┘ + │ resolve → lock → build + v + ┌─────────┐ + │ Built │ <── rebuild() + └──┬──┬───┘ + enter() │ │ │ freeze() + v │ v + ┌─────────┐ ┌─────────┐ + │ Running │ │ Frozen │ + └─────────┘ └────┬────┘ + │ archive() + v + ┌──────────┐ + │ Archived │ + └──────────┘ +``` + +State transitions are validated in `karapace-core/src/lifecycle.rs`. Invalid transitions return `CoreError`. + +### Build pipeline + +`Engine::build(manifest_path)` executes: + +1. Parse manifest (`karapace-schema::parse_manifest_file`) +2. Normalize (`ManifestV1::normalize`) — sort packages, deduplicate, lowercase backend +3. Select runtime backend (`karapace-runtime::select_backend`) +4. Resolve — backend downloads base image, computes content digest, queries package manager for exact versions → `ResolutionResult` +5. Create lock file (`LockFile::from_resolved`) with pinned versions and content digest +6. Compute identity (`LockFile::compute_identity`) → `env_id` (blake3) +7. Store manifest as object, create layers, write metadata +8. Backend builds the environment filesystem +9. Write lock file to disk + +### Identity computation + +Defined in `karapace-schema/src/lock.rs::LockFile::compute_identity()`. + +Input fed to blake3 in order: +- `base_digest:` +- `pkg:@` for each resolved package (sorted) +- `app:` for each app (sorted) +- `hw:gpu` / `hw:audio` if enabled +- `mount: