Skip to content
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
11 changes: 11 additions & 0 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -221,3 +221,14 @@ rfc.md

# Markdown/mermaid lint tooling deps
scripts/lint-mermaid/node_modules/

# Devenv
.devenv*
devenv.local.nix
devenv.local.yaml

# direnv
.direnv

# pre-commit
.pre-commit-config.yaml
28 changes: 26 additions & 2 deletions CONTRIBUTING.md
Original file line number Diff line number Diff line change
Expand Up @@ -96,7 +96,27 @@ New issues opened by users without `write`, `maintain`, or `admin` repository pe

## Prerequisites

Install [mise](https://mise.jdx.dev/). This is used to set up the development environment.
If you use Nix, enter the checked-in devenv shell first. It provides `mise`,
Z3, `pkg-config`, platform-specific VM build tools, the Docker CLI, and local
workflow CLIs without using host package managers for those tools:

```bash
devenv shell
mise trust
mise install --locked
```

On macOS, keep Xcode Command Line Tools installed for host developer tools. The
devenv shell configures Nix's Darwin clang wrapper and Apple SDK paths so C++
builds can see SDK and libc++ headers.

The Docker daemon/machine still has to be configured on the host. Podman is not
installed by the devenv shell; if you use the Podman driver, install it on the
host and configure rootless UID/GID mapping (`newuidmap`, `newgidmap`,
`/etc/subuid`, and `/etc/subgid`).

Without Nix/devenv, install [mise](https://mise.jdx.dev/). This is used to set
up the development environment.

```bash
# Install mise (macOS/Linux)
Expand Down Expand Up @@ -127,7 +147,11 @@ Project requirements:

### Z3 installation

The `openshell-prover` crate links against the system Z3 library via pkg-config.
The Nix/devenv shell provides Z3 and `pkg-config` for local builds. If you are
not using Nix/devenv, install Z3 through your platform package manager.

The `openshell-prover` crate links against the system Z3 library via
`pkg-config`.

```bash
# macOS
Expand Down
21 changes: 17 additions & 4 deletions architecture/build.md
Original file line number Diff line number Diff line change
Expand Up @@ -33,10 +33,13 @@ Binary staging is driven by `tasks/scripts/stage-prebuilt-binaries.sh`, which
runs `cargo build` natively on a matching host or `cargo zigbuild` when
cross-compiling. Local Docker image tasks infer the target architecture from
`DOCKER_PLATFORM` when set, otherwise from the container engine host metadata
with the kernel architecture as the fallback. CI invokes the same staging step
via the `rust-native-build.yml` workflow (per-architecture, per-component) and
uploads the result as an artifact that the image build job downloads back into
the staging directory before running Buildx.
with the kernel architecture as the fallback. Static musl supervisor builds use
target C/C++ linker settings from the environment when present, and otherwise
create Zig-backed wrappers so native Nix shells do not link musl Rust artifacts
with the host glibc compiler. CI invokes the same staging step via the
`rust-native-build.yml` workflow (per-architecture, per-component) and uploads
the result as an artifact that the image build job downloads back into the
staging directory before running Buildx.

Runtime layout:

Expand All @@ -60,6 +63,16 @@ the macOS user's shared home directory.
Local image work should use `mise` tasks rather than direct Docker commands so
the same staging and tagging assumptions are used locally and in CI.

Local development can run inside the optional Nix/devenv shell. `devenv.nix`
provides host-native prerequisites such as Z3, `pkg-config`, libclang,
e2fsprogs, container client CLIs, and platform-specific VM build dependencies
(`elfutils`/`libcap-ng` on Linux, `lld`/`dtc` on macOS); `mise` remains the task
runner and installs the version-pinned language/toolchain entries from
`mise.toml`. On macOS the shell configures Nix's Darwin clang wrapper and Apple
SDK paths so C++ probes build against SDK and libc++ headers. The shell also
raises the soft open-file limit when possible so parallel Cargo and `sccache`
workloads do not inherit macOS's low interactive default.

## CI and E2E

Required checks run on GitHub Actions. E2E and GPU workflows use NVIDIA
Expand Down
18 changes: 14 additions & 4 deletions crates/openshell-driver-vm/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -229,11 +229,21 @@ Each table is created atomically via `nft -f` on VM start and torn down atomical
## Prerequisites

- macOS on Apple Silicon, or Linux on aarch64/x86_64 with KVM
- On macOS and Linux, `devenv shell` provides the native build prerequisites
without installing them through Homebrew, `apt`, or `dnf`
- Xcode Command Line Tools on macOS for host developer tools; the devenv shell
configures Nix's Darwin clang wrapper and Apple SDK paths for C++ builds
- Rust toolchain
- e2fsprogs (`mke2fs` or `mkfs.ext4`, plus `debugfs`) for root and overlay disk image creation and QEMU environment injection
- Guest-supervisor cross-compile toolchain (needed on macOS, and on Linux when host arch ≠ guest arch):
- Matching rustup target: `rustup target add aarch64-unknown-linux-gnu` (or `x86_64-unknown-linux-gnu` for an amd64 guest)
- `cargo install --locked cargo-zigbuild` and `brew install zig` (or distro equivalent). `vm:supervisor` uses `cargo zigbuild` to cross-compile the in-VM `openshell-sandbox` supervisor binary.
- e2fsprogs (`mke2fs` or `mkfs.ext4`, plus `debugfs`) for root and overlay disk
image creation and QEMU environment injection
- Guest-supervisor cross-compile toolchain (needed on macOS, and on Linux when
host arch differs from guest arch):
- Matching rustup target: `rustup target add aarch64-unknown-linux-gnu` (or
`x86_64-unknown-linux-gnu` for an amd64 guest)
- `mise install --locked` installs Zig and `cargo-zigbuild` where needed.
Non-mise setups can install `cargo-zigbuild` and Zig through the host
package manager. `vm:supervisor` uses `cargo zigbuild` to cross-compile the
in-VM `openshell-sandbox` supervisor binary.
- [mise](https://mise.jdx.dev/) task runner
- Docker or Podman socket on the local CLI/gateway host when using
`openshell sandbox create --from ./Dockerfile` or `--from ./dir`; the CLI
Expand Down
65 changes: 65 additions & 0 deletions devenv.lock
Original file line number Diff line number Diff line change
@@ -0,0 +1,65 @@
{
"nodes": {
"devenv": {
"locked": {
"dir": "src/modules",
"lastModified": 1778893748,
"narHash": "sha256-VqVecb+w+gCD/hNAUI+3xcJEI4aHThCoq8jr0yBKODU=",
"owner": "cachix",
"repo": "devenv",
"rev": "768261db727bdd30d6a70af0dc9938a79830e515",
"type": "github"
},
"original": {
"dir": "src/modules",
"owner": "cachix",
"repo": "devenv",
"type": "github"
}
},
"nixpkgs": {
"inputs": {
"nixpkgs-src": "nixpkgs-src"
},
"locked": {
"lastModified": 1778507786,
"narHash": "sha256-HzSQCKMsMr8r55LwM1JuzIOB+8bzk0FEv6sItKvsfoY=",
"owner": "cachix",
"repo": "devenv-nixpkgs",
"rev": "8f24a228a782e24576b155d1e39f0d914b380691",
"type": "github"
},
"original": {
"owner": "cachix",
"ref": "rolling",
"repo": "devenv-nixpkgs",
"type": "github"
}
},
"nixpkgs-src": {
"flake": false,
"locked": {
"lastModified": 1778274207,
"narHash": "sha256-I4puXmX1iovcCHZlRmztO3vW0mAbbRvq4F8wgIMQ1MM=",
"owner": "NixOS",
"repo": "nixpkgs",
"rev": "b3da656039dc7a6240f27b2ef8cc6a3ef3bccae7",
"type": "github"
},
"original": {
"owner": "NixOS",
"ref": "nixpkgs-unstable",
"repo": "nixpkgs",
"type": "github"
}
},
"root": {
"inputs": {
"devenv": "devenv",
"nixpkgs": "nixpkgs"
}
}
},
"root": "root",
"version": 7
}
195 changes: 195 additions & 0 deletions devenv.nix
Original file line number Diff line number Diff line change
@@ -0,0 +1,195 @@
# SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
# SPDX-License-Identifier: Apache-2.0

{ pkgs, lib, ... }:

let
isLinux = pkgs.stdenv.isLinux;
isDarwin = pkgs.stdenv.isDarwin;

z3Dev = lib.getDev pkgs.z3;
z3Lib = lib.getLib pkgs.z3;
opensslDev = lib.getDev pkgs.openssl;
opensslLib = lib.getLib pkgs.openssl;
elfutilsDev = lib.getDev pkgs.elfutils;
elfutilsLib = lib.getLib pkgs.elfutils;
libcapNgDev = lib.getDev pkgs.libcap_ng;
libcapNgLib = lib.getLib pkgs.libcap_ng;
libclangLib = pkgs.llvmPackages.libclang.lib;

pkgConfigInputs = [
z3Dev
opensslDev
] ++ lib.optionals isLinux [
elfutilsDev
libcapNgDev
];

nativeLibs = [
z3Lib
opensslLib
libclangLib
] ++ lib.optionals isLinux [
elfutilsLib
libcapNgLib
];

libclangSharedLibrary = if isDarwin then "libclang.dylib" else "libclang.so";

pythonWithPyelftools = pkgs.python3.withPackages (ps: [
ps.pyelftools
]);

zigMuslWrapper = name: tool: target:
pkgs.writeShellScriptBin name ''
set -euo pipefail

args=()
for arg in "$@"; do
case "$arg" in
--target=*) ;;
*) args+=("$arg") ;;
esac
done

exec zig ${tool} --target=${target} "''${args[@]}"
'';

aarch64MuslCc = zigMuslWrapper "openshell-zig-aarch64-linux-musl-cc" "cc" "aarch64-linux-musl";
aarch64MuslCxx = zigMuslWrapper "openshell-zig-aarch64-linux-musl-cxx" "c++" "aarch64-linux-musl";
x86_64MuslCc = zigMuslWrapper "openshell-zig-x86_64-linux-musl-cc" "cc" "x86_64-linux-musl";
x86_64MuslCxx = zigMuslWrapper "openshell-zig-x86_64-linux-musl-cxx" "c++" "x86_64-linux-musl";
in
{
packages = with pkgs; [
# Project task runner. mise installs the version-pinned Rust, Python, Node,
# Kubernetes, documentation, and SBOM tools from mise.toml.
mise

# Core source-control and scripting tools.
cacert
coreutils
curl
file
findutils
gawk
git
gnugrep
gnused
gnutar
gzip
jq
patch
perl
xz
zstd

# Native build prerequisites used by Rust crates and VM runtime builds.
bc
bison
cmake
cpio
e2fsprogs
flex
gnumake
openssl
pkg-config
z3

# Local workflow CLIs. The daemon or machine still needs host setup.
# Podman is intentionally not installed here: Linux rootless Podman also
# needs host uidmap helpers and /etc/subuid + /etc/subgid entries, so use a
# host-installed Podman when that driver is required.
docker-client
gh
] ++ lib.optionals isLinux [
# Linux-only VM runtime build dependencies.
elfutils
gcc
libcap_ng
llvmPackages.libclang
] ++ lib.optionals isDarwin [
# macOS VM runtime build dependencies that are otherwise documented as
# Homebrew prerequisites. Darwin builds use Nix's clang wrapper and Apple
# SDK paths so C++ probes can see SDK and libc++ headers.
dtc
llvmPackages.lld
] ++ [
pythonWithPyelftools
];

env = {
PKG_CONFIG_PATH = lib.concatStringsSep ":" [
(lib.makeSearchPathOutput "dev" "lib/pkgconfig" pkgConfigInputs)
(lib.makeSearchPathOutput "dev" "share/pkgconfig" pkgConfigInputs)
];
LIBRARY_PATH = lib.makeLibraryPath nativeLibs;

LIBCLANG_PATH = "${libclangLib}/lib";
OPENSHELL_LIBKRUNFW_PYTHON = "${pythonWithPyelftools}/bin/python3";
OPENSHELL_SKIP_SYSTEM_DEPS = "1";

Z3_SYS_Z3_HEADER = "${z3Dev}/include/z3.h";
Z3_LIBRARY_PATH_OVERRIDE = "${z3Lib}/lib";

# Match CI's Zig-backed musl toolchain so native Nix shells do not leak
# glibc-built C objects into static supervisor/CLI links.
CC_aarch64_unknown_linux_musl = "${aarch64MuslCc}/bin/openshell-zig-aarch64-linux-musl-cc";
CXX_aarch64_unknown_linux_musl = "${aarch64MuslCxx}/bin/openshell-zig-aarch64-linux-musl-cxx";
CARGO_TARGET_AARCH64_UNKNOWN_LINUX_MUSL_LINKER = "${aarch64MuslCc}/bin/openshell-zig-aarch64-linux-musl-cc";
CARGO_TARGET_AARCH64_UNKNOWN_LINUX_MUSL_RUSTFLAGS = "-Clink-self-contained=no";

CC_x86_64_unknown_linux_musl = "${x86_64MuslCc}/bin/openshell-zig-x86_64-linux-musl-cc";
CXX_x86_64_unknown_linux_musl = "${x86_64MuslCxx}/bin/openshell-zig-x86_64-linux-musl-cxx";
CARGO_TARGET_X86_64_UNKNOWN_LINUX_MUSL_LINKER = "${x86_64MuslCc}/bin/openshell-zig-x86_64-linux-musl-cc";
CARGO_TARGET_X86_64_UNKNOWN_LINUX_MUSL_RUSTFLAGS = "-Clink-self-contained=no";
} // lib.optionalAttrs isLinux {
LD_LIBRARY_PATH = lib.makeLibraryPath nativeLibs;
} // lib.optionalAttrs isDarwin {
DYLD_FALLBACK_LIBRARY_PATH = lib.makeLibraryPath nativeLibs;
};

enterShell = ''
desired_open_files=8192
current_open_files="$(ulimit -Sn 2>/dev/null || ulimit -n 2>/dev/null || echo 0)"

case "$current_open_files" in
""|*[!0-9]*)
;;
*)
if [ "$current_open_files" -lt "$desired_open_files" ]; then
ulimit -Sn "$desired_open_files" 2>/dev/null \
|| ulimit -n "$desired_open_files" 2>/dev/null \
|| echo "Warning: could not raise open file limit to $desired_open_files"
fi
;;
esac

mise_trust_status="$(mise trust --show 2>/dev/null || true)"
if printf '%s\n' "$mise_trust_status" | grep -q ': untrusted$'; then
echo "OpenShell devenv ready. Run 'mise trust' once."
else
mise_missing_tools="$(mise ls --current --missing --no-header 2>/dev/null || true)"
if [ -n "$mise_missing_tools" ]; then
echo "OpenShell devenv ready. Run 'mise install --locked'."
else
echo "OpenShell devenv ready."
fi
fi
'';

enterTest = ''
pkg-config --exists z3
test -f "$Z3_SYS_Z3_HEADER"
test -d "$Z3_LIBRARY_PATH_OVERRIDE"
test -e "$LIBCLANG_PATH/${libclangSharedLibrary}"
"$OPENSHELL_LIBKRUNFW_PYTHON" -c 'from elftools.elf.elffile import ELFFile'
test -x "$(command -v mise)"
test -x "$(command -v mke2fs)"
test -x "$(command -v docker)"
test -x "$CC_aarch64_unknown_linux_musl"
test -x "$CXX_aarch64_unknown_linux_musl"
test -x "$CC_x86_64_unknown_linux_musl"
test -x "$CXX_x86_64_unknown_linux_musl"
'';
}
Loading
Loading