Requirements for shared-gateway multi-tenancy

[pdettori]

Context

We have built Kagenti, an open source multi-tenant AI agent orchestration platform for Kubernetes (Kind, OpenShift, HyperShift). We integrate OpenShell as our sandbox runtime for interactive and headless agent execution. Today we deploy an OpenShell gateway-per-tenant model (one gateway StatefulSet per namespace) which provides hard isolation but creates operational overhead: each tenant is a full gateway + driver + credentials-driver deployment, TLS certificate set, and ingress resource.

We'd like to understand the requirements and constraints for a shared-gateway model — a single gateway instance serving multiple tenants with logical isolation — as an eventual upstream capability. This issue documents the requirements from a downstream integrator's perspective, building on the discussion in #1145 and the "virtual gateway" concept.

We are not proposing a specific implementation — the upstream team is better positioned for that. We're describing what multi-tenancy means to us as adopters and what invariants we need to hold.

---

Requirements

R1: Authenticated caller identity must flow through the entire request lifecycle

Today, once OIDC authentication succeeds (PR #935), the gateway processes requests without carrying the caller's identity context into downstream operations (sandbox creation, listing, deletion, provider access). For shared multi-tenancy, the identity established at authentication must be threaded through so that authorization decisions can be made at each resource boundary.

What we need: After a user authenticates, every operation they perform should be attributable to that identity. The gateway should know "who is asking" at every decision point, not just at the front door.

R2: Sandbox resources must be scoped to a tenant or owner

When multiple tenants share a gateway, sandbox list must only return sandboxes belonging to the caller's tenant (or the caller themselves). Today all sandboxes are visible to any authenticated user of a given gateway.

What we need:

• A sandbox has an owner (user or tenant/group)
• List operations are filtered by ownership by default
• A user in tenant A cannot see, connect to, or exec into tenant B's sandboxes
• Admin roles may have cross-tenant visibility (audit use case)

R3: Mutating operations must enforce ownership

sandbox delete, sandbox exec, SSH connections, and policy mutations must verify the caller owns (or has delegation rights to) the target sandbox. This is related to #1354 (per-sandbox secret binding) — the shared sandbox secret today allows any holder to act as any sandbox.

What we need: A sandbox operation fails with permission denied if the caller is not the owner or a delegated admin. This must hold even if the caller knows the sandbox name/ID.

R4: Provider credentials must be isolated per tenant

Providers (LLM API keys, OAuth2 clients) stored in the gateway's credential system must be scoped. Tenant A's providers are invisible and inaccessible to Tenant B. Today all providers on a gateway instance are globally visible.

What we need:

• Provider CRUD is scoped to the caller's tenant
• Provider resolution during sandbox execution only returns credentials belonging to the sandbox's tenant
• No cross-tenant credential leakage even with a valid auth token for the wrong tenant

R5: Resource accounting should be possible per tenant

For chargeback and capacity planning, we need to attribute sandbox resource consumption (CPU, memory, GPU, storage, session count) to a tenant.

What we need: The gateway exposes enough metadata (labels, annotations, or an API) to aggregate resource usage per tenant. This doesn't necessarily mean the gateway enforces quotas — Kubernetes ResourceQuota on namespaces or the compute driver can handle enforcement — but the gateway's data model must support the accounting.

---

Non-requirements (for this issue)

These are explicitly not part of what we're asking for here:

• Specific implementation mechanism — whether this is "virtual gateways," middleware, or a tenant-aware data layer is an upstream design decision
• Cross-gateway federation — we're talking about a single gateway instance
• SPIFFE/SVID-based identity — useful future direction but not a prerequisite for logical tenancy
• Policy hierarchy (shared base policies + tenant overrides) — useful but can come later
• Gateway HA — related (feat(k8s, helm): Enable running OpenShell Gateway with multiple replicas #1021) but orthogonal; multi-tenancy and multi-replica are separable

---

Our current workaround

We deploy one full gateway stack per tenant namespace. This provides hard process isolation (compromising one gateway gives zero access to others) and avoids all the shared-state problems. The tradeoffs:

Benefit	Cost
Zero shared state between tenants	N gateway pods for N tenants
Namespace-scoped RBAC (K8s-native)	N TLS certificate sets
Simple failure domain (one tenant, one gateway)	N ingress resources
No upstream changes required	Per-tenant Helm releases
Works today	Scaling to 50+ tenants becomes operationally heavy

This works well for 2-10 tenants. At 50+ tenants with heterogeneous resource needs, the per-gateway overhead becomes significant and a shared model would be valuable.

---

Relevant upstream work

• Multi-tenant support roadmap? #1145 — Multi-tenant roadmap discussion (drew's "virtual gateway" concept)
• feat(auth): add OIDC/Keycloak authentication with RBAC and scope-based permissions #935 — OIDC/Keycloak authentication (merged — provides the identity foundation)
• security: per-sandbox secret binding for sandbox→gateway RPCs #1354 — Per-sandbox secret binding (open — identity at the sandbox↔gateway RPC layer)
• feat(k8s, helm): Enable running OpenShell Gateway with multiple replicas #1021 — Multi-replica gateway (open — shared state is a prerequisite)
• feat(gateway): add DB-backed resource_version CAS for stored objects #1255 — DB-backed resource_version CAS (open — concurrent access to stored objects)

---

Questions for the maintainers

1. Does the "virtual gateway" concept from Multi-tenant support roadmap? #1145 align with these requirements, or is the team thinking about a different isolation model?
2. Is there a preferred identity claim for tenant membership (e.g., a JWT claim like tenant or org, a Keycloak group, an OIDC scope)?
3. Would the team prefer requirements like these filed as separate issues per area (R1-R5), or kept together as a cohesive set?