To evaluate an open-source agent platform for air-gapped AI, begin with the mission workflow and the operating boundary. Turn those conditions into pass-or-fail tests, then examine the platform's license, dependencies, inference interfaces, permissions, telemetry, software provenance, evaluation tooling, maintenance burden, and exit path.
The repository is only one part of the decision. A platform can publish source code yet depend on hosted authentication, cloud telemetry, extension registries, remote documentation, or a single inference provider. Review the broaderair-gapped agentic AI architecturefirst, then test whether the candidate fits each layer without weakening the boundary.
What should the platform prove before selection?
The platform should prove five outcomes: it performs the bounded mission workflow, operates inside the approved boundary, limits every agent action, produces reviewable evidence, and can be sustained without the original provider. Feature counts matter only after these gates are met.
| Decision gate | Evidence to require | Failure signal |
|---|---|---|
| Mission fit | Representative tasks, users, measures, and failure cases | A polished workflow that avoids hard cases |
| Isolation | Successful install, operation, recovery, and update with egress denied | Undocumented calls or cloud-only control services |
| Control | Identity-bound permissions, policy decisions, approvals, and intervention | Shared credentials or unrestricted tool execution |
| Evidence | Versioned traces, logs, configurations, evaluations, and audit export | Outputs without reconstructable context |
| Ownership | Source, build path, artifacts, documentation, skills, and transition plan | A nominal fork that the program cannot build or sustain |
Use the same decision frame you would use for any consequential AI acquisition. The guide to evaluating an AI vendor beyond the demoexplains how to separate claims from evidence, while theAI Technology Evaluation Scorecardcan make criteria, weights, and disqualifying conditions visible.
How do licensing and governance affect control?
Verify both the license and the project's ability to govern change. The Open Source Initiative'sOpen Source Definition contains ten criteria, including access to source, permission to modify, and permission to distribute derived works. Source availability alone does not establish those rights.
Record the license for the core, interface, extensions, client libraries, and bundled components. Review attribution, reciprocal obligations, patent terms, trademarks, and any separately licensed enterprise features with counsel. Then inspect governance: release ownership, maintainer concentration, security reporting, contribution rules, release cadence, and the process for accepting or rejecting changes.
Ask a practical question: if the upstream project changed direction tomorrow, could the program legally and technically maintain its chosen version? The answer should be supported by license text, a reproducible build, internal skills, and an identified decision authority. A fork without those conditions is not meaningful independence.
Can the agent platform actually operate offline?
Prove offline operation in a representative environment with network egress denied. Do not accept an architecture slide or a cached demo as evidence. Exercise the lifecycle from installation through recovery, including the moments when software often reaches for an external identity, registry, license, package, model, or telemetry service.
- Inventory: Map runtime, build, package, model, extension, identity, documentation, time, certificate, and observability dependencies.
- Stage: Import approved artifacts and metadata through the authorized transfer process.
- Deny: Block external resolution and egress rather than assuming the system is disconnected.
- Exercise: Install, start, authenticate, run agents and tools, collect telemetry, back up, restore, and restart after failure.
- Update: Import and validate a controlled platform, dependency, and model change.
Capture every attempted connection and unresolved dependency. Also test clocks, certificates, secrets rotation, storage growth, and degraded internal services. The goal is not just to make one workflow run. It is to establish a supportable disconnected operating model.
How portable are the model and inference layers?
Model portability means the agent workflow can use an approved inference choice without rewriting the platform's core. Require documented adapters and contract tests for authentication, streaming, structured output, tool calling, context limits, errors, retries, cancellation, and resource accounting.
Test at least two materially different inference configurations when portability affects the acquisition decision. Keep the tasks and acceptance criteria fixed. Compare output quality, latency, memory, throughput, failure behavior, and tool-call conformance. A nominally compatible endpoint may still behave differently at the tokenizer, schema, sampling, or error-handling layer.
Separate the platform release from the model release. Each should have its own version, provenance, evaluation results, rollback package, and approval record. Theair-gapped AI model update lifecycledescribes how to move model changes through intake, testing, approval, deployment, monitoring, and rollback without creating an informal path around the boundary.
How should tools, permissions, and identity be bounded?
Treat each agent as a software principal with explicit authority. NIST SP 800-207 states that trust should not be granted solely because of network location or ownership. ItsZero Trust Architectureguidance supports resource-focused authentication and authorization, which remains relevant inside an isolated network.
- Bind a distinct workload identity to each agent or bounded agent role.
- Allowlist tools, operations, data scopes, destinations, and execution environments.
- Use short-lived credentials where the environment supports them, and never place shared secrets in prompts.
- Require human approval for high-consequence actions and make denial the safe default.
- Provide pause, revoke, terminate, and quarantine controls that work during degraded conditions.
Test confused-deputy scenarios, prompt injection, cross-agent delegation, stale authorization, tool-output poisoning, and attempts to exceed data scope. The question is not whether an agent usually follows instructions. It is whether the platform enforces authority when the model does not.
What observability and audit evidence should be preserved?
Preserve enough evidence to reconstruct what happened without collecting more sensitive content than the mission requires. Record platform, model, prompt-template, policy, tool, and configuration versions alongside identities, decisions, approvals, timestamps, outcomes, and errors.
Trace the causal chain from request through inference, tool selection, policy enforcement, tool result, human intervention, and final output. The OpenTelemetry Specificationdefines vendor-neutral traces, metrics, and logs that can support this instrumentation, but the program must still define mission-specific event semantics, retention, access, and redaction.
Export evidence to an approved store that does not depend on the agent platform's own interface. Test audit retrieval during an incident and verify that privileged users cannot silently alter the record. Sensitive prompts and tool outputs may require field-level controls or hashes rather than indiscriminate full-content logging.
How should supply-chain provenance be evaluated?
Require a component inventory and verifiable origin for every imported artifact. CISA'sSBOM Resources Librarydistinguishes source, build, analyzed, and deployed views of software. For an air gap, those views help connect what was approved, built, transferred, and installed.
Ask for source commits, dependency locks, build instructions, SBOMs, vulnerability disposition, signatures, and provenance attestations. Verify them before transfer and again inside the boundary. NIST'sSecure Software Development Frameworkrecommends provenance data and integrity mechanisms for third-party components. OpenSSF'sSLSA guidance provides an incrementally adoptable way to reason about artifact provenance and build integrity.
An SBOM is evidence, not approval. The program still needs a process to assess vulnerabilities, expired keys, compromised maintainers, unmaintained packages, license changes, and emergency updates. Test that process with a simulated component recall before relying on it.
What evaluation should run before the platform advances?
Run a layered evaluation that covers mission outcomes, model behavior, agent control, system integration, disconnected operations, security, recovery, and sustainment. NIST'sAI Resource Center organizes testing, evaluation, verification, and validation resources around the AI Risk Management Framework. Tailor those practices to the specific use and consequences.
Use representative users and tasks. Include ordinary, ambiguous, edge, and adversarial cases. Record versions, configurations, interventions, failed runs, resource use, and uncertainty. A platform should not pass because its best demonstration worked. It should pass because the evidence supports the next bounded decision.
How do maintainability and exit rights change the score?
Score the platform's ownership conditions as part of technical merit. Require documented builds, configuration export, data export, infrastructure definitions, extension interfaces, test suites, upgrade procedures, rollback, security response, and an internal skills plan.
Then execute the exit scenario. Export agent definitions, policies, evaluations, audit records, tool contracts, and operational data. Build the approved version from controlled source. Replace one inference adapter or tool integration. Estimate the people, time, infrastructure, and licensing needed to maintain a fork.
Forward-deployed engineers can help translate these conditions into a working system, evidence package, and sustainment handoff. The guide toforward-deployed engineering for defense AIexplains how embedded delivery connects mission users, software, integration, evaluation, and transition. Decision Terrain also providesair-gapped agentic AI implementationfor teams that need the platform adapted to their environment and inference choice.
Frequently asked questions
Does open source mean an agent platform will work in an air gap?
No. Open-source rights can permit use, inspection, modification, and redistribution, but they do not remove hosted dependencies. Verify installation, identity, inference, tools, extensions, telemetry, documentation, recovery, and updates with external network access denied.
Should the program require support for more than one inference provider?
Require it when model choice, hardware change, cost, or supply risk could affect the mission. Portability should be demonstrated with contract tests and representative tasks. An adapter name or compatible endpoint is not enough evidence of equivalent behavior.
Is an SBOM sufficient for software supply-chain approval?
No. An SBOM identifies components and relationships, but approval also requires origin, integrity, vulnerability review, license review, build evidence, exception handling, and an update process. Verify the artifact that enters the boundary against the evidence that was approved.
What is the strongest sign that the program truly owns the solution?
The strongest sign is an exercised transition path. The program can build the software, operate it disconnected, export its data and configuration, replace a bounded component, investigate an incident, apply an approved update, and sustain the system with identified people.