Server

In-place updates

Run sudo ivoai update --dry-run before a server update. This stages and executes the checksum-verified candidate for bounded preflight but commits no managed-state change. The server transaction uses /var/lib/ivoai/updates, snapshots only explicit IVOAI server assets and the managed executable, runs setup --mode server, and verifies with ivoai server doctor. It does not create client XDG state merely to update a server. On failure, sudo ivoai update --rollback restores the previous compatible checkpoint and reconciles systemd through the restored server setup. Live Qdrant, memory, enrollment, and Web OAuth stores are not blindly copied; a future format change to any of them must declare a quiesced reversible migration. Follow the two-production canary procedure; never update both production instances together.

Server administration is available both through ivoai server ... subcommands and the Server Administration section of the interactive menu. Mutating local server actions are shown as unavailable unless ivoai is running on a supported Linux host with root privileges. Restore, stop, enrollment revocation, and connector removal require explicit confirmation in the menu.

Supported systems

Initial server support targets Ubuntu 22.04, Ubuntu 24.04, and Debian 12 on Linux amd64 and arm64. Both architectures use their reviewed, immutable embedding-runtime OCI digest. Run ivoai setup --mode server as root. The operation is idempotent.

Server setup requires Docker Engine 28.0.0 or newer and Docker Compose 2.33.1 or newer. Engine 28 introduced the gateway-priority support used to give the embedding container temporary model-download egress without giving the backend networks permanent Internet access. Setup validates OS, architecture, container/LXC state, systemd, privilege, Docker CLI, daemon, Engine and Compose before committing server assets. On Debian 12, a missing Engine is provisioned idempotently from Docker's official signed APT repository after verifying the reviewed signing-key fingerprint; package candidates are resolved and installed at exact versions. Existing unknown repository files or keys are never overwritten. Ubuntu and non-official legacy installations receive a fail-closed diagnostic and use the official Debian or Ubuntu operator path.

Inside LXC, an unreachable daemon is diagnosed separately. Host-side nesting, keyctl, cgroup or device permissions must be enabled on the Proxmox/LXC host; the guest setup never pretends it can change host policy. After fixing the prerequisite, rerun sudo ivoai setup --mode server. Doctor and Memory/Context status report SERVER_SETUP=INCOMPLETE and the prerequisite root cause when setup did not commit, instead of treating missing generated .env files as the primary failure.

When the Engine is compatible but Compose is missing or too old, ivoai installs the architecture-specific official Docker Compose plugin at /usr/local/lib/docker/cli-plugins/docker-compose after verifying its pinned SHA-256 checksum. Progress is printed every 10 seconds and the bounded download window is 30 minutes for slow links. An existing compatible Compose installation is preserved.

Layout

Purpose	Path
Configuration	/etc/ivoai
Secrets	/etc/ivoai/secrets
Persistent authoritative data	/var/lib/ivoai
Backups	/var/lib/ivoai/backups
Application assets	/opt/ivoai
Logs	journald

Product documentation service

Server setup enables ivoai-docs.service. It serves the immutable Docusaurus production build embedded in the installed IVOAI binary; Node.js, the Docusaurus dev server and a repository checkout are not used at runtime. The service runs as the unprivileged ivoai-docs account, restarts on failure, starts at boot, writes no application state, and logs through journald.

The managed default is stored in /etc/ivoai/docs.json:

{"listen_address":"0.0.0.0:7780"}

sudo ivoai server docs status reports health, listen address and loopback URL. The wildcard bind is deliberate so an external reverse proxy can reach it; restrict TCP 7780 at the host/network firewall to the Nginx Proxy Manager source whenever possible. Bind address and firewall exposure are separate controls.

Choose a different conflict-free listener idempotently when required:

sudo ivoai server docs configure --listen 0.0.0.0:7781

For an external Nginx Proxy Manager, create a dedicated Proxy Host:

Internet --HTTPS--> Nginx Proxy Manager --HTTP/LAN--> ivoai-server:7780

Domain:          docs.example.com
Forward scheme:  http
Forward host:    <IVOAI_SERVER_LAN_IP>
Forward port:    7780
WebSockets:      not required
TLS:             terminated by Nginx Proxy Manager

Do not install Nginx Proxy Manager inside IVOAI. ivoai setup --mode server and subsequent updates refresh the embedded build and managed unit idempotently.

The gateway and context services run as distinct unprivileged accounts (ivoai-gateway and ivoai-context) in a shared read-only group. Their systemd units hide unrelated processes, deny each service the other's private state, use Restart=on-failure, NoNewPrivileges=yes, and narrow write allowlists. Qdrant's unprivileged image, the embedding runtime, and ai-memory use the non-login ivoai container identity. Dependency ports bind only to loopback, require separate generated internal credentials, and are not public. After TEI downloads the pinned model and passes health checks, its download network is disconnected.

Rerunning setup changes ownership only on managed mount roots; it does not traverse application-created content. This preserves legitimate Hugging Face cache symlinks. Qdrant readiness uses its unauthenticated /readyz endpoint, while its data API continues to require the generated internal credential. Qdrant storage remains at /var/lib/ivoai/qdrant; its writable snapshot workspace and initialization marker live at /var/lib/ivoai/qdrant-snapshots and /var/lib/ivoai/qdrant-init. These separate mounts allow the pinned image to run as the ivoai non-root container identity without making /qdrant writable.

Operations

ivoai server setup
ivoai server status
ivoai server doctor
ivoai server start
ivoai server stop
ivoai server restart
ivoai server logs
ivoai server gateway configure --public-url https://ai.example.com
ivoai server web-access create --ttl 10m
ivoai server web-access list
ivoai server web-access revoke <id>
ivoai server backup [--output <path>]
ivoai server restore --input <backup>
ivoai server remote status
ivoai server remote doctor
ivoai server remote connector list

The gateway exposes liveness at /health, readiness at /ready, and non-sensitive protocol discovery at /.well-known/ivoai. Protocol version 1 is checked before a client persists connection state.

Each server instance issues its own scoped enrollment credential. Multi-server pooling, purpose selection, explicit read federation and primary/standby routing are client-side responsibilities; an ivoai server does not learn about peer instances, replicate another organization's data or implement a hidden quorum. See Multi-server knowledge sources.

Public HTTPS gateway

Setup listens on 127.0.0.1:7744 without TLS. For the usual deployment, keep that loopback listener behind an administrator-managed HTTPS reverse proxy and record its public origin without editing a file:

sudo ivoai server gateway configure --public-url https://ai.example.com

If the HTTPS reverse proxy runs on another host or container, explicitly bind the gateway to the server's private address and allow only the proxy's source address:

sudo ivoai server gateway configure \
  --public-url https://ai.example.com \
  --listen 192.0.2.10:7744 \
  --trusted-proxy 192.0.2.20/32

Use the real private address of the ivoai server and the real source IP/CIDR of the proxy. Requests from other peers, and proxy requests without X-Forwarded-Proto: https, are rejected. Do not use 0.0.0.0/0.

Alternatively, let ivoai serve TLS directly. The certificate and key are copied into /etc/ivoai/secrets/tls as service-owned 0600 files, and a non-loopback listener is accepted only when both are supplied:

sudo ivoai server gateway configure \
  --public-url https://ai.example.com \
  --listen 0.0.0.0:7744 \
  --tls-cert /absolute/path/fullchain.pem \
  --tls-key /absolute/path/privkey.pem

Certificate issuance and renewal remain operator responsibilities. Re-run the configure command to refresh managed certificate copies. Qdrant, embeddings, and ai-memory stay on loopback mappings; only the gateway or reverse proxy is public. The dependency containers join a non-internal network only while Docker establishes the loopback bindings; systemd disconnects that transient network after startup. After systemd loads the backend environment files, the context service cannot access the managed secrets tree.

Nginx Proxy Manager

Create one Proxy Host for the public hostname with scheme http, the ivoai server's private address, and port 7744. Enable a valid certificate and Force SSL. Do not attach an NPM Access List, Basic Authentication, or another login challenge to the host: Web connectors must reach the ivoai OAuth metadata and authorization flow.

When NPM runs on another host or container, configure the gateway listener and the narrow source CIDR as shown above. NPM must preserve Authorization, Host, and the original HTTPS scheme. The following Advanced configuration is suitable for the Streamable HTTP MCP route:

proxy_set_header Authorization $http_authorization;
proxy_set_header Host $host;
proxy_set_header X-Forwarded-Proto https;
proxy_buffering off;
proxy_request_buffering off;
proxy_read_timeout 3600s;

Do not use 0.0.0.0/0 as a trusted proxy and do not publish ports 6333, 6334, 8080, or 49374. Only the HTTPS proxy origin should be internet-facing.

Web MCP access

The public connector URL is the configured origin plus /mcp, for example https://ai.example.com/mcp. Before connecting a Web client, create a one-time code:

sudo ivoai server web-access create --ttl 10m

The command prints the activation code once. During the connector's OAuth browser flow, review the requested scopes and enter that code. The default grant contains context:read, memory:read, and memory:write. The destructive memory:delete scope remains available only when it is selected explicitly. Select a narrower set when mutation is not needed. list displays identifiers, scopes, expiry, and revocation state without tokens. revoke invalidates the selected Web grant and its refresh-token family.

The unified MCP exposes context search/read and bounded memory CRUD. Deleting memory requires the memory:delete scope and an explicit confirmation for the normalized page path. Context remains read-only. The gateway does not proxy administrative ai-memory tools or arbitrary MCP calls.

Enrollment

ivoai server enrollment create --ttl 10m
ivoai server enrollment list
ivoai server enrollment revoke <id>

Only the create command displays the one-time code. The server persists a cryptographic digest, expiry, and state, never the original code. Consumption issues a scoped client credential; replay is rejected. The v0.1 authentication backend is the owner-only, cross-process-locked local record store at /var/lib/ivoai/enrollment/state.json; it contains hashes and metadata, not plaintext codes or issued bearer tokens.

Context connectors

Filesystem and Git connectors normalize text, reject sensitive and unsafe paths, chunk documents, produce local embeddings, and upsert a versioned Qdrant collection. Connectors are managed with explicit commands such as ivoai server connector add --name docs --type filesystem --path /srv/docs, ivoai server connector list, and ivoai server connector remove docs. The core remains healthy with zero connectors and zero documents. Connector definitions are loaded when the context service starts. Removal purges the connector's catalog and vector entries before removing its registry definition, then restarts the context service so its active configuration matches the registry.

Agent-facing context tools are read-only: context_search, context_get_document, context_recent, and context_health. Ingestion and connector administration are separate authenticated operations.

Backup and restore

Backups include configuration without unnecessary plaintext secrets, connector and corpus metadata, context metadata, ai-memory persistent data, and index rebuild metadata. Original corpus and memory data are authoritative; vector indexes are rebuildable. Restore validates bounded entries, rejects links and traversal, excludes secrets, and writes regular files atomically. The CLI automatically stops the managed gateway, context, and dependency services around each backup/restore operation and starts them again afterward. Restore merges validated files into the managed roots and does not delete stale files that are absent from the archive. Before restart it reapplies dedicated service ownership only to the validated restored trees, without traversing normal application caches.