Deployment¶

Lethe runs as a self-hosted Docker Compose stack of nine services. This page covers the service topology, model requirements, configuration, and development workflow. For the software architecture and iteration cycle, see their respective pages.

Operational characteristics:

• Multi-user queue --- requests are enqueued in a FIFO queue and processed one at a time. Callers receive immediate streaming responses with real-time queue position updates and heartbeats. The queue depth is configurable (LETHE_QUEUE_SIZE, default 64).
• API key authentication --- Bearer token auth with per-key concurrency limits (LETHE_PER_KEY_MAX_QUEUED, default 3). When no keys are configured, auth is disabled for development.
• GPU sharing --- the reasoning model (~17 GiB), embedding model (~2.4 GiB), and DeBERTa NLI (~0.4 GiB int8) share one GPU via ROCm. Peak VRAM: ~23.7 GiB of 24 GiB, with ~300 MiB margin managed by sub-batching and KV cache quantization.
• CPU services --- Whisper runs CPU-only (no VRAM).
• Session graph --- each session creates a fresh Neo4j graph that persists after completion. The graph is the audit trail of the agent's reasoning process.
• Streaming --- responses stream via Server-Sent Events with sequence-numbered events and resumable reconnection. Multi-hour sessions survive proxy drops and network interruptions. The client receives real-time iteration progress, and the full answer arrives after the assembly stage completes.
• Dual API --- both the Open Responses API (/v1/responses) and OpenAI Chat Completions API (/v1/chat/completions) are supported, making Lethe compatible with any OpenAI-compatible client (Open WebUI, LobeChat, LibreChat, ChatBox, NextChat, AnythingLLM, etc.).

Service Topology¶

flowchart TB
    subgraph client ["Client"]
        App["OpenAI Client<br/><small>any compatible library</small>"]
    end

    subgraph stack ["Docker Compose Stack"]
        Lethe["lethe<br/><small>:8000</small><br/>Python app server"]
        LLM["llama-cpp<br/><small>:8080</small><br/>Reasoning LLM (GPU)"]
        Embed["llama-cpp-embed<br/><small>:8090</small><br/>Embedding model (GPU)"]
        Neo["neo4j<br/><small>:7474 :7687</small><br/>Graph database"]
        SearX["searxng<br/><small>:8888</small><br/>Metasearch engine"]
        Cloak1["cloak-1<br/><small>:9222</small><br/>Stealth Chromium"]
        Cloak2["cloak-2<br/><small>:9223</small><br/>Stealth Chromium"]
        Cloak3["cloak-3<br/><small>:9224</small><br/>Stealth Chromium"]
        Whisper["whisper<br/><small>:9095</small><br/>Speech-to-text (CPU)"]
        NLI["nli<br/><small>:8070</small><br/>DeBERTa NLI (GPU int8)"]
        Tor1["tor-1<br/><small>:9050</small><br/>Tor SOCKS5"]
        Tor2["tor-2<br/><small>:9050</small><br/>Tor SOCKS5"]
        Tor3["tor-3<br/><small>:9050</small><br/>Tor SOCKS5"]
    end

    App -->|"HTTP"| Lethe
    Lethe -->|"HTTP"| LLM
    Lethe -->|"HTTP"| Embed
    Lethe -->|"Bolt"| Neo
    Lethe -->|"HTTP"| SearX
    Lethe -->|"CDP"| Cloak1
    Lethe -->|"CDP"| Cloak2
    Lethe -->|"CDP"| Cloak3
    Lethe -->|"HTTP"| Whisper
    Lethe -->|"HTTP"| NLI
    Cloak1 -->|"SOCKS5"| Tor1
    Cloak2 -->|"SOCKS5"| Tor2
    Cloak3 -->|"SOCKS5"| Tor3

All services share a Docker network. The Lethe app service depends on Neo4j, both llama.cpp instances, SearXNG, and CloakBrowser — it waits for their health checks before starting.

Services¶

lethe (Application Server)¶

Exposes these endpoints:

• POST /v1/responses — Open Responses API (resumable streaming, background execution)
• GET /v1/responses/{id} — Reconnect to an in-progress response stream
• POST /v1/responses/{id}/cancel — Cancel a queued or in-progress response
• POST /v1/chat/completions — OpenAI Chat Completions compatibility shim
• GET /v1/queue — View current queue positions (scoped to caller's API key)
• POST /v1/reset — Clear session graph (also cancels queued items for that session)
• GET /v1/models — Lists available models
• GET /health — Health check (no auth required)

FIFO queue: requests are processed sequentially by a single worker. When the queue is full or per-key limits are reached, new requests receive HTTP 429.

llama-cpp (Reasoning LLM)¶

The reasoning model runs in thinking mode — it produces extended reasoning traces before the final structured JSON response. The thinking budget is dynamically adjusted by the budget manager based on task complexity.

llama-cpp-embed (Embedding Model)¶

Embedding requests use instruct prefixes (document or query) following the Qwen3 embedding convention. The CoalescingEmbedClient driver batches concurrent single-embedding requests with a 50ms debounce window.

neo4j (Graph Database)¶

Neo4j stores the task graph, evidence nodes, and edge relationships. At session start, only task nodes are cleared (reset_tasks) — evidence nodes and SIMILAR_TO edges persist across sessions, building cumulative knowledge. All graph algorithms (PPR, cycle detection, similarity sync) are implemented in application code using NumPy rather than via Neo4j's GDS plugin. This avoids a plugin dependency and keeps the deployment to the Community edition, at the cost of transferring node embeddings to the application for computation.

searxng (Metasearch)¶

Custom configuration enables curated search engines across general, academic, discussion, video, and news categories. Rate limiting is disabled for development use.

CloakBrowser Pool (Stealth Browser × 3)¶

Three stealth Chromium instances with independent Tor circuits provide 24 concurrent rendering slots (8 tabs × 3 instances). Playwright connects via CDP through the BrowserPool round-robin distributor.

whisper (Speech-to-Text)¶

Multi-stage build compiles whisper.cpp from source with ffmpeg and OpenMP support. Transcribes audio extracted from video content during research. The whisper service is not in Lethe's depends_on chain --- if it starts slowly or is unavailable, video transcription silently degrades while text-based research continues unaffected.

nli (DeBERTa NLI)¶

Bidirectional entailment classification for SIMILAR_TO edge enrichment. Each evidence pair is classified in both directions; the 6 raw probabilities reduce to 3 scores (support, contradiction, confidence) that feed the NLI kernel in PPR transitions. Runs on GPU with int8 quantization (bitsandbytes) and adaptive sub-batching that halves batch size on OOM. Highest-similarity edges are enriched first when the NLI cap per iteration is reached.

proxy-pool (Tor)¶

Tor SOCKS5 proxy used by CloakBrowser and the fallback HTTP crawler for anonymized web fetching. SearXNG does not route through Tor --- it queries search engines directly. Circuit isolation prevents cross-domain tracking.

Models¶

Lethe requires two GGUF model files, placed in the .models/ directory (gitignored):

Model paths are configured via environment variables LLM_MODEL and EMBEDDING_MODEL, which Docker Compose maps to volume mounts.

Configuration¶

All configuration is managed through environment variables with the LETHE_ prefix. The .env.example file documents every setting with defaults.

The most commonly adjusted deployment variables are:

For the complete parameter reference --- budget tuning, embedding algebra thresholds, research engine settings, and validation rules --- see Configuration. All settings are validated and frozen at startup via Config.from_env(). Before each session, the server probes /health on both model endpoints; if either is unreachable, the session fails immediately with a clear error.

Development¶

Prerequisites¶

• Python 3.12+
• uv package manager
• Docker and Docker Compose (for the full stack)
• ROCm-compatible AMD GPU (for reasoning model) or adapted llama.cpp build

Setup¶

# Install Python dependencies
uv sync --all-extras

# Copy and configure environment
cp .env.example .env
# Edit .env with your Neo4j credentials and model paths

# Download GGUF models to .models/
# (see model table above for recommended files)

# Start infrastructure services
docker compose up -d

# Install Playwright browser (for local development outside Docker)
uv run playwright install chromium

Quality Gates¶

make check    # Run all gates: lock + format + lint + typecheck + boundary-check + test
make fix      # Auto-format and auto-fix lint issues
make test     # Run unit tests only (excludes integration and LLM markers)

Project Layout¶

reasoning/
├── src/lethe/          # Main package (11 subpackages, 56 modules)
├── tests/              # 11 test modules (124 test cases)
├── config/             # Service configurations (SearXNG, whisper)
├── docs/               # This documentation (Zensical)
├── .env.example        # Environment variable template
├── docker-compose.yaml # Full 9-service stack
├── Dockerfile          # Lethe app container
├── Makefile            # Quality gate automation
├── pyproject.toml      # Package metadata and tool config
└── zensical.toml       # Documentation site config

API Usage¶

Lethe exposes two compatible API endpoints:

1. Open Responses API (/v1/responses) — full-featured resumable streaming with background execution, queue position visibility, and cancellation.
2. Chat Completions API (/v1/chat/completions) — standard OpenAI-compatible interface for any chat application (Open WebUI, LobeChat, LibreChat, ChatBox, NextChat, AnythingLLM, etc.).

Both endpoints require Bearer token authentication (unless LETHE_API_KEYS is empty).

Authentication¶

All API endpoints (except /health) require a Bearer token:

curl -H "Authorization: Bearer YOUR_API_KEY" http://localhost:8000/v1/models

Generate keys for your .env:

python -c "import secrets; print(','.join(secrets.token_urlsafe(32) for _ in range(10)))"

Chat Completions API (Universal Compatibility)¶

This is the endpoint to use with Open WebUI and other chat applications:

from openai import OpenAI

client = OpenAI(base_url="http://localhost:8000/v1", api_key="YOUR_KEY")
response = client.chat.completions.create(
    model="lethe",
    messages=[{"role": "user", "content": "Analyze Aave V3 security"}],
    stream=True,
)
for chunk in response:
    if chunk.choices[0].delta.content:
        print(chunk.choices[0].delta.content, end="")

curl -N -X POST http://localhost:8000/v1/chat/completions \
  -H "Authorization: Bearer YOUR_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "lethe",
    "messages": [
      {"role": "system", "content": "Format with academic citations"},
      {"role": "user", "content": "Analyze the security model of Aave V3"}
    ],
    "stream": true
  }'

The Chat Completions shim auto-generates an isolated session_id per request. For session continuity, pass X-Session-Id header.

Creating a Response (Streaming)¶

import httpx

async with httpx.AsyncClient(timeout=httpx.Timeout(30, read=None)) as client:
    async with client.stream(
        "POST",
        "http://localhost:8000/v1/responses",
        json={
            "model": "lethe",
            "input": [
                {"role": "user", "content": "Analyze the security model of Aave V3"}
            ],
            "instructions": "Format with academic citations",
            "stream": True,
            "background": True,
            "session_id": "my-session-001",
        },
        headers={"Authorization": "Bearer YOUR_KEY"},
    ) as resp:
        async for line in resp.aiter_lines():
            if line.startswith("data: "):
                print(line[6:])

curl -N -X POST http://localhost:8000/v1/responses \
  -H "Authorization: Bearer YOUR_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "lethe",
    "input": [
      {"role": "user", "content": "Analyze the security model of Aave V3"}
    ],
    "stream": true,
    "background": true,
    "session_id": "my-session-001"
  }'

Resuming a Disconnected Stream¶

If the connection drops, reconnect using the response ID and last seen sequence number:

curl -N -H "Authorization: Bearer YOUR_KEY" \
  "http://localhost:8000/v1/responses/resp_abc123?stream=true&starting_after=15"

The server replays all events after sequence number 15, then continues streaming live events. This makes multi-hour sessions resilient to transient network failures.

Event Wire Format¶

Each SSE event includes a type and sequence number:

event: response.created
data: {"type":"response.created","sequence_number":0,"response":{"id":"resp_...","status":"queued"}}

event: response.in_progress
data: {"type":"response.in_progress","sequence_number":1,"response":{"id":"resp_...","status":"in_progress"}}

event: lethe:progress
data: {"type":"lethe:progress","sequence_number":3,"detail":"Iteration 12/50 — processing evidence"}

event: response.output_text.delta
data: {"type":"response.output_text.delta","sequence_number":5,"delta":"...content..."}

event: response.completed
data: {"type":"response.completed","sequence_number":42,"response":{"id":"resp_...","status":"completed"}}

data: [DONE]

The lethe:progress events provide real-time visibility into the research loop (iteration counts, headings being written, sources found). These use the Open Responses vendor-prefix extension mechanism.

Non-Streaming (Synchronous)¶

Set "stream": false to block until the response completes:

curl -X POST http://localhost:8000/v1/responses \
  -H "Authorization: Bearer YOUR_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "lethe",
    "input": [{"role": "user", "content": "Analyze Aave V3 security"}],
    "stream": false,
    "session_id": "my-session-001"
  }'

Message Role Semantics¶

The system processes the last user message as the research goal --- the content that drives task decomposition, embedding algebra, and convergence. System messages are separated and preserved as output formatting instructions for the final composition LLM call only; they never enter the embedding algebra or influence research navigation.

Streaming Resilience¶

The server emits SSE comment heartbeats (: ping) every 15 seconds during idle periods. This prevents Nginx and other reverse proxies from killing the connection due to read timeouts. Combined with sequence-numbered events and the GET /v1/responses/{id} reconnection endpoint, multi-hour sessions survive proxy restarts, network blips, and client reconnections without losing any data.

Response state is held in-memory for 30 minutes after completion, providing a generous window for late reconnections. After TTL expiry, the response buffer is garbage-collected.

Concurrency¶

The server uses an in-memory FIFO queue with a configurable capacity (LETHE_QUEUE_SIZE, default 64). A single worker drains the queue sequentially — one research session at a time — ensuring exclusive access to the LLM, embedding model, and Neo4j database. Multiple clients can submit requests concurrently; each receives an immediate streaming response that shows queue position, then transitions to live progress when the job starts.

Per-key limits (LETHE_PER_KEY_MAX_QUEUED, default 3) prevent any single client from monopolizing the queue. When the queue is full, HTTP 429 is returned.

The GET /v1/queue, GET /v1/models, and GET /health endpoints are always available regardless of queue state.

Queue Management¶

Check queue status:

curl -H "Authorization: Bearer YOUR_KEY" http://localhost:8000/v1/queue

Response shows all queued items with positions and ownership:

{
  "queue": [
    {"position": 1, "response_id": "resp_abc...", "status": "in_progress", "own": true},
    {"position": 2, "response_id": "resp_def...", "status": "queued", "own": false}
  ],
  "total": 2
}

Cancel a queued or in-progress response:

curl -X POST -H "Authorization: Bearer YOUR_KEY" \
  http://localhost:8000/v1/responses/resp_abc123/cancel

Open WebUI Configuration¶

To use Lethe with Open WebUI, add it as an OpenAI-compatible connection:

1. Go to Admin Panel → Connections → OpenAI API
2. Set Base URL: http://lethe:8000/v1
3. Set API Key: one of your LETHE_API_KEYS values
4. The lethe model will appear in the model selector

Open WebUI uses the Chat Completions API, which Lethe fully supports including streaming, system prompts, and proper SSE formatting.

Nginx Reverse Proxy Configuration¶

When deploying Lethe behind Nginx, use the following configuration to support long-lived SSE streams:

location /v1/responses {
    proxy_pass http://lethe:8000;
    proxy_http_version 1.1;
    proxy_set_header Connection "";
    proxy_set_header Host $host;
    proxy_set_header X-Real-IP $remote_addr;
    proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;

    # Disable buffering — SSE events must flow immediately
    proxy_buffering off;
    proxy_cache off;
    chunked_transfer_encoding off;
    add_header X-Accel-Buffering no;

    # 24h read timeout — server heartbeats every 15s keep the connection alive;
    # this is just a safety net for truly idle connections
    proxy_read_timeout 86400s;
    proxy_send_timeout 86400s;
}

location /v1/chat/completions {
    proxy_pass http://lethe:8000;
    proxy_http_version 1.1;
    proxy_set_header Connection "";
    proxy_set_header Host $host;
    proxy_set_header X-Real-IP $remote_addr;
    proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;

    proxy_buffering off;
    proxy_cache off;
    chunked_transfer_encoding off;
    add_header X-Accel-Buffering no;

    proxy_read_timeout 86400s;
    proxy_send_timeout 86400s;
}

location /v1/ {
    proxy_pass http://lethe:8000;
    proxy_http_version 1.1;
    proxy_set_header Connection "";
    proxy_set_header Host $host;
}

location /health {
    proxy_pass http://lethe:8000;
    proxy_read_timeout 5s;
}

Key points: - proxy_buffering off and X-Accel-Buffering no ensure SSE events are not buffered by Nginx and reach the client immediately. - proxy_read_timeout 86400s (24 hours) prevents Nginx from killing the connection during long research sessions. The server's 15-second heartbeats ensure the connection is never truly idle from Nginx's perspective. - proxy_http_version 1.1 with an empty Connection header enables HTTP keep-alive between Nginx and the upstream, allowing proper SSE delivery.

Troubleshooting¶

For GPU compatibility beyond AMD ROCm, rebuild the llama.cpp images with the appropriate backend (CUDA, Vulkan, Metal). The HSA_OVERRIDE_GFX_VERSION environment variable in docker-compose.yaml is AMD-specific and should be removed or replaced for other GPU vendors.