Research Engine¶

Lethe's research engine searches, crawls, and evaluates web content without any LLM calls. URL discovery, content extraction, page scoring, and termination all operate in embedding space --- using the same algebra that drives the agent loop. This makes web research fast (no LLM latency per page), deterministic (no stochastic model decisions), and structurally parallel to the main reasoning process.

Architecture¶

flowchart TB
    subgraph input ["Input"]
        Query["Search Query<br/>+ Goal Embeddings"]
    end

    subgraph discovery ["Discovery"]
        Provider["SearXNG Provider<br/><small>Google, Brave, DuckDuckGo,<br/>arXiv, Semantic Scholar</small>"]
        Rank["GA-Ranked URL Queue<br/><small>title+snippet embedding</small>"]
    end

    subgraph pipeline ["Continuous Pipeline"]
        Stream["Sliding Window<br/><small>max_inflight fetch+embed</small>"]
        Score["Page Scoring<br/><small>GA, novelty, IG</small>"]
        Converge["Spectral Convergence<br/><small>SVD, EMA, ψ</small>"]
    end

    subgraph fetch ["Content Fetch"]
        Browser["CloakBrowser<br/><small>stealth CDP</small>"]
        HTML["HTML Adapter<br/><small>trafilatura</small>"]
        PDF["PDF Adapter<br/><small>pypdf</small>"]
        Video["Video Adapter<br/><small>PyAV + whisper</small>"]
    end

    subgraph output ["Output"]
        Evidence["DiscoveredPage Tuple<br/><small>chunked, scored by GA</small>"]
    end

    Query --> Provider --> Rank
    Rank --> Stream
    Stream --> Browser
    Browser --> HTML & PDF & Video
    HTML & PDF & Video --> Score
    Score --> Converge
    Converge -->|"ψ < threshold"| Stream
    Converge -->|"ψ ≥ threshold"| Evidence

Research Loop¶

The engine runs a four-phase continuous pipeline:

class ResearchEngine:
    async def research(
        self,
        query: str,
        goal_query_emb: np.ndarray,
        goal_doc_emb: np.ndarray,
        session_pressure: float = 0.0,
        time_budget_s: float = 1e18,
    ) -> ResearchResult

Phase 1: Initial Search + Suggestion Ranking¶

Query SearXNG, collect search hits and engine suggestions, embed all suggestions, rank by goal-alignment with soft weighting. Low-GA suggestions are filtered via soft_gate with a relative cutoff (55% of best GA).

Phase 2: Multi-Query Discovery¶

The original query's hits are seeded into the URL pool. Up to 3 additional suggestion queries expand the pool. All hit titles and snippets are embedded as documents, and GA against the goal query embedding determines quality. All discovered URLs are sorted by GA descending — highest-quality URLs are processed first. No hard GA floor; convergence decides when to stop.

Phase 3: Continuous Processing¶

A sliding window of max_inflight (default 16) concurrent fetch+embed coroutines streams pages in completion order. Each page is scored, and convergence signals are updated incrementally:

1. Fetch + embed --- CloakBrowser renders the URL, the appropriate content adapter extracts text, and the concatenated chunks are embedded as a single document vector.
2. Score --- GA and novelty are computed per page (information gain is derived from them inside the convergence step).
3. Convergence --- best-evidence GA, IG saturation (1 − ig_ema), and average-GA reconstruction are blended into the convergence score and tracked for a plateau. Termination potential ψ is evaluated.
4. Refill --- As tasks complete, new URLs from the ranked queue fill the window until the queue is exhausted or ψ triggers termination.

Phase 4: Final Ranking¶

All discovered pages are sorted by goal alignment and returned as a ResearchResult. Telemetry (coverage p10, top-5 GA/novelty, stability, recon error) is logged.

Information Gain¶

Each page contributes a per-step information gain, the harmonic mean of its novelty and goal alignment:

The harmonic mean ensures both novelty and goal alignment must be present --- a novel but irrelevant page scores low, as does a relevant but redundant page. IG feeds a smooth EMA (ig_ema); its complement, IG saturation \((1 - \text{ig\_ema})\), is the leading convergence signal: as new pages stop adding information, saturation rises and the convergence score plateaus. IG is consumed directly by the convergence step and is not stored per page.

Termination¶

Research uses its own \(\psi\) with threshold 0.7 (vs. agent loop's 0.82), reflecting that web exploration should run longer before declaring exhaustion:

where:

• budget_term --- page budget fraction combined with session pressure and wall-clock time pressure via budget_horizon and soft_gate. Session pressure from the agent loop propagates via OR-combination: late-session research terminates faster even with pages remaining.
• plateau --- convergence tracker plateau signal over the convergence score. No warmup gate; plateau_signal's built-in data_confidence = 1 − exp(−n/window) prevents premature termination before the score has accumulated enough samples.

The design intent is that research stops for exactly two reasons: it has explored enough (the score plateaus) or it has run out of budget/URLs. Diminishing returns are not a separate exit — they are captured by IG saturation inside the convergence score, so an exhausted evidence space manifests as a stable, plateaued score (genuine convergence) rather than a premature abandonment of still-ranked URLs.

Exit reasons are classified by dominant ψ factor: converged (plateau dominates), budget_limit (budget term dominates), or urls_exhausted (all discovered URLs processed before reaching the page budget).

Content Extraction¶

CloakBrowser¶

All web fetching goes through CloakBrowser --- a stealth Chromium instance accessible via CDP (Chrome DevTools Protocol). Playwright connects to it and renders pages with full JavaScript execution, bypassing bot detection that would block direct HTTP requests.

Concurrency: Bounded to MediaConfig.max_concurrent (default 16) simultaneous browser tabs, preventing CloakBrowser from being overwhelmed.

HTML Adapter¶

Uses trafilatura for main content extraction with lxml for metadata (title, description, Open Graph tags). Trafilatura provides high-precision text extraction that strips navigation, ads, and boilerplate.

PDF Adapter¶

Direct HTTP download (bypassing the browser's PDF viewer) followed by pypdf extraction. Falls back gracefully if extraction fails --- the PDF is skipped rather than crashing the research engine.

Video Adapter¶

A multi-stage pipeline:

1. CDP intercept --- Capture video/audio stream URLs from network requests
2. PyAV extraction --- Extract audio track from media container
3. whisper.cpp transcription --- Send WAV audio to whisper server for STT
4. HTML fallback --- If media extraction fails, use the page's text content

Video content is chunked and embedded identically to text content. The transcript preserves temporal structure through segment boundaries.

Structural Isomorphism with Agent Loop¶

The research engine mirrors the agent loop's architecture:

This parallel structure is deliberate --- both loops solve the same problem (explore an information space efficiently and stop when returns diminish) at different scales. The embedding algebra provides a common language for both.

SearXNG Configuration¶

Lethe configures SearXNG as a metasearch engine aggregating results from multiple backends (see Deployment for service topology and SearchConfig for tuning parameters):

Engine weights and timeouts are tuned per category. The combination provides broad coverage --- academic papers, developer discussions, official documentation, and current news all contribute to the evidence pool.

Results are returned as SearchHit objects with title, URL, and snippet. Snippets are embedded for initial ranking before any page is fetched.

References¶

• Pirolli, P. and Card, S. "Information Foraging." Psychological Review, 106(4), 1999 --- Information scent and patch models for web navigation; foundation for embedding-based relevance scoring.
• Volz, J. et al. "Chunking Methods in RAG." ECIR, 2026 --- Paragraph-group chunking without overlap.
• Qian, H. et al. "InForage: Scent of Knowledge." NeurIPS, 2025 --- IFT-grounded retrieval-augmented reasoning with RL-based foraging optimization. Lethe implements similar IFT principles but substitutes RL with deterministic embedding scoring, avoiding the need for training data of search trajectories.
• Zhang, Z. et al. "Scout: Active Information Foraging with Decoupled Epistemic States." 2026 --- Formalizes active foraging under partial observability: exploration trace vs. compact epistemic state, with gap-diagnosed convergence for stopping. Mirrors Lethe's separation of navigation history from the evidence subgraph packed into context, and its ψ-based sufficiency control.
• Shao, J. et al. "SeekBench: Do LLM Agents Know How to Ground, Recover, and Assess?" ICLR, 2026 --- Process-level benchmark for search agents evaluating groundedness, recovery from bad retrieval, and calibration. Relevant for evaluating Lethe's convergence-based stopping and evidence-grounded composition --- dimensions that answer-only QA metrics miss.
• Caesar. "Deep Agentic Web Exploration for Creative Answer Synthesis." 2025 --- Graph-based memory for agentic web exploration; related stagnation detection via knowledge graph analysis. Lethe differs in using zero-LLM navigation (no model calls during crawl).
• Fan, W. et al. "Graph Retrieval-Augmented Generation: A Survey." 2024 --- Comprehensive survey of GraphRAG approaches; positions PPR-based retrieval (HippoRAG, Lethe) within the broader landscape of graph-enhanced retrieval.