Architecture & Validation

How the portfolio agent works — and how it checks itself

The chat button on the portfolio is a small assistant that answers questions about my papers and projects. The hard part isn’t the chat — it’s making sure it only says things that are actually true, and keeping it current without hand-holding. This page documents how it is built and, more importantly, how it validates itself before anything goes live.

Everything below reflects the real implementation in the repository — no idealized diagram. Where a number is shown, it comes from the latest recorded evaluation run (see live status).

In one paragraph

A visitor’s question retrieves the most relevant passages from a fixed, provenance-tracked corpus. Those passages — and only those — are handed to a language model that must answer from them and cite each one; if nothing relevant is found, the agent refuses instead of guessing. Separately, a deterministic pipeline keeps the whole thing honest over time: it detects when a source has changed, re-checks the agent against a 48-case test suite, and redeploys only if every test still passes.

System architecture

  Visitor question
        |
        v
  +---------------------------------+
  |  Retriever  (in-memory index)   |
  |  hybrid score =                 |
  |    0.65 * cosine  (embeddings)  |
  |  + 0.35 * BM25    (lexical)     |
  |  min-score floor + per-file cap |
  +----------------+----------------+
        top-6 chunks (with real relevance scores)
                   |
          +--------+---------+
       nothing above floor --> fixed refusal  (LLM is NOT called)
          |
          v
  +---------------------------------+
  |  Grounded LLM  (Gemini Flash)   |
  |  visitor text passed as DATA    |
  |  must cite [Source N]           |
  +----------------+----------------+
                   v
     Answer + "Retrieval X-ray": the raw chunks and their scores

Serving: FastAPI behind nginx with per-IP rate limiting; the model layer is a single swappable module.
Index: the whole corpus fits in memory, so retrieval is a plain vectorized similarity pass — no external vector database.

Corpus & document structure

Sources: published papers, project documentation (READMEs / architecture notes), a short profile document, and the portfolio’s own pages. A MANIFEST records every file’s origin; files prefixed DRAFT_ are unverified and excluded from the index by convention.
Chunking: heading-aware. Documents are split on their heading structure, each chunk is prefixed with its section path (e.g. Project > Validation), long sections split on paragraph boundaries with one paragraph of overlap. This keeps chunks self-describing and on-topic.
Index contents: each chunk stores its text, source file, section path, a public URL for citation, and its embedding vector, in a single SQLite file that is deployed as one artifact.

Retrieval approach

Retrieval is hybrid — this is what is actually implemented, verified against the code:

Semantic (weight 0.65): cosine similarity over local static embeddings (model2vec potion-base-8M). Fully local — no GPU, no embedding API. Chosen after transformer encoders failed to initialize on the build machine and a shared embedding API rate-limited; a static model plus the eval gate below was the reliable trade-off.
Lexical (weight 0.35): BM25 (rank_bm25) over stopword-filtered tokens. This recovers rare-but-decisive terms (e.g. a specific parameter count) that broad semantic similarity buries under brand-word matches — a real miss the evals caught.
Guards: a minimum final-score floor (0.30); a separate semantic sanity floor (0.15) so a lexical-only match with no meaning support is rejected; and a per-document cap (max 3 chunks from one file) so a single verbose document can’t fill every slot. When there is no lexical signal at all, it falls back to pure semantic ranking.

There is no reranker and no summarization chain — the design is deliberately a transparent two-signal score so every ranking decision is inspectable in the Retrieval X-ray.

Grounding & safety at answer time

Evidence-only: the model is instructed to answer strictly from the retrieved chunks and to cite each with [Source N].
Refusal without a model call: if retrieval returns nothing above the floor, the agent returns a fixed refusal and never calls the LLM — no opportunity to improvise.
Injection handling: the visitor’s message is delivered to the model wrapped as quoted data, never as instructions, with an explicit clause against following embedded commands or appending dictated text.
Credentials rule: credentials are never asserted unless stated verbatim in the evidence — the agent will say a qualification is not present rather than invent one.

Validation checks & metrics

Two layers of checks run: version/drift checks (deterministic, cheap) decide whether anything even needs re-testing; the evaluation suite decides whether it is safe to ship. Results below are from the latest recorded run.

Check	What it verifies	Result
Version verification	The live PyPI / GitHub version of each tracked project matches what the site and corpus claim (a project deliberately not version-stamped is treated as a living doc, not drift).	current
Drift detection	Every corpus file is hashed against the last synced baseline; any change flags a re-index + re-evaluation before deploy.	in sync
Retrieval evaluation (hit@6)	For each answerable question, the correct gold source appears in the top-6 retrieved chunks.	27/27
Answerability tests	Recruiter-style questions with known-correct sources; the answer must cite `[Source N]` and contain the gold facts.	27/27
Out-of-scope tests	Questions the corpus cannot answer (finance, weather, personal data) must be refused, never improvised.	8/8
Prompt-injection tests	Embedded instructions — role hijacks, system-prompt extraction, forced slogans, dictated falsehoods — must be ignored.	8/8
Credential-trap tests	Attempts to make the agent invent degrees or certifications must fail; it states only what the evidence supports.	5/5
Deployment gate	The corpus deploys to the live agent only if every category above is green. A single failure blocks the deploy.	enforced

Scored automatically: gold-source matching for retrieval, required-keyword and citation checks for answers, refusal-pattern detection for out-of-scope, and forbidden-pattern regexes for injection and credential cases. LLM: Gemini Flash. The suite is re-run every time the corpus changes — see the eval report for the failures it has caught (a real injection breach, a live model regression, and an error in one of my own published papers).

Automation flow — the self-maintaining pipeline

A scheduled job keeps the deployed agent in step with reality. It is a plain Python pipeline; the safety comes from the eval gate, not from any model deciding what to ship.

  PyPI + GitHub  (real versions / release history)
        |  harvest
        v
  fact-tiering  ->  VERIFIED / SUPPORTED / REVIEW      (assigned by rules, never by a model)
        |
        v
  corpus changed since last sync?  --no-->  stop      (cheap: no re-index, no LLM calls)
        | yes
        v
  re-index corpus  (local embeddings)
        |
        v
  EVALUATION GATE  (48 cases)  --any red-->  DEPLOY REFUSED, logged for review
        | all green
        v
  deploy corpus  ->  restart agent  ->  snapshot to versioned history

Confidence is labelled by rule, never by the model: VERIFIED a deterministic check passed (e.g. version matches live PyPI); SUPPORTED the claim is in a source document with real provenance; REVIEW a source changed or two sources disagree.

Deployment safety model

The gate is deterministic. A test suite — not a language model — decides whether to deploy. A red result refuses the deploy and records why; nothing half-checked reaches visitors.
Cheap when idle. If nothing changed, the run does a few version/hash checks and stops — no re-index, no model calls, no cost. The expensive path only runs on a real change.
Honest boundary. The pipeline detects, validates, gates, deploys and versions. It does not author claims: when a project changes, the sentences describing it are written by a human and then validated — a model is never allowed to invent version claims about the work.
Versioned history. Every sync is snapshotted, so the state of the knowledge base is auditable over time.
Room to extend. The trigger is a single scheduled entry point, so an event-driven trigger (run on a new release instead of on a timer) or a failure notification can be added later without changing the pipeline itself.

Live status

Loaded from the pipeline’s own records. If it can’t be fetched, the last known values are shown with their timestamp.

Evaluation

48/48

Eval gate

PASS

Sync state

in sync

Last sync (UTC)

—

Snapshot generated —.

Try it: open the portfolio and click the chat button — then ask it “what retrieval do you use?” or “did Panos finish his degree?” and click a source chip to see the exact evidence behind the answer.