Personal project by Ulf Fäldt Reifenberg, built on 2026-11-16 with coding agents for a job application to Konsert. Unofficial and not affiliated with Konsert or Rouse. All data is synthetic; the graph schema is my own sketch and says nothing about how Sapivia works.

Does graph context help an LLM answer questions about a company?

Loading results.json …

What was tested

Accuracy per retrieval arm

Exact match against ground truth that is computed from the graph. Intervals are 95% bootstrap intervals over the 40 questions (10,000 resamples). With 40 questions an interval is about ±15 points wide, so small differences cannot be seen.

F1 is the set-overlap score for list answers (1 or 0 for numbers and strings). Tokens are the sum of what the provider reported for all model calls for one question (input + output).

Differences between arms (same questions)

Paired bootstrap: the same question indices are resampled for both arms, so the interval reflects how often one arm wins on a question where the other does not. "Wins" counts questions where only one of the two arms was right.

Accuracy per question class

Step budget for the agent

The agent calls search_nodes, get_node and neighbors as prompted JSON. A malformed or unknown call uses one step. When the budget is spent, tools stop being executed and the model must answer.

Why the wrong answers were wrong

For each wrong answer: were the nodes needed for the answer in the model's context (reading miss), or not (retrieval miss)? "Needed nodes" are listed per question in data/questions.json (support). For list questions they are the answer nodes, not the intermediate hops, and for counts they are the counted nodes, so the retrieval-miss label is strict for counts that can be read from one node's edge list.

Access control: filter before retrieval, or ask the model to behave

Four worked questions

One per class, picked automatically as the question where the four 8B arms disagreed most.

Cost and metering

What this does not show

Contact: linkedin.com/in/ulffaldt · other projects: ai-release-gate, injection-gate-bench, mcp-scope-gate, euro-asr-bench