Personal project by Ulf Fäldt Reifenberg, built on 2026-11-16 with coding agents for a job application to Konsert. Unofficial and not affiliated with Konsert or Rouse. All data is synthetic; the graph schema is my own sketch and says nothing about how Sapivia works.
Does graph context help an LLM answer questions about a company?
Loading results.json …
What was tested
Accuracy per retrieval arm
Exact match against ground truth that is computed from the graph. Intervals are 95% bootstrap intervals over the 40 questions (10,000 resamples). With 40 questions an interval is about ±15 points wide, so small differences cannot be seen.
F1 is the set-overlap score for list answers (1 or 0 for numbers and strings). Tokens are the sum of what the provider reported for all model calls for one question (input + output).
Differences between arms (same questions)
Paired bootstrap: the same question indices are resampled for both arms, so the interval reflects how often one arm wins on a question where the other does not. "Wins" counts questions where only one of the two arms was right.
Accuracy per question class
Step budget for the agent
The agent calls search_nodes, get_node and neighbors as prompted JSON. A malformed or unknown call uses one step. When the budget is spent, tools stop being executed and the model must answer.
Why the wrong answers were wrong
For each wrong answer: were the nodes needed for the answer in the model's context (reading miss), or not (retrieval miss)? "Needed nodes" are listed per question in data/questions.json (support). For list questions they are the answer nodes, not the intermediate hops, and for counts they are the counted nodes, so the retrieval-miss label is strict for counts that can be read from one node's edge list.
Access control: filter before retrieval, or ask the model to behave
Four worked questions
One per class, picked automatically as the question where the four 8B arms disagreed most.
Cost and metering
What this does not show
Contact: linkedin.com/in/ulffaldt · other projects: ai-release-gate, injection-gate-bench, mcp-scope-gate, euro-asr-bench