How accurate is a call graph built without a type checker?

Why measure it

Half of agent-lens stands on one data structure. hubs, impact, layers, risk, reach and the function graph viewer all read the same static call graph, built by a resolver that parses each file (syn for Rust, oxc for TypeScript, ruff's parser for Python, tree-sitter for Go) and never runs a type checker, a build, or a language server. That is what keeps the binary fast and dependency-free — and it is also a guess. A graph that invents edges tells an agent that a harmless change has a blast radius; one that drops edges tells it dead code is safe to delete.

So the guess needs a number. Hand-labelled suites such as PyCG's micro-benchmark are small and use their own edge definitions, so instead each language gets a mechanical oracle: the type checker the ecosystem already trusts. Humans only look at the pairs where the two sides disagree.

How it is scored

1. Pin small, well-tested projects by full commit SHA, two per language.
2. Run the oracle: go-vta over the packages and their tests; pyright's language server asked for the definition at every call site; rust-analyzer's call hierarchy; the TypeScript checker walking every call expression.
3. Normalise both sides to function → function pairs: the innermost named function around the call site, and the function that runs. Builtins, stdlib and third-party code are out; constructors map to __init__ / constructor.
4. Score. Precision is the share of agent-lens resolved edges the oracle confirms. Recall is over the oracle's static dispatch edges — a direct call or a method on a concrete type — since interface and trait calls are not expected to resolve from syntax; agent-lens reports those as candidate sets instead.
5. Adjudicate disagreements in a TOML file keyed by qualified name, so a verdict survives line shifts and is reused on every run.

Everything below counts every unadjudicated agent-lens-only pair as a false positive, so the precision is a lower bound. Full edge definition, oracle contracts and scoring rules: docs/callgraph-accuracy.md; the harness: scripts/callgraph-accuracy.

Accuracy

Measured 2026-09-28 on agent-lens 2a26836 with mise run callgraph-accuracy.

"Left as candidates" are static calls agent-lens saw but would not commit to: it emitted an ambiguous edge listing the possible callees, the right one among them. That is the resolver's deliberate trade: when a name is shared, report the set rather than guess, so precision stays high and an agent can still follow the candidates.

What the oracles found

The point of an oracle is the disagreement list, not the headline. Reading it turned up concrete resolver bugs in every language; each fix landed with a metamorphic test and the numbers moved accordingly (first run against each oracle vs today, same targets):

• Go. Calls inside func literals were dropped; a method on a call's result (NewDecoder(r).Decode(v)) was read as the package-level function of that name; wg.Add(1) could bind to a free function Add.
• Python. The biggest jump. import more_itertools as mi over a from .more import * barrel left almost every test call unresolved; from itertools import count was bound to an in-repo count; A() did not reach A.__init__; super().m() could bind to the caller's own override.
• TypeScript. Functions inside a namespace were named without it, new C() recorded no call site, and calls through a local closure (const parse = (s) => …) fell through to name matching.
• Rust. Calls in macro arguments (assert_eq!(f(x), …), format!("{}", g())) were invisible, and two V::fmt in one file (Display and Debug) lost all their edges. Making macro arguments visible is also why log's precision fell: it exposed calls to methods a macro generates, which have no node to resolve to.

Speed against accuracy

Wall-clock for the whole repository, warm caches, on a 4-vCPU Xeon @ 2.10GHz: median of 10 runs of agent-lens analyze function-graph --format json, median of 3 runs of each oracle. The Go oracle is timed as a prebuilt binary, so its compile is excluded; pyright and rust-analyzer include starting the server and indexing, which is what a caller pays.

The two sides sit in opposite corners. agent-lens keeps F1 at 0.87–0.99 on seven of the eight projects (rust-lang/log, at 0.61, is the outlier its macros explain above) while spending one to three orders of magnitude less time; the oracle buys the last few points of accuracy with a type checker or language server. Every project finishes in under 160 ms, which is what lets the graph analyzers run inside a hook or on every agent turn. The gap is smallest against go-vta and tsc (11–50×), which type-check a module in a single process, and largest against pyright and rust-analyzer (450–760×), where the oracle is a language server answering one request per call site. Construction cost on synthetic 1,024-function corpora for all four languages is gated in CI by the function_graph Criterion benchmark, so a slowdown fails the pull request that causes it.

Limits

• Dynamic dispatch is not resolved. Interface, trait and virtual calls are under 3% recalled as resolved edges in Go, Rust and TypeScript; they show up as candidate sets, which contain the right callee 73–100% of the time.
• Types still matter. The remaining static misses are method chains on returned values (ok(1).andThen(…)), methods inherited from a base class, and re-exports through pub use m::* — things only a type or a class hierarchy answers.
• Eight projects is a sample. The discovery runs behind each fix covered 20 more repositories (listed in the doc), where precision held at 0.97–0.99 per language; they are not in the pinned set and none of their disagreements are adjudicated.

Reproduce with mise run callgraph-accuracy [target…]. It needs network access for the checkouts, the Go module proxy, PyPI and npm; output lands in target/callgraph-accuracy/.