Same Score, Different Confidence: Why a 40 Out of 100 With Thin Disclosure Is Not a 40 With Three Quarters of Segment Data
A low score can mean a weak company or a company that simply disclosed little, and a single number hides which; here is how LacunaIndex reports how well-evidenced each score is, separately from the score itself.
A diligence analyst looks at two issuers. Both score 40 out of 100 on execution. On the face of it, they are the same.
They are not. One was scored from three quarters of segment disclosure, verbatim management quotes, and reconciling filings. The other was scored from a thin investor deck and one earnings call, with most dimensions marked "not disclosed." The first score is a measurement. The second is closer to a shrug that happens to be expressed as a number.
LacunaIndex treats this as a first-class problem: the score and the confidence in the score are two different outputs. This post explains how the second one is built, and how to use it.
The problem with a single number
A governance or execution score compresses a lot of evidence into one figure. That is useful for ranking and screening. It is dangerous for diligence, because the compression throws away the one thing you need before relying on it: how much evidence stood behind it.
Two failure modes follow:
- Sparse disclosure reads as weakness. An issuer that discloses little can score low simply because there is little to credit.
- Sparse disclosure reads as safety. The opposite risk: no contradicting evidence was found, so nothing was marked down. Silence gets mistaken for a clean bill of health.
Neither is a conclusion about the company. Both are conclusions about the public record.
How the confidence layer works
In the codebase, the confidence layer is deliberately described as decoupling "what we observed" from "how well it was evidenced." It is computed deterministically from the report's existing structure, with no extra model prompt, and the rules are intentionally simple so they can be defended line by line.
Execution dimensions are scored by counting evidence items that carry a real verbatim quote (more than a few characters, so empty placeholders do not count):
| Evidence items with a verbatim quote | Dimension confidence |
|---|---|
| 0 | 0 |
| 1 | 35 |
| 2 | 55 |
| 3 or more | 75 |
There is an additional rule: if a dimension explicitly records missing disclosure and has one or fewer evidence items, its confidence is capped at 25. A dimension that says "we looked and could not find it" cannot also claim middling confidence.
Vision dimensions (how clearly management articulates strategy) are scored on the depth of the supporting evidence text. Very short evidence yields 15; progressively longer, more substantive evidence steps up through 45 and 65 to a maximum of 80.
Section confidence is the weighted mean of its dimension confidences, using the same weights that produce the section score. If a heavily weighted dimension is thinly evidenced, it drags section confidence down in proportion, exactly as it would drag the score.
The result is banded: 65 or above is high, 40 or above is medium, and anything lower is low.
The gap between execution and vision is bounded by the weaker side
LacunaIndex compares what management says it will do (vision) against what the record shows it doing (execution). That gap is only as trustworthy as its weaker input. The confidence in the gap is therefore the minimum of the two section confidences, and the report records which side is the binding constraint. A dramatic gap built on a well-evidenced vision and a barely-evidenced execution section is flagged as bounded by execution confidence, not presented at face value.
The caveat is built in
When confidence is low, the product does not leave the reader to infer it. The execution caveat reads, in substance: the score was derived from sparse disclosure, so interpret it as a floor, not a forensic assessment; the absence of evidence is recorded as low confidence, not as a clean bill of health. The vision caveat makes the same point about strategic articulation.
"Floor, not forensic assessment" is the right reading. A low-confidence score tells you the minimum the record supports. It does not tell you where the truth sits.
Synthesis confidence has a floor
The overall verdict also carries an "analyst confidence in this synthesis" figure. This is explicitly not confidence in the issuer's narrative. It is how well-evidenced the synthesis itself is. A deterministic floor is calculated as the average of execution and vision confidence, plus small bonuses for source depth (a modest bonus at four or more sources, a larger one at eight or more) and a bonus for having at least three links in the evidence chain. The language model's own stated confidence may not undercut this floor by more than a repair threshold, which prevents a fluent but under-evidenced summary from sounding more certain, or less certain, than the evidence warrants.
Six states for what the record shows
Confidence sits on top of a second control: every claim a report touches is classified into one closed vocabulary, so two analysts looking at the same source material reach the same label.
- Verified: explicit, measurable disclosure directly substantiates the claim (management states a revenue figure and the segment table reports the same one).
- Supported: indirect operational evidence makes the claim credible without proving it.
- Partial: disclosure exists but is incomplete.
- Claimed: a narrative assertion with no measurable proof attached.
- Unverified: no public evidence found. The definition is explicit that this records absence of proof, not absence of truth.
- Contradicted: public evidence directly conflicts with the claim.
Unrecognised or malformed values fall back to Unverified rather than being waved through, so an error cannot accidentally upgrade a claim.
How to use this in diligence
- Read confidence before score. If confidence is low, treat the score as a screening floor and move straight to the questions the report flags as missing.
- Convert low confidence into a request list. Every "missing disclosure" note is a ready-made question for management or an expert call.
- Do not rank across confidence bands without saying so. Two issuers at 40, one high-confidence and one low-confidence, belong in different conversations.
- Treat "Unverified" as an open item, not a finding. It means nobody has shown the claim to be true or false yet.
Limits
These heuristics are intentionally simple. Counting quotes and measuring evidence text length are proxies; they do not judge whether a quote is persuasive. A dimension can reach 75 with three weak quotes. The layer tells you how much evidence was found, and the six-state classification tells you what kind. Neither replaces reading the underlying filings, which is what the verbatim citations are there to make fast.
The principle is the one to carry away: a score tells you where the record points, and a confidence tells you how far to trust the pointing.