Should AI Tell You When It's Guessing?

Why This Question Matters

Most LLM tools respond in a confident tone regardless of whether they are pulling from solid sources or making aggressive inferences from scant evidence. Users usually can’t tell which is happening under the hood. As these tools move deeper into everyday work, that difference has increasingly high stakes.

One common proposal is to have the tool provide a confidence rating next to each response, such as a score from 1 (low confidence) to 10 (high confidence) that depends on the level of direct, clear, and verifiable evidence in support of its response.

That sounds like a simple feature to ship, but it’s actually pretty unclear how users would respond. In my dissertation research and beyond, I’ve published several peer-reviewed studies on how people interpret and respond to uncertainty. Disclosing uncertainties can do some strange things, with some types having positive responses like increasing trust and credibility while other types have negative responses like increasing skepticism and backlash (for a short summary, see How to Communicate Uncertainty Without Losing Credibility.)

But it’s pretty important to get this right. This feature could change how hundreds of millions of people read and respond to AI output. If done right, it could simultaneously prompt more judicious and responsible user behaviors, while also building stronger trust in AI overall! So let’s dive in.

A Note on Self-Report vs Actual Behavior

If we ask people whether they want a confidence indicator (and why) and whether they would use it, we can get some interesting information. But we need to keep in mind that it’s cheap and easy for survey participants to say “yes I’d want that” and “yes I’d definitely adjust my decisions based on it.” This is why it’s important to observe what users actually do.

So, I tested both. Here, Study 1 is a survey that captures stated demand for a “confidence indicator” feature, how people rank the feature against other things an assistant could add, and how they say they would act if they were given an answer that was flagged as “low-confidence.” Then, Study 2 is a controlled experiment with a behavioral component. It measures whether people actually decide to verify answers when they are flagged as “low confidence” and also how the use of a confidence indicator affects both the accuracy of beliefs and the subjective ratings of the tool.

As we go through, note the contrast between the two studies. The survey tells us what users think they want or do, while the experiment gives us a look at what actually happens.

Study 1 · SurveyWhat People Say They Would Want/Do

The first study is a short survey run on Prolific. After some data cleaning and quality screening, it covers 215 respondents—86% of whom use AI tools at least once per week. The survey first measured how a confidence indicator ranks in perceived usefulness compared to other potential features, then asks about reasons why it would or would not be useful, and also how likely they would be to externally verify information that was flagged as “low confidence.”

Most people say confidence indicators would be very useful

When asked directly how useful a confidence rating would be on a five-point scale from “not at all” to “extremely useful,” the answers skewed strongly positive. Two thirds of participants rated it at the top end (40% “very useful” and 26% “extremely useful”). Another 27% called it “moderately useful.” And only 8% in total picked “only a little” or “not at all useful.” These are overwhelmingly positive responses to a feature that they haven’t even seen or tried yet.

But it’s just too easy for a participant to rank features as maximally useful! So I decided to force them to make tough choices by ranking five possible additions to an AI tool against each other. This is often a good strategy for mitigating demand effects or acquiescence bias.

In the forced-rank measure, the confidence indicator landed near the top. Figure 1 shows that its rank distribution is front-loaded, with most people placing it first or second and only a handful ranking it last. It tied with source citations as the feature with the highest % of respondents (56%) ranking it in their top 2 places. This is still strong positive feedback about this proposed feature.

Most people say they would externally verify flagged responses

I also presented participants with a specific scenario in which an AI answer is labeled “2/10 - Low Confidence.” I then asked them how likely they would be to verify it against an external, official source before relying on it.

Figure 2 shows that the response was nearly unanimous, as 92% said they would be likely to verify (83% “very likely”; 9% “somewhat likely”). Taken at face value, the self-report data says that a low-confidence flag is useful and effective. It appears that if you just warn people about tenuous inferences, they will find it useful and 83% of them will be “very likely” to verify with an external source. But keep those optimistic numbers in mind for when we move on to Study 2.

The reasons why

Next, I gave an open-ended question to people who said this feature would be useful. This question asked them to explain in their own words why they think a confidence indicator would be useful. Using these qualitative responses, I conducted a thematic analysis and coded them into seven main themes, letting each response carry more than one theme. The treemap in Figure 3 sizes each of these categories by how many responses expressed it.

Two things stand out from the qualitative coding. The single most common reason, which is present in nearly half (48%) of responses is triage. People want the rating as a signal for when to give further scrutiny to an answer. The second largest cluster is related but inverted. This 26% of responses mentioned that they would want a confidence indicator precisely because it would let them trust the AI more and check less. Both of these leading themes come from the same root. We’ve known for a long time in cognitive psych that our brains are efficiency-maximizing prediction machines. We don’t mind working hard if it’s needed, but also we’d love to conserve energy when possible. Applied here, a confidence indicator is a cue that tells us when further analysis is needed versus when we can relax and just trust what we’re seeing.

It’s also worth noting that a meaningful minority (11%) were openly skeptical even while rating the feature useful. They pointed out that an AI assessing its own confidence is still just the AI making fallible judgments.

Study 1 leaves us with a clear picture. People (say they) want the feature, rank it as highly useful, and almost without exception (say they) would take action if they received a low-confidence warning. The obvious next question is whether any of that survives contact with a live decision environment.

Study 2 · ExperimentWhat People Actually Do

Study 2 was a 2×2 between-subjects experiment with 970 participants recruited on Prolific (778 after data cleaning). Everyone was paid a baseline amount for their time, but to incentivize careful information gathering they were told they would earn a cash bonus for each correct answer on a knowledge quiz at the end of the study.

Participants were presented with a simulated conversation thread between them (the user) and a LLM interface analogous to ChatGPT, Claude, etc. The conversation showed the user asking a simple factual question and the LLM providing a response. To guard against any bias from any specific topic, I randomized the topics such that a participant either saw information about vitamin D and sleep, remote work and productivity, or earthquake prediction.

The main experimental manipulation occurred by randomizing participants into one of four conditions that were formed by crossing two factors:

The Verification Decision

After reading the response, each participant chose what to do next. They could either go straight to take the knowledge quiz (to try earn their bonus cash), or they could first open a secondary source to check the LLM’s claims. They were told that checking the external source would likely cost them about two extra minutes of time. This was done because that tradeoff mirrors the real decision people make whenever they consider fact-checking an AI answer.

That choice—verify or not—was the primary outcome measure. It is a real behavior (not a survey response) that is made under competing motives of costs and benefits, and it answers the question product teams care about: does the indicator actually get people to check the AI’s work?

As shown in the example in Figure 5, the secondary source was written so that anyone in an inaccurate condition who checked would find the correct information from an authoritative entity. This also gave them an advantage on the quiz, so the decision to verify would have real downstream consequences for their cash payout.

After the quiz, participants rated their satisfaction, willingness to reuse the AI, perceived accuracy, and how useful a confidence feature would be in general. The full study design is visualized in Figure 6.

Finding #1. A low-confidence flag drove people to verify.

Across all conditions, about 21% of participants chose to fact-check the response by clicking into the secondary source. The chart below in Figure 7 shows the verification rates in each pair of conditions (accurate responses and inaccurate responses). The left pair isolates effect of adding a 9/10 flag on a correct answer, and the right pair isolates the effect of adding a 5/10 flag on an incorrect answer.

In terms of verification rates, the low-confidence flag did exactly what it is supposed to do. It raised verification by 13 points (from 17.3% to 30.2%) an effect that is both practically meaningful and statistically significant effect (p = .003). As you might expect, the high-confidence flag did much less for verification, because users are less motivated to verify an answer that is already high confidence. The high-confidence indicator nudged verification up about 4 points, although this difference is not statistically significant (p = .27).

It’s also interesting to see that in conditions with no confidence indicators, the inaccurate info had the same verification rates as the accurate info (17.3% versus 16.8%). Basically, when left to themselves, people did not catch the error. The extra verification appeared only when the 5/10 low-confidence flag was added.

One honest limit. The formal test of whether the 2/10 flag lifts verification more than the 9/10 flag, the indicator-by-accuracy interaction, does not quite reach significance (p = .22), because the high-confidence flag also drifted upward and the study cannot cleanly separate the two simple effects. But the headline does not rest on that contrast. The 2/10 flag’s effect against its own no-flag baseline is large and significant on its own.

Finding #2. When people did verify, it boosted their knowledge scores.

Verifying paid off in the knowledge quiz scores. People who clicked to verify with a secondary source scored higher on the comprehension quiz (M = 1.78 out of 2) than people who did not (M = 1.28, p < .001). Intuitively, this gap was even more dramatic in the inaccurate conditions. Non-verifiers who saw wrong information scored 0.61 on their knowledge test, while verifiers in those same conditions averaged 1.71 out of 2.

Finding #3. The low-confidence flag harmed subjective ratings.

Even though the low-confidence indicator prompted people verify and thereby increased their knowledge accuracy, it created less positive attitudinal ratings of the tool overall. Figure 9 shows that users in the “inaccurate + indicator” condition (i.e., those who saw a 5/10 low confidence flag) reported significantly lower satisfaction, willingness to reuse the tool, and accuracy ratings. Basically, people rated the tool more highly when they simply saw inaccurate information without a warning flag.

Finding #4. Users still say confidence indicators would be useful, regardless of which condition they were in.

Lastly, I ended Study 2 with the same measure used in Study 1 for assessing perceived usefulness of a confidence indicator feature. The purpose of this was just to see how useful people think confidence indicators would be—now that they were just in a scenario where they either were or were not given one.

Across every condition, participants rated a confidence indicator feature as very useful (means between 3.9 and 4.1 out of 5), and no condition stood out as meaningfully different. People who had just used the indicator rated it about the same as people who never saw it.

Interestingly, those in the “inaccurate + indicator” condition who had rated the tool more negatively gave about the same usefulness ratings as everyone else. One way to interpret this is that people get frustrated when a tool gives them a low-accuracy answer even if it’s flagged, but overall they like the idea of confidence indicators in theory.

Main Takeaways

There are some important practical implications to unpack here.

Study 3?

Study 2’s main limitation is ecological validity. Paid participants were viewing a static mockup and were reading an AI tool’s answers to questions that they did not ask. This means they did not have an intrinsic, genuine incentive to pursue accuracy (other than the cash bonus for accurate quiz scores). In a real session, a person’s own motivation to get the answer right would be stronger. It’s hard to know how that would affect behavior and attitudes. It could be that real users in a real interface would be more appreciative of careful confidence calibrations if they are intrinsically motivated to get the right answer. Or it could be that real users would

Either way, this is a valuable feature to continue testing. With some strategic framing, perhaps a product team could introduce users to this feature in way that emphasizes its benefits and mitigates some of the negative response to low-confidence flags. If users understand clearly that a low-confidence flag is actually the tool doing its job (and benefiting the user) then maybe we can arrive at a place where users verify low-confidence results AND reflect on that experience positively. Another potential angle for future development is to reduce the friction and cost of verification. Perhaps tools can integrate in-line source links, evidence summaries, or one-click fact-checking to streamline the verification process. That way, users may see the prompt to verify less as an inconvenient obstacle and more as a helpful enhancement.

The best next step is to run Study 3 in a live product setting with real users. Let’s do it!