A convincing answer can still be wrong

Some expected results are exact: a note still exists, a forbidden request is refused, a count has the right value. When a feature uses a language model, an answer can satisfy its format and still fail the task. We need to carry the same care about expected results into its contents.

Imagine an app that turns event announcements into calendar drafts. You paste this into it:

Mending circle. 24 October 2026. Community room, Willow Library. Bring a garment you would like to repair.

A moment later, a tidy card appears: Mending circle, 24 October, 4–6 pm. There is a little calendar icon and an Add button. The request returned 200, the JSON parsed, and every field fits the screen. Where did four o'clock come from?

Look back at the announcement. It has a date and a place, but no start time or duration. A finished-looking card has made an unsupported detail look like a decision someone already made.

Decide what the answer is allowed to contain

Before changing the prompt, decide what this feature promises. Here it extracts information from the supplied announcement. It does not choose a convenient meeting time or search for a different event with a similar name.

That gives us an answer to the missing field. Keep the time empty and show “Time not given”. Ask the person to confirm it before creating a timed event. Do not quietly convert it into an all-day event either; that would add another interpretation the announcement did not supply.

A response format can define fields for the title, date and start_time while allowing missing values to stay empty (often represented as null). The same rule applies to a missing date as to a missing time. Conflicting values need a review flag rather than a guess. That helps your code handle incomplete information, but it cannot establish that a filled-in field is true. A valid value can still be unsupported. The instruction and the interface have to agree that an incomplete draft is an acceptable result.

Try changing just one part of the announcement: add “Starts at 16:00”. Now the draft should include 16:00. Remove that sentence and the time should disappear again. This pair checks whether the answer follows the input, rather than whether you like how either card looks.

Keep the evidence close to the field

For a feature like this, let the user inspect the source beside the draft. The date can point to “24 October 2026”; the location can point to “Community room, Willow Library”. The time has nothing to point to.

You can ask the model to return a supporting excerpt for each extracted field. Then check that the excerpt really occurs in the supplied text. That catches invented quotations, but finding the words is only the first check. “Last year's session started at 16:00” contains a time without supporting this year's start time. Someone still needs to check what the excerpt says about the field.

Keep separate states for a missing detail, an ambiguous detail and a failed request. If the provider times out, your app has not learned that the announcement contains no time. It has failed to produce an answer. The user should be able to retry without losing the announcement, rather than receiving an empty calendar card disguised as a result.

Make a small comparison you can repeat

You can evaluate this feature before building an evaluation platform. Make six short announcements and write their expected fields before running the model:

A complete announcement with an explicit date, start time and place.
The same announcement with its time removed.
An announcement with a start time but no end time.
One that mentions last year's time without giving this year's.
One with two conflicting start times and no explanation of which is current.
One with no event date at all.

For the conflict case, the expected behavior is to flag uncertainty for review, not choose whichever time makes a complete card. For each result, mark supported fields, missed fields and unsupported additions separately. Also record whether a draft that needs confirmation was presented as ready to add.

Run each announcement twice. These twelve answers let you inspect variation across runs, but they still cover only six situations. Repeating the complete announcement a hundred times would not test the missing-time case.

In lulumelon's reporting code, repeated answers remain grouped under their prompt when estimating uncertainty. Another rule excludes questions that already name a brand from the rate intended to measure its unprompted appearance. The two choices address the same evaluation problem we have here: a score should answer the question it claims to answer. Counting a name supplied by the question would measure something different.

Now change one thing in your calendar feature: its instruction, response schema or model. Run the same cases again and compare the actual fields. Keep the announcements, expected behavior, settings and outputs together. Reserve a few additional announcements for a later check, so revising around these six does not become your only evidence.

Do not hide request failures inside a neat accuracy figure. If two of twelve attempts time out, report ten completed answers and two failures, then describe the errors among the completed answers. lulumelon's offline tests likewise distinguish an unsuccessful provider call from a successful answer that mentions no brand.

Six cases are a starting set, not a claim that the feature is reliable everywhere. When a real announcement exposes another ambiguity, add that situation. The useful result of this exercise is concrete: you can point to the field that changed, the source that supports it, and the missing information the interface must still ask for.