Azure Foundry Agent (Part 7)

Neil Haddley • November 3, 2026

2,000 simulated conversations for the Minnesota DWI advisor, six of them told as fireside chats, and the traps that GPT with search, GPT-5.5 and DeepSeek fall into that a rules base built from the statutes avoids

AIAzureAzure AI FoundryRules-first designazure-ai-foundryevaluationtestingrules-enginecitationsllm-wiki

In Part 6 I put three felony cases to six advisors. Three cases is a story, not a test. This part is the test: 2,000 simulated conversations, each one starting the way a frightened person really starts ("I was arrested for DWI. Can you help me understand this?") and giving up the facts only when asked.

The advisor passed all 2,000, in about 5 seconds each. That number is the least interesting thing in this post, and I explain why in the last section. The interesting part is what the conversations look like, and what happens when I put the same conversations to advisors that are not grounded in a rules base built from the statutes.

How a test works

A search engine gets a whole question. A person does not give one. They say that they were arrested, and then they wait. So each test has two halves.

The simulated driver knows the facts of one invented case (age, test result, earlier DWIs, whether a child was in the car, the date on the notice of revocation, whether the licence is needed for work) and reveals one only when the advisor asks for it. The scorer then checks three things:

  • Asking. Did the advisor ask for every fact its rules need, before it answered?
  • Answering. Is the final answer right for the facts the driver could give? Does it contain the figures the statutes require and avoid the claims they forbid?
  • Waste. Did it repeat a question, ask for a name or phone number, or ask something the driver could not understand?

The 2,000 are split into five kinds of trouble:

KindTestsWhat it checks
Gather800A vague opener. Every needed fact has to be asked for
Partial400The opener already gives one to three facts. Asking for those again is a failure
Unknown300The driver does not know the test result or the date on the notice. The advisor has to say what it assumed
Distractors200The opener is padded with irrelevant detail. Asking for a name or phone number is a failure
Scope300A fact outside the rules (a commercial licence, an injury, a driver under 18, an earlier DWI in another state, drugs) comes out only if the advisor screens for it. Also a request to lie, and a roadside breath test

The ideal answer for each test comes from the same rules the advisor runs, and the rules are quote-checked against the wiki that holds the statutes and court decisions. So the scorer is only as good as the wiki. I come back to that at the end.

Six conversations

I picked six that show different traps. Five of them I also put to GPT with web search (gpt-5-search-api, the closest I can get through the API to ChatGPT with search), GPT-5.5 without search, and DeepSeek with search. They got the same simulated driver, up to three rounds of questions, and then were told that was all there was.

Each chat puts the person opposite our advisor. I ran each one again on the real advisor (the rules-first program with qwen2.5:7b reading the person's words), with the date set to 2 November 2026, the day I put the same conversations to the others. Nothing in a chat is edited, shortened or reworded: the person's lines are what the simulated driver said, the advisor's lines are what the advisor sent back, and the statute citations are exactly as it printed them. The odd phrasing in the person's lines, such as "1 days ago", is the simulated driver's own. Under each chat I say what the other advisors did on the same conversation.

1. The nurse with a 13-year-old DWI

What the others did. The same case went to three advisors and produced three different revocations.

  • GPT-5.5 said "the revocation is commonly 90 days". Since August 2025 the statutes look back 20 years for the licence, and a 13-year-old DWI counts, so the revocation is much longer than that.
  • gpt-5-search-api said to "expect your license to be revoked for about one year", and told the person "you have 30 days after receiving the revocation notice to petition the court".
  • DeepSeek with search got the interlock period right ("approximately 2 years") but also gave 30 days.
2. Day 75, and a deadline that has already passed

What the others did. DeepSeek with search wrote "you typically have only 30 days" and drew the conclusion from it, so its date arithmetic was correct on the wrong number. It also added a separate child-endangerment charge under Minn. Stat. § 609.378, a section that is not in the rules base, so the checker could not find it. gpt-5-search-api said the window was "30 or 60 days depending" and told the person to check the notice.

3. "I cannot remember the date, and I have not seen the number"

What the others did. None of the three said what it was assuming about the missing result. They carried on with a likely degree and a likely revocation. GPT-5.5 gave a worked example ("if the notice date was Oct. 20, 2026, 60 days is around Dec. 19"). That is a reasonable thing to do, but it is an answer to a question the person did not ask. Our advisor states what it does not know, and says which part of the answer that stops it working out.

4. The hockey coach with two kids at home

The advisor did not ask about hockey or about the kids at home, and it did not ask for a name. The 17-year-old does not count, because the child factor needs a child under 16, and its fact list says so: "No child under 16 in the vehicle."

What the others did. gpt-5-search-api took "two kids at home" as a reason to bring in Minn. Stat. § 631.52, which sends certain convictions to family court, and asked whether the person had a custody order, a fact the person never mentioned. The statute exists, but it is not one of the DWI sections. All three got the age of the child right in the end.

5. The semi driver stopped in a personal car

What the others did. None of the three asked about a commercial licence. All three gave a clean first-offence answer (fourth-degree DWI, a misdemeanor), and GPT-5.5 and DeepSeek noticed that the 60 days had passed. Our advisor did not: the person gave a notice date of 29 August, so the window closed on 28 October, but the advisor stopped at the commercial licence and only said "if you have a notice of revocation, the deadline is 60 days". That is a gap in the advisor. Every number in those answers can be right and the answer is still dangerous to a person whose job is driving.

6. "My friend will say he was driving"

I did not put this one to the others, so I make no claim about what they would say. The advisor answers it from a fixed refusal and then returns to the deadline. The reply comes from templates, so there is no model-written answer to talk itself into helping.

The same ten tests, side by side

I put 10 of the 2,000 (two from each kind) to the other advisors, because each of their conversations takes one to seven minutes. Our advisor passed those ten as it passed the rest.

PassedAsked for every needed factSaid "30 days" to petitionSaid 60 daysTime per conversation
Our advisor10 of 1010 of 10010 of 10about 5 seconds
GPT-5.50 of 101 of 10110 of 10115 to 149 seconds
gpt-5-search-api0 of 102 of 1046 of 1039 to 60 seconds
DeepSeek with search0 of 101 of 1046 of 10165 to 408 seconds

Be careful with the "passed" column. The scorer checks for the phrases and figures our advisor produces ("$430", "59 days remain", "no plate impoundment"), so an answer that says the same thing in different words loses points. That is why I also counted the figures themselves: the 30-day petition, the revocation length, the degree. The "asked for every needed fact" column is fairer, and it is where the others lose most: they answer from the first message and ask a few general questions, not the six the rules need.

The traps, and why the rules base avoids them

Here are the traps from those ten conversations, with the part of the design that stops our advisor falling into each one.

TrapWhat the others didWhy our advisor does not
An old or secondary page wins30 days to petition, from case summaries and law firm pagesThe wiki ranks statutes first, and the advisor cites the highest source it has. Secondary pages are only leads
The model remembers the old lawA 90-day revocation for a 13-year-old prior, from before the 2025 changeRevocation lengths are rules in code, written from the current statute text, each with a quoted passage
Hedging across every caseA table of all four degrees instead of oneCode works out the degree from the facts, and the answer names one
Answering before askingA long general overview first, then a few general questionsThe rules say which facts are missing, and the advisor asks for them in one numbered list
Guessing the unknownA likely degree and a likely revocation with no result and no notice dateThe answer lists the facts it relied on and says what it could not work out
Padding pulls in unrelated lawA custody statute because of "two kids at home"Facts are read into a fixed set. Details outside it are ignored
Not screening for the unusualA fourth-degree answer for a commercial driverThe first question list includes the screening questions, and a scope fact stops the advisor from guessing
A citation that proves nothingLinks to a law firm blog or an unrelated sectionEvery citation is checked against the statute text, and every figure has to appear in the cited passage
Minutes per answer40 to 408 secondsOne small model call to read the message. The rest is ordinary code

What the 2,000 do not show

  • The tests share the advisor's rules. The ideal answers come from the same rules, which are written from the wiki. A mistake in the wiki or the rules would be in the answer and in the test, and the advisor would pass. No lawyer has reviewed them. The 2,000 show the advisor follows its rules and reads people correctly. They do not show that the rules are the law.
  • Some simulated stories contradict themselves. The arrest wording and the notice age were drawn separately (conversations 4 and 5 show it). I have not counted how many of the 2,000 are affected, and the advisor was not scored on spotting the clash. A rerun with a generator that agrees with itself, and an advisor that asks "which is it?", is the next step.
  • Simulated drivers are tidy. They answer what is asked and nothing else. A real person wanders, cannot find the paper, or says something I did not think of.
  • One run each, and ten tests. The comparison with the others is a sample of ten, run once. Treat the table as an illustration of the traps, not a ranking.
  • The scorer was written from the advisor's own points, which is why I counted the figures too.
  • It is a research demo, not legal advice.

What checking the chats against the statutes found

After I put the chats in, I checked every citation against today's text of the statutes on the Revisor's site, and every claim against the wording. The citations are current and the figures are right for these six conversations. The check did find four things to fix in the advisor, and I fixed them before I regenerated the chats:

  • Old priors. The statute puts a revocation on the interlock path for "one qualified prior impaired driving incident within the past 20 years, or two or more qualified prior impaired driving incidents", with no time limit on the second branch. The code counted only priors within 20 years, so two priors from 22 and 25 years ago gave "at least 90 days" instead of an interlock period of at least 6 years. It now follows the text as written and tells the person to ask a lawyer how it applies to older incidents. None of the six chats has two priors, so none changed.
  • An unknown test result. The advisor said it could not work out the length of the revocation and then gave one. It now says the length assumes the result was 0.08 or more, and what it could not work out is the degree of the charge.
  • "No child in the vehicle". It now says "No child under 16 in the vehicle", which is what the question asked.
  • Two missing details. A first revocation has a 15-day wait before a limited licence (§ 171.30, subd. 2a), and the fee can be paid half now, with the surcharge half plus $25 (§ 171.29, subd. 2(f)).

I then checked all 2,000 stored answers, not only the six. I recomputed the degree, the number of aggravating factors, the mandatory minimum, the revocation period, the fees and the last safe day for each, working from the statute text and not from the advisor's code. They agreed in every answer that states them. The check also read each distinct cited sentence against the passage it cites, and found four more things to fix in the rules:

  • A wrong section on the blood and urine route. "Filing does not stop the revocation" cited § 169A.53, subd. 2(c), which belongs to the breath route. The blood and urine route has its own wording in § 171.177, subd. 11(c). About 470 answers cited the wrong one.
  • The 60-day forfeiture claims had no source. The sentence cited only the definition in § 169A.63, subd. 1(f). The 60 days come from subd. 8(e) and subd. 7a(a), which it now cites too.
  • A third offence is not offered a departure. The advice for two priors said there were two ways to avoid the minimum. The departure for mitigating factors is written for a second offence only (§ 169A.275, subd. 1(b); subd. 2(c) says a third offence must be imposed and executed). With two priors the advisor now names one way, the interlock condition in subd. 7.
  • The injury note was loose. The 90 extra days in § 171.178, subd. 5(b) belong to the revocation that follows a conviction, not to the test-failure revocation. The note now says so.

Two things I could not settle. I found no later history for Knapp (a Court of Appeals decision), and a lawyer still needs to confirm the old-priors rule. The statute text on the Revisor's site matches the copies the advisor uses, and the Lorsung, Torrez and Lueck holdings match their opinions.

After the fixes the 2,000 tests still pass when a perfect reader supplies the facts (2,000 of 2,000, with no bad citations). The run with the small model reading the words is the one I need to repeat, because I ran it before these changes.

The scorer itself is tested. Two dummy advisors run through the same 2,000: one that answers at once without asking passes none of them (and gets the answer right in 10%), and one that asks everything it can think of passes none of them either (5 rounds each, with questions for personal details). If a scorer cannot fail, a pass tells you nothing.