Part 1 covered the research; Part 2 and Part 3 covered the build — RAG core, frontend, MCP server. This post is different: no new code, just nsw-legal-research-assistant run live against a demo script modeled directly on the interview where Thomson Reuters' Valerie McConnell walked Legal IT Insider through CoCounsel's "Search a Database" skill — the McConnell interview documented back in Part 1's research. Same six beats, same idea — grounded answers, pinpoint citations, trend analysis across a whole matter, skill-chaining into a draft — run against this app instead, with real output, not illustrative text.

New since Part 3: Phase 6.5, and where "unverified" citations come from

Everything in Part 3 covered Phases 4 through 6. What's on screen in this post — named matters, the Review skill, trust-tier citations, skill-chaining drafting — is Phase 6.5, added afterward because single-question retrieval over one undifferentiated document pool genuinely can't reproduce the trend-analysis or multi-database parts of McConnell's demo. Closing that gap meant named collections, an aggregate Review skill, the not-found bridge, skill-chaining drafting, and a trust-tier system for lower-provenance sources — that last one is worth explaining properly, since it's the reason Beat 2 below shows a citation with a warning icon on it.

Two different Hugging Face datasets showed up while closing that gap, doing two different jobs. isaacus/open-australian-legal-qa — 2,124 synthetic question-answer pairs with citation metadata, generated by GPT-4 rather than human-written, CC BY 4.0 — got identified as a candidate for spot-checking citation-grounding accuracy against, in the same spirit as CoCounsel's team of licensed lawyers writing "law school type exams for a machine" from Part 1. It hasn't actually been wired into an automated eval run yet — filtering it to NSW entries and scripting a comparison is still just a reasonable next step, not something to claim credit for having done.

corto-ai/nsw-caselaw is a different animal: 27,453 scraped NSW Caselaw decisions, empty README, no stated license or permission from NSW Caselaw at all. Part 1 already covered why this app never scrapes NSW Caselaw itself — its robots.txt unconditionally disallows automated access to /search and /decision for every user agent, no exception for small or targeted requests. corto-ai/nsw-caselaw's own url field points straight at that disallowed /decision/... path on every row, about as direct a sign as you'll get that it was built by disregarding robots.txt rather than through anything negotiated.

So why is it in the app at all? Specifically because it's the wrong kind of source — pulled in on purpose, into a clearly separate, clearly labeled "NSW Caselaw Bulk Import (unverified)" collection, to give the trust-tier mechanism something real to demonstrate rather than only ever showing citations that already passed every check. Every document from it carries source_trust: "bulk_import" instead of "verified", and a fixed label — corto-ai/nsw-caselaw (Hugging Face) -- no stated license or NSW Caselaw permission, unverified provenance — gets attached to every citation it produces. The system prompt itself is told to hedge its language around unverified excerpts, not just the API response, and the collection is kept structurally isolated from every real matter so a scoped question can never silently draw on it.

There's a real, separate reason to believe permission is obtainable rather than just theoretical: a related Hugging Face dataset, the Open Australian Legal Corpus, states in its own acknowledgements that NSW Caselaw granted its author permission to scrape their data. That's concrete precedent, not a hopeful guess — enough to actually draft a written request modeled on it, addressed to NSW Caselaw's own listed contact, asking for the same kind of narrow, rate-limited, per-user-request access this app would actually use, and explicit that it's asking before building anything that would need it, not retroactively. That draft hasn't been sent yet. Until it has — and until it's actually granted — every document behind a real matter in this app got there the way Part 1 described: a human manually finding and downloading a specific decision through NSW Caselaw's own search, then uploading it by hand. corto-ai/nsw-caselaw stays exactly where Beat 2 shows it: isolated, labeled, and never trusted.

Beat 1 — Grounded Q&A, scoped to a matter

McConnell's demo queried named, scoped databases — a 200-contract set, a 1,200-contract set — rather than one undifferentiated pile. That's the direct reason this app has a Matter dropdown at all: asking a question with nothing selected searches every uploaded document; selecting Dog Attack Case Law scopes it to just the five documents filed under that matter.

> What did the court decide in Mason v Demasi regarding the appellant's dog bite claim?

Real output, this run:

> The primary judge awarded the appellant damages totalling $76,459.50 as follows: Non-economic loss $32,532.50 [1], Past out-of-pocket expenses $3,927.00 [1], Future out-of-pocket expenses $5,000.00 [1], Past economic loss $10,000.00 [1], and Future economic loss $25,000.00 [1]. However, the Court of Appeal set aside that earlier judgment and ordered a new trial limited to damages: Mason v Demasi [3] and [4].

>

> It is unclear from the provided excerpts what the final decision was regarding the appellant's dog bite claim.

Every dollar figure carries a bracket marker. That's the whole trust mechanism in one screen — click one.

Beat 2 — Pinpoint citations, and the state where they can't be trusted blindly

Click [1] and the PDF opens in a new tab, scrolled to the exact page the figure came from — the same "show its work" pattern McConnell described as CoCounsel's core trust mechanism, a hyperlink plus an excerpt backing every claim so a professional can verify without redoing the research. The citation metadata — case name, page number — is read directly off the retrieved database row, never generated by the model, so which excerpt gets cited is the LLM's call, but the citation itself can't be hallucinated.

Every citation in the run above is a verified source: a human found, downloaded, and uploaded the document, so its provenance is known — which is why those markers render as plain bold text with no warning. The trust mechanism has a second state — the corto-ai/nsw-caselaw bulk-import collection described above, kept apart from any real matter specifically so a scoped question can never silently draw on it:

The absence of a badge on a verified citation is the trust signal, working silently in the background. Worth showing once for completeness — not something to expect against a matter that only ever contains sources someone actually vetted.

Beat 3 — The "not found" bridge

Still scoped to the matter, ask something the five documents don't cover:

> What is the process for filing a trademark opposition in Australia?

Real output:

> I couldn't find anything relevant to this question in the uploaded documents. Try rephrasing, or upload a document that covers this topic.

>

> Unverified suggestions worth searching for manually (not citations — go find and upload if relevant):

> - Attorney-General (NSW) v Bryson [1897] 2 Ch 48

> - Trade Marks Act 1995 (Cth)

> - Australian Broadcasting Corporation v Lenah Game Meats Pty Ltd (2001) 208 CLR 199

> - Registrar of Trade Marks v Woolworths Limited [2013] FCAFC 42

The suggestions come from a separate, ungrounded LLM call and are labeled that way so they can never be mistaken for the citation-backed answer in Beat 1. That's a deliberate honest-dead-end design, not a missing feature — NSW Caselaw's reuse policy excludes automated crawling regardless of how targeted it is, so the bridge out is a human-in-the-loop suggestion, not a bot that goes and fetches the answer itself. The exact case names differ run to run, same as they will if you try this yourself — they're general-knowledge output, deliberately unverified, not a fixed list.

Beat 4 — Review: one row per document, not top-k across the pool

This is the piece plain question-answering architecturally can't do. McConnell's demo used "Search a Database" for trend and risk analysis across a whole repository — 38,000 SEC filings, a 1,200-contract set — which single-question retrieval can silently get wrong by only looking at the handful of chunks that score highest overall. The Review skill asks every document individually instead, so a document that's a poor match for one particular question still gets its own honest answer rather than being crowded out by the others.

Testing this beat earlier surfaced something worth reporting honestly rather than smoothing over: a specific figure from the Handbook, confirmed to be the top-ranked retrieved chunk for a damages question, wasn't showing up in the answer. Pulling a stronger local model to test the hypothesis gave the same result — which disproved the model-quality theory instead of confirming it. Re-reading the actual source text showed the Handbook states a severity percentage, not a dollar figure, so the original question was genuinely ambiguous against that document; a precisely-phrased question got the correct answer from both models, identically. That's a better trust signal than a smooth answer with no visible reasoning — the diagnostic mistake was in the question, not the tool, and the only way to know that was to go check.

Beat 5 — Skill-chaining: draft from an answer, no re-supply

McConnell's most complex demoed move: approve a database answer, then say "draft a letter from that," with no re-upload and no re-running the search. Same move here — click Draft from this… underneath the Beat 1 answer:

> Draft a short memo to a colleague summarizing what the research above establishes.

Real output:

> To: Colleague

>

> From: Researcher

>

> Subject: Mason v Demasi - Damages Awarded

>

> A review of the case law in Mason v Demasi ([1] NSWCA 210) reveals that this Court set aside an earlier judgment and ordered a new trial limited to damages. The primary judge awarded the appellant damages totaling $76,459.50, broken down as follows:

>

> * Non-economic loss: $32,532.50

> * Past out-of-pocket expenses: $3,927.00

> * Future out-of-pocket expenses: $5,000.00

> * Past economic loss: $10,000.00

> * Future economic loss: $25,000.00

>

> The Court noted concerns regarding the appellant's emotional condition and whether her evidence was false or exaggerated.

>

> Please note that this decision does not address the merits of the dog bite claim itself.

No new retrieval happens here — the exact answer and citations already on screen get passed straight back to a second LLM call, explicitly forbidden from introducing new facts or citations. It's deliberately stateless: the frontend, not the server, holds the "conversation" by re-sending what it already has, rather than a session table on the backend. That statelessness is also why the guardrail is worth testing honestly rather than cherry-picked: asking it to draft from an answer that hadn't actually established an outcome produced a correct refusal instead of a fabricated finding — a clean refusal on ungrounded ground is a stronger trust signal than a draft that always complies.

Where this actually runs

Every beat above ran against the full stack live on a Docker Compose setup on the M4 MacBook Air from Part 2 — Postgres/pgvector, the FastAPI backend, and the React frontend, all local, no cloud dependency for any of it:

Beat 6 — Beyond the original demo: MCP from Claude Code

Not part of McConnell's original demo at all — MCP wasn't a documented concept yet at the time of that interview. Part 3 already covered ask_nsw_caselaw, the MCP tool matching Beat 1's Q&A skill. Phase 6.5 shipped two more, mirroring the two skills it added: review_nsw_caselaw_collection for Beat 4, and draft_from_nsw_caselaw_answer for Beat 5. All three are thin wrappers calling the exact same functions the REST API and the frontend call — same skill, multiple interfaces, not separate implementations to keep in sync.

A real MCP client/server round trip — spawning app/mcp_server.py as an actual subprocess over stdio, the same transport Claude Code uses, not a direct Python call — against the live stack:

Same shape as the browser — the same mix of grounded and not-found rows across the five documents, the same reused-figures-only drafting behavior — because both interfaces are calling review_collection() and draft_from_answer() directly, not a second reimplementation of them. The exact wording differs from Beats 4 and 5's screenshots, same as it does between any two runs of this app; the guarantee is architectural, not word-for-word reproducibility.

Honest comparison to the original demo

What this app isn't trying to be: a reproduction of CoCounsel at enterprise scale. What it is: the same architectural pattern — curated, scoped RAG with a non-negotiable citation-grounding trust mechanism — proven against a real matter, with the actual gaps between "toy demo" and "useful tool" identified, closed where reasonable, and reported honestly where not.