Can AI Improve Public-Service Quality?

The problem, in numbers

When the quality of public guidance is measured, the results can be sobering. Between February and May 2025, Canada's Auditor General placed 167 test calls to the Canada Revenue Agency's contact centres. The auditors measured agents answering general questions about individual tax accurately only 17% of the time, and benefits and business tax questions accurately 56% and 54% of the time.^{Source 1} Only 9% of an agent's performance evaluation rested on the accuracy and completeness of answers, against 45% for keeping to schedule and call handling time.^{Source 1}

That is the floor this review is about. Our separate reviews cover speed, unit cost and eligibility decisions. Here we ask: does AI make the guidance people receive from government more accurate and more consistent? We found no published Australian evaluation of the accuracy of an AI assistant answering the public, so the evidence here comes from Canada, the UK and the US.

What improves, and why

In our reading, the case rests on one mechanism: a system drawing only on published guidance gives everyone the same source, at any hour, whoever picks up the phone. The Canadian audit shows the mechanism is not automatic. The auditors also put six tax questions to the agency's own chatbot, Charlie, which answered two accurately, while other publicly available AI tools answered five.^{Source 1} Six questions are far too few to rank them. According to a Canadian Press report, the agency told MPs it would bring in standardised training and automation to improve accuracy.^{Source 1}

Outside government, the large field study of AI help for customer support agents covered in our small business customer service review found the biggest gains among less experienced staff, a higher floor rather than a higher ceiling.

Plain-language letters are a second route. A Dutch workshop study tested ChatGPT on the six most complicated letters from a corpus of 200 government letters, combining automated readability metrics, a legal and linguistic expert review and a randomised test with 72 readers.^{Source 5} As a workshop paper, we cite it for that method, not as proof of effect. In our reading, its lesson is that a simpler letter needs an expert check for legal accuracy, not just a readability score.

What the evidence actually says

Grounded assistants can improve under sustained testing. The UK Government Digital Service's first GOV.UK Chat experiment, a private pilot to which 1,000 users were invited, did not reach the accuracy GDS required. It produced a few hallucinations, mostly for ambiguous or inappropriate questions, and some users dismissed the risk of error partly because they trusted the GOV.UK brand.^{Source 2} By March 2026, after two public pilots in which more than 10,000 users asked 26,000 questions, GDS reported self-assessed accuracy rising from 76% at its earliest benchmark to 90%.^{Source 3} These are the team's own assessments, made by subject experts and automated tools, and an answer counts as accurate only if it meets all the standards of published content.^{Source 3}

Untested assistants fail confidently. New York City's MyCity chatbot, built on Microsoft's Azure AI services, told business owners they could take a cut of workers' tips and that landlords need not accept housing vouchers. The Markup found it inconsistent as well as wrong: it told one reporter that landlords did have to accept vouchers, then told all ten staff who separately asked the same question that they did not.^{Source 4} In January 2026 the new mayor called it "functionally unusable", and the city has since taken it down.^{Source 4}

Edge cases are where the floor gives way. GOV.UK Chat declines questions outside published guidance by design and answers 88% of in-scope questions.^{Source 3} Some users wanted an adviser who could see their personal circumstances, which it cannot do. Longer term, GDS plans to hand users to departmental customer support within the same interface.^{Source 3} In our reading, the questions a general assistant cannot answer are often the ones that matter most to the person asking.

Where it falls short

• The best accuracy figure is self-assessed. The 90% GOV.UK Chat result is the team's own measurement.^{Source 3} The only independent test we found, the Canadian auditor's six questions to the tax agency's chatbot, is too small to settle anything.^{Source 1}

• An official badge invites over-trust. Some GOV.UK Chat pilot users dismissed the risk of inaccuracy partly because of the site's credibility.^{Source 2}

• Partial answers look complete. Inaccurate answers often covered some aspects of a question but not all.^{Source 3}

• A weak baseline is not a licence. In our reading, beating agents who were right 17% of the time on individual tax would clear a low bar, not a high one.^{Source 1}

• Cost per accurate answer is unknown. New York's mayor put the chatbot's cost at around half a million dollars when he moved to end it.^{Source 4}

Where governance decides the outcome

Australian rules so far focus on registering and assessing AI, not on the route to a person. In NSW, the AI Operational Policy requires agencies to appoint an accountable official, register every AI use case on the AI Assessment Framework Platform (required for all new registrations from 30 September 2026) and refer high-risk and critical-risk use cases to the AI Review Committee.^{Source 6} We found no requirement on the policy's page that members of the public be able to reach a person instead. In our reading, a public-facing assistant needs four things before launch: grounding in published guidance only, accuracy testing against a strict standard on real questions, a visible hand-off to a person for questions about individual circumstances, and an error log that feeds back into both the content and the system. Where an answer may shape a decision about entitlements, the safeguards in our benefits review apply.

How to think about adoption

Measure the current floor first: test-call your own contact centre, as the Canadian auditor did, before claiming AI improves on it. Ground the assistant only in guidance you publish and maintain, because it can be no more consistent than the content beneath it. Score answers against a strict standard, as GDS does, and publish the method. Ask the same question many times, from different people, because inconsistency is itself an error, as the New York test showed. Design the hand-off to a person for personal circumstances, complaints and anything outside published guidance. When simplifying letters, have a legal reviewer confirm the meaning has survived. And treat a pilot's own accuracy score as a reason to commission an independent test, not as the test.