Deloitte, a leading UK-based accounting firm, has agreed to partially refund the Australian Labor Department for inaccuracies in a government report it produced using AI. The report, which was initially published in July, contained multiple factual errors, prompting its replacement with a revised 237-page version. The errors included fabricated quotes and non-existent academic references, highlighting concerns about the reliability of AI-generated content in official documents.
The Department of Employment and Workplace Relations (DEWR) quietly replaced the original report, which was published in July, with a revised 237-page version last Friday, just ahead of a long weekend. Officials initially said the update added new information and corrected “some footnotes and references.”
Sydney University academic Chris Rudge had earlier flagged numerous apparent “hallucinations” typical of large language models, prompting Deloitte to launch an internal review in August.
The updated report includes a new disclosure confirming that AI – specifically Microsoft’s Azure OpenAI GPT-4o model – had been used in its preparation. It also corrects over a dozen errors, including references to a non-existent court ruling and academic papers, as well as a fabricated quote attributed to Justice Jennifer Davies (misspelled “Davis” in the first version), the deputy president of the Australian Competition Tribunal.
Rudge told the Financial Review that Deloitte’s admission transformed what was previously “a strong hypothesis” into certainty, even if its confession was “buried in the methodology section.”
A DEWR spokesperson confirmed that Deloitte had “agreed to repay the final installment under its contract,” though the amount was not disclosed. The full study on the computerized application of automated penalties in Australia’s welfare system cost 440,000 Australian dollars (about $290,000).
Rudge, a welfare expert, reportedly first noticed something was amiss when the report cited a book supposedly written by his Sydney University colleague Lisa Burton Crawford. The title seemed outside her field of expertise and turned out to not exist at all.