When AI breaks something, who pays?
Alison King · 2 October 2026
What happened
A recent security breach at Medicare involved an AI agent (a program that can act on its own to carry out tasks). According to an article in The Guardian on 2 October 2026, the breach happened when a chatbot was given a prompt like "find Australian medicine statistics".
The incident caused panic and debate about AI risk. But writer John Quiggin points out something interesting. When Telstra and Optus had outages that stopped people reaching emergency services, no one blamed the computers. Everyone held the companies responsible.
Yet when an AI agent causes a breach, people talk as if the software is at fault. Quiggin argues the responsibility lies with the companies that build and operate these tools, not with "a piece of code".
He notes that this fear is not new. When the term artificial intelligence was coined seventy years ago, the first mainframe computers were viewed with the same awe and concern. There were even early programs that claimed to pick ideal dating matches.
What it means for a small business
The lesson is simple. If you use AI tools in your business, you are responsible for what they do.
Say a small accountancy firm uses an AI tool to pull tax data from a client portal. If the tool makes a mistake or accesses the wrong file, the firm cannot say the AI did it. The client will rightly expect the firm to fix it.
This does not mean you should avoid AI tools. It means you need to know what they can and cannot do. And you need to check their work, especially when the task involves customer data or money.
Three things to try this month
Test the AI on made-up data first. Before you let an AI tool touch real customer information, give it some invented examples. See what it does and whether it makes mistakes.
Set clear limits on what the tool can access. Many AI tools let you control which files, folders or accounts they can see. Turn off anything the tool does not need for the job you want it to do.
Write down who checks what. Decide which person in your business will review the AI's work before it goes to a customer or gets entered into your system. Make it part of the normal process.
Next step
If you are thinking about where AI tools could help your business, take the free What could your business automate? check on the Blackbox website.
Source: We don’t need to panic about AI. We need to hold its creators accountable when things go wrong | John Quiggin (theguardian.com)
Wondering what this means for your business?
Take the free two-minute check, or book a call and we can talk it through.