Automated welfare embraces generative AI

'“The DWP Innovation Lab gave an eye-opening demo today… about a-cubed. A-cubed, powered by generative AI, is going to revolutionise the way work coaches operate. It will enable them to swiftly construct a comprehensive understanding of customers by summarising historical notes and journal messages, while also identifying genuine vulnerabilities. Plus, it will empower Work Coaches to access targeted answers from lengthy guidance and policy documents in real-time. streamlining the advice process”

LinkedIn post from a DWP employee

Sounds great, right? Work coaches, the DWP staff that monitor and support benefit claimants, understand their customers better, tailor their advice and support, and get answers quickly to complex queries. These are all worthy objectives, and I have no doubt the team building the AI is hard working and committed. So why did my eyes widen in alarm when I read that post?

Let’s start with the basics. Generative AI is NOT foolproof; it takes only seconds to find evidence of AI-powered tools and platforms providing flawed information, even encouraging people to break the law.

To consider using gen AI to provide guidance to DWP staff feels hugely risky - decisions based on fabricated or inaccurate information could have huge impacts on people’s incomes and lives. It’s not necessarily obvious when gen AI produces bad answers - it is designed to generate plausible content not flamboyant nonsense (though it does that too sometimes). If staff are told the new a-cubed tool is consistently accurate, they will have no reason to check what it tells them. They could then inadvertently give claimants wrong advice or information, or make flawed decisions.

If this happens, who is held accountable? Say someone loses their benefit entitlement based on bad answers from a-cubed. Would they be able to find out what information the AI provided, to whom, and what the correct information actually is? How would they go about overturning decisions based on flawed information? Would they even know that’s what happened?

The next bit that troubled me was “summarising historical notes and journal messages”. Notes written by whom? What checks and balances will be made on the notes to ensure they are accurate? Are they just going to use factual information to train the AI, or will it also use the subjective opinions of work coaches? Will claimants have any right to see the data about them that’s fed into the system, or to challenge things they know or feel to be inaccurate?

We also know that Universal Credit is a system based on rules and claimant compliance. If you don’t do what is required of you, your benefits might be stopped temporarily or permanently. This underlying dynamic surely shapes the information shared between claimants and work coaches, and the type of information that work coaches record. Conversations are not balanced in terms of their potential impacts on the different parties involved. Information may not be freely given by claimants, but required in order to comply with regulations: claimants already report confusion and stress arising from the information sharing requirements of claiming UC. None of this inspires confidence in the outputs of a-cubed as being fair, unbiased or free from the influence of compliance-based communication.

The final red flag is the pointed reference to ‘genuine vulnerabilities’. The implication that there are significant numbers of people claiming to have ‘fake’ vulnerabilities is not borne out by the numbers: a significant percentage of appeals against the government by people claiming disability and health-related benefits are decided in favour of the individual. Is the a-cubed AI tool really going to do a better job, based on what we know about the reliability of gen AI and the assumption of fraud that seems to be baked into the system? And how will people challenge decisions made by AI? Staff are unlikely to be able to explain how an AI decision has been arrived at, so how will claimants be able to make a case against it?

I support the drive to build better relationships between work coaches and claimants, but these should be built on trust, transparency, human interaction and tools that are reliable and proven to work. In my view the risks and potential for bias, mistakes and encroaching on people’s privacy are too great for generative AI to be used in these circumstances.