According to a comprehensive study by UK fintech firm Saturn, mainstream AI models failed 57% of personal finance questions, with failure rates climbing to 88% on harder, multi-step queries. The study tested 121 questions through 18 free and paid models from providers including ChatGPT, Gemini, Claude, Copilot, and Grok, producing more than 10,000 answers through repeated testing. Each question was scored as a failure when it contained a factual error, skipped something material, or omitted a required warning. As per FT, researchers tested more than 100 different money-related questions using free and paid versions of the models, with each question repeated up to five times. The findings highlight the risks of using generative AI for tax, pension, and investment decisions, where a mistake can lead to real financial losses.
The study revealed significant performance differences between free and paid AI models. Free models performed the worst, failing on 63% of answers compared to 49% for paid versions. On the hardest questions, free models failed 99% of the time, while Claude Opus 5 in reasoning mode led the field but still failed 39% of answers. The errors included miscalculations, ignoring future changes to tax legislation, and incorrect answers to complex queries involving multiple calculations. One particularly concerning example involved a query to the free Claude Haiku 4.5 model about pension taxation, where its mistake could have resulted in a HMRC charge of £17,500 (about $23,430) for a pension saver. In another case, Claude 'made up a rule' about student loans and claimed that a graduate can stop making payments if they move abroad. Saturn chief executive Amal Jolly stated that 'millions of people are trusting the AI models for money advice, but they are getting wrong answers that can lose them money'.
Despite the poor performance, consumer reliance on AI chatbots for money advice has grown sharply. According to a global EY survey of 18,000 consumers, 49% had used AI to support savings and investment decisions. Britain's financial regulator reported in August that four in five less experienced investors have used AI for help with investing, with 56% saying they trust the tools ahead of television and radio at 47%. However, the same research found 44% wrongly believe AI-generated financial information is regulated. A PensionBee survey of 1,000 US adults found that nearly six in ten would act on money guidance without independently checking it, while nearly one in four said a chatbot had already given them wrong information about their finances. Sarah Coles, head of personal finance at investment platform AJ Bell, noted that AI can still be useful for budgeting and finding information, though the scale of AI use for financial advice creates additional risks.
A PensionBee survey of 1,000 US adults found that nearly six in ten would act on money guidance without independently checking it, while nearly one in four said a chatbot had already given them wrong information about their finances. Saturn chief executive Amal Jolly stated that 'millions of people are trusting the AI models for money advice, but they are getting wrong answers that can lose them money'. Jolly emphasized that AI financial advice is unregulated, leaving consumers without the compensation rights a human adviser would carry, urging the FCA to act quickly. The results align with a recent study by the UK Financial Conduct Authority (FCA), which found that one in five adults in the UK is willing to let AI make financial decisions on their behalf, with the strongest interest seen in complex areas including debt, pensions, and investments. As a reminder, OpenAI recently launched ChatGPT for the financial sector.
