A startup tested if ChatGPT and other AI chatbots could understand SEC filings. They failed about 70% of the time and only succeeded if told exactly where to look.

Financial companies’ hopes of using AI for analyzing SEC filings have been dampened due to poor performance of AI models from tech giants like OpenAI, Meta, and Anthropic. The study by Patronus AI indicates that these AI models are prone to providing incorrect answers or hallucinations when fed SEC document data. The financial services industry needs to refine AI testing methodologies and implement best practices to minimize errors. Even though some companies are developing AI tools for financial analysis, doubts remain over their security and accuracy, especially when dealing with crucial customer information. While AI performance improves when fed entire documents, difficulties arise with lengthy and complex financial texts.