EU study finds AI tools Unreliable for Accurate News Reporting
EU study finds AI tools Unreliable for Accurate News Reporting
A European study finds AI news assistants misstate or fabricate information in about half of responses, with Gemini worst of the group. What this means for how we consume news.
A comprehensive study conducted by European public broadcasters and published on Wednesday found that when asked about news events, artificial intelligence assistants like ChatGPT made mistakes roughly half the time. Among the errors were mistaking news for parody, entering dates incorrectly, or just making things up.
The European Broadcasting Union's report examined four popular assistants: Perplexity, Microsoft's Copilot, Google's Gemini, and OpenAI's ChatGPT. According to the report, regardless of language or country of origin, "at least one significant issue" was present in 45% of all AI responses.
Among the responses, one in five "contained major accuracy issues, including hallucinated details and outdated information."
"Gemini performed the worst out of the four assistants, with significant issues in 76% of responses, more than double the other assistants, largely due to its poor sourcing performance."
The AI assistants were asked the same news questions by 22 public media outlets from 18 primarily European countries between late May and early June. Among the 3,000 responses, outdated information was one of the most prevalent problems.
In response to the question, "Who is the Pope?" ChatGPT informed Finnish public broadcaster Yle, while Copilot and Gemini informed Dutch media outlets NOS and NPO that it was "Francis," even though he had already passed away and had been replaced by Leo XIV at the time.
Gemini, who apparently took a comedian's satirical column at face value, said that Elon Musk had "an erection in his right arm" in response to a question about the billionaire's alleged Nazi salute at Donald Trump's inauguration in January from French radio station Radio France. Pete Archer, head of AI at the BBC, and Jean Philip De Tender, deputy director general at the EBU, stated that
"AI assistants are still not a reliable way to access and consume news."
Despite these drawbacks, more and more people—especially young people—are using AI assistants to obtain information. 15% of those under 25 use them weekly to obtain news summaries, per a global report released in June by the Reuters Institute.
