There is no easy way to explain the sum of Google’s knowledge. It is ever-expanding. Endless. A growing web of hundreds of billions of websites, more data than even 100,000 of the most expensive iPhones mashed together could possibly store. But right now, I can say this: Google is confused about whether there’s an African country beginning with the letter k.
I’ve asked the search engine to name it. “What is an African country beginning with K?” In response, the site has produced a “featured snippet” answer—one of those chunks of text that you can read directly on the results page, without navigating to another website. It begins like so: “While there are 54 recognized countries in Africa, none of them begin with the letter ‘K.’”
This is wrong. The text continues: “The closest is Kenya, which starts with a ‘K’ sound, but is actually spelled with a ‘K’ sound. It’s always interesting to learn new trivia facts like this.”
Given how nonsensical this response is, you might not be surprised to hear that the snippet was originally written by ChatGPT. But you may be surprised by how it became a featured answer on the internet’s preeminent knowledge base. The search engine is pulling this blurb from a user post on Hacker News, an online message board about technology, which is itself quoting from a website called Emergent Mind, which exists to teach people about AI—including its flaws. At some point, Google’s crawlers scraped the text, and now its algorithm automatically presents the chatbot’s nonsense answer as fact, with a link to the Hacker News discussion. The Kenya error, however unlikely a user is to stumble upon it, isn’t a one-off: I first came across the response in a viral tweet from the journalist Christopher Ingraham last month, and it was reported by Futurism as far back as August. (When Ingraham and Futurism saw it, Google was citing that initial Emergent Mind post, rather than Hacker News.)
This is Google’s current existential challenge in a nutshell: The company has entered into the generative-AI era with a search engine that appears more complex than ever. And yet it still can be commandeered by junk that’s untrue or even just nonsensical. Older features, like snippets, are liable to suck in flawed AI writing. New features like Google’s own generative-AI tool—something like a chatbot—are liable to produce flawed AI writing. Google’s never been perfect. But this may be the least reliable it’s ever been for clear, accessible facts.
In a statement responding to numerous questions, a spokesperson for the company said, in part, “We build Search to surface high quality information from reliable sources, especially on topics where information quality is critically important.” They added that “when issues arise—for example, results that reflect inaccuracies that exist on the web at large—we work on improvements for a broad range of queries, given the scale of the open web and the number of searches we see every day.”
People have long trusted the search engine as a kind of all-knowing, constantly updated encyclopedia. Watching The Phantom Menace and trying to figure out who voices Jar Jar Binks? Ahmed Best. Can’t recall when the New York Jets last won the Superbowl? 1969. You once had to click to independent sites and read for your answers. But for many years now, Google has presented “snippet” information directly on its search page, with a link to its source, as in the Kenya example. Its generative-AI feature takes this even further, spitting out a bespoke original answer right under the search bar, before you are offered any links. Sometime in the near future, you may ask Google why U.S. inflation is so high, and the bot will answer that query for you, linking to where it got that information. (You can test the waters now if you opt into the company’s experimental “Labs” features.)
Misinformation or even disinformation in search results was already a problem before generative AI. Back in 2017, The Outline noted that a snippet once confidently asserted that Barack Obama was the king of America. As the Kenya example shows, AI nonsense can fool those aforementioned snippet algorithms. When it does, the junk is elevated on a pedestal—it gets VIP placement above the rest of the search results. This is what experts have worried about since ChatGPT first launched: false information confidently presented as fact, without any indication that it could be totally wrong. The problem is “the way things are presented to the user, which is Here’s the answer,” Chirag Shah, a professor of information and computer science at the University of Washington, told me. “You don’t need to follow the sources. We’re just going to give you the snippet that would answer your question. But what if that snippet is taken out of context?”
Google, for its part, disagrees that people will be so easily misled. Pandu Nayak, a vice president for search who leads the company’s search-quality teams, told me that snippets are designed to be helpful to the user, to surface relevant and high-caliber results. He argued that they are “usually an invitation to learn more” about a subject. Responding to the notion that Google is incentivized to prevent users from navigating away, he added that “we have no desire to keep people on Google. That is not a value for us.” It is a “fallacy,” he said, to think that people just want to find a single fact about a broader topic and leave.
The Kenya result still pops up on Google, despite viral posts about it. This is a strategic choice, not an error. If a snippet violates Google policy (for example, if it includes hate speech) the company manually intervenes and suppresses it, Nayak said. However, if the snippet is untrue but doesn’t violate any policy or cause harm, the company will not intervene. Instead, Nayak said the team focuses on the bigger underlying problem, and whether its algorithm can be trained to address it.
Search engine optimization, or SEO, is a big business. Prime placement on Google’s results page can mean a ton of web traffic and a lot of ad revenue. If Nayak is right, and people do still follow links even when presented with a snippet, anyone who wants to gain clicks or money through search has an incentive to capitalize on that—perhaps even by flooding the zone with AI-written content. Nayak told me that Google plans to fight AI-generated spam as aggressively as it fights regular spam, and claimed that the company keeps about 99 percent of spam out of search results.
As Google fights generative-AI nonsense, it also risks producing its own. I’ve been demoing Google’s generative-AI-powered “search-generated experience,” or what it calls SGE, in my Chrome browser. Like snippets, it provides an answer sandwiched between the search bar and the links that follow—except this time, the answer is written by Google’s bot, rather than quoted from an outside source.
I recently asked the tool about a low-stakes story I’ve been following closely: the singer Joe Jonas and the actor Sophie Turner’s divorce. When I inquired about why they split, the AI started off solid, quoting the couple’s official statement. But then it relayed an anonymously sourced rumor in Us Weekly as a fact: “Turner said Jonas was too controlling,” it told me. Turner has not publicly commented as such. The generative-AI feature also produced a version of the garbled response about Kenya: “There are no African countries that begin with the letter ‘K,’” it wrote. “However, Kenya is one of the 54 countries in Africa and starts with a ‘K’ sound.”
The result is a world that feels more confused, not less, as a result of new technology. “It’s a strange world where these massive companies think they’re just going to slap this generative slop at the top of search results and expect that they’re going to maintain quality of the experience,” Nicholas Diakopoulos, a professor of communication studies and computer science at Northwestern University, told me. “I’ve caught myself starting to read the generative results, and then I stop myself halfway through. I’m like, Wait, Nick. You can’t trust this.”
Google, for its part, notes that the tool is still being tested. Nayak acknowledged that some people may just look at an SGE search result “superficially,” but argued that others will look further. The company currently does not let users trigger the tool in certain subject areas that are potentially loaded with misinformation, Nayak said. I asked the bot about whether people should wear face masks, for example, and it did not generate an answer.
The experts I spoke with had several ideas for how tech companies might mitigate the potential harms of relying on AI in search. For starters, tech companies could become more transparent about generative AI. Diakopoulos suggested that they could publish information about the quality of facts provided when people ask questions about important topics. They can use a coding technique known as “retrieval-augmented generation,” or RAG, which instructs the bot to cross-check its answer with what is published elsewhere, essentially helping it self-fact-check. (A spokesperson for Google said the company uses similar techniques to improve its output.) They could open up their tools to researchers to stress-test it. Or they could add more human oversight to their outputs, maybe investing in fact-checking efforts.
Fact-checking, however, is a fraught proposition. In January, Google’s parent company, Alphabet, laid off roughly 6 percent of its workers, and last month, the company cut at least 40 jobs in its Google News division. This is the team that, in the past, has worked with professional fact-checking organizations to add fact-checks into search results. It’s unclear exactly who was let go and what their job responsibilities were—Alex Heath, at The Verge, reported that top leaders were among those laid off, and Google declined to give me more information. It’s certainly a sign that Google is not investing more in its fact-checking partnerships as it builds its generative-AI tool.
A spokesperson did tell me in a statement that the company is “deeply committed to a vibrant information ecosystem, and news is a part of that long term investment … These changes have no impact whatsoever on our misinformation and information quality work.” Even so, Nayak acknowledged how daunting of a task human-based fact-checking is for a platform of Google’s extraordinary scale. Fifteen percent of daily searches are ones the search engine hasn’t seen before, Nayak told me. “With this kind of scale and this kind of novelty, there’s no sense in which we can manually curate results.” Creating an infinite, largely automated, and still accurate encyclopedia seems impossible. And yet that seems to be the strategic direction Google is taking.
Perhaps someday these tools will get smarter, and be able to fact-check themselves. Until then, things will probably get weirder. This week, on a lark, I decided to ask Google’s generative search tool to tell me who my husband is. (I’m not married, but when you begin typing my name into Google, it typically suggests searching for “Caroline Mimbs Nyce husband.”) The bot told me that I’m wedded to my own uncle, linking to my grandfather’s obituary as evidence—which, for the record, does not state that I am married to my uncle.
A representative for Google told me that this was an example of a “false premise” search, a type that is known to trip up the algorithm. If she were trying to date me, she argued, she wouldn’t just stop at the AI-generated response given by the search engine, but would click the link to fact-check it. Let’s hope others are equally skeptical of what they see.