This article is part of a series in which I reflect on every chapter in the book The AI Con by Emily M. Bender and Alex Hanna.
"Very useful, no doubt, that was to Saruman; yet it seems that he was not content. Further and further abroad he gazed, until he cast his gaze upon the Barad-Dûr. Then he was caught!" (The Two Towers, p. 203). In The Lord of the Rings the wizard Saruman used a palantír, a Seeing Stone, to achieve his personal goals but by doing so lost his freedom. In themselves palantíri are not evil. They are tools to see into the realm to acquire information. However, once the Dark Lord Sauron got hold of one, they became his instrument of power and corrupted anyone who used them. Lord Denethor, Steward of the kingdom of Gondor was one of them: "The knowledge he obtained was, doubtless, often of the service to him; yet the vision of the great might of Mordor that was shown to him fed the despair of his heart until it overthrew his mind." (The Return of the King, p. 132). Although Generative AI is not exactly like a palantír, there are some troubling resemblances between the two: manipulation, power, and corruption under the guise of improvement.
AI is hype. This doesn't mean it is fake, useless, or a complete scam. It means Big Tech gives a false narrative about what Generative AI is and what it can do. Hype creates FOMO and drives investment. Tens of billions of dollars have already been invested in a promise that cannot be fulfilled in a foreseeable future. To keep investors satisfied and attract new ones, the fantasy needs to be sold ever more fervently. But at the end of the day, Artificial Intelligence is automation, not thinking. It is a smoke screen, pulled up in an attempt to obtain (political) power and unprecedented wealth. When the smoke has cleared and the mirrors wiped clean, how much harm will have been done to society?
With every new Large Language Model (LLM) we get illusive benchmark graphs on how much better that model performs compared to other models. Not only are these benchmarks questionable, new models hallucinate more, if they even get released. OpenAI's GPT-5 [Gobi], Meta's Llama 3.5 [Behemoth], and Anthropic's Claude 3.5 [Opus] all have been delayed. The reason: insufficient progress in their capabilities. Eventually they will be released, with amazing benchmarks, but once the party hats are taken off, the bunting is removed, and the cake has either been eaten or thrown away, the question remains, how much real progress have they made and at what cost?
LLMs are quite clunky and for this reason they cannot reach AGI (AI that thinks like a human) unless a major breakthrough will be discovered and significant changes will be made in their architecture. LLMs don't learn in real time. A model is trained, delivered, and used, but it can't learn from that usage. Actually, new models are trained from scratch, over and over again. Many LLMs now have memory and they can be used to locally and temporarily alter GenAI output for a specific user, but this information cannot used to update the model itself because of privacy issues, data quality, and data manipulation. A specific model is a finished product.
The well-known hallucinations are not a bug but an unintended feature for the LLM to speak coherently. LLMs produce bullshit as they are indifferent to truth. They add 'hallucinations' to their output to keep the flow of the conversation. Also, LLMs are trained on polluted data, which is data that contains inconsistencies, lies, and misconceptions, or they simply have too little information on a topic to provide truthful information. Finally, LLMs are prediction machines. They are supposed to guess the next word. Researchers at Anthropic have found out that the default response by Claude to questions about unknown information is “I don’t know”. However, if it does know something about the topic, it overrides the default response and the prediction game is triggered. As LLMs are trained on vast amounts of data, they probably override their default 'I don't know' response continuously. So, rather than stating that they don't know the answer, LLMs have the tendency to be helpful and provide the user with false information. You can mitigate hallucinations by using clear and specific prompts, but those can't make the hallucinations go away completely.
Although AI companies admit they don't know what's happening in the black box of LLMs, they can influence output. These can be guard rails to prevent abuse or malicious intent. But Grok, Elon Musk's AI on X, bluntly showed more nefarious use cases. On May 14 a modification was made to Grok that made the model add comments about supposedly white genocide in South-Africa, which is a myth, after every response. Interestingly enough, this was at the same time Donald Trump claimed Afrikaners were victims of genocide and had a publicity stunt to have them move over to America as refugees. The day after Grok went gobbledygook, it put the number of Jews killed in the Holocaust into question. It is unclear who made these changes to Grok, but that is irrelevant when we look at the implications. Although models themselves largely remain a mystery, outcomes can be manipulated, either by their owners, employees, or hackers and far more subtle and inconspicuously than what happened to Grok.
A different but perhaps a more troublesome example of Generative AI output manipulation is OpenAI's sycophant debacle. On April 25 OpenAI wanted to update the model's default personality which turned it into a sycophant that was excessively agreeable to its users. This resulted in unsettling conversations. Now whether this was a deliberate act gone wrong or a genuine mistake is not the issue. The fact is that Generative AI can be altered in such a way that it can influence personal opinions and thereby shape public debate. The more we become used to AI, the more we inadvertently trust AI output.
People also have the tendency to anthropomorphise AI, something a shocked Joseph Weizenbaum at MIT already discovered with ELIZA, one of the earliest 'chatbots', released in the 1960s. The Eliza effect is now a term used to indicate people attributing human traits to a computer program that communicates in text. Anthropomorphism and the sycophant nature of Generative AI (albeit less prevalent than the OpenAI's failed update) may have added to the fact that the main reason for people to use GenA is 'therapy' and 'companionship'; it is a dystopia in the making.
Generative AI is a powerful, manipulative tool that "connects a commercial goal with a popular fantasy of sentient machines." (The AI Con, p. 9) It rakes in enormous amounts of money and influence while promising bliss is just around the corner. Like Saruman and Denethor we are lured into the palantír. We think the stone can be of service to us, but actually it is us serving the dark lords of Silicon Valley.
However, AI is not inherently evil and can be used effectively as long as we acknowledge its flaws and dangers. We need to understand how these systems work, dispel the myth around them, and know how to approach them. We should be distrustful to the snake oil that is being sold to us for if we are not vigilant, we will be slowly but steadily be caught and corrupted.