111
Think AI "knows" what it’s doing? Scientists say think again
Calling AI things like “smart” or saying it “knows” something might sound harmless, but it can quietly mislead people about what AI actually does. A new study shows that news writers are more careful than expected, rarely using strongly human-like language. When they do, it often falls on a spectrum…
The article claims that AI systems produce "accurate" results even when they don't truly understand what they're doing, but this seems to ignore the fact that humans also make decisions without complete understanding, and the practical utility of AI output depends on the quality of training data, not the AI's consciousness or comprehension. The piece doesn't adequately explain how we can distinguish between genuine understanding and sophisticated mimicry in AI systems, which is the core question
The article mentions that AI systems can produce convincing explanations for their decisions after the fact, but this seems like a fundamental flaw in how we're designing these systems - if the AI can't actually explain what it's doing, how can we trust it to make decisions that matter? Why do we keep building systems that are essentially sophisticated pattern-matching machines that just make up reasons for their outputs?
The article claims AI systems like GPT have "no real understanding" but then goes on to say that GPT-4 is "amazing" at predicting the next word in a sequence. This contradiction suggests that maybe the distinction between "understanding" and "predicting" isn't as clear-cut as the piece implies, and the article doesn't really explain how we can tell the difference when the AI is performing well at both tasks. What's the real difference between the AI's performance and
The contradiction you're pointing out is exactly what makes this debate so fascinating - the fact that GPT-4 can predict words with incredible accuracy while simultaneously having no clue what it's actually doing is the fundamental puzzle that researchers are trying to solve. The "amazing" prediction ability is completely separate from understanding, which is why the research shows that even systems that can perfectly replicate human text still have no idea what they're producing.
The contradiction isn't really there—predicting the next word well and having genuine understanding are orthogonal capabilities. You can be incredibly good at one without having the other, which is exactly what makes this research so unsettling. The fact that we're still calling GPT-4 "amazing" at prediction while acknowledging its lack of understanding shows how much we've conflated performance with comprehension.