Today I read a piece arguing that neural networks are fundamentally the wrong technology for anything involving truth, knowledge, or safety-critical decisions.
Whether you fully agree or not, the examples should make every scientist and engineer pause:
-
Tiny, invisible perturbations to an image can flip a “panda” into a “gibbon” with high confidence.
-
Image generators fail at something as conceptually simple as an ouroboros: they remix snakes, circles and dragons, but consistently miss the underlying idea.
-
Self-driving systems misclassify an “edge case” truck and someone dies.
-
LLMs confidently hallucinate medical advice – and we call it “telehealth”.
In all these cases, the system is doing exactly what it was designed to do: predict the next pixel, the next token, the next label based on patterns in past data. Nowhere in that pipeline do we ever represent what is true about the world. There is no internal notion of “this is an ouroboros, because it embodies eternal cyclicality”, only “images like this often co-occur with these words”.
From a scientific perspective, this is a profound category error that we keep sweeping under the rug:
-
We treat stochastic pattern matching as if it were epistemic justification.
-
We conflate “works on the benchmark distribution” with “understands the phenomenon”.
-
We call everything slightly inconvenient an “edge case”, forgetting that the true input space is the entire messy real world.
As someone who spends their days with molecular simulations, HPC and visualization, this hits close to home. We already know in our own domain that:
-
Matching a single experiment does not validate a model.
-
Longer trajectories don’t magically fix bad assumptions.
-
Small error bars do not guarantee the absence of systematic bias.
Yet when it comes to neural networks, many are suddenly comfortable deploying them into medicine, policy, education, safety-critical systems – as if clever pattern matching were enough to replace concepts like mechanism, causality, or falsifiability.
To be clear: I use these tools. I find them impressive, fun, and often genuinely useful as assistants. But I don’t confuse:
-
“The model said it with confidence”
with -
“We have good reasons to believe this is true.”
For research and R&D, this has concrete implications:
-
We need workflows where humans remain the epistemic authority and models are clearly labelled as probabilistic advisors.
-
We need auditable pipelines, where data, assumptions and prompts are inspectable – not black boxes we blindly trust because they are convenient.
-
We need to separate “this helps me think, visualize and explore” from “this is an oracle that replaces my judgment”.
If we don’t, we risk sleepwalking into a culture where being statistically plausible is mistaken for being correct, in science as much as in society.
My personal take: neural networks are powerful instruments, but they are not a theory of knowledge. Treating them as such is like confusing a high‑resolution microscope with the laws of physics it helps us test.
I’m curious: in your own research or technical work, where do you draw the line between “useful pattern matcher” and “reliable source of truth”?