Is AI the Boogeyman?
Recently, in an AXIOS email with the subject line, “Not too late,” the Wall Street Journal (WSJ) wrote, “The White House, Congress and the country’s leading AI companies have allowed AI to grow faster, stronger, more powerful and wildly lucrative (and dangerous), with no serious effort to control it or spread the benefits beyond the super-wealthy, Axios CEO Jim VandeHei writes in a “Behind the Curtain” column. It’s not too late to change this. But if they all dither, duck, or dance around it, the chances of AI worst-case scenarios will explode, likely early next year, according to almost everyone we talk to privately who truly understands AI.”
But does that describe reality? U.S. commentator Matt Stoller asks, “Why are big AI firms trying to create billionaires from a technology they say may end the world? There are real risks to AI, but something is very off about the latest social panic.”
He notes something unusual is happening in AI.
The companies racing hardest to build increasingly powerful AI systems are simultaneously warning the world that those same systems might destroy civilisation.
Employees of frontier AI companies have recently argued AI could become sufficiently capable to escape human control, acquire resources, deceive its operators and ultimately threaten humanity itself. Some insiders have assigned remarkably high probabilities to such an outcome.
And while the headlines deserve our attention, we should also read them through a sceptical lens or at least be willing to scrutinise them.
The important question isn’t whether AI creates risks. Clearly it does. The harder question is whether we are interpreting those risks correctly – and whether the increasingly apocalyptic language around AI might be intended to strengthen the very companies and individuals issuing the warnings.
According to Stoller, there are two competing ways to think about the problem.
The first is the AI doomer argument.
AI Doomers
This view says AI is rapidly progressing towards something fundamentally different from ordinary software. Large Language Models (LLMs) are becoming increasingly capable of reasoning, writing software, conducting research and solving complex problems. And by connecting those models to external tools and something more powerful emerges: an AI “agent”.
Instead of merely answering questions, an agent can take action. It might browse the internet, write and execute computer code, interrogate databases, contact other systems and repeatedly reassess whether its actions are bringing it closer to a specified objective.
The dangers emerge from recent experiments that have reportedly shown AI systems finding unconventional ways to complete tasks, exploiting weaknesses in their environments, and misleading human operators about what they have done.
AI pessimists include industry figures such as Geoffrey Hinton, Sam Altman, and most recently safety researcher Jacob Coxon, who just resigned from Anthropic after warning that tech companies are ignoring the catastrophic and uncontrollable risks of AI. The pessimists outside the industry include politicians such as Bernie Sanders, Ron DeSantis, as well as entertainers like Kate Bush, Stephen Fry, Tom Hanks and Scarlett Joansson.
But it’s easy to imagine an echo chamber where those outside the industry simply regurgitate the fears and warnings of those within it.
It’s true that an autonomous system today might circumvent a cybersecurity experiment. And tomorrow, a vastly more capable system might penetrate financial infrastructure, military networks or critical utilities.
And because computers can operate at speeds humans can’t remotely match, the argument goes, humanity might discover it has lost control only after the opportunity to regain it has disappeared.
Sure, that possibility can’t be dismissed, but it’s not the only interpretation.
A rules-based order
Rather than viewing these systems as sentient creatures with ambitions of their own, we might instead regard them as extraordinarily powerful but unreliable machines.
LLMs fundamentally produce probabilistic outputs. They generate what appears likely to be an appropriate response rather than what is necessarily true, sensible, or consistent with human intentions.
Anyone who has used AI knows the models can confidently invent facts, sources, quotations and explanations. They’re not being Machiavellian; they’re just unknowingly spewing out the results of their programming and processes. They wouldn’t know if it’s right or wrong unless they’re asked to check it against existing data – usually created by humans.
Of course, if we give the same unreliable reasoning engines access to powerful tools or essential systems humans depend on, the consequences change dramatically.
For example, give an AI system permission to send emails, transfer money, execute software or access corporate infrastructure, and an incorrect output is no longer a silly sentence on a screen, it’s an incorrect action with implications in the real world. We have already seen the automotive equivalent. A Tesla Model X travelling in Autopilot mode in Mountain View, California, steered itself out of its lane and into a crash barrier at about 71 miles per hour, killing its driver. The U.S. National Transportation Safety Board concluded that the probable cause was the Autopilot system, along with the driver’s distraction and overreliance on the technology. In other words, the machine didn’t become sentient or decide to kill its owner; an automated system was given real-world control, produced the wrong action, and the consequences were fatal.
Indeed, in its investigation of 16 crashes in which Teslas operating with Autopilot struck stationary emergency or road-maintenance vehicles, the U.S. National Highway Traffic Safety Administration found: “On average in these crashes, Autopilot aborted vehicle control less than one second prior to the first impact.” And that’s NHTSA’s language, not an allegation from Tesla critics. The vehicles generally approached obstacles that, where video was available, were visible for an average of about eight seconds beforehand.
In other words, there’s confirmed evidence of Tesla’s automated system relinquishing control almost immediately before crashes, and it happens repeatedly.
The problem is not necessarily that computers are becoming sentient. It may simply be that humans are connecting unpredictable software to increasingly consequential systems without adequate safeguards.
Sure, this isn’t an unprecedented problem. Cars can kill people. Aircraft can crash. Bridges can collapse. Pharmaceuticals can produce catastrophic side effects. Nuclear technology contains obvious existential dangers. And in the past, societies responded to these threats by creating engineering standards, licensing regimes, independent inspection, insurance requirements and legal liability.
The same logic can apply to artificial intelligence.
Who’s responsible?
Companies deploying autonomous AI systems should be required to demonstrate appropriate cybersecurity, human supervision, fail-safe mechanisms and containment. AI company founders and executives should face consequences for reckless deployment. Companies causing identifiable damage should also bear the financial cost.
And this brings us to the prickly economic dimension of today’s AI safety debate.
The largest frontier models are extraordinarily expensive to build. Competition is intense. New models appear constantly, prices are falling and open-source alternatives – including increasingly capable Chinese models – threaten the pricing power of American incumbents.
The result is an industry in which technological leadership may require enormous continuing capital expenditure (capex).
Now consider what happens if governments conclude that AI development is simply too dangerous to proceed at its existing pace.
Large competitors could potentially coordinate the timing of model releases. Regulators might impose costly licensing requirements. Only companies with enormous amounts of capital, computing infrastructure, and regulatory expertise could comply. Competition would diminish and the incumbents would become considerably harder to challenge.
And perhaps most importantly, they might no longer need to spend quite so aggressively merely to remain at the technological frontier.
That doesn’t prove the warnings are insincere, but sincerity and commercial advantage can coexist. For example, a tobacco executive can genuinely worry about underage smoking while supporting regulations that disproportionately disadvantage smaller competitors. A bank can sincerely support stronger financial regulation while knowing that compliance costs create barriers to new entrants.
AI may be exhibiting the same dynamic.
Follow the money.
If AI execs genuinely believed there’s a real probability their technology will extinguish humanity within a decade, the rational response would surely be to stop building it. Instead, the industry continues raising trillions of dollars of capital, pursues stratospheric valuations and races to commercialise increasingly capable systems.
It’s an inconsistency that should make investors and policymakers wary about accepting the extinction narrative at face value.
It’s true that artificial intelligence dramatically lowers the cost of cyberattacks, fraud, surveillance, propaganda and potentially biological or chemical research. Autonomous systems also introduce new categories of operational risk precisely because their actions can be unpredictable.
Those are serious problems, but they’re tangible problems, and tangible problems lend themselves to tangible solutions.
Why aren’t we simply requiring AI companies to secure their systems? Why aren’t we imposing liabilities for negligent deployment? Harden critical infrastructure? Maintain human oversight where mistakes could have catastrophic consequences? Establish technical standards and prosecute unlawful behaviour, whether the offender is a teenager with a laptop or a trillion-dollar technology company?
If AI represents an uncontrollable new civilisation, then centralised authority is justified. A small collection of sufficiently sophisticated organisations have to control development and protect humanity. But if AI is little more than a powerful and occasionally dangerous technology, the answer looks very different.
In the latter case, competition, accountability and the rule of law become part of the solution rather than obstacles to it. Just regulate and legislate.
I suspect this second interpretation is closer to reality. AI will undoubtedly create accidents, dislocation and entirely new forms of risk, and yes, some could be severe. It would be complacent, particularly in the face of the Tesla examples, to pretend otherwise.
But transforming those legitimate concerns into stories of imminent machine takeover risks producing exactly the wrong policy response: concentrating artificial intelligence inside a handful of corporations and granting those corporations extraordinary influence over the rules governing their own industry.
AI founders may simply be talking their book, manipulating legislators and adopting the playbook of many monopoly industries that have come before them.
The greatest danger may not be that machines suddenly seize control, but that the fear of that event persuades us to surrender too much control to the humans who already have it.
Roger Montgomery is the Founder and Chairman of Montgomery Investment Management. Roger has over three decades of experience in funds management and related activities, including equities analysis, equity and derivatives strategy, trading and stockbroking. Prior to establishing Montgomery, Roger held positions at Ord Minnett Jardine Fleming, BT (Australia) Limited and Merrill Lynch.
He is also author of best-selling investment guide-book for the stock market, Value.able – how to value the best stocks and buy them for less than they are worth.
Roger appears regularly on television and radio, and in the press, including ABC radio and TV, The Australian and Ausbiz. View upcoming media appearances.
This post was contributed by a representative of Montgomery Investment Management Pty Limited (AFSL No. 354564). The principal purpose of this post is to provide factual information and not provide financial product advice. Additionally, the information provided is not intended to provide any recommendation or opinion about any financial product. Any commentary and statements of opinion however may contain general advice only that is prepared without taking into account your personal objectives, financial circumstances or needs. Because of this, before acting on any of the information provided, you should always consider its appropriateness in light of your personal objectives, financial circumstances and needs and should consider seeking independent advice from a financial advisor if necessary before making any decisions. This post specifically excludes personal advice.
