Oxford summit rings the alarm on corporate AI

Meta Platforms

Mon 13:00

The author is a Reuters Breakingviews columnist. The opinions expressed are her own.
By Yawen Chen
LONDON, Oct 19 (Reuters Breakingviews) - Last week's AI selloff highlighted the financial strains facing the boom. Meanwhile at Oxford's Generative AI Summit, which on Thursday and Friday brought over 250 researchers, startups and investors to the ancient university town, attendees worried about other obstacles. Making AI agents safe for widespread use looks trickier than once assumed, according to fresh studies that were widely discussed at the conference. If true, businesses' fears of rogue bots will slow corporate adoption.
The stakes are high. Microsoft MSFT.O, Meta Platforms META.O, Oracle ORCL.N, Amazon.com AMZN.O and Alphabet GOOGL.O have over $1 trillion in future total lease commitments, largely for data centres. Justifying those sums depends on getting corporate clients to find profitable uses for AI agents, which handle tasks from coding to customer service without constant human oversight. Anthropic's forthcoming float essentially makes the same bet. The question, given added urgency by the OpenAI Hugging Face attack this year, is whether agents will always do what they're told. The disconcerting answer in Oxford: not necessarily.
Researchers including conference speaker Christian Schroeder de Witt studied situations where agents use other agents as tools to get jobs done. Even if each agent was individually protected against a certain type of risk, where malicious instructions are hidden in text, the group as a whole was still vulnerable. That's because, when collaborating, the agents lost track of whether data was trustworthy or not. The researchers developed a new safeguard that worked well, though it came with a performance cost.
Companies also need to know what their agents are doing. Attendees had more bad news on this front. Rebecca Finlay of nonprofit Partnership on AI discussed another study, which identified shortcomings in how four popular AI agent software systems recorded their activities. Only one of them made it clear when there has been a shift in the "approval model", which refers to whether human intervention is needed. The implication is that agents might start bypassing people without a company noticing. A bank using AI to process mortgage applications, for example, could suddenly find that loans had been moved forwards without a human check.
Despite the risks, corporate enthusiasm remains strong. Giles Pratt, who heads law firm Freshfields' global IP, Data and Tech practice, said that board directors are increasingly questioning who will take the fall for AI-led decisions that go wrong. On the other hand, he said that clients still want safeguards that let them proceed rather than pause rollouts.
Another theory is that competition can fix safety flaws. In other words, OpenAI, Anthropic, Meta and others have an incentive to solve these agent misuse problems at source, in order to win risk-averse corporate clients. But in private, AI safety executives argue that developers have less incentive to prevent issues that harms outsiders, likening AI safety to pollution: a market failure calling for government intervention.
Natalie Black of Ofcom, Britain's communications watchdog, was optimistic that regulation could work, citing international cybersecurity cooperation. Yet Huw Roberts of the Oxford Internet Institute questioned who could force powerful AI companies to open their systems to independent inspectors. Reuters reported that OpenAI and Anthropic had sought control over what outside evaluators could publish. Both firms say they support independent reviews while seeking to protect trade secrets.
The danger is that competitive pressure encourages deployment before safeguards catch up. In that case, problems like the Hugging Face attack may be a taste of what's to come.
     
Follow Yawen Chen on Bluesky and LinkedIn.
     
CONTEXT NEWS
The Oxford Generative AI Summit took place on October 15 and 16.


(Editing by Liam Proud; Production by Streisand Neto)
((For previous columns by the author, Reuters customers can click on CHEN/yawen.chen@thomsonreuters.com))