US Military Awards Multi-Million Dollar Contract to Scale AI for AI-Driven Defense Initiative - Major Digest

The Rise of Military-Grade AI: A New Frontline in the AI Arms Race For the first two-and-a-half years of the generative AI revolution, the AI arms race has been waged between competing companies seeking to make bank from the promise and potential of the technology. But things are maturing in the AI world—and with it, there’s another frontline for AI: the military. Scale AI, the company set up by Alexandr Wang, has been awarded what CNBC reports is a multimillion-dollar deal to help develop Thunderforge, which the U.S. Department of Defense calls “an initiative designed to integrate artificial intelligence into military operational and theater-level planning, and fusing cutting-edge modeling and simulation tools.”

The AI Arms Race Goes Military

The move is unsurprising—militaries are always keen on keeping at the cutting edge of technology, trying to eke out an advantage against competitive armies—but disappointing, says Margaret Mitchell, researcher and chief ethics scientist at Hugging Face, an AI company. “We already know we’re moving forward to push AI systems farther and farther out from our control,” she says. “Many in the industry and in the media are treating more and more powerful systems as if they are inevitable, and therefore making it so.” Technology has always relied on military clientele to act as a crucible for, and accelerant of, new innovation. “Military use has long been a staple of technological development,” she says. “Massively destructive outcomes are fully predictable based on history and how the tech marketplace works.” (Scale AI declined to comment. The Defense Department did not respond to Fast Company’s request for comment.)

Existential Risks and International Collaboration

That level of destruction could be catastrophic, argues David Krueger, assistant professor at the University of Montreal, studying AI safety and risk. “I think it’s likely to lead to the end of humanity, to human extinction,” he says, speaking generally about the use of AI for military purposes, calling the military use of AI “one of the most obvious ways in which AI poses an existential risk to humanity.” Krueger says that AI is being used in many areas to hand off human control and outsource it to AI systems. “I think this is a risk in every domain, and I think in the military, it’s particularly concerning, and something which will require international collaboration to avoid getting out of hand and risking human extinction.”

Human Oversight and Enterprise Functions

Scale AI has said that the Thunderforge program will operate with human oversight, and Noah Sylvia, a research analyst at the Royal United Services Institute (RUSI), points out that “as AI functions go, I would say it is not as controversial as a lot of other ones, because this is what you could term an enterprise function.” Indeed, the press release by the Defense Innovation Unit announcing Scale AI’s deal for Thunderforge points out the same program will also include Anduril’s Lattice software platform and state-of-the-art Large Language Models (LLMs) enabled by Microsoft.

A Way Out?

“I struggle to see a way out of it,” says Hugging Face’s Mitchell. Even if individual countries or companies were to decide to step aside from using AI for military purposes, or to decline to provide support to countries that are seeking military AI—as Hugging Face has refused to do in the past—others would likely step into the breach.

Cross-Country Guidelines and Safety Mechanisms

Developing cross-country guidelines for how to consider the use of AI in military environments will be vital in the future, says Mitchell. She suggests a multipoint plan that includes keeping AI systems within strict operational boundaries, making it impossible for systems to autonomously deploy weapons, introducing safety mechanisms, and advancing what’s deemed state-of-the-art in input data analysis and output evaluation to gain a deeper understanding into what systems can and cannot do. She also has two simpler suggestions. “Do not deploy technology whose actions you cannot reasonably foresee,” she says. And secondly: “Do not fully cede human control.” **