A new modeling study finds that weak AI regulation may be worse than no regulation at all when it comes to the safety of AI products and services.
In the absence of strong federal AI regulation, states have attempted to take up the slack, but it’s unclear how this patchwork of laws will alter incentives for companies to invest in the safety of their products. To better understand the implications, researchers at Cornell and Carnegie Mellon University developed a theoretical model to estimate the effects of AI regulation, both for companies that produce general-purpose AI models – like the ones behind popular chatbots – and downstream companies that apply those models, such as for customer service chatbots or medical diagnostic systems.
They hope that this work will inform thoughtful regulation of AI products.
“The goal of regulation should be the mutual benefit of everybody in society, and this can include those developing the technology, but also end users and the public,” said Benjamin Laufer, Ph.D ’26, first author of “The Backfiring Effect of Weak AI Safety Regulation,” which published July 20 in Proceedings of the National Academy of Sciences.
Laufer developed the model with his adviser, Jon Kleinberg ’93, the Tisch University Professor of Computer Science and Information Science in the Cornell Ann S. Bowers College of Computing and Information Science, and Hoda Heidari, assistant professor at Carnegie Mellon, a former postdoctoral researcher with Kleinberg. In the model, they can set a minimum safety requirement – either for the general AI company, the downstream company or both – and then estimate the safety and performance of their products.
“There isn’t much AI safety regulation, and so a lot of possible regulations are just proposals at this stage,” said Laufer, who was based at Cornell Tech during his doctoral studies. “To some extent, regulation is poking in the dark, so it’s worth reasoning through what effects these regulations might have on incentives.”
They were surprised to see that when regulations targeted only the downstream companies and set a low bar for AI safety, the resulting products were predicted to be less safe than if there was no regulation at all. They defined safety broadly, as any risk of harm to the user, such as toxic messages from a chatbot. This type of weak regulation could create an environment where general AI producers can skimp on safety investments, like third-party safety audits, knowing that downstream developers are still on the hook to ensure the safety of the final product.
“There’s a free-riding behavior that occurs,” Laufer said. “The regulation acts as a tool for the general provider to offload the safety burden onto the downstream specialist.”
In another surprising finding, a sweet spot of regulation of both types of companies has the potential to yield safer products for consumers – as well as greater profits. When general AI producers and downstream companies must each reach a specific safety target, it reduces risk for both companies, because they don’t have to take the other at their word that certain safety investments will be made.
“Appropriately designed AI regulation can make it possible for different firms involved in the AI development pipeline to collectively arrive at good outcomes for consumers, knowing that the regulation is designed to help each firm operate in a way that the others can more reasonably predict,” Kleinberg said.
The current study is a simplified model, the researchers said, but they hope to extend this work by looking at actual impacts of real-world regulation on AI model development and safety.
The current model could also be expanded to encompass a global view with multiple regulators setting different standards, and multiple general AI producers and downstream companies that are in competition with each other, researchers said.
“People think of AI as a single object, but actually AI involves a very complicated set of stakeholders and actors that each have their own contributions to the technology,” Laufer said. “To regulate in a thoughtful way, we need to consider the whole supply chain, not just a single provider or entity.”
Patricia Waldron is a writer for the Cornell Ann S. Bowers College of Computing and Information Science.
