Amazon partnership establishes Cornell AI security initiative

Artificial intelligence agents – also known as agentic AI – can now build and launch software based on a few prompts. But AI agents can produce incorrect, misleading, and even malicious code that hackers can exploit.

Thanks to a gift from Amazon, a pair of computer scientists from the Cornell Ann S. Bowers College of Computing and Information Science and Cornell Tech will lead the development of safety protocols to shore up AI agents and the code they produce. The gift also establishes a new Cornell-based initiative, AI4AI, that brings together university leaders in machine learning, security, formal methods and verification to improve agentic AI.

Alexandra Silva, professor of computer science at Cornell Bowers, and Vitaly Shmatikov, professor of computer science at Cornell Tech and Cornell Bowers, are the principal investigators behind the project, “Assured Integrity for AI-Based Software.” Among participating Cornell faculty are: Saikat Dutta, assistant professor of computer science; Kevin Ellis, assistant professor of computer science; Greg Morrisett, the Jack and Rilla Neafsey Dean and Vice Provost of Cornell Tech and professor of computer science, and Andrew Myers, professor of computer science and Class of 1912 Professor of Engineering.

“Agentic AI has great potential to change how software is written, but we need stronger assurance that agents will not generate harmful code,” Myers said. “In this project we are exploring how to get the real benefits of agentic AI but in a safe and secure way.”

“With Amazon’s support, the launch of AI4AI at Cornell establishes a research home to address one of the most pressing problems in how we produce code today,” said Thorsten Joachims, the Jacob Gould Schurman Professor of Computer Science and Information Science, vice provost for AI Strategy and director of the Cornell AI Initiative. “AI4AI brings together our strengths in AI and in programming languages, and we’re very excited about this partnership with Amazon.”

Prompted by a few keystrokes, AI agents powered by large language models work across different software tools, comb the internet, interact with code repositories, and write code. But agentic AI’s revolutionary ability to simply follow directions may create vulnerable software if security expectations are unspecified or manipulated with malicious intent. Agents lack skepticism. They can generate code with security weaknesses with potentially serious downstream consequences, researchers said.

In effect, the Cornell team’s work will make AI agents more cautious with their code outputs by developing a security framework with rules and verification checks.

“We’re proud to support Cornell’s researchers as they build the foundational frameworks for secure AI code generation. And the opportunity extends further,” said Debashis Das, principal in the Office of the Chief Information Security Officer at Amazon Web Services. “AI security agents are emerging to protect software applications well beyond the coding phase, across the entire lifecycle. Industry-academia collaboration like this is essential to advancing the tools and frameworks the developers need to innovate with confidence.”

Louis DiPietro is a writer for the Cornell Ann S. Bowers College of Computing and Information Science.