The change was obvious – but in the end, things stayed about the same.
Cornell researchers took advantage of online petition website Change.org‘s gradual rollout of an artificial intelligence writing tool to gauge AI-enhanced language’s impact on petition outcomes.
The effect on outcomes? Negligible, at best.
Analysis of 1.5 million petitions showed that the AI writing assistance altered the wording of petitions, increasing their homogeneity and length, but did not improve petition outcomes, according to a research team that included Mor Naaman, the Don and Mibs Follett Professor of Information Science at Cornell Tech, the Jacobs Technion-Cornell Institute at Cornell Tech and the Cornell Ann S. Bowers College of Computing and Information Science.
“That was a surprise for us,” said Isabel Corpus, a doctoral student in information science and lead author of “Introducing AI to an Online Petition Platform Changed Outputs but Not Outcomes,” which published Sept. 30 in Nature Human Behaviour.
“There is this existing research that shows AI-generated text is persuasive, it’s believable and it can be creative, sometimes more so than people writing alone,” Corpus said. “And then on top of that, we would expect that a platform would integrate a tool that would improve outcomes, so we were surprised that it didn’t.”
In addition to Corpus and Naaman, co-authors included Allison Koenecke, assistant professor of information science at Cornell Tech and Cornell Bowers; and Eric Gilbert, professor in the School of Information at the University of Michigan.
Naaman said he heard about Change.org’s introduction of an AI writing assistant a few years ago from an employee of the site. When they looked into it, Corpus discovered that the tool was being rolled out gradually in English-speaking countries – first in the U.S., Great Britain and Canada, and 11 weeks later in Australia. That stagger allowed the team to measure the AI impact by comparing petitions written before and after the introduction of the tool across the different countries.
The researchers hypothesized that, since all four countries followed similar trends in petition style and outcomes prior to the AI tool’s release, differences in trends during and after the 11-week rollout period – Oct. 2 through Dec. 15, 2023 – could be attributed to the AI tool.
Although he couldn’t say for sure, Naaman suspects the AI writing assistant Change.org employed was trained, in part, on successful petitions the site hosted previously. “We saw in the data that those [AI-enhanced] petitions had the markers of what was successful in the past,” he said, noting that Change.org was one of the first online platforms to directly employ an AI writing assistant.
The team measured the effect of AI on lexical style using three main features: lexical diversity, by calculating the ratio of unique words to total words; readability, determined by measuring the number of words per sentence and syllables per word; and petition length, including text and title.
What stood out was the consistent effect on lexical features: Petitions written with AI’s help were longer, with more complicated and varied words.
“We also saw a lot of homogeneity in the language of the titles,” Corpus said. “Petitions started calling for people to ‘implement,’ ‘mandate’ or ‘urge,’ whereas before we rarely saw that these words were used.”
Using two metrics to gauge petition effectiveness (the share of petitions that reach one comment in 30 days; and the share that reach 10 signatures at the time of data collection), the researchers found that outcomes did not improve compared to pre-AI levels, and in some cases worsened. The share of petitions that reached one comment after 30 days post AI in fact decreased by around 5% relative to the pre-AI baseline.
Naaman said the team had a few theories as to why AI-assisted petitions didn’t fare any better than exclusively human-written ones. For one thing, people are suspicious when they feel something is written by a “robot,” although he admitted this was very early in the era of AI writing helpers such as ChatGPT. “People didn’t have their AI radars as high in 2023,” he said.
More likely, he said, AI-generated petitions tended to be less specific and detailed, and the petitioners likely are less committed to sharing their plea more broadly if it was, at least in part, written for them.
“From our research, it seems that people don’t sign a petition because they’re browsing on Change.org,” Corpus said. “They sign a petition or engage with a cause because someone they trust, or an organization they trust, shares it with them. So if there’s a decline in shares, then we would expect there to also be like a decline in engagement.”
Support for this research came from a Cornell Bowers Deans’ Excellence and Hopper-Dean Fellowship.
