Wikipedia talk:Signs of AI writing

Discovery

How are potential articles discovered eg. Category:Articles containing suspected AI-generated texts? Seems like a great job for AI, based on the rules from this page. -- GreenC 16:20, 17 July 2025 (UTC)

I'm sure many of the articles in that category were discovered manually. When I see an article with LLM-generated text, I tag it with {{AI-generated}} using Twinkle just as I would tag any other kind of content issue.

But thanks to the new filters described at WT:AIC § New edit filters, which scan for some of the signs described here, we can now check the logs of edit filters 1346 (hist · log) (Possible AI-sourced citations), 1369 (hist · log) (Markdown-formatted text), and 1370 (hist · log) (Unusual action at AfC) to help find new additions of LLM-generated content. — Newslinger talk 19:19, 17 July 2025 (UTC)

User:Newslinger: OK. As a side project, sometime, I'd like to choose 3,000 articles at random. Build a prompt from rules on this page. Then make queries on those articles. The prompt can be requested to score from 0-100. The average score of the known/probables, in the "suspected texts" category, could be a benchmark score, to help determine a threshold for human attention (no automatic tagging of pages). What do you think of this design? -- GreenC 20:52, 17 July 2025 (UTC)

I think that's an interesting idea, and it reminds me of how ORES and ClueBot NG use machine learning to evaluate edits and generate scores that represent the likelihood that they contain vandalism or other issues. The project sounds like it would be worth a try, and would have both practical and academic value.​ — Newslinger talk 21:08, 17 July 2025 (UTC)

I don't think AI detecting LLMs such as GPTZero are very reliable. I think our best bet for detecting AI is experienced humans and edit filters. –Novem Linguae (talk) 21:41, 17 July 2025 (UTC)

Artificial intelligence content detection does not have a great track record, and I generally don't use it. Having said that, I am curious as to whether a detection system for AI that that is specifically tailored to Wikipedia edits can perform well enough to be merged into or used alongside tools like ORES. — Newslinger talk 22:20, 17 July 2025 (UTC)

I can't imagine 100% fully automated AI detection would ever work. Yet like a meta detector it can give signals, some false, some real, that a little digging can confirm. It's better than no metal detector if your search area is very large. -- GreenC 05:12, 18 July 2025 (UTC)

I discovered that the article Eugenio Duse was AI-generated by sheer luck: I clicked on the random article button (as an IP editor; I didn't have an account yet) and found an article that had many signs from the list. The article has since been fixed, and the bad version is now in the list of signs as an example. SuperPianoMan9167 (talk) 01:30, 22 July 2025 (UTC)

Discussions on project pages

I don't know if WP:LLMTALK is a better venue for this but I write it here because it more has to do with identifying LLM.

We've identified a tendency to hallucinate shortcuts as well as the tendency to explain the shortcut, see Wikipedia:Administrators' noticeboard/Incidents § EditorSage42 bludgeoning and likely LLM use.

• the pattern described in WP:BOOMERANG, where
• WP:BLUDGEON specifically targets
• WP:CONSENSUS building relies
• Per WP:ADMINACCT, administrative attention should be

Curious if this fits the project page, but I would understand if it's out of scope and this would rather focus on article content rather than discussions. Bobby Cohn 🍁 (talk) 15:36, 21 July 2025 (UTC)