That’s a good idea in general, although they’re kinda lying about the existing use on the front page: Fix "used by" overestimate on the web pages by webknjaz · Pull Request #30 · openai/agents.md · GitHub
I pointed Claude at my gist This Gist collects opinions and pointers on fighting LLM spam on GitHub · GitHub (with links to all discussions/policies I’ve seen) in research mode and it started scanning 357 sources.
This is what it composed: https://claude.ai/public/artifacts/de2ffa40-98dc-4458-b1b5-f3e645bf7840.
It’s rather comprehensive (and a bit long). But, there’s a policy template and coping strategies inside. I shared this on the internal Slack and @gundalow asked me to re-post here.
3 Likes
Fedora’s AI has been published now:
Seems reasonable to me. There would benefits to align with that for people who contribute to both projects.
People are trying to align on stuff @ wg-ai-alignment/moderation at main · chaoss/wg-ai-alignment · GitHub.
I’ve also started a discussion in pip-tools as I’m looking to come up with something to be copied into many projects I’m involved in: [policy] Good-faith agentic contributions and LLM use, and avoiding death by a thousand AI slops · jazzband/pip-tools · Discussion #2278 · GitHub.
3 Likes
UPD: I’m happy to announce that we’ve accepted the initial policy in pip-tools 3 days ago, it’s a part of the contrib guide — Contributing - pip-tools documentation v7.5.4.dev32. The corresponding PR (in addition to the discussion I posted earlier) holds some interesting context regarding the decisions agreed upon and improvements postponed over the course of the debate: https://github.com/jazzband/pip-tools/pull/2318.
P.S. This forum post has also been mentioned in the article that attempts to visualize how different projects handle slop: The Generative AI Policy Landscape in Open Source – console.log().
5 Likes
I (finally!) took the time to read that article properly. The bottom line there, as I see it, is to ensure there are markers (any of Co-Authored-by:, Assisted-by:, etc…) and to be aware that using AI incurs in a risk in the realm of licensing and copyrighting, but that risk is not dissimilar from the risk we have had with plain humans (says Mr 
) in the past.
To what I would add a grain of salt: scale. While the risk is really very similar, a human could produce a certain amount X of code (in whatever unit you choose: LOC, files, modules, PRs, …). And humans on the other side of the equation, would incur in a certain cost Y (also in whatever unit you choose: monies, effort, organizational time, personal time, CPU cycles of tools performing analysis, etc…) to consume that code, in terms of: processing the risk, mitigating, solving, ensuring things are alright.
With AI-assisted coding, the new X is now 3X, 10X, potentially-too-many X. For example, yesterday I had one contributor in c.g. who submitted 6 PRs, clearly vibe-coded, within a window of 30mins, give and take. This has been the first time we’ve seen that in c.g., but I am sure it won’t be the last. And then it won’t be only one person. It is not realistic to believe that we can scale Y up to match the demand using only humans on the other side. The only solution I can see right now, is use AI itself to review the issues and PRs. It feels like a cold war arms race: they use it more, we use it more to counter that, and as it seems to be working, they will use even more after that, and so on so forth.
The one thing we can try to do here is to take some control of the producer side of the equation by providing standard rules/prompts/skills/agents that “know” how to behave, what is acceptable and what is not, and will attempt to smooth the path before it even makes it to the consumer side. As expected, this will not solve 100% of the cases, but it could greatly reduce the risk and effort spent on the consumer side.
And those are my two cents.
2 Likes
Many people don’t realize it is harder and takes more time to review code well than to write it well, if it is written badly, it multiplies the reviewer’s cost.
I would recommend a throttle rule, limit submissions, specially of new contributors, x2 so if ‘vibing’. The limit can be enforced in the form of ‘we only review one PR, we do not get to the 2nd until after the first one is merged’. This is something we used to do in core, pre collections, when modules/plugins were all in the same place and we were overwhelmed with new submissions.
This allows the authors to learn what to look for, how to adjust the PRs to get a feel of what is required to get a merge, so their subsequent PRs can be presented in a much better way.
5 Likes
Likely not the first one, but there is an Openclaw agent which has raised a PR in the ansible/ansible-documentation repo: fix(docs): clarify SSH connection plugin default behavior by money-pit-bot · Pull Request #3497 · ansible/ansible-documentation · GitHub
@oranod and @Andersson007 were discussing there about how to proceed with the PR. As @Andersson007 mentioned in that PR, there is a AI policy in the works which we hope to share with everybody in the next few weeks.
What are people’s thoughts on fully autonomous AI agents submitting these kind of PRs. Do we accept them? or do we not accept them at all? @webknjaz I see that Contributing - pip-tools documentation v7.5.4.dev46 the approach is the latter:
Autonomous Code Submissions
The use of agents which write code and submit pull requests without human review is not permitted.
IMO, the main concern for me is also maintainer burden as @felixfontein mentioned earlier on. Fully autonomous agents can churn out code and PRs and this can put tremendous stress on the maintainer(s). The maintainer(s) could use AI tools themselves to help with the review but I’m not sure that is sustainable long term as the human review would be lost due to cognitive overload and could also have negative affects on the overall health of the codebase.
@Andersson007 and I were discussing the other day about a PR which was raised by an autonomous agent, he spent significant time reviewing it and then the agent closed the PR and the agent’s GitHub account was deleted from GitHub entirely.
Interested to hear others thoughts on this area.
1 Like
I think it is not realistic to suppose that we will be able to distinguish a fully autonomous from a semi-autonomous. If that’s feasible today, it is only a matter of time (and not a lot of it) for it to become impossible to distinguish.
The path I see that could work would:
- Provide the skillset for the AIs to consume and use, tilted towards ours standards, conventions, rules, etc. Instead of trying to fight the wave, we should harness its strength in our favour.
- Forbid PRs larger than some arbitrary size, except when coming from the maintainers and/or some trusted contributors. Reduce the size of the change → reduce the review effort → reduce the risk → reduce the blast radius. Agile methodology for the win.
- Update: that could (should?) be combined with some sort of throttling as suggested above by @bcoca.
- Actively use AI to triage and review issues and PRs using that skillset mentioned above (and possibly more artefacts. E.g. if a PR is larger than the established limit, the agent itself can comment and close the issue. They will use AI, and we can use it too.
- Constantly review the interactions in issues/PRs and accrue knowledge to the skills/rules/agents. Constant improvements.
- Update: Add one (or more, as needed) AI agents specialized in scanning the changes for security vulnerabilities or attacks. Get ahead of the curve, accept that malicious code will be submitted eventually, and analyse it earlier on.
- It follows that we could use these agents to review the security of the entire codebase, but that would have to be executed independently from the issues/PRs mechanics.
- Update: Conversely we could have other agents looking over specialised aspects of the project:
- an agent to review docs
- an agent to review the architectural soundness of the code
- etc…
I don’t believe we can stop that wave from crashing. We might as well surf it. 
I think that for now, we should reject contributions which do not seem to have human involvement. As long as we don’t have proper sentient AIs who can be trusted similarly to humans, letting a bot act autonomous without checking its output is both a liability and basically offloading all the work to maintainers. This is particularly problematic as maintainer time is a very scarce resource.
(If maintainers have their own autonomous bots for their repos, that’s something they have to make out among the maintainers team. Or if they want to grant specific users allowance for specific bots of them. My main problem is with general, unsolicited autonomous bot contributions.)
2 Likes
Regarding not being able to determine whether a bot is fully autonomous, semi-autonomous, or a bot at all: that’s indeed an unsolvable problem, I fully agree with @russoz here. For the moment, I think the best approach similar to duck typing: if it walks like a duck and quacks like a duck, it’s probably a duck. This definitely has false positives and false negatives, but I can’t think of a method that doesn’t…
2 Likes
Good Egg: Trust Scoring PRs · Actions · GitHub Marketplace · GitHub <= had been looking for this, if not this something similar can be done via bot
I’ve personally viewed the AI policy and Code of Conduct more generally as “is this interaction adding value is a meaningful and polite way”.
I personally don’t want us to get towards auto detecting (or rejecting) contributions due to auto detection. I think it might be good to ask what level of AI assistance was used to generate a contribution.
I’ve personally used Claude to autonomously create collection PRs to bulk fix Read the Docs links. PRs were created in draft mode, reviewed by myself then ready for review & merge. Worked well and I’ve got other improvements I want to make.
Although not common, we gave had people blindly run security scanners and submit reports or PRs without understanding what they are suggesting, same problem. The creator of the contribution needs to agree that they are acting in good faith and understands what they are suggesting.
2 Likes
Hi, everyone
We’ve created a forum post with an AI policy proposal. There’s also a link to a PR with the policy.
Please task a look.
2 Likes
The policy has been merged 
Thanks everyone for your feedback! 
3 Likes