Interesting policy… I suspect it falls apart in the details though. Like, there are plenty of creative elements in responding to questions (and any of the other categories). That mean there will be lots of weird edge cases that are hard to adjudicate. And those GenAI components of “non-creative” categories will have ripple effects into the broader project and community (as they are everywhere, so maybe not that important for a.policy to grapple with, IDK).
That mean there will be lots of weird edge cases that are hard to adjudicate.
Humans a pretty decent at that. Source: Of all things, moderation, where even if you have nice rules written up, it feels like half the reports are edge cases and you just have to make a call.
I suspect they’ll be able to handle making those calls. Get some wrong, get most right, that’s probably good enough.
It’s not like it is a legally binding agreement that needs a high level of quality in adjucation. If things seem off then it can be cited and you don’t have to prove it. Conversely, if something passes human review despite being codegen, well then I guess the codegen isn’t that bad
I’ve seen a pull request that was ultimately nonsensical but plausible and short, and purported to be needed in a configuration none of the maintainers had access to. It was a pretty harmless change either way so they were inclined to accept it. But they sought a little clarification and the clearly LLM response in comment got it shut down. No human stepped up to actually explain an understanding and the LLM sounded like a TV show writer writing tech, sentences that sounded credible but to anyone with understanding was nonsensical bullshit.
LLM code can sometimes be hard to tell in a short code change, but the prose when things are anything vaguely needing explanation is just so very very obvious.
The codegen is a particularly big problem for famous open source projects. All of a sudden you have a flood of people trying to get their name on your project and flooding you with low quality stuff. Codegen enables folks without good ideas to generate submissions that demand a lot of review. Being about to shoot down obviously AI content without deeper review is pretty critical to keep the project viable. There may be good things tossed out with the bathwater, but it is a ton of bathwater. Besides, if there’s something real behind it, a good human is likely to emerge to take it forward more credibly.
Closed source or personal projects can enjoy a bit more judgement. You can evaluate the people independent of the submissions and may be able to have an even more nuanced take.
From the article:
Seems pretty level-headed to me.
Interesting policy… I suspect it falls apart in the details though. Like, there are plenty of creative elements in responding to questions (and any of the other categories). That mean there will be lots of weird edge cases that are hard to adjudicate. And those GenAI components of “non-creative” categories will have ripple effects into the broader project and community (as they are everywhere, so maybe not that important for a.policy to grapple with, IDK).
Humans a pretty decent at that. Source: Of all things, moderation, where even if you have nice rules written up, it feels like half the reports are edge cases and you just have to make a call.
I suspect they’ll be able to handle making those calls. Get some wrong, get most right, that’s probably good enough.
It’s not like it is a legally binding agreement that needs a high level of quality in adjucation. If things seem off then it can be cited and you don’t have to prove it. Conversely, if something passes human review despite being codegen, well then I guess the codegen isn’t that bad
I’ve seen a pull request that was ultimately nonsensical but plausible and short, and purported to be needed in a configuration none of the maintainers had access to. It was a pretty harmless change either way so they were inclined to accept it. But they sought a little clarification and the clearly LLM response in comment got it shut down. No human stepped up to actually explain an understanding and the LLM sounded like a TV show writer writing tech, sentences that sounded credible but to anyone with understanding was nonsensical bullshit.
LLM code can sometimes be hard to tell in a short code change, but the prose when things are anything vaguely needing explanation is just so very very obvious.
“refine” vs “create” is already a super grey area
Agreed… although this still leaves the horrible environmental effects and losing any kind of ownership ofethe code
But solely in terms of using “ai” as a tool, it is a sensible approach
nah, it’s a dumb policy which will be impossible to maintain in the years to come.
Maybe in the years to come but the shit code flood gates are open now when the tech isn’t good enough for the job.
Of that comes to pass, they can change it.
The codegen is a particularly big problem for famous open source projects. All of a sudden you have a flood of people trying to get their name on your project and flooding you with low quality stuff. Codegen enables folks without good ideas to generate submissions that demand a lot of review. Being about to shoot down obviously AI content without deeper review is pretty critical to keep the project viable. There may be good things tossed out with the bathwater, but it is a ton of bathwater. Besides, if there’s something real behind it, a good human is likely to emerge to take it forward more credibly.
Closed source or personal projects can enjoy a bit more judgement. You can evaluate the people independent of the submissions and may be able to have an even more nuanced take.
That’s fair :)