I include a rule in the initial PR prep that it runs. So there's a hard limit to the length of the comments and what type of comments it can write and they get flagged in the review. It also reads the comments and checks for consistency with the following code so they don't drift. And the reviewer is a completely different agent/different model.

So all of that happens before the manual review and usually catches a lot of the 3-5 line comments it inevitably adds.

You could probably add linting rules to your tool of choice and tell claude that your lint tool has to pass.

but then no human generated comments allowed either, right?

I'd do this by only running the linter locally and using this as a custom rule for the LLM, but I need to think about it a bit harder.

There is probably something you can do to only apply some rules to the actual changed lines in a git diff.

You may have to write your own linter for that specifically.

you'd probs want a commit hook for that

Does "don't write comments" not work?

about as effective as don't make mistakes... For anthropic's models at least

It’ll work sometimes.

You’re using a non-deterministic algorithm to generate output. If you want deterministic rules applied to it, you have to use deterministic systems to do it.

Most lint tools allow exceptions if you add a special marker. I think even allow custom user exceptions.

E999: human generated comment :-)

But then the LLMs sees the syntax and is likely able to mimic it.

Shouldn't the "human generated comment" part be a strong hint not to?

As if LLMs would ever not ignore explicit instructions if they stand in the way of a goal...

I just delete them most of the time. But I keep my LLMs on a short leash. No 40000 line PRs.