How do you prevent the agent from reward-hacking the hidden eval? e.g. writing t... | Hacker News

Hacker Newsnew | past | comments | ask | show | jobs | submit

azertyvode 39 days ago | parent | context | favorite | on: Show HN: I RL-trained an agent that trains models ...

How do you prevent the agent from reward-hacking the hidden eval? e.g. writing training data that effectively leaks the eval distribution rather than teaching a general skill?

Danau5tin 39 days ago [–]

Great question. Today the answer is that there is nothing stopping this, but you are right this is a 100% valid reward hacking possibility.

No signs of it in the run I did, but for any future larger scale runs, it would be wise to dedicate some focus to prevent this.

Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact