RLHF preference label for reward modelling, crowdsourced

“Collecting 50,000 preference labels for reward modeling typically costs $30,000 to $150,000 depending on whether you use crowdsourcing ($0.50 to $2 per ranking) or domain experts ($5 to $10 per ranking).”

• RLHF preference label for reward modelling, by domain experts
• Human-written SFT demonstration dataset of 10,000 demonstrations

• TaskMonk
• Expert and human work

• All price observations
• As a table, with the rest of the market
• How every number on the site traces to its records