agent-platform-autorater-customization

Customizes an autorater (evaluation metric) for an AI agent so its grades match the user's own judgment, starting from the agent's real trajectories. Use when a user wants a metric tailored to their agent, wants to turn failing trajectories into rubrics, says an existing autorater grades their agent wrong, or wants an autorater to agree with human labels (autorater hillclimbing, judge calibration). Don't use for quick evals with built-in metrics or for improving the agent itself (use agent-platf

Install

KitScout test notes

Not tested by KitScout yet. When our harness runs it, pros, cons and the tasks we tried appear here, with the date and evidence.

Similar tools

• ask-matt 323,485 ★
  Ask which skill or flow fits your situation. A router over the skills in this repo.
• chief-of-staff 323,485 ★
  Pursue a long-running goal in a single session by co-ordinating subagents.
• claude-handoff 323,485 ★
  Hand the current conversation off to a fresh background agent that picks up the work immediately.
• grill-me 323,485 ★
  A relentless interview to sharpen a plan or design.
• grilling 323,485 ★
  Grill the user relentlessly about a plan, decision, or idea. Use when the user wants to stress-test their thin