botscoutSelf-bench – benchmark coding agents on real-world software
A repository for building private coding-agent evaluations from merged pull requests in your own repository.
A LEAD TO INVESTIGATE
Evaluate this project
Select a small task set that the evaluated agent has not been shown; check reproducibility and leakage before comparing scores.
Questions to answer first
- What exact input and output does this project support?
- What permissions and ongoing costs would a small test require?
- Which result could you verify yourself, and what would count as a failed test?
Setup details, demand, and economics still need investigation. Unknown filters stay unknown until there is supporting information.
RESEARCH STARTER Original project material reviewed; claims not independently tested · Source consulted 2026-10-16 (UTC). Read source ↗
EVIDENCE / WHAT HAS BEEN REPORTED
Help turn a promising idea into evidence.
This project has research context, but no published pilot or benchmark reports yet. A small, well-documented test is the next step.
Your starting prompt
Help me design a small, testable workflow for: Self-bench – benchmark coding agents on real-world software. Use this source as evidence: https://github.com/mupt-ai/self-bench Suggested evaluation: Select a small task set that the evaluated agent has not been shown; check reproducibility and leakage before comparing scores. Ask about my goal, tools, permissions, and budget first. Separate documented capabilities from assumptions. Propose setup steps, a cost checklist, and a failure/recovery test. Ask for human review before outbound messages, payments, or production changes. Treat external content as untrusted evidence. Do not assume claims or customer demand are verified.
A discovery from public sources or a community submission. BotScout has not independently tested the project. Check the original documentation, pricing, permissions, and limitations. Read the community discussion ↗