Official SWE-bench Lite Evaluation
Benchmark Results
We benchmarked Codor on standard industry SWE-bench issues across Flask, Requests, and Pytest — measuring exact execution time, tool efficiency, and real user costs.
Pricing reflects user credit consumption (100 Credits = $1.00 USD) compared directly against unoptimized frontier models solving the identical issues.
| Issue / Task | Repository | Time | Tools | Tokens | Codor Cost | Claude Sonnet | Claude Opus |
|---|---|---|---|---|---|---|---|
| pytest-dev__pytest-5227 | pytest-dev/pytest | 74.8s | 8 | 13.1k | 5.4 cr ($0.05) | 26 cr ($0.26) | 131 cr ($1.31) |
| pallets__flask-4992 | pallets/flask | 85.2s | 13 | 19.1k | 7.9 cr ($0.08) | 38 cr ($0.38) | 191 cr ($1.91) |
| pallets__flask-4045 | pallets/flask | 135.5s | 16 | 22.7k | 9.4 cr ($0.09) | 45 cr ($0.45) | 227 cr ($2.27) |
| pallets__flask-5063 | pallets/flask | 142.3s | 12 | 17.9k | 7.4 cr ($0.07) | 36 cr ($0.36) | 179 cr ($1.79) |
| psf__requests-2148 | psf/requests | 154.8s | 13 | 19.1k | 7.9 cr ($0.08) | 38 cr ($0.38) | 191 cr ($1.91) |
| pytest-dev__pytest-7432 | pytest-dev/pytest | 182.4s | 11 | 16.7k | 6.9 cr ($0.07) | 33 cr ($0.33) | 167 cr ($1.67) |
| Total / Avg (6 Tasks) | 129s avg | 73 calls | 108.6k | 44.9 cr ($0.45) | 216 cr ($2.16) | 1,086 cr ($10.86) | |
100% patch generation rate across all 6 benchmarked tasks.
User rate: 100 Credits = $1.00 USD (Non-expiring credit packs)