I Measured dev-orchestra: Time per Request, and Findings by Number of Reviewers
Introduction
dev-orchestra is a plugin for Claude Code / Codex / Antigravity that takes a single-sentence request and runs design, implementation, testing, reviews by multiple AIs, and fixes. I covered what it does in the introduction post.
In the series I said it “costs more time and tokens,” but I never gave measured numbers. So this time I measured two things:
- How long one request takes
- How the number of findings, and how many of them I could accept, change with 1, 2, and 4 reviewers
Up front: the sample is very small (one end-to-end request, and two reviews per reviewer count), on one small Python package. Read this as a record for getting a feel for the trends.
dev-orchestra is open source on GitHub (MIT license). If it looks useful, a star would mean a lot.
Setup
| Item | Details |
|---|---|
| dev-orchestra | 1.2.1, preset standard, optimization.level at the default balanced |
| Host | Claude Code 2.1.309 (model: Opus 5.5). I sent the request with claude -p and gave the plan approval in a second claude -p --resume |
| Models per role | architect: Claude Fable; implementer and review_fixer: Claude Opus |
| Reviewers | claude-general (Opus), codex-general (Codex CLI 0.163.0, gpt-6.1-sol), claude-security (Sonnet), claude-test (Sonnet) |
| Subject | textkit, a small Python package I made for the test (one function and three tests) |
| Environment | Windows 11, Python 3.14.7. The configuration lived in a separate folder (DEV_ORCHESTRA_HOME) |
The request, which I wrote in Japanese, says:
Add truncate(text, width, suffix="…") to textkit. Truncate by display width, counting full-width characters (East Asian Width W or F) as 2 and everything else as 1. When truncating, fit the result, suffix included, within width. Update the README and the tests too.Time for One Request
From the request to the final report took 6 minutes 12 seconds, not counting the time the plan waited for my approval.
| Span | Time |
|---|---|
| Request → plan ready and approval requested | 2 min 19 s |
| Approval → final report | 3 min 53 s |
| Total | 6 min 12 s |
dev-orchestra’s record (state.json) had the breakdown by stage. Each role’s run took:
| Stage | Role | Time |
|---|---|---|
| Design | architect (Fable) | 75 s |
| Implementation | implementer (Opus) | 45 s |
| Review | claude-general (Opus), one reviewer | 19 s |
| Fix (1st) | review_fixer (Opus) | 27 s |
| Fix (2nd) | review_fixer (Opus) | 45 s |
The five runs add up to 3 min 31 s. The remaining ~2 min 40 s went to the host Claude Code (the orchestrator) choosing stages, running tests, triaging findings, and writing the report.
The design review was skipped automatically, since the change was small and touched no high-risk paths.
One Reviewer, Not Four
standard has four reviewers, but for this request the panel was cut down to one. The review log says:
note: low-risk change (4 file(s), 76 line(s)): 1 reviewer instead of the full panel (claude-general). Cross-model disagreement is what a second reviewer buys; raise optimization.level or the low_risk thresholds to keep it.Under balanced, a change of up to 5 files and 150 lines that touches no high-risk path gets a single reviewer (optimization.low_risk_max_files and optimization.low_risk_max_lines). That’s why small requests stay light. If you want another vendor’s eyes even on small changes, raise optimization.level or lower those two values.
Why the Fix Ran Twice
The review had two findings; the orchestrator accepted one and rejected one. The rejected one said combining marks and zero-width characters should have width 0, which contradicts “everything else as 1” in the request.
The accepted one was about tests, but it also suggested adding tests for combining marks. The orchestrator noted “accepted with a narrower scope,” but the brief handed to the fixer (fix-brief.md) carried the reviewer’s suggestion as written, so the first fix added the very tests that had been rejected. The orchestrator caught that and ran the fixer again with a brief that spelled out the scope.
In 1.2.1, triage notes don’t make it into the fixer’s brief. When you accept a finding only in part, check the final diff yourself.
Tokens and Cost
tokens show lists the five runs dev-orchestra made:
stage meas. input output total billed cost tools tool out
architect 1/1 34 5,890 - 14,703 $0.4737 5 2,368
implementer 1/1 14 4,103 - 27,614 $0.3380 14 10,107
review 1/1 2 1,517 - 7,364 $0.0778 0 0
review_fixer 2/2 26 5,495 - 43,959 $0.5371 14 6,369
ALL 5/5 76 17,005 - 93,640 $1.4265 33 18,844Separately, the host Claude Code reported $1.74 for its two claude -p calls, for about $3.2 in total. Both are API-price equivalents reported by the CLIs; on a subscription they indicate how much of your quota was used, not what you’re billed.
Number of Reviewers vs. Findings and Accepted Findings
Next, I took the review on its own and varied the panel. I took the diff from the run above at the point implementation finished (4 files, 75 lines added, 1 deleted), applied it to a fresh repository each time, and had 1, 2, or 4 reviewers review the same diff. Two runs per count, six in total.
dev-orchestra review snapshot
dev-orchestra review run --only claude-general codex-general --json
dev-orchestra review show --jsonNaming reviewers with --only overrides the cut to one reviewer described above. The panels were:
| Count | Reviewers |
|---|---|
| 1 | claude-general |
| 2 | claude-general, codex-general |
| 4 | claude-general, codex-general, claude-security, claude-test |
How I Triaged
I sorted the findings by the same rule the orchestrator used in the end-to-end run:
- Accepted: something genuinely missing within the request and the approved plan. If only part of it was in scope, accepted with a narrower scope
- Rejected: anything that contradicts “everything else as 1” in the request (giving combining or zero-width characters width 0, never cutting inside a grapheme, and so on), or that doesn’t need fixing
- Duplicate: the same main point as another accepted finding
dev-orchestra merges duplicates conservatively; across these six runs it merged none automatically. It does list “possible duplicate” pairs, which I used when sorting.
Results
| Reviewers | Run | Findings | Accepted | Rejected | Duplicate | Time | Tokens |
|---|---|---|---|---|---|---|---|
| 1 | 1 | 2 | 1 | 1 | 0 | 25 s | 6,992 |
| 1 | 2 | 2 | 1 | 1 | 0 | 23 s | 6,751 |
| 2 | 1 | 4 | 2 | 2 | 0 | 53 s | 16,529 |
| 2 | 2 | 4 | 1 | 3 | 0 | 48 s | 14,837 |
| 4 | 1 | 8 | 4 | 3 | 1 | 25 s | 27,795 |
| 4 | 2 | 9 | 4 | 4 | 1 | 38 s | 27,964 |
Time runs from the start to the end of review run; tokens are the reviewers’ billed tokens summed (Codex included).
What I Found
The number of findings grew roughly in proportion to the number of reviewers: 2 with one, 4 with two, 8–9 with four.
Accepted findings grew too, but the extra ones were about tests. Most of what four reviewers added came from claude-test: “no test for when the suffix width exactly equals width (truncate("abc", 1) gives "…"),” “nothing checks that the default suffix … counts as width 1,” and so on. All were real gaps in the tests. On the other hand, no reviewer count produced a finding that was a bug in the code.
The most common finding contradicted the request. “It counts combining marks and zero-width characters as width 1” came up in all six runs. Claude (claude-general) and Codex (codex-general) said it independently, and with four reviewers claude-test raised the same point as a missing test. It’s a fair point in general, but the request had fixed “everything else as 1,” so I rejected it. The same finding came up in the end-to-end run too. Something you decided on purpose can look like a bug to reviewers.
Codex reported exactly one finding every time, and it was always this width-counting point. On this diff, adding another vendor didn’t surface anything new. The one run where two reviewers gave an extra accepted finding (run 1) came from Claude: “the docstring doesn’t match the behavior.”
Review time is set by the slowest reviewer. Reviewers run in parallel, so four of them don’t take four times as long. Two reviewers took longer than four because in those runs either Codex or Claude took over 45 seconds. Tokens, though, grow with the number of reviewers (about 7,000 for one, about 28,000 for four).
claude-security had zero findings both times, which is expected for a diff that handles no external input or secrets.
Limits of These Numbers
- One subject and one request. The diff was small (76 lines) and the spec was clear. Larger changes or vaguer requests will likely give different results
- Only two runs per reviewer count. Even at the same count, the number and content of findings shifted a little between runs
- I made the accept/reject calls. The rule is stated above, but someone else might sort them differently
- A case where the other-vendor reviewer (Codex) finds a real bug Claude missed didn’t come up with this diff. That doesn’t show it “doesn’t help,” only that I “couldn’t measure it this time”
The Downsides
- It costs time and tokens. This request took about 6 minutes not counting my approval, and about $3.2 at API prices. I didn’t compare it with asking a single AI directly, but it called AIs five times, and that time and those tokens add up
- More reviewers means more findings to reject. With four, 4–5 of the 8–9 findings were rejected or duplicates. The orchestrator triages, but the final check is yours
- Your code is sent to each reviewer’s vendor. With two or more reviewers, the diff went to both Anthropic and OpenAI. Reviewers can also read files around the diff, so it may not be just the diff that’s sent. I covered handling code that can’t leave your machine in the introduction post
Summary
- One small feature request with dev-orchestra 1.2.1 took 6 min 12 s not counting approval, about $3.2 at API prices
- 3 min 31 s of that was the roles’ runs; the rest was the orchestrator’s decisions, tests, and reporting
- For small changes,
balancedcuts the panel from four reviewers to one - On the same diff, findings were 2 with one reviewer, 4 with two, 8–9 with four; accepted were 1, 1–2, and 4
- The extra accepted findings from more reviewers were about tests; no reviewer count produced a code-bug finding
- The most common finding contradicted a spec decided in the request; to reviewers it looks like a bug
- A small measurement on one subject — a record for getting a feel for the trends
GitHub: istb16/dev-orchestra (a star would be very encouraging)
Related Posts
Giving dev-orchestra Standing Instructions per Role
I tried instructions, added in dev-orchestra 1.2.1. Standing instructions for the implementer or reviewers written in the repository go into every prompt. I check, with real output, how they are shown, where they go, and how a change to the instructions is treated.
Rearranging dev-orchestra's Reviewers and Roles
How to set the model for each role in dev-orchestra, how to choose reviewer focuses, and config suggest-roles, which proposes domain specialists. I also showed a change with deliberately planted bugs to 9 reviewers to see what changes as you add more of them.
Comparing dev-orchestra's quality / standard / fast presets
I expanded dev-orchestra's setup presets quality / standard / fast in three environments (Claude Code only, Codex only, and both) and compared them. Here is how the per-role models, the number of reviewers, and how much cost is cut change, and which one to pick.