I Had dev-orchestra Review the Whole Change Once More at the End

Introduction

Last time, I measured how long dev-orchestra takes and how many findings it reports with different numbers of reviewers.

dev-orchestra is a plugin for Claude Code / Codex / Antigravity that takes a one-sentence request through design, implementation, testing, review by several AIs, and fixes.

This time I try the closing review of the whole change (review.whole_change_recheck) added in 1.3.0. When a review round that looked only at the fix finds nothing more, the review ends with one more round over the whole change. The other changes in 1.3.0 are in the release post.

dev-orchestra is open source on GitHub (MIT license). If it looks useful, a star would mean a lot.

→ github.com/istb16/dev-orchestra

Why it is needed

From the second round on, dev-orchestra’s code review sends reviewers only the diff of the fix. It records the working tree the previous round looked at and diffs against that, so a one-line fix does not mean sending the whole change again. The reviewers also get the findings the fix was meant to address.

Looking only at the fix, though, a review can miss a fix that clashes with an earlier part of the change. The fix itself may be right while another part of the same change still rests on the old assumption. The details are in the repository docs, under “The second round only diffs the fix” and “Closing on the whole change”.

How the loop goes

With review.whole_change_recheck on, a review ends on a round that looked at the whole change. The first round already looks at the whole change, so it counts as a whole-change round.

The round was	It found	Next (next_review)
Either	Something above Low, rounds left	Fix, re-test and review again: fix only (fix) if findings were accepted, otherwise the whole change (whole)
Fix only	Nothing above Low	Fix the accepted Lows, re-test, then review the whole change (whole)
Whole change	Nothing above Low	Fix the accepted Lows, re-test, and finish without another round (null)
Either	Something above Low, no rounds left	A final fix and re-test, with no further round (null)

“Above Low” means findings at Medium or higher, plus any severity added to review.re_review_severities, that were not rejected or marked as duplicates. Up to 1.2.1 only Critical and High called for another round, so a Medium finding now calling for another round is one of the changes in 1.3.0 (in this post’s examples, I did not get a case where a Medium alone leads to another round).

The cap on rounds (review.max_review_iterations, default 2) still applies. The one exception: the whole-change round right after a fix-only round with nothing above Low may run as the round after the cap (round max + 1).

Of the rows in the table, the first comes up in step 2 (only its “findings accepted, fix only” half; I did not try the “nothing accepted, whole change” half), the second in step 3, and the fourth in steps 4 and 5. The third comes up only in step 6, where the first round (a whole-change round) simply ends the review. In this post I did not get to try a whole-change round after a fix that comes back with nothing above Low. That case comes up in the 15th post of the series (scheduled for November 13, 2026).

Setup

Item	Details
dev-orchestra	1.3.0, preset standard, optimization.level at the default balanced
How I ran it	For steps 1 to 5 I ran review snapshot, review run and review status by hand, and did the triage and fixes myself. For step 6 I left the whole request to the orchestrator
Reviewers	balanced judged it a small change and cut the panel to one (claude-general, Claude Opus)
Subject	A small Python package, textkit, made for the test
Environment	Windows 11, Python 3.14.7, Claude Code 2.1.312, Codex CLI 0.163.0. Settings kept in a separate folder (DEV_ORCHESTRA_HOME)

I ran the loop by hand so that I could decide what each fix did, and set up a fix that clashes with the earlier change.

1. Prepare a change to review

textkit starts with just display_width, which counts display width, and three tests. I made a change that adds two functions (5 files, 63 lines).

# textkit/width.py
def truncate(text: str, width: int, suffix: str = "…") -> str:
    """Cut ``text`` to at most ``width`` columns, ending with ``suffix`` when cut."""
    if display_width(text) <= width:
        return text
    budget = width - display_width(suffix)
    out = ""
    for ch in text:
        if display_width(out + ch) > budget:
            break
        out += ch
    return out + suffix

# textkit/table.py
SEPARATOR = " | "
SUFFIX = "..."


def render_row(cells: list[str], width: int) -> str:
    """Cut and pad every cell to ``width`` columns and join them with ``SEPARATOR``."""
    if width < 1:
        raise ValueError("width must be at least 1")
    parts = []
    for cell in cells:
        cut = truncate(cell, width, SUFFIX)
        parts.append(cut + " " * (width - display_width(cut)))
    return SEPARATOR.join(parts)

I left a bug in truncate on purpose: when width is narrower than suffix, it returns the whole suffix, which is wider than width. render_row accepts any width of 1 or more, but the SUFFIX it uses is 3 columns wide. The tests passed with python -m unittest, and I recorded that with dev-orchestra state record test ok.

2. The first round (the whole change)

dev-orchestra review snapshot
dev-orchestra review run
dev-orchestra review status

Snapshot: .ai/workflows/a47656e712c1/reviews/review-target.diff
  strategy: git diff HEAD
  files:    5
  size:     3124 bytes (sha256 a4f653908d1b)

note: claude-test (when: relevance) added: tests/test_width.py is not documentation
note: low-risk change (5 file(s), 63 line(s)): 1 reviewer instead of the full panel (claude-general). Cross-model disagreement is what a second reviewer buys; raise optimization.level or the low_risk thresholds to keep it.
ok      claude-general     claude   opus               4 finding(s)

1 successful, 0 failed
Consolidated: .ai/workflows/a47656e712c1/reviews/consolidated.md

iteration 1/2
accepted findings: 0
blocking (critical/high): 2 F1, F2
re-review recommended: yes
next: triage the findings above Low (review triage), then run review status again
coverage: round complete, change complete

The next: line is new in 1.3.0. Findings above Low are waiting for triage, so it says to triage them first. There were four findings.

ID	Severity	What it said	Triage
F1	High	render_row accepts widths 1 and 2, but SUFFIX is 3 columns, so a cell overflows its width	Duplicate (same cause as F2)
F2	High	When width is narrower than suffix, the whole suffix comes back and exceeds width	Accepted
F3	Medium	It re-measures from the start for every character, so long text is slow	Accepted
F4	Low	No tests for a width narrower than the suffix, for zero or less, or for a wide character on the cut	Accepted

dev-orchestra review triage F2 F3 F4 --status accepted
dev-orchestra review triage F1 --status duplicate --note "same root cause as F2"

After triage, review status says to fix, re-test, and review what changed.

next: fix the accepted findings, re-test, and re-review what changed

When I had the same diff reviewed several times, the render_row finding came back as Medium on some runs and High on others. Severity varies from run to run.

3. Fix it, then a fix-only round (round 2)

I changed truncate as follows. When width is narrower than suffix, it cuts without the suffix, and it adds up widths as it goes. I added tests too.

    if display_width(suffix) > width:
        suffix = ""
    budget = width - display_width(suffix)
    used = 0
    for i, ch in enumerate(text):
        used += display_width(ch)
        if used > budget:
            return text[:i] + suffix
    return text + suffix

After recording the tests again, running review snapshot once more takes only the diff of the fix.

Snapshot: .ai/workflows/a47656e712c1/reviews/review-target.diff
  strategy: git diff <previous round> <now>
  scope:    what changed since the last reviewed round, not the whole change
            whole change kept at .ai/workflows/a47656e712c1/reviews/review-target-full.diff
            reviewers also get the findings the fix was meant to address
  files:    3
  size:     2159 bytes (sha256 a7c6656a948f)

This round found nothing. review status said:

iteration 2/2
accepted findings: 0
blocking (critical/high): 0 
re-review recommended: yes
next: re-review the whole change (review snapshot takes it)
coverage: round complete, change complete

The cap of 2 rounds has been reached, yet it says re-review recommended: yes and that the next step is to review the whole change. Up to 1.2.1, the review would have ended here.

4. The whole-change round (round 3)

Just running review snapshot again took the whole change. I did not pass --full.

Snapshot: .ai/workflows/a47656e712c1/reviews/review-target.diff
  strategy: git diff HEAD
  scope:    whole change (review.whole_change_recheck)
  files:    5
  size:     3866 bytes (sha256 44824ffac2ab)

ok      claude-general     claude   opus               5 finding(s)

There were five findings, one of them Medium.

### F1 [MEDIUM] correctness

- Location: `textkit/width.py`:18-20
- Problem: When `width` equals the suffix width, or the budget left after the suffix is too small for the next wide character, the result is only the suffix with no text. A wider column can therefore show less text than a narrower one.
- Impact: `truncate("abcdef", 2, "...")` returns `"ab"`, but `truncate("abcdef", 3, "...")` returns `"..."`. In `render_row`, `render_row(["日本語"], 4)` gives `"... "`, so a width-3 or width-4 column shows no cell text at all.

Cutting "abcdef" with "..." gives "ab" at width 2 but only "..." at width 3. The branch my fix added (drop the suffix when the width is narrower than it) disagrees with what the earlier change already did (when the width equals the suffix, no width is left for text, so only the suffix comes back), so the narrower width shows more text. In a table, since render_row uses a 3-column SUFFIX, a 3-column cell shows just "..." for any text longer than 3, and a 4-column cell shows just "... " when the next character is full-width. Round 2, which looked only at the fix, did not report it.

The other four findings were Low: the README does not mention the new functions, render_row lacks tests, control characters count as width 1, and there is a return that can never run. I accepted F1 plus the README and unreachable return findings, and rejected the other two as out of scope.

This was round 3, the round after the cap. Since something above Low remained, it says not to review again but to do a final fix and re-test (I left the final fix out of this post).

iteration 3/2
accepted findings: 3
blocking (critical/high): 0 
re-review recommended: no
next: do not re-review -- the iteration budget is spent
coverage: round complete, change complete
iteration budget exhausted -- fix the accepted findings once more (review fix-brief, then run review_fixer), re-test and record it (state record test ok|failed), and do not re-review; then report what is still open

dev-orchestra status says the same in one line.

Review: round 3/2, 3 accepted, 0 blocking -- final fix pending (fix, re-test, do not re-review)

From start to finish of review run, the three rounds took 28, 21 and 43 seconds. The reviewer’s billed tokens were 6,674, 9,187 and 13,666, 29,527 in all ($0.34 at API prices). The whole-change round took more time and tokens than the fix-only round, since its diff was bigger.

5. Try a fix that clashes on purpose

Next I tried a fix that plainly clashes with the earlier change. I had the same change reviewed again from a clean state with .ai deleted, and this time fixed truncate like this.

    budget = width - display_width(suffix)
    if budget < 0:
        raise ValueError("width must be at least the width of suffix")

It is one of the fixes the reviewer suggested. But render_row still accepts widths 1 and 2 and calls truncate with a 3-column SUFFIX, so a long cell in a 1- or 2-column row now raises ValueError.

The first round’s findings were much the same as in step 2 (render_row at High, truncate at Medium). Again I accepted the truncate finding and the test finding, marked the render_row one as a duplicate, and rejected the slowness finding (Low). Here is what the fix-only round 2 (2 files, 12 lines) reported.

### F1 [MEDIUM] correctness
- Location: `textkit/table.py`:11-15
- Problem: The new `ValueError` in `truncate` reaches `render_row`. `render_row` accepts `width >= 1` but always truncates with `SUFFIX = "..."`, which is 3 columns wide. So with `width` 1 or 2, any cell wider than `width` makes it raise.

The fix-only round caught it first. render_row was not in the diff, but the reviewer was given “the findings the fix was meant to address”, including the render_row finding I had marked as a duplicate. Reviewers can read files, so I think it went and looked at render_row from there.

I accepted this F1 and a Low saying render_row has no tests, and rejected the other Low. This round was round 2, which is the cap, and a Medium remained, so no whole-change round ran.

iteration 2/2
accepted findings: 2
blocking (critical/high): 0 
re-review recommended: no
next: do not re-review -- the iteration budget is spent
coverage: round complete, change complete
iteration budget exhausted -- fix the accepted findings once more (review fix-brief, then run review_fixer), re-test and record it (state record test ok|failed), and do not re-review; then report what is still open

Compared with step 4, the clash in step 5 was almost exactly what an earlier finding (the render_row one) described, while the one in step 4 was not described by any earlier finding. Perhaps the whole-change round helps when the clash cannot be traced from an earlier finding. But with one run each, I cannot say for sure why the fix-only round missed it in step 4 and caught it in step 5.

6. On and off, compared over one whole request

Finally, I left the same request as last time (add truncate to textkit) to the orchestrator from start to finish, once with the option on (the default) and once with it turned off in the global configuration. The subject was textkit before the change in step 1, and the caller was claude -p in Claude Code 2.1.312 (model: Opus 5.5). I gave the plan approval in a second claude -p --resume call, and the time spent waiting for my approval is not counted.

	On	Off
Request → plan ready and approval asked for	2 min 41 s	2 min 9 s
Approval → final report	2 min 2 s	3 min 21 s
Total	4 min 43 s	5 min 30 s
Review rounds	1	1
Findings (Medium or higher)	1 (0)	4 (1)
Accepted findings	0	1 (Low, tests)
Review fixer runs	None	1
What dev-orchestra called (tokens show)	$0.95	$0.99
The calling Claude Code	$1.45	$2.11

The amounts are the API-price equivalents the CLIs reported.

Both ended after the first review round. The first round looks at the whole change, so no whole-change round is added. With the option on, there was a single Low finding, and review status said to finish.

next: finish -- nothing above Low is open; do not re-review

With it off, one Medium came up, but it contradicted the request’s “everything else is width 1”, so it was rejected. Rejected findings do not count as “above Low”, so it would not have led to another round with the option on either.

In other words, for this request the option made no difference. The differences in time and cost come from the plans and findings differing between runs (the “on” run put truncate in a separate file, while the “off” run planned to raise ValueError when the suffix does not fit). With one run each, this comparison does not measure how many extra rounds the option adds. Extra rounds come after a finding above Low is accepted and fixed, as in step 4.

7. Turning it off

To go back to the old loop (re-review only on Critical and High, no whole-change round), turn the option off. The fast preset has it off from the start.

dev-orchestra config set review.whole_change_recheck false --scope global

This writes the global configuration. You can also turn it off in the project configuration (.dev-orchestra.yaml), but a project file can arrive with the branch under review, so it is treated as loosening the review and you get a warning.

$ dev-orchestra config validate
Configuration is valid.
Warnings:
  - review.whole_change_recheck: the project config (.dev-orchestra.yaml) turns it off; that file can come with the branch under review, so check that this is intended

doctor gives the same warning.

The downsides

• More rounds, time and tokens. In step 4, the whole-change round 3 took 43 seconds and 13,666 tokens, the heaviest of the three. The bigger the change, the heavier the whole-change round
• A Medium now calls for another round. Up to 1.2.1, fixing a Medium finding ended the review; in 1.3.0 there is one more round. A run of small findings makes it easier to hit the cap
• The whole-change round also brings findings unrelated to any clash. Four of the five in step 4 were Lows that had nothing to do with the fix, which means more triage
• Once the cap is spent, the final fix is not reviewed. If the whole-change round finds something above Low after the cap is used up, it is fixed and re-tested, and that is the end. Raise review.max_review_iterations if you want more rounds
• The subject was small and I tried only a few runs. The whole-change round found one issue, and in another case the fix-only round got there first. How often it pays off, I cannot say yet

Summary

• In 1.3.0, review.whole_change_recheck makes the review end with one more round over the whole change once a fix-only round has nothing above Low
• It is on by default under standard (including no preset) and quality. A Medium finding now calls for another round
• The whole-change round may run as the round after the cap (round max + 1). The next: line in review status and scope: whole change (review.whole_change_recheck) show which round you are in
• In my example, after a fix-only round found nothing, the whole-change round found one clash between the fix and the earlier change
• With a fix that clashed on purpose, the fix-only round caught it first. I think that is because the earlier findings were passed along, but with one run each I cannot be sure
• Running last time’s request once with the option on and once with it off, both ended after the first review round: the option made no difference
• To turn it off, set review.whole_change_recheck: false. Turning it off in the project file gives a warning

Next time (scheduled for November 9, 2026), I plan to use the review on its own.

GitHub: istb16/dev-orchestra (a star is always appreciated)

You can read every post in this series in the “dev-orchestra” list.