Rearranging dev-orchestra's Reviewers and Roles

Introduction

Last time, I compared dev-orchestra’s presets. A preset decides, all at once, “where to spend money across the whole machine.”

This time, I’ll cover how to rearrange what’s inside it yourself.

• Which CLI and which model to use for each role
• How many reviewers, and with which focuses, to line up
• Having config suggest-roles propose domain specialists that fit the repository

At the end, I showed a change with deliberately planted bugs to 9 reviewers and measured what changes as you add more of them.

Everything in this post was checked with dev-orchestra 1.0.0. For the latest information, see the configuration docs.

dev-orchestra is open source on GitHub (MIT license). If it looks useful, a star would mean a lot.

→ github.com/istb16/dev-orchestra

Steps

1. Get to know the roles

dev-orchestra has five roles.

Role	Config name	What it does
Lead	orchestrator	Decides which stages are needed and summarizes the results
Design	architect	Investigates and writes a plan (read-only)
Implementation	implementer	Writes code and tests from the plan
Review	reviewers	Read the same diff without seeing each other’s results (read-only). You can have any number
Fix	review_fixer	Fixes only the findings accepted in triage

The four roles other than review have one agent each, but you can line up any number of reviewers.

2. Specify models as “family + latest”

In dev-orchestra’s config, you don’t write a model name (a dated ID). You write a family name and “use the latest in that family.”

implementer:
  provider: claude
  model:
    family: opus        # model family
    version: latest     # latest in that family

This way, when a new model comes out, it follows along without any config changes. On every run, it checks with the installed CLI which model is the latest opus before using it, so you never think you’re running on a model name that doesn’t exist. If it can’t confirm, it stops instead of guessing.

dev-orchestra model list shows which family names you can use. Here’s what I got (excerpt).

claude: installed
  fable                        family=fable                source=cli-help
  opus                         family=opus                 source=cli-help
  sonnet                       family=sonnet               source=cli-help
codex: installed
  CLI default (recommended coding model) family=recommended-coding   source=cli-default
  GPT-6.1-Sol                  family=gpt-6.1-sol          source=cli-catalog
  GPT-6-Astra                  family=gpt-6-astra          source=cli-catalog

If you really need to pin a single model, you can write version: pinned with an ID. It will no longer follow new models, though, so only do it on purpose.

3. Change the CLI or model per role

The easiest way to rearrange things is to ask the agent. Behind the scenes, commands like these run.

What you ask	Command that runs
“Use Claude’s opus for implementation”	config set implementer.model.family opus
“Have Codex do the design”	config set architect.provider codex (+ a family name Codex accepts)
“Change it for this project only”	Adds --scope project to the writing command

Settings take effect in the order project → global → built-in defaults. You override only what you want to change on top of the preset.

4. Give one role a cheaper model and a second opinion

With model tiers, you can give one role other models as alternates.

implementer:
  provider: claude
  model:
    family: opus
    version: latest
  model_tiers:
    light:                 # a cheaper model for things like one-line fixes
      model:
        family: sonnet
        version: latest
    second-opinion:        # another pass with a different company's AI
      provider: codex

The orchestrator picks which tier to use based on how hard the request is (run implementer --tier light). Specifying a tier that doesn’t exist is an error; it never silently runs on the default model.

5. Choose reviewer focuses

Each reviewer gets a focus (role). Even on the same diff, the focus changes what it looks at.

Focus	Mainly looks at
general	Correctness, edge cases, error handling, and so on
security	Authentication/authorization, injection, handling of secrets
performance	Complexity, N+1 queries, unnecessary I/O
test	Tests for changed behavior, meaningless assertions
architecture	Separation of responsibilities, coupling, consistency with existing conventions
database	Migration safety, locks, constraints, indexes
frontend	State handling, rendering, accessibility
backend	API contracts, validation, status codes

You add, remove, and change reviewers like this (you can ask the agent for this too).

dev-orchestra reviewer add --provider codex --role security   # add a Codex security reviewer
dev-orchestra reviewer set 2 --provider codex --role general  # replace the 2nd one
dev-orchestra reviewer remove performance                     # remove
dev-orchestra reviewer list                                   # check

With reviewer set <id> --when high-risk, you can also make a reviewer that runs only on risky changes.

6. Have config suggest-roles propose domain specialists

This is the feature I most wanted to introduce this time.

config suggest-roles looks at the repository’s file names and decides whether database, frontend, or backend specialists are needed. It doesn’t call a model, so it uses no tokens.

For this post, I made a small TODO app repository with a frontend, an API, and DB migrations, and tried it.

Proposed:
  claude-database  claude / sonnet  role database
    when.paths: migrations/*, */migrations/*, *.sql
    evidence: migrations/: 2 files, e.g. db/migrations/001_create_users.sql, db/migrations/002_create_todos.sql
  claude-frontend  claude / sonnet  role frontend
    when.paths: *.tsx
    evidence: *.tsx: 2 files, e.g. frontend/src/TodoItem.tsx, frontend/src/TodoList.tsx
  claude-backend  claude / sonnet  role backend
    when.paths: api/*, */api/*, routes/*, */routes/*
    evidence: api/: 3 files, e.g. api/db.py, api/routes/todos.py, api/routes/users.py
Each joins only code rounds touching its paths, and every design review round.

It even shows why it proposed each one (evidence).

On the other hand, running it on this blog’s repository (Astro) proposed nothing.

Not proposed:
  database: no signal (no migrations/, migrate/, alembic/ or prisma/ directory; fewer than 2 *.sql files; no schema.prisma)
  frontend: no signal (fewer than 2 *.tsx, *.jsx, *.vue or *.svelte files)
  backend: no signal (no api/, server/, backend/, handlers/, routes/ or controllers/ directory holding 2 files outside a frontend)
Nothing to add.

It also shows why it didn’t propose anything, so you’re never left wondering why nothing was added.

To accept the proposal, add --write. This was added to the project config file (.dev-orchestra.yaml).

reviewers_extra:
  - id: claude-database
    provider: claude
    model:
      family: sonnet
      version: latest
    role: database
    when:
      paths:
        - "migrations/*"
        - "*/migrations/*"
        - "*.sql"

The key is when.paths. A specialist joins code reviews only when files under those paths change. The database reviewer isn’t called when you only fix CSS, so everyday code reviews don’t get heavier. However, as the last line of the output says, specialists join every design review. Design reviews cost that much more.

	Presets	suggest-roles
What it decides	How to spend money across the whole machine	Adding specialists that fit the repository
Where it’s saved	Global config	Project config
When it applies	Always	Code reviews only when those paths change; design reviews every time

7. Measuring what changes as you add reviewers

Finally, I measured how the results change with the number of reviewers.

Method

• I made a change with 6 deliberately planted bugs (4 files) in the TODO app above
• I reviewed it only once with 9 reviewers: the quality preset (6 reviewers across Claude and Codex) plus the 3 specialists from suggest-roles
• Since reviewers run without seeing each other’s results, I counted results for setups like “1 reviewer only” or “2 companies” by combining the 9 reviewers’ results
• I triaged every finding myself and split them into the planted bugs (B1–B6) and real problems I hadn’t planted (E1–E6)

	Planted bug
B1	Embeds the search term into SQL with an f-string (SQL injection)
B2	Delete doesn’t check the owner (you can delete someone else’s TODO)
B3	+ 1 on the pagination offset (off by one item)
B4	Adds a NOT NULL column with no default (the migration fails)
B5	Renders the title with dangerouslySetInnerHTML (XSS)
B6	The delete test doesn’t verify the owner check

Results

Grouped by setup, it came out like this.

Setup	Problems found (out of 12)	Findings (before merging duplicates)
Claude fable (general) alone	10	10
Codex (general) alone	5	5
2 companies (the 2 above)	10	15
The 6 in quality	12	43
6 + 3 specialists	12	60

What I learned

• Obvious bugs were found by almost everyone. B1–B5 were found by all 8 reviewers except the frontend specialist (whose focus is the frontend)
• Going from 2 to 6 reviewers found 2 more problems (10 → 12). The 2 added were E5 and E6
• Going from 6 to 9 added no problems, only findings. The 6 found all 12, and with 9 the findings went from 43 to 60. After dev-orchestra merged duplicates, the 60 became 51
• The difference showed up in the unplanted problems. “A function’s arguments changed and broke its callers” (E5) was found only by the architecture focus, and “no tests for search” (E6) only by the test focus
• The model difference was also large. With the same general focus, Claude fable alone found 10 of the 12, while Codex found 5. In this run, I couldn’t separate the effect of the model from the effect of the focus
• The 2 Codex reviewers found 5 of the planted bugs with no false positives, but in this run they didn’t raise any problem that the others missed
• The cost for all 9 was about 110,000 tokens in total (because the diff was small, just 4 files)

This is the result of a single run on a small sample. The planted bugs were all obvious ones, and treating the unplanted problems as “real” was my own judgment. On more complex changes in real development, the effect of mixing in a different company’s AI may look different (an earlier post showed a different tendency on real code).

From these results, here’s how I’m thinking about my own usage.

• Rather than more reviewers, try adding different focuses. In this one run, architecture and test each raised one problem nobody else found. However, I didn’t try a setup with several general reviewers, so I can’t go as far as saying “it works better than adding more general reviewers”
• Add specialists with when.paths. In code reviews, they join only when those paths change
• More findings mean more triage. dev-orchestra merges duplicates (this time, 60 became 51), but with reviewers it’s not “the more the better”

Summary

• There are 5 roles (lead, design, implementation, review, fix). Only reviewers can be lined up in any number
• Specify models as family + latest. It follows new models as they come out and won’t run on a name it can’t confirm
• With model tiers, you can give one role a cheaper model or a second opinion
• Reviewers have 8 focuses. Rearrange them with reviewer add / set / remove
• config suggest-roles proposes database, frontend, and backend specialists from file names. It uses no tokens. Specialists join code reviews only when those paths change, and every design review
• Measured once with 9 reviewers: going from 2 to 6 found 2 more problems, and going from 6 to 9 only added findings
• In this one run, architecture and test each raised one problem nobody else found. Even with the same general focus, results differed a lot by model

Next time, I’ll write about how to connect your own agent (a local LLM) to dev-orchestra.

All posts in this series are in the list of “dev-orchestra” posts.

GitHub: istb16/dev-orchestra (a star is always appreciated)