tk3.biz
Back to Blog

I Had dev-orchestra Ask Me Questions Before the Design

Introduction

If you ask an AI only to “let tasks have a due date”, what feature do you get? Is the date set when a task is added, or later? A date only, or a time as well? Is the list sorted by due date?

Whatever the person asking leaves undecided, the AI decides quietly and writes the plan. Its calls are usually reasonable. But sometimes you only notice “that’s not what I meant” once it is built.

Since 1.1.0, dev-orchestra has a feature that makes it ask about the undecided points before the design (design.ask_questions). This time I tried it. The biggest surprise was that even a request I thought I had made clearly turned out to have four undecided points.

dev-orchestra is a plugin for Claude Code / Codex / Antigravity that takes a one-sentence request and runs the design, implementation, tests, reviews by several AIs, and fixes. Last time, I triaged the findings from several AIs and had the fixer address them.

dev-orchestra is open source on GitHub (MIT license). If it looks useful, a star would make me happy.

→ github.com/istb16/dev-orchestra

Setup

I tried two settings, auto and true. The default is false (no questions). auto asks only when the request is vague. true always adds a round where the architect (the design role) reads the code and asks questions.

dev-orchestra config set design.ask_questions auto
ItemDetails
dev-orchestra1.3.0, preset standard
CallerClaude Code 2.1.312. I made the request with claude -p and passed my answers with claude -p --resume
ArchitectClaude Fable
SubjectA small Python package todo I made for this (a Store that keeps tasks in JSON, a CLI with add, done, and list, and 2 tests)
EnvironmentWindows 11, Python 3.14.7. The settings were in a separate folder (DEV_ORCHESTRA_HOME), not my usual one

There were two requests, a vague one and a clear one (I wrote them in Japanese; these are translations).

  • The vague request was “Let tasks have a due date.”
  • The clear request was “Add rename(task_id, title) to Store. Raise KeyError if the ID doesn’t exist. Add todo rename <id> <title> to the CLI, and update the README and tests.”

I ran the vague request under auto, and the clear one under both auto and true. Every run was stopped where it asked me to approve the plan.

A vague request got three questions

When I asked “Let tasks have a due date.” under auto, it stopped before the design and asked three questions (after 1 minute 6 seconds). Here they are, translated from Japanese:

How "due date" works changes the code depending on how it is read, so let me check 3 points before the design (recorded as Q1–Q3).

Q1. How is a due date set?
- A: Only when adding — `todo add buy milk --due 2026-11-01`
- B: With a separate command on an existing task — `todo due 1 2026-11-01`
- C: Both A and B (B's command can also change or clear it)
- Recommended: C — tasks made before this change can get a date, and a wrong date can be fixed

Q2. What date format is accepted?
- A: Date only, `YYYY-MM-DD`
- B: Date with an optional time, `YYYY-MM-DD[ HH:MM]`
- C: `YYYY-MM-DD` plus relative words such as `today` / `tomorrow`
- Recommended: A — simplest and unambiguous, and it can be stored in `todo.json` as an ISO string

Q3. How does `todo list` show it?
- A: Show the date next to the title, keep the current order (by id)
- B: Show the date and sort by due date (tasks without one last)
- C: As B, and mark overdue tasks with something like `(overdue)`
- Recommended: C — a due date helps you see what is next and what is late

The doubts I listed at the start came back almost exactly as questions. With options and a recommendation attached, answering takes little effort. I replied like this (in Japanese):

Q1: C. Q2: "Only YYYY-MM-DD dates. Past dates are accepted too." Q3: I'll leave it to you.

Until the questions are answered, no plan is written. Even when I ran the command that writes the plan myself, it stopped before calling a model (and --force does not get past it).

$ dev-orchestra run architect --prompt-file request.md --output .ai/plan.md
refusing to run architect: 3 design question(s) still open (Q1, Q2, Q3) (design.ask_questions).
Ask the user; record their words with `dev-orchestra design questions answer <ID> --option <key>` or `--text "..."`, or `--delegate` when they leave it to the architect.
Never answer for the user. Exit 6.

After the answers, the plan had a ## Decisions section that recorded who decided what (excerpt):

## Decisions
- **Q1 (user): both `add --due` and a `due` command** that sets, changes or clears. ...
- **Q2 (user): strict `YYYY-MM-DD` only, past dates accepted.** ...
- **Q3 (me): Option C, sort by due date with undated last and mark overdue.** ...
- **Clear syntax is an explicit `--clear` (me).** ...
- **Unknown id in `due` is an error, exit 1 (me).** ...

(user) is what I decided, and (me) is what the architect decided. Q3, which I left to it, became C, as recommended. Smaller points that did not come up as questions (how to clear a due date, the exit code for an unknown ID, and so on) were decided by the architect.

Even a request I thought was clear got four questions

Next was the request to add rename. I thought I had written out the whole spec.

Under auto, no questions came. The request spells out the spec, so it apparently decided there was nothing to ask.

Under true, the architect read the code and then asked four questions (55 seconds). Before the questions, it wrote:

The code settles most of the mechanics: title joining from argv, usage docstring and README updates, `Store` already exported, KeyError test via assertRaises, and no argument validation anywhere in the CLI. Four decisions remain for you.

It does not ask what the code answers, only what has to be decided. Here are the four questions next to what the architect decided on its own under auto:

QuestionWhat I answered under trueWhat the architect decided under auto
An unknown ID in the CLIno task <id> to stderr, exit code 1 (the recommendation)The same
Return value of Store.renameLeft to it → None (like complete())The updated Task (like add())
CLI output on successLeft to it → nothing (like done)Prints renamed <id>
An empty title“Treat a whitespace-only title as empty, and reject it in rename. Leave add as it is.”Accepted (like add)

Now that it was pointed out, I had not thought about what to do with an empty title. Every call the architect made under auto was reasonable too. Still, the same request was decided two different ways.

The interesting one was the empty title. In the auto run, a design reviewer had flagged that it “should be rejected”. The orchestrator did not take that finding, to keep it in line with add. So a spec decision was being made while triaging review findings. Under true, I got to make it myself at the start.

That said, the return value and the success output were points I left to the architect under true as well. Those two differed not because of the questions but because the architect decided differently from run to run. The only thing that changed because I decided it myself was the empty title.

The downsides

  • It costs one round trip. The questions for the vague request came after 1 minute 6 seconds, and the architect question round under true took 55 seconds ($0.34 at API prices). Once questions come, the run stops until you answer, so if you are away from your desk, it waits that much longer
  • For a clear request, auto is often enough. Always using true can get you asked about points you don’t care about
  • You are the one who answers. This time I left two of the four to the architect; if you leave everything to it, there is little point in being asked

Summary

  • If most of your requests are vague, auto is enough. Even a request you think is clear leaves points undecided, so if you want to decide those yourself too, true suits you
  • If answering feels like a chore, “leave it to the architect” is fine. You can check those points in the (me) lines of the plan’s ## Decisions
  • How much passing the decisions changes the design review is something this test couldn’t tell

Next time (scheduled for November 23, 2026), I’ll try ways to keep tokens and cost down.

The fine print

From here on are the finer points that are good to know when you use it. The full details are in the docs, “Design questions”.

The setting’s values and the two stages of questions

ValueBehavior
false (default)Does not ask. The architect decides and writes the plan
autoAsks only when the orchestrator (the calling Claude Code, for example) judges the request ambiguous, a new spec, or open to several approaches
trueAsks every time. An architect round, where the architect reads the code and asks, always happens once (the orchestrator also asks first if it has something to ask)

The questions come in two stages. The first is the orchestrator’s requirement-level questions: “what do you want built” cannot be learned from the code, so it asks the user. The second is the architect’s questions with run architect --questions: having read the code, it asks only about the points the plan depends on, and it does not write a plan in this run. Under auto, the second stage is not required. For the vague request this time, the orchestrator’s questions were all it took.

Recording the answers

Questions are recorded with design questions record and numbered Q1, Q2, and so on. Answers are recorded with design questions answer in one of three ways: the option chosen (--option), your own words (--text), or “leave it to you” (--delegate). In the first run, the orchestrator recorded my answers like this (the --text answer was kept in my Japanese words; here it is translated):

dev-orchestra design questions answer Q1 --option C
dev-orchestra design questions answer Q2 --text "Only YYYY-MM-DD dates. Past dates are accepted too."
dev-orchestra design questions answer Q3 --delegate

The recorded answers are written out to decisions.md. The question file was recorded in English; the orchestrator seems to have written it in English for the architect (the questions it showed me were in Japanese).

- Answer: C: Both of the above (the separate command can also change or clear it)
- Answer: (my Japanese words for Q2, verbatim)
- Answer: left to the architect

While questions are open, status says so too.

Design questions: 3 open of 3 (Q1, Q2, Q3), round 0/2 -- ask the user; the plan run is refused until they are answered

How the decisions reach the design review

The decisions are also passed to the design reviewers as premises. Reviewers do not report the plan for following them. If a reviewer thinks a decision itself is wrong, it reports a finding with Category: decision, and the orchestrator does not triage that one on its own but puts it to the user.

In the two runs where I recorded answers, the design review ran twice each, and no Category: decision finding came up. No finding contradicted the decisions either. But I did not compare against a run without passing the decisions, so I can’t tell how much it helped.

Changing an answer after approval

If you change an answer after approving the plan, the approval becomes stale. In the true run, I approved the plan with the CLI’s design approve and then changed the answer to Q3 (the success output). Changing an answer needs --revise.

$ dev-orchestra design questions answer Q3 --option B
Q3 is already answered; pass --revise to change the answer
$ dev-orchestra design questions answer Q3 --option B --revise
Q3: B: Print `renamed <id>`, following the `added <id>` pattern from `add`.
$ dev-orchestra status
...
Plan approval: stale -- a recorded decision changed after the plan was approved; revise the plan and ask again ...

In status --json, stale_reason becomes decisions-changed. The plan did not change, but its premise did, so it needs approving again. The earlier answer stays in the question’s history.

A cap on question rounds

Architect question rounds are capped at 2 by default (design.max_question_rounds). To check this without calling a model, I recorded a “no questions” round by hand three times.

$ dev-orchestra design questions record --by architect --file no-questions.md
recorded architect round 1 with no questions in .ai/workflows/capdemo/decisions.md
$ dev-orchestra design questions record --by architect --file no-questions.md
recorded architect round 2 with no questions in .ai/workflows/capdemo/decisions.md
$ dev-orchestra design questions record --by architect --file no-questions.md
refusing to record another architect round: round cap reached (2 of 2, design.max_question_rounds): present the open points together and ask how to proceed.
Pass --force for one more round only when the user asks for it. Exit 6.

Running question rounds also has its own budget (budgets.architect_questions, default 3), separate from the runs that write the plan. Questions cannot go on forever.

When nobody is watching

A run with plan approval turned off (design.require_approval: false) is assumed to have nobody watching. Then the questions are recorded but not waited for; the remaining questions are left to the architect and the run goes on. I checked this without calling a model, by giving the architect the mock provider.

note: design.require_approval is false, so nobody is there to answer: Q1 left to the architect (recorded in .ai/workflows/capdemo/decisions.md)

decisions.md keeps left to the architect: nobody was there to answer (design.require_approval is false), so you can look back later at the points that were asked but never answered.

GitHub: istb16/dev-orchestra (a star would mean a lot)

All posts in this series are on the “dev-orchestra” list.

Related Posts