Solo AI Lab
← Back to the notebook

Experiment guides / Practical experiment guide

Design a pilot that can tell you to stop

Define an accepted result, keep a baseline, and preserve failures. A small trial should help you make a decision before it becomes a product commitment.

On this page

Use a pilot to answer one decision

A useful pilot can tell you that a proposed service should remain smaller, change direction or stop. This guide walks through a planning example: can a three-shot product brief give a reviewer an understandable handoff before any video is rendered? The trial below is proposed and NOT_RUN. The public reference files were produced by an existing text-planning tool; they are not customer research or rendered-video results.

Google PAIR's user-needs guidance asks designers to connect an AI feature to a concrete need and define success. Here, the uncertain decision is whether to test a narrow planning service, rather than commit to selling finished video production.

Gather the inputs before choosing more software

Fill the decision and boundary sections

The following is a partially filled example you can adapt. It describes a future trial, not a completed result:

  • Decision: whether a bounded brief-handoff trial is worth another iteration.
  • Hypothesis: a reviewer can identify each shot's purpose, camera direction and preservation constraint from the text plan without mistaking it for a rendered clip.
  • Included task: inspect a three-shot plan and record unclear or missing instructions.
  • Excluded tasks: generate video, assess actual motion fidelity, promise a model's compatibility or deliver work to a customer.
  • Permission boundary: public reference files and private research notes only; any later external delivery requires its own authorization.

Choose your time and spending cap before running the trial. As a planning example, one might allocate a single 60-minute review session and no paid model calls. Those are illustrative limits, not a recommended sample size or evidence that the job fits within an hour.

Define an output a second person can inspect

Use one note per shot: identifier, intended purpose, motion/camera instruction, continuity constraint, unresolved question and reviewer disposition. The published reference has three shot records and separates settings suggestions from prompt text. That structure helps you ask precise questions; it does not prove a reviewer will understand it or that a video provider will follow it.

  1. Inspect the reference plan without modifying it and record confusing instructions.
  2. Choose one ambiguity to revise. Keep the original beside the revision so the change remains visible.
  3. Ask the agreed reviewer to explain the intended next action using the same brief. Do not supply the answer before recording their interpretation.
  4. Keep active handling time, corrections and unresolved items in the observation section. Leave observed fields blank until the work occurs.

Write stop conditions that address the actual risk

For this example, pause or narrow the offer if the reviewer expects a finished video, if the task requires image analysis the planning tool did not perform, or if essential shot instructions remain unresolved after the allowed revision. Do not label a polished prompt a successful production outcome. A missing dependency should leave an honest incomplete result and a recoverable original, consistent with PAIR's graceful-failure guidance.

Microsoft's human–AI interaction guidelines offer another lens: inspect initial expectations, ordinary use and errors separately. The proposed trial applies that distinction to a text handoff; it is not an evaluation of a deployed model.

Close with a decision and the evidence that supports it

Your finished planning artifact is the card with a task, boundary, acceptance rule, stop condition and data plan. It remains NOT_RUN until actual execution. After an authorized trial, record what happened, what was reviewed and why the next decision is continue narrowly, revise or stop. Preserve the original plan so the success rule cannot quietly change after disappointing observations.

Use the cost calculator for an explicit estimate and the support-budget guide to account for operator work. When the artifact is honest enough to show a relevant person, continue with one permitted route to a research conversation.

Suggest a correction

Continue reading