Running evaluations

Once you’ve run your first evaluation, the next questions are operational: how do I check this into source control, inject secrets, block CI on completion, retry failures, and compare runs? This page is the playbook.

For the exhaustive command and flag list, see dn evaluation.

File-backed manifests

Keep the evaluation definition in evaluation.yaml when you want it in source control, when the request grows past a readable command line, or when you need per-row inputs.

evaluation.yaml

name: nightly-regression
project: sandbox
task_names:
  - corp-recon
  - local-enum
model: openai/gpt-4.1-mini
secret_ids:
  - 11111111-2222-3333-4444-555555555555
concurrency: 4
max_steps: 50
cleanup_policy: always

dn evaluation create --file evaluation.yaml

While the request is being submitted, interactive terminals show a progress indicator and redirected output prints one plain status line. --json suppresses this status so scripts receive only the evaluation payload.

Explicit CLI flags override values from the file. Use secret_ids in the manifest for exact source-controlled configuration; use repeatable --secret flags to resolve names against your user-configured secrets at runtime. max_steps limits agent tool and generation steps for each sample; task_timeout_sec still applies as the wall-clock limit.

Which capability version runs

--capability accepts NAME or NAME@VERSION, and the ref is re-resolved on every run.

# tracks the highest published version — bump the version and re-run to pick up edits
dn evaluation create my-eval --task corp-recon --capability acme/web-security

# pinned: always runs 1.0.0, even once newer versions exist
dn evaluation create my-eval --task corp-recon --capability acme/[email protected]

An unpinned ref resolves to the highest version number visible to you at the moment the run starts — resolution is by semantic version, not by publish time. So the edit → bump the version → dn capability push → re-run loop picks up your changes without any extra step. Publishing a lower version than one that already exists (e.g. a 1.2.4 hotfix while 2.0.0 is out) will not be picked up by an unpinned ref — pin to that version explicitly to run it. A pinned ref always runs that exact version; if it doesn’t exist or isn’t visible to you, the run fails rather than falling back to another version.

The evaluation’s runtime is bound to whichever version resolved. Switching between pinned and unpinned refs, or pinning a lower version, re-points that binding — a runtime carries one version of a given capability at a time.

Injecting secrets

--secret injects user-configured secrets into both the runtime sandbox and the task environment sandbox.

# exact name: strict, must exist
dn evaluation create my-eval --task corp-recon --model openai/gpt-4.1-mini \
  --secret OPENROUTER_API_KEY

# glob: best-effort, zero matches is allowed
dn evaluation create my-eval --task corp-recon --model openai/gpt-4.1-mini \
  --secret 'OPENROUTER_*'

Selector	Behavior
Exact name	Strict — fails fast when the secret isn’t configured.
Glob pattern	Best-effort — silently skips when nothing matches.
Duplicates	De-duplicated before the request is submitted.

Pointing a task at a different host

--egress adds outbound destinations for this run’s sandboxes. The common case is retargeting: the task declares the host it was authored against, and the run points it somewhere else.

# run the task against staging instead of the host it declares
dn evaluation create my-eval --task corp-recon --model openai/gpt-4.1-mini \
  --egress staging.example.com

# repeatable; wildcards and CIDRs work the same as in a task manifest
dn evaluation create my-eval --task corp-recon --model openai/gpt-4.1-mini \
  --egress '*.staging.example.com' --egress 10.20.0.0/16

Targets union with the task’s own egress.allow and are checked against the deployment identically, so a run can widen relative to the task but never past what the deployment permits. A target the deployment denies is refused when you create the run, naming the target, and checked again when each sandbox provisions. Overrides are additive, so a run cannot narrow a task below what it declared.

If the task declares nothing, supplying --egress makes this run a declaration: it reaches what you named plus the platform’s own destinations, and not the open internet. See Scoping what a sandbox can reach.

Blocking on completion

Use --wait on create or the standalone wait command to gate CI or scripts on results. Both exit non-zero if the evaluation didn’t complete successfully.

# block at creation time
dn evaluation create my-eval --task corp-recon --model openai/gpt-4.1-mini --wait

# or wait on an existing evaluation
dn evaluation wait 9ab81fc1 --timeout-sec 3600

Cleanup policy

--cleanup-policy is easy to ignore until compute is left running.

• always (default) — clean up after every terminal outcome. Hosted E2B evaluations require this policy because the runtime and task sandbox form a restricted security pair.
• on_success — retain failed sandboxes for inspection on Docker and OpenSandbox deployments. Hosted E2B deployments reject evaluation creation with this policy. Clean up retained sandboxes with dn sandbox.

Retry and cancel

# requeue failed, timed-out, cancelled, and infrastructure-error samples
dn evaluation retry 9ab81fc1

# cancel a running evaluation (terminates active sandboxes)
dn evaluation cancel 9ab81fc1

retry is most useful after a terminal run when you want to requeue only the samples that ended in failed, timed-out, cancelled, or infrastructure-error states. It does not requeue judge errored samples: those mean verification was unavailable for the recorded trajectory, and rerunning the agent would create a different sample.

Export and compare

# export samples as JSONL (optionally include transcripts)
dn evaluation export 9ab81fc1 --format jsonl

# compare two evaluations side by side
dn evaluation compare 9ab81fc1 b2c34de5

Use compare to see how a different model, prompt, or task version performs against the same workload.

In the App, the evaluation view has a Download all N logs button in the header that bundles every sample’s trajectory into a single ZIP — one <sample>.json per linked sample plus a manifest.json. It uses the same format (ATIF, OpenAI, or Native) you last picked in the per-sample export menu. Samples that never ran (no linked session) are listed in the manifest as skipped rather than failing the download.

Transcripts

dn evaluation get-transcript 9ab81fc1/75e4914f

The transcript is available mid-run — the session link is established as soon as the runtime creates it, before the agent begins streaming. Samples without a linked session return 404 (old evaluations, or runtime session-registration failures); export --transcripts skips those with a warning instead of failing. For the payload shape, see dreadnode.sessions.

Sample references use eval/sample slash syntax (for example 9ab81fc1/75e4914f). Both IDs support prefix matching — the first 8 characters are enough.

Shared scope

Evaluation commands use the standard platform context from Authentication: --profile, --server, --api-key, --organization, --workspace, --project.

Note

Many “evaluation bugs” are scope mistakes. Confirm the workspace and project before assuming the evaluation didn’t exist.

When a run feels stuck

dn evaluation get 9ab81fc1 --json
dn evaluation list-samples 9ab81fc1
dn sandbox list --state running

That triangulates whether you’re looking at a control-plane problem, a task failure, or a cleanup-policy surprise.