Skip to content
Dreadnode Dreadnode Dreadnode

Quickstart

Launch your first hosted evaluation against a published task, inspect results, and debug a failure.

Launch a hosted evaluation, watch it run, and drill into a failing sample — all from the CLI.

Prerequisites

Section titled “Prerequisites”
  • The Dreadnode CLI authenticated (dn login) — see Authentication
  • A published task (scaffold with dn task init, validate, then dn task push)
  • A model identifier like openai/gpt-4.1-mini

1. Launch the evaluation

Section titled “1. Launch the evaluation”
Terminal window
dn evaluation create flag-file-check \
  --task [email protected] \
  --model openai/gpt-4.1-mini \
  --concurrency 1 \
  --wait

--wait blocks until the evaluation finishes and prints a summary. Hosted E2B evaluations clean up the runtime and task sandboxes after every terminal outcome.

2. Check overall results

Section titled “2. Check overall results”
Terminal window
dn evaluation get 9ab81fc1
● completed  flag-file-check
ID  9ab81fc1-...


Model        openai/gpt-4.1-mini
Concurrency  1
Cleanup      always


Progress  ████████████████████████████  1/1  pass: 100.0%
          passed=1


Results   100.0%  ✓ 1 passed
          [email protected]  100.0% (1/1)
          durations: p50=34s  p95=34s  max=34s

UUID prefix matching works everywhere — the first 8 characters are enough.

3. List samples and read a transcript

Section titled “3. List samples and read a transcript”
Terminal window
dn evaluation list-samples 9ab81fc1
dn evaluation get-transcript 9ab81fc1/75e4914f

list-samples shows status, task, and duration per sample. get-transcript returns the full agent conversation — every user message, assistant response, and tool call. Sample references use eval/sample slash syntax.

4. Debug a failure

Section titled “4. Debug a failure”
Terminal window
dn evaluation list-samples 9ab81fc1 --status failed
dn evaluation get-sample 9ab81fc1/75e4914f

get-sample adds the lifecycle breakdown — when the item was queued, provisioned, started, and finished — plus the error message and any verification result.

Hosted evaluations remove failed sandboxes, so use the transcript and lifecycle details captured by the evaluation instead of connecting to the task environment after completion.

5. Retry or compare

Section titled “5. Retry or compare”
Terminal window
# requeue failed, timed-out, and errored samples
dn evaluation retry 9ab81fc1


# or launch a new evaluation with a different model
dn evaluation create flag-file-check-v2 \
  --task [email protected] \
  --model openai/o4-mini \
  --wait


dn evaluation compare 9ab81fc1 b2c34de5

What to reach for next

Section titled “What to reach for next”