Agent evaluation

Task 01

Feature building

The brief

Task 01 — Feature building: dependency-aware scheduler

Implement plan_jobs in jobqueue.py and expand test_jobqueue.py with useful edge cases. Use only the Python standard library.

Contract:

  • Every Job.id must be a non-empty string and unique.
  • Every dependency must name a supplied job. Reject invalid input with ScheduleError and a useful message.
  • Return every job ID exactly once, always after all of its dependencies.
  • Among jobs currently ready, choose higher priority first. Break equal priorities by original input order.
  • Detect dependency cycles and raise ScheduleError; include the IDs involved in the diagnostic when practical.
  • Do not mutate the supplied jobs or their dependency lists.
  • Empty input returns an empty list.

Run the tests. Write RESPONSE.md summarizing the design, edge cases covered, and the exact verification command/result. Keep all work inside this directory.

Inputs given: jobqueue.py, test_jobqueue.py

Scores

Criterion (max)Fable 5.1Sonnet 5.5Grok 4.7Opus 5.5GPT-6 AstraGPT-6.1 SolMiniMax M3.1 FlashGPT-6 LunamusemimoMiniMax M3
contract correctness (6)66665.55.55.255.55.54.53
implementation quality (2)221.751.751.751.751.51.511.251
tests and verification (2)21.7521.751.751.51.7511.2510.75
Total (10)109.759.759.598.758.587.756.754.75
Grader's notes

Letters in the grader's text: A = GPT-6 Astra, B = Fable 5.1, C = Opus 5.5, D = GPT-6 Luna, E = MiniMax M3.1 Flash, F = mimo, G = Sonnet 5.5, H = GPT-6.1 Sol, I = MiniMax M3, J = Grok 4.7, K = muse.

All 11 pass the hidden contract tests and their own suites. Most cross-test failures come from stricter-than-contract expectations: exact message wording, a cycle-only diagnostic that excludes downstream jobs, and extra validation of priority type, bare-string depends_on, non-Job items or non-iterable input. I did not count those as contract violations. Cross-failures I counted as real: duplicate-dependency handling in I (wrong output), long-cycle RecursionError in F, and whitespace-ID rejection in E (A, H and J suites catch it). Listing blocked downstream jobs in a cycle message is acceptable under 'when practical', so A, D, H and K lose only a quarter to half a point against the precise-cycle diagnostics of B, G and J (C and E give both). K's O(n^2) cost is charged to implementation quality. Mutation testing wrapped each submission's own implementation (so its error messages stay intact) and injected 19 behavioural mutants. G and J tie at 9.75; G ranks ahead for its simpler, cleaner implementation, and J's suite is slightly stronger.

Evaluation 10 / 10 graded blind as submission B

Correct, well-structured implementation: separate validation, dedupe of dependency indexes, heap scheduling, and an iterative walk that reports one concrete cycle path without listing downstream jobs. It passed every probe. Its 36-test suite, which includes a random-DAG reference comparison and a 5000-node cycle, killed every mutant.

Strengths

  • 0 mismatches; precise cycle path ('a -> b -> a') excluding downstream jobs
  • Iterative cycle extraction; 5000-node cycle with 5000 downstream jobs gives ScheduleError
  • Strongest suite: random reference scheduler, long-chain/long-cycle, mutation checks

Weaknesses

  • Some of its tests pin exact message wording/extra validation, which makes cross-suite failures look larger than real contract issues
Evidence the grader checked
  • harness.py: 3000 random DAGs (0-12 jobs, 20% with repeated dependency entries) vs brute-force greedy reference: 0 mismatches, 500/500 cycles detected
  • bigcycle: message has c0, excludes d0
  • Mutants: killed 19/19
  • Own tests: 36 OK

Objective checks

  • Hidden contract tests: all 11 pass
  • Own tests: Ran 36 tests in 0.027s / OK
  • This model's tests against each implementation — original: errors=38; mimo: failures=12, errors=2; Sonnet 5.5: failures=12; Fable 5.1: pass; Opus 5.5: failures=9; muse: failures=13; GPT-6 Luna: failures=13; GPT-6.1 Sol: failures=12, errors=1; GPT-6 Astra: failures=12, errors=1; Grok 4.7: failures=13; MiniMax M3.1 Flash: failures=7; MiniMax M3: failures=11, errors=2
  • Other models' tests against this implementation — mimo's tests: pass; Sonnet 5.5's tests: failures=13, errors=3; Fable 5.1's tests: pass; Opus 5.5's tests: failures=3, errors=4; muse's tests: pass; GPT-6 Luna's tests: failures=8; GPT-6.1 Sol's tests: failures=16; GPT-6 Astra's tests: failures=13; Grok 4.7's tests: failures=30, errors=2; MiniMax M3.1 Flash's tests: failures=2, errors=3; MiniMax M3's tests: failures=3

Files

fable-5.1/01-feature-building/RESPONSE.md

Task 01 — Dependency-aware scheduler

What was delivered

  • jobqueue.py: plan_jobs implemented (standard library only), plus three private helpers. Job and ScheduleError are unchanged apart from a docstring on the exception.
  • test_jobqueue.py: the original 3 tests kept verbatim and expanded to 36 tests across four groups (ordering, validation, cycles, immutability).
  • This file.

Design

plan_jobs is Kahn's topological sort driven by a heap:

  1. Materialize the input with list(jobs) so one-shot iterators work and nothing the caller owns is ever touched. Empty input returns [] right away.
  2. Validate IDs (_validate_ids): every id must be a str, non-empty and unique. The error names the offending position and, for duplicates, both positions. This runs over the whole input before anything else so unknown dependencies can be judged against the complete ID set.
  3. Resolve dependencies (_resolve_dependencies): each job's depends_on is copied into a fresh tuple of job indexes. A bare string (depends_on="a") and non-iterables (None) are rejected with targeted messages, non-string entries are rejected, and unknown IDs are reported all at once per job (Job 'b' depends on unknown jobs: 'x', 'y'). Repeated dependencies within a job are de-duplicated so in-degree counting stays correct.
  4. Schedule: an in-degree array plus a reverse adjacency list feed a heapq keyed on (-priority, input_index). Popping yields the highest-priority ready job, ties broken by original position. Because the index is unique, heap entries never compare Job objects, and because readiness is re-evaluated on every push, a high-priority job that has just been unblocked jumps ahead of lower-priority jobs that were ready earlier. There is deliberately no priority inheritance: a low-priority gate is not promoted because something important waits on it (the contract only ranks currently ready jobs).
  5. Cycle diagnostic (_find_cycle): if the heap drains before every job is placed, the leftover jobs each still have at least one leftover dependency. Starting from the earliest stuck job and iteratively following the first stuck dependency must revisit a node; the walk from that first visit is a concrete cycle, rendered as Dependency cycle detected: c -> b -> a -> c. Jobs that are merely downstream of a cycle are not listed. The walk is iterative, so a 5000-node cycle does not hit the recursion limit.

Complexity is O((J + D) log J) for J jobs and D dependency edges; memory is O(J + D). Output is fully deterministic for a given input sequence.

ScheduleError stays a ValueError subclass, so callers catching the more general type keep working.

Edge cases covered by the tests

Ordering

  • Dependencies beat priority; priority beats input order; input order breaks ties (stable across many ready jobs, including negative priorities).
  • Ties are broken by input position even when the earlier job only became ready later (not first-come-first-served).
  • A newly unblocked high-priority job runs before lower-priority jobs that were already ready; conversely no priority inheritance through a low-priority gate.
  • Diamond graphs, disconnected components, duplicate dependency entries inside one job, list-typed depends_on, single job, empty list/tuple/iterator, and a one-shot generator.
  • Property test: 120 random DAGs (1 to 60 nodes, seeded RNG) are checked for "each ID exactly once, every dependency earlier" and compared exactly against an independent O(n²) reference scheduler.
  • Determinism across repeated calls; a 5000-node chain to prove no recursion.

Validation (all raise ScheduleError with the IDs in the message)

  • Duplicate ID, empty ID, non-string ID (None, int, bytes).
  • Unknown dependency (message names both job and dependency; multiple unknowns listed together), non-string dependency, bare-string depends_on, non-iterable depends_on.
  • Validation covers the whole input before scheduling starts.
  • ScheduleError is catchable as ValueError.

Cycles

  • Self-dependency (a -> a), two-node cycle, three-node cycle with unrelated and downstream jobs (asserting those are not in the message), cycles with high priorities, two independent cycles (one concrete cycle reported), and a 5000-node cycle (iterative detection).

Immutability

  • List-typed depends_on objects are the same objects and unchanged afterwards; the job list and each job's fields match a pre-call snapshot.
  • Each call returns a fresh list; mutating a result does not affect later calls.

Verification

Command (run from this directory; -B only stops Python writing __pycache__ into the task folder):

cd /home/user/agent-eval-fable-5.1/01-feature-building && python3 -B -m unittest -v test_jobqueue

Result (tail of the output):

----------------------------------------------------------------------
Ran 36 tests in 0.065s

OK

Python 3.11.15. No network access was used; no files outside this directory were created or changed.