Agent evaluation

Task 05

Technical writing

The brief

Task 05 — Technical writing: webhook delivery guide

Turn SOURCE_NOTES.md into WEBHOOKS.md, a developer guide for customers integrating with the service. Target 700–1,100 words.

The guide must include a minimal verification example, delivery and retry semantics, ordering/idempotency guidance, key rotation, operational advice, and a troubleshooting table. Clearly distinguish guarantees from recommendations. Resolve apparent contradictions conservatively and call out anything that still requires product confirmation instead of inventing behavior.

Write RESPONSE.md with your editorial choices and unresolved questions. Work only in this directory.

Inputs given: SOURCE_NOTES.md

Scores

Criterion (max)Opus 5.5Sonnet 5.5GPT-6.1 SolGPT-6 AstraFable 5.1GPT-6 LunaGrok 4.7mimomuseMiniMax M3.1 FlashMiniMax M3
factual fidelity (3)2.52.52.752.752.252.52.51.751.50.750.75
information design (3)32.752.52.52.52.252.252.252.2521.5
clarity/examples (3)32.52.52.252.252.2521.751.751.251.25
handling uncertainty (1)11110.750.7510.750.50.250.5
Total (10)9.58.758.758.57.757.757.756.564.254
Grader's notes

Letters in the grader's text: A = GPT-6.1 Sol, B = Sonnet 5.5, C = mimo, D = GPT-6 Luna, E = MiniMax M3.1 Flash, F = muse, G = Opus 5.5, H = MiniMax M3, I = GPT-6 Astra, J = Grok 4.7, K = Fable 5.1.

A and B tie at 8.75; A ranked higher for fewer unsupported statements and a more robust verifier. K, D and J tie at 7.75: K ranked first for structure and a multi-secret example, D second for robust but single-secret code, J third for choppy prose despite the best open-questions list. A and I are near-duplicates in approach and code; I is slightly wordier and has a thinner troubleshooting table. All WEBHOOKS.md files are within 700-1,100 words by wc -w (G at 1,094 is closest to the ceiling). Code was executed against signatures built from the notes' scheme (timestamp + '.' + raw body, HMAC-SHA256 hex), including previous-secret, old-timestamp, uppercase-hex, non-ASCII-header, missing-header, malformed-timestamp and body-only cases.

Evaluation 9.5 / 10 graded blind as submission G

The strongest submission. It uses Guarantee/Recommendation/Unconfirmed labels, a retry-outcome table, a robust multi-secret verifier, and a detailed troubleshooting table. It reasons conservatively about freshness: log the timestamp age rather than reject on it, since replays can come 7 days later. One minor overreach: it states that retries reuse the event ID as a guarantee.

Strengths

  • Verifier never raised in tests (missing headers, non-ASCII, uppercase hex) and accepts the previous secret
  • Retry table (timeout, retryable codes, other 4xx, 2xx) is very scannable
  • Explains the failure modes: a 401 isn't retried so alert and replay; dedupe swallows replays, so clear the record to reprocess; event ID is unsigned
  • Unconfirmed items: interval reference point, Retry-After, post-retry behavior, redirects, rotation signing key

Weaknesses

  • Header table states 'Retries and manual replays of an event reuse it' as a Guarantee; the notes only say this for replays
  • UTF-8 secret encoding is assumed in code without flagging it in the guide
  • Close to the 1,100-word ceiling (1,094 by wc -w)
Evidence the grader checked
  • 'log timestamp age instead of rejecting on it' handles the notes' unknown tolerance conservatively
  • 'about 14.6 h in total' is placed under Unconfirmed
  • 'X-Orbit-Event | Event ID. Retries and manual replays of an event reuse it.' vs notes line 15 (replay only)

Objective checks

  • WEBHOOKS.md ≈ 911 words (700–1,100)

Files

opus-5.5/05-technical-writing/RESPONSE.md

Response: Webhook Delivery Guide

Deliverable

WEBHOOKS.md is 1,094 words by wc -w, or about 1,030 without table pipes, so it falls inside the 700–1,100 target. It covers every required element:

  • a request/header reference
  • a minimal verification example
  • response rules
  • retry semantics
  • ordering and idempotency
  • key rotation
  • operations advice
  • a troubleshooting table

I ran the Python verifier locally against these cases:

  • A valid signature made with the second of two secrets is accepted.
  • A signature made with the wrong secret is rejected.
  • A body changed by a single byte is rejected.
  • A non-ASCII signature header is rejected; it does not raise an exception.

Editorial choices

  1. Three explicit labels. Every section separates Guarantee (behavior stated in the notes), Recommendation (our advice), and Unconfirmed (open questions). Readers can then tell what they may build on.
  2. Signature conflict resolved toward the current implementation. The guide documents timestamp + "." + raw body as current behavior. A callout says the older "parsed JSON" description is stale. I did not describe the old scheme as a supported alternative.
  3. Conservative stance on timestamp freshness. The notes give no tolerance window. Retries can arrive up to about 15 h late and replays up to 7 days late. Rejections return a non-retried 4xx. For those reasons I did not recommend a typical 5-minute window, which could silently drop legitimate events. The guide tells readers to log timestamp age until behavior is confirmed.
  4. Unsigned event ID pointed out. The signature covers only the timestamp and body. The X-Orbit-Event header can therefore be altered, and the guide says so.
  5. Consequences drawn from stated facts, not new behavior. Some advice follows directly from the notes; none of it describes new behavior:
    • Signature failures return 401, which is not retried, so readers should alert and replay.
    • Deduplication will swallow manual replays, because replays reuse the event ID.
    • Readers should return 503 for transient errors.
    • The rotation procedure is robust to either signing behavior during the overlap.
  6. "Roughly" kept as "approximate". Retry times are not presented as exact. The count is stated precisely: 5 retries, 6 attempts in total.
  7. Redirects are treated as unconfirmed. The advice to register the final URL is safe whatever the answer turns out to be.
  8. HTTPS is given only as a recommendation, because the notes don't say whether plain HTTP is allowed.
  9. Payload schema omitted. The notes don't describe body fields, so the guide invents none. That includes an in-body event ID or a creation time for ordering.
  10. Logging warning. The guide warns against logging secrets, signature headers, or full sensitive payloads, as the notes request.

Unresolved questions for product/engineering

  1. Signing scheme. Confirm that the old "sign parsed JSON" scheme is gone for every endpoint, including legacy ones. Confirm that the hex is lowercase.
  2. Timestamp on retries and replays. Is the request signed again with a new X-Orbit-Timestamp? What freshness window should we recommend?
  3. Does X-Orbit-Timestamp reflect event creation or send time?
  4. Event ID in the body. Does the signed body contain the event ID, so that deduplication can rely on a signed value?
  5. Retries and the event ID. Confirm that retries reuse the event ID. The notes state this only for replays; the guide assumes it for retries.
  6. Retry intervals. Are they relative to the previous attempt or to the first one? How much jitter is there? Is Retry-After honored?
  7. After the final retry. Is the endpoint disabled or the customer notified? Can the event still be replayed?
  8. 3xx and 1xx responses. How are they classified: failure, retried, or not retried?
  9. Redirects. Are they followed? Security says no; the implementation owner has not confirmed.
  10. Rotation overlap. During the 24 h window, does Orbit sign with the current secret, the previous one, or both? Can X-Orbit-Signature contain multiple v1= values? Is the new secret shown once, like the original?
  11. Replay window. Are the 7 days measured from event creation or from the first delivery? Are replays subject to the same retry policy?
  12. Oversized payloads. What happens to events over 512 KiB: are they dropped, truncated, or sent as a reference?
  13. Plain HTTP endpoints. Are they permitted?
  14. Content-Type. What is the exact Content-Type header value?