add cancel request protocol for ZMQ env client-server

[mikasenghaas]

Summary

• Adds a CancelRequest message type to the ZMQ client-server protocol so the client can notify the server to stop processing cancelled rollout/group requests
• Client sends cancel messages on CancelledError (e.g. scheduler timeout) and from cancel_all_pending() — fire-and-forget, failures are logged but don't affect the cancellation flow
• Server tracks request_id → asyncio.Task mappings and cancels the corresponding task when a cancel message arrives, handled inline in the serve loop for minimal latency

Before: Cancellation was one-directional and local-only. The server continued burning inference compute on cancelled requests until the response was either silently ignored or hit a ZMQError.

After: The server receives a cancel message and cancels the in-flight asyncio task, stopping inference work promptly.

Test plan

• CancelRequest serialization/deserialization roundtrip (Pydantic + msgpack)
• Client send_cancel() sends properly formatted message, no-ops on empty list, swallows errors
• Client send_request() catches CancelledError, cleans up pending entry, sends cancel to server
• Client cancel_all_pending() sends cancel for all pending request IDs
• Server _handle_cancel() cancels tracked tasks, ignores unknown/done IDs, handles invalid requests
• Existing crash recovery tests pass (updated mock to skip cancel messages)
• Full test suite passes (723 passed, only pre-existing alphabet_sort env failure)

🤖 Generated with Claude Code

---

Note

Medium Risk
Adds a new cross-process cancellation path and changes the ZMQ server receive loop to parse and branch on message type, which can affect in-flight request handling and concurrency edge cases.

Overview
Adds a new CancelRequest message type and exports it via verifiers.workers to support server-side cancellation of in-flight work.

Updates ZMQEnvClient to fire-and-forget send_cancel() calls when send_request() is externally cancelled and when cancel_all_pending() clears pending futures.

Updates ZMQEnvServer to track request_id → asyncio.Task, detect cancel messages inline in the serve loop, and cancel/remove the corresponding task; includes new tests covering serialization, client send behavior, and server cancellation, plus a crash-recovery test tweak to ignore cancel frames.

^{Written by Cursor Bugbot for commit fd55552. This will update automatically on new commits. Configure here.}

[cursor]

Cursor Bugbot has reviewed your changes and found 1 potential issue.

^{Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.}

[cursor]

Deserialization failure silently drops request without responding

Low Severity

Moving msgpack.unpackb from process_request (where failures were caught by the except Exception handler that sends an error BaseResponse back to the client) into the serve loop (where failures just continue without sending any response) changes error behavior. Previously, a deserialization failure produced an immediate error response to the client; now the client's future is never resolved and it silently hangs until timeout. While rare in practice, this makes debugging serialization mismatches significantly harder.