32 lines
4.8 KiB
JSON
32 lines
4.8 KiB
JSON
{
|
|
"task_id": "RP24-02",
|
|
"commit": "3fa7c0b1f982ecf2d55a772c909dd4aa0a0409ba",
|
|
"status": "changes_requested",
|
|
"summary": "I checked acceptance item RP24-02 against the spec, the diff, the full sources, the Runner validation result, and the coordinator's replay of a real run. Status: changes requested.\n\nWhat passed:\n- The scope stays within runner/core.py and tests/test_runner.py.\n- No tests were deleted or disabled. The formal suite ran 56 tests and all passed (51 baseline plus 5 new).\n- The trusted self-test pin for RP24-02 checks the frozen base commit and the spec hash, and pairing a task with the wrong project is rejected.\n- The inner `--sandbox` flag was removed from the Antigravity command line, and no `--dangerously-skip-permissions` was added.\n- The recovered-error audit is written to `recovered-read-errors.json` in the run artifact, alongside the stdout hash. The original stdout is not modified.\n\nWhy it is not a pass:\n1. The fix fails on the real data it was meant to handle. The coordinator replayed the actual run 2be30fb95a1b4898bc7be87ad072f8ec through `agy_result` and it failed with \"GateError: Antigravity unknown event\". Real Antigravity output begins with an `init` event, and `agy_result` rejects every event before the final result unless it is a `step_update`. The spec's central case (recovered missing-file read errors followed by a valid result) is therefore still rejected in production.\n2. The test fixtures are synthetic and do not match the real output shape. They have no `init` event and no ACTIVE, `agent_response`/`text_delta`, `checkpoint` or `finish` steps, which is why the suite is green while the real replay fails.\n3. Removing the inner sandbox is only checked by looking at the command line. The spec requires a real shell run under the actual Antigravity outer sandbox profile that succeeds within the allowed write scope, while writes outside it and credential reads stay denied. That test is missing.\n\nNot done yet, as the spec expects at this stage: the isolated bootstrap, the full host suite before production adoption, and integration/final validation (`production_adopted` is false).",
|
|
"acceptance_ids": [
|
|
"RP24-02"
|
|
],
|
|
"findings": [
|
|
{
|
|
"severity": "high",
|
|
"message": "The real replay fails. Running the actual immutable stdout from run 2be30fb95a1b4898bc7be87ad072f8ec (input sha256 f7e5ad9f...) through agy_result raises 'GateError: Antigravity unknown event'. Real Antigravity streams start with an {\"event\":\"init\",...} line, and agy_result requires every event before the final result to be step_update. The spec's main case, five view_file ENOENT errors inside the workspace followed by SUCCESS/submitted, is still rejected. The fix should explicitly accept a single leading init event, after strictly checking its shape and that its cwd equals the task workspace, while still rejecting any other unknown event types."
|
|
},
|
|
{
|
|
"severity": "high",
|
|
"message": "There is no real outer-sandbox shell test for the Antigravity profile. The only check on removing --sandbox is test_agy_single_outer_boundary_argv, which inspects argv. The spec requires real tests under seatbelt(..., service='agy'): a bounded shell command such as git status in the worktree succeeds; writes inside the allowed paths succeed; writes to README.md, git metadata, CONTROL or provider-auth are denied; and Keychain/credential reads are denied. None of these exist. The observed failure ('sandbox_apply: Operation not permitted') has not been shown to be fixed under the outer profile."
|
|
},
|
|
{
|
|
"severity": "medium",
|
|
"message": "The recovered-read fixtures do not match the real stream format. test_agy_observed_recovered_read_and_rejects and test_agy_recovered_result_schema_and_identity leave out the init event, the ACTIVE→ERROR step pairs, agent_response steps with text_delta and usage, and the checkpoint and finish steps. Add a positive fixture shaped like the real stream (a trimmed copy of the real log structure is enough), plus negative cases: init not first, a duplicate init, an init cwd outside the workspace, an unknown step_type with an error, and a provider failure after a recovered error."
|
|
},
|
|
{
|
|
"severity": "low",
|
|
"message": "test_trusted_authorization_matrix only covers RP24-01. There is no positive or negative matrix case for the new RP24-02 pin, for example the RP24-02 project with the wrong base or spec hash, the wrong spec_ref, or the RP24-01 task id under the RP24-02 project. The code rejects these cases, but no test proves it."
|
|
},
|
|
{
|
|
"severity": "low",
|
|
"message": "The stderr permission-denied pattern now also matches 'operation not permitted' and 'access.*denied'. That is a reasonable fail-closed choice, but 'access.*denied' is broad and could reject benign provider log lines. No test covers the new patterns."
|
|
}
|
|
]
|
|
}
|