docs: pin model routing and current P24-02 acceptance gaps

2026-10-13 11:52:48 +09:00
parent 020d6b014e
commit a791117e98
24 changed files with 3369 additions and 10 deletions
@@ -0,0 +1,10 @@
+{
+  "id": "e61cce59721949aab3817d44e5eeeaa1",
+  "status": "failed",
+  "artifact": "/Users/korizatou114/Library/Application Support/agent-control/the-tree-runner-agy-execution/runs/e61cce59721949aab3817d44e5eeeaa1",
+  "pid": null,
+  "exit_code": null,
+  "formal_acceptance": false,
+  "process_started": false,
+  "reason": "Installed AGY CLI1.3.3; submitted adapter pins1.3.2. Compatibility probe rejected before provider subprocess launch. No repeated probe, no pin bypass or production adoption."
+}
@@ -0,0 +1,31 @@
+{
+  "task_id": "RP24-02",
+  "commit": "3fa7c0b1f982ecf2d55a772c909dd4aa0a0409ba",
+  "status": "changes_requested",
+  "summary": "I checked acceptance item RP24-02 against the spec, the diff, the full sources, the Runner validation result, and the coordinator's replay of a real run. Status: changes requested.\n\nWhat passed:\n- The scope stays within runner/core.py and tests/test_runner.py.\n- No tests were deleted or disabled. The formal suite ran 56 tests and all passed (51 baseline plus 5 new).\n- The trusted self-test pin for RP24-02 checks the frozen base commit and the spec hash, and pairing a task with the wrong project is rejected.\n- The inner `--sandbox` flag was removed from the Antigravity command line, and no `--dangerously-skip-permissions` was added.\n- The recovered-error audit is written to `recovered-read-errors.json` in the run artifact, alongside the stdout hash. The original stdout is not modified.\n\nWhy it is not a pass:\n1. The fix fails on the real data it was meant to handle. The coordinator replayed the actual run 2be30fb95a1b4898bc7be87ad072f8ec through `agy_result` and it failed with \"GateError: Antigravity unknown event\". Real Antigravity output begins with an `init` event, and `agy_result` rejects every event before the final result unless it is a `step_update`. The spec's central case (recovered missing-file read errors followed by a valid result) is therefore still rejected in production.\n2. The test fixtures are synthetic and do not match the real output shape. They have no `init` event and no ACTIVE, `agent_response`/`text_delta`, `checkpoint` or `finish` steps, which is why the suite is green while the real replay fails.\n3. Removing the inner sandbox is only checked by looking at the command line. The spec requires a real shell run under the actual Antigravity outer sandbox profile that succeeds within the allowed write scope, while writes outside it and credential reads stay denied. That test is missing.\n\nNot done yet, as the spec expects at this stage: the isolated bootstrap, the full host suite before production adoption, and integration/final validation (`production_adopted` is false).",
+  "acceptance_ids": [
+    "RP24-02"
+  ],
+  "findings": [
+    {
+      "severity": "high",
+      "message": "The real replay fails. Running the actual immutable stdout from run 2be30fb95a1b4898bc7be87ad072f8ec (input sha256 f7e5ad9f...) through agy_result raises 'GateError: Antigravity unknown event'. Real Antigravity streams start with an {\"event\":\"init\",...} line, and agy_result requires every event before the final result to be step_update. The spec's main case, five view_file ENOENT errors inside the workspace followed by SUCCESS/submitted, is still rejected. The fix should explicitly accept a single leading init event, after strictly checking its shape and that its cwd equals the task workspace, while still rejecting any other unknown event types."
+    },
+    {
+      "severity": "high",
+      "message": "There is no real outer-sandbox shell test for the Antigravity profile. The only check on removing --sandbox is test_agy_single_outer_boundary_argv, which inspects argv. The spec requires real tests under seatbelt(..., service='agy'): a bounded shell command such as git status in the worktree succeeds; writes inside the allowed paths succeed; writes to README.md, git metadata, CONTROL or provider-auth are denied; and Keychain/credential reads are denied. None of these exist. The observed failure ('sandbox_apply: Operation not permitted') has not been shown to be fixed under the outer profile."
+    },
+    {
+      "severity": "medium",
+      "message": "The recovered-read fixtures do not match the real stream format. test_agy_observed_recovered_read_and_rejects and test_agy_recovered_result_schema_and_identity leave out the init event, the ACTIVE→ERROR step pairs, agent_response steps with text_delta and usage, and the checkpoint and finish steps. Add a positive fixture shaped like the real stream (a trimmed copy of the real log structure is enough), plus negative cases: init not first, a duplicate init, an init cwd outside the workspace, an unknown step_type with an error, and a provider failure after a recovered error."
+    },
+    {
+      "severity": "low",
+      "message": "test_trusted_authorization_matrix only covers RP24-01. There is no positive or negative matrix case for the new RP24-02 pin, for example the RP24-02 project with the wrong base or spec hash, the wrong spec_ref, or the RP24-01 task id under the RP24-02 project. The code rejects these cases, but no test proves it."
+    },
+    {
+      "severity": "low",
+      "message": "The stderr permission-denied pattern now also matches 'operation not permitted' and 'access.*denied'. That is a reasonable fail-closed choice, but 'access.*denied' is broad and could reject benign provider log lines. No test covers the new patterns."
+    }
+  ]
+}
@@ -0,0 +1,31 @@
+{
+  "task_id": "RP24-02",
+  "commit": "09262f37c14397118d51b6b43c35ddbee14fcfa2",
+  "status": "changes_requested",
+  "summary": "受入項目RP24-02を、仕様・差分・ソース・Runner試験結果と照合しました。結論は差し戻しです。\n二重sandboxの除去は満たしています。外側Seatbeltは維持されており、`--sandbox`だけを外し、`--dangerously-skip-permissions`も追加していません。self-test pinの拡張も満たしています。対象はproject/task/base/spec SHAの完全一致に限られ、RP24-01の互換性と既存の拒否経路も残っています。Runner試験runner-unitは成功(57件)しています。\n一方、中核要件である「実際に観測された回復済みview_file ENOENTを、正当な結果として分類する」が実データで成立していません。coordinatorが実際のstdout(2be30fb9…, sha256 f7e5ad9f…)を再生したところ、提出コードは step_index 0 の `user_input`/DONE を `Antigravity unknown step` として拒否しました。このstdoutは、本タスクの起点になった実際の失敗ログです。\n新規試験は、この`user_input`ステップを含まない合成fixtureで書かれていたため、この欠陥を検出できませんでした。読み取り専用の修正案(許可するstep_typeに`user_input`を追加し、正例と負例の試験を追加)は再生に成功しています。ただし採用も正式検証もされていません。\nこのため、現在の提出コミットでは受入を満たしたと判断できません。",
+  "acceptance_ids": [
+    "RP24-02"
+  ],
+  "findings": [
+    {
+      "severity": "high",
+      "message": "中核要件が実データで未達です。agy_result(runner/core.py)が許可するstep_typeは (tool, agent_response, checkpoint, finish) だけです。実際のAntigravity出力の先頭 step_index 0 `user_input` DONE が拒否され、観測された5件のview_file ENOENTを含む元ログ(2be30fb9…)の再生が `Antigravity unknown step` で失敗します。次の2点を対応してください: (1) 通常の`user_input`(stateはDONEのみ、error・tool_infoなし)を許可する。(2) 実ログと同じ構造(各stepのconversation_id、step0のuser_input、ACTIVE→ERRORの対、結果レコードのresponse/structured_output)の正例fixtureをtests/test_runner.pyに追加する。"
+    },
+    {
+      "severity": "medium",
+      "message": "検証が不足しています。新規の正例fixture(test_agy_observed_recovered_read_and_rejects)は、実ログの構造を再現しない合成データでした。そのため、提出時点の57件成功は実際の転送形式で受入できることを示しません。observedの実ストリームと同じ形の正例と、user_inputがERRORの場合・tool_infoやerrorを持つ場合などの負例を追加してください。"
+    },
+    {
+      "severity": "medium",
+      "message": "権限拒否の検出がstderrに限られています。実ログでは、`run_command` が state=DONE のまま tool_info.output に `sandbox-exec: sandbox_apply: Operation not permitted` を返していました。現在のagy_resultは、DONEのtool出力に含まれる拒否文言を検査しません。sandbox除去後は再発しにくいものの、仕様の「permission/authorization系エラーは拒否を維持し、曖昧な結果を隠さない」に照らすと抜けがあります。tool出力中のsandbox_apply/Operation not permitted を拒否するか監査に記録し、その試験を追加することを推奨します。"
+    },
+    {
+      "severity": "low",
+      "message": "trusted_self_test では、SELF_TEST_PINS を参照した直後に RP24-01 の pin を再代入する分岐が重複しています。既存試験が SELF_TEST_BASE / SELF_TEST_SPEC_SHA を patch するための措置と読めますが、意図をコメントで明記するか、pinを毎回動的に組み立てる形に統一してください。"
+    },
+    {
+      "severity": "low",
+      "message": "実シェルの試験(test_agy_actual_outer_shell_and_boundaries)は、service=agy の外側profileで git status の成功、許可範囲への書き込み、範囲外の拒否を確認しており妥当です。ただし、実際のagy CLIからrun_commandを経由したシェル実行は含まれません。正式採用前に、root trusted host での実運用確認の結果を証跡として残してください。"
+    }
+  ]
+}
@@ -0,0 +1,66 @@
+{
+  "formal_acceptance": false,
+  "actual_healthy_stream_proven": false,
+  "proposal_adopted": false,
+  "synthetic_control": true,
+  "derived_from_original_sha256": "f7e5ad9fbaaf547816b83745d62b28bad893ac8ad0de324458d2ac4c6d6802c5",
+  "proposal_sha256": "0fa718fbe28e7e0c341b1718cde76838d43a9dadb00bc5f41f29a41074aad824",
+  "explicit_fixture_modifications": [
+    {
+      "record_index": 4,
+      "field": "step_update.tool_info.output",
+      "from": "sandbox-exec: sandbox_apply: Operation not permitted\r\n",
+      "to": "",
+      "reason": "Synthetic control only. Never classify this modified stream as actual healthy provider evidence."
+    }
+  ],
+  "positive_control_recovered_reads": 5,
+  "negative_cases": [
+    {
+      "case": "user_input_ERROR",
+      "rejected": true,
+      "reason": "Antigravity malformed user input"
+    },
+    {
+      "case": "user_input_tool_info",
+      "rejected": true,
+      "reason": "Antigravity malformed user input"
+    },
+    {
+      "case": "user_input_error",
+      "rejected": true,
+      "reason": "Antigravity malformed user input"
+    },
+    {
+      "case": "user_input_bool_index",
+      "rejected": true,
+      "reason": "Antigravity malformed user input"
+    },
+    {
+      "case": "unknown_step",
+      "rejected": true,
+      "reason": "Antigravity unknown step"
+    },
+    {
+      "case": "duplicate_init",
+      "rejected": true,
+      "reason": "Antigravity malformed init"
+    },
+    {
+      "case": "outside_init_cwd",
+      "rejected": true,
+      "reason": "Antigravity malformed init"
+    },
+    {
+      "case": "failed_terminal",
+      "rejected": true,
+      "reason": "Antigravity terminal result不足"
+    },
+    {
+      "case": "restore_actual_shell_denial",
+      "rejected": true,
+      "reason": "Antigravity shell permission denied"
+    }
+  ],
+  "remaining_required": "Fresh healthy actual CLI1.3.3 stream, full submitted-suite validation, independent Opus review and ordinary integration before adoption."
+}
@@ -0,0 +1,31 @@
+{
+  "scope": "Read-only proposal negative checks, not Runner formal validation",
+  "cases": [
+    {
+      "case": "user_input_error",
+      "rejected": true,
+      "error": "Antigravity failed step"
+    },
+    {
+      "case": "unknown_step",
+      "rejected": true,
+      "error": "Antigravity unknown step"
+    },
+    {
+      "case": "duplicate_init",
+      "rejected": true,
+      "error": "Antigravity malformed init"
+    },
+    {
+      "case": "outside_init_cwd",
+      "rejected": true,
+      "error": "Antigravity malformed init"
+    },
+    {
+      "case": "terminal_failure",
+      "rejected": true,
+      "error": "Antigravity terminal result不足"
+    }
+  ],
+  "passed": true
+}
@@ -0,0 +1,24 @@
+{
+  "formal_acceptance": false,
+  "proposal_adopted": false,
+  "worker_or_production_source_changed": false,
+  "proposal_source_sha256": "0fa718fbe28e7e0c341b1718cde76838d43a9dadb00bc5f41f29a41074aad824",
+  "actual_original_stream": {
+    "path": "/Users/korizatou114/Library/Application Support/agent-control/the-tree-p24-02-operations/runs/2be30fb95a1b4898bc7be87ad072f8ec/stdout.log",
+    "sha256": "f7e5ad9fbaaf547816b83745d62b28bad893ac8ad0de324458d2ac4c6d6802c5",
+    "rejected": true,
+    "reason": "Antigravity shell permission denied",
+    "shell_denials": [
+      {
+        "record_index": 4,
+        "step_index": 2,
+        "state": "DONE",
+        "output": "sandbox-exec: sandbox_apply: Operation not permitted\r\n",
+        "command_parameters": {
+          "CommandLine": "git status"
+        }
+      }
+    ]
+  },
+  "correction": "Original full stdout is not a positive fixture: it contains actual shell permission denial. Earlier user_input-only replay acceptance is insufficient. Do not remove denial events and describe the result as an actual positive stream. A fresh healthy actual stream and full negative replay are required."
+}
@@ -0,0 +1,26 @@
+--- a/runner/core.py
+++ b/runner/core.py
+@@ -107,12 +107,22 @@
+         require(not event.get("error"), "Antigravity provider error")
+         step = event.get("step_update")
+         require(isinstance(step, dict), "Antigravity malformed step")
+-        require(step.get("step_type") in ("tool", "agent_response", "checkpoint", "finish")
+        require(step.get("step_type") in ("user_input", "tool", "agent_response", "checkpoint", "finish")
+                 and step.get("state") in ("ACTIVE", "DONE", "ERROR"), "Antigravity unknown step")
+         require(not step.get("error"), "Antigravity provider error")
+        if step.get("step_type") == "user_input":
+            require(step.get("state") == "DONE" and "tool_info" not in step
+                    and "error" not in step and type(step.get("step_index")) is int
+                    and step["step_index"] == 0, "Antigravity malformed user input")
+            continue
+         info = step.get("tool_info", {})
+         require(isinstance(info, dict), "Antigravity malformed tool")
+         error = info.get("error")
+        if info.get("name") == "run_command" and "output" in info:
+            output = info["output"]
+            require(isinstance(output, str), "Antigravity malformed shell output")
+            require(not re.search(r"(?im)^\s*(?:sandbox-exec: sandbox_apply: Operation not permitted|(?:error: )?(?:permission denied|operation not permitted|access denied))\s*$", output),
+                    "Antigravity shell permission denied")
+         if error is None:
+             require(step.get("state") not in ("ERROR", "FAILED", "CANCELLED"), "Antigravity failed step")
+             continue
@@ -0,0 +1,14 @@
+{
+  "scope": "Read-only concrete proposal and actual transport replay, not formal acceptance. Production and worker sources untouched.",
+  "submitted_source": "09262f37c14397118d51b6b43c35ddbee14fcfa2",
+  "actual_original_error": "Antigravity unknown step: actual user_input DONE at step0",
+  "proposed_change": "Allow observed ordinary user_input step type; existing ERROR/failure/path/tool/result schema guards remain. Add positive real envelope fixture and negative error cases in worker.",
+  "source_before_sha256": "b17b51cd48cf45b56d45238043376e010f89472e3a1697f49022dec0d1b8fc31",
+  "proposal_sha256": "482f8ffd26e9456c6fe37891181db55b0941a022823e4fe2a30d1c1b3e626c46",
+  "input_sha256": "f7e5ad9fbaaf547816b83745d62b28bad893ac8ad0de324458d2ac4c6d6802c5",
+  "proposal_replay_passed": true,
+  "recovered_exact_errors": 5,
+  "result_schema_passed": true,
+  "runner_formal_acceptance": false,
+  "adopted": false
+}
@@ -0,0 +1,11 @@
+--- a/runner/core.py
+++ b/runner/core.py
+@@ -107,7 +107,7 @@
+         require(not event.get("error"), "Antigravity provider error")
+         step = event.get("step_update")
+         require(isinstance(step, dict), "Antigravity malformed step")
+-        require(step.get("step_type") in ("tool", "agent_response", "checkpoint", "finish")
+        require(step.get("step_type") in ("user_input", "tool", "agent_response", "checkpoint", "finish")
+                 and step.get("state") in ("ACTIVE", "DONE", "ERROR"), "Antigravity unknown step")
+         require(not step.get("error"), "Antigravity provider error")
+         info = step.get("tool_info", {})