Strengthen grading transparency and reproducibility evidence

README.md

@@ -11,158 +11,273 @@ tags:
 
 # Cloud DevOps RLEnv
 
-Cloud DevOps RLEnv is an OpenEnv benchmark for real incident-response workflows in cloud production systems.
+Cloud DevOps RLEnv is an OpenEnv-compatible cloud incident-response benchmark designed for agentic SRE and DevOps workflows.
 
-It is designed to evaluate whether an agent can triage noisy telemetry, identify root cause, apply a safe fix, and do all of this efficiently under cost and pressure.
+This environment rewards correct diagnosis and safe remediation, not blind action execution. It is deterministic, reproducible, and optimized for hackathon evaluation.
 
 ## Judge-Aligned Snapshot
 
-This section maps the environment directly to the scoring rubric.
-
 | Parameter | Weight | How this environment addresses it |
 | --- | --- | --- |
-| Real-world utility | 30% | Simulates practical SRE outage response loops: triage logs, map failing dependency, remediate safely, verify health. |
-| Task and grader quality | 25% | Three deterministic tasks (easy/medium/hard), explicit objectives, reproducible reward logic, strict success/failure gates. |
-| Environment design | 20% | Typed actions and observations, deterministic reset, shaped rewards, action cost, clear episode boundaries and timeout. |
-| Code quality and spec compliance | 15% | OpenEnv-compatible layout, typed models, Dockerized runtime, strict inference logging contract, local validators included. |
-| Creativity and novelty | 10% | Multi-hop metadata dependency, cascading failure drift, high-decoy infrastructure search, efficiency-aware reward shaping. |
+| Real-world utility | 30% | Models practical SRE outage response loops: telemetry triage, dependency mapping, and safe remediation. |
+| Task & grader quality | 25% | Three deterministic tasks with explicit objectives, strict success gates, and reproducible scoring behavior. |
+| Environment design | 20% | Typed action/observation/state models, clean reset semantics, shaped rewards, action-cost efficiency pressure, clear boundaries. |
+| Code quality & spec compliance | 15% | OpenEnv-compliant project layout, Dockerized runtime, strict inference output contract, validation scripts. |
+| Creativity & novelty | 10% | Multi-hop metadata dependency, cascading failures, high-decoy search space, and safety-aware penalties. |
+
+## Why This Environment
+
+Real incidents are multi-step and noisy. Good agents must:
+- gather context before changing systems
+- identify root cause from logs and topology
+- apply minimal, correct fixes
+- verify resolution
+
+Cloud DevOps RLEnv simulates that behavior with realistic failure patterns, decoy resources, shaped rewards, and anti-shortcut guardrails.
+
+## Why It's Hard
+
+This benchmark is intentionally designed to resist brute-force policies and reward disciplined SRE reasoning:
 
-## Real-World Utility
+- Needle-in-a-haystack discovery: 20+ decoy compute nodes and 20+ decoy security groups increase search complexity.
+- Ambiguous telemetry: noisy, raw operational logs surface symptoms (including IP-only clues) rather than direct root-cause labels.
+- Action-penalty heuristics: every action has a small negative cost, so efficient remediation beats command spamming.
+- Multi-hop dependency resolution: agents must map IP addresses to resource IDs via metadata lookup before applying fixes.
+- System drift under pressure: in hard mode, delayed remediation triggers cascading failures that worsen observability and reward dynamics.
 
-The benchmark models incidents that closely mirror day-2 operations in production:
+## Environment Scope
 
-- Security group misconfiguration blocks service traffic.
-- Service-to-database communication fails with telemetry-first diagnosis required.
-- Load balancer upstream failures require dependency mapping and targeted restart under time pressure.
+- Domain: Cloud SRE / DevOps incident response
+- Difficulty tiers: easy, medium, hard
+- Max environment steps per episode: 20
+- Runtime health states: CRITICAL, DEGRADED, HEALTHY
+- Decoy resources: 20 backend instances + 20 backend security groups
+
+## OpenEnv Compliance
+
+Core files:
+- openenv.yaml
+- env.py
+- models.py
+- inference.py
+- server/app.py
+- server/cloud_devops_env_environment.py
 
-This is not a toy command simulator. The hard task requires reasoning over partial evidence and acting safely under drift.
+Validator command:
 
-## Why It Is Hard
+```bash
+..\\.venv\\Scripts\\openenv validate
+```
 
-- Needle-in-a-haystack discovery: 20+ decoy instances and 20+ decoy security groups.
-- Ambiguous telemetry: logs expose symptoms and IPs, not always direct resource names.
-- Action-cost pressure: every action incurs a negative reward, penalizing brute-force behavior.
-- Multi-hop dependency: agent must use metadata resolution before remediation in medium/hard.
-- Cascading failures: unresolved hard incidents degrade additional components after step 8.
+## Action Space
 
-## Task Suite And Difficulty Progression
+Model: CloudAction
 
-| Task | Objective | Required reasoning depth | Typical strong trajectory |
+| Field | Type | Required | Description |
 | --- | --- | --- | --- |
-| easy | Restore web access by opening port 80 on sg-web | Single-hop config diagnosis | list_resources -> update_security_group(sg-web, allow 80) |
-| medium | Restore DB connectivity by opening port 5432 on sg-db | Multi-hop: logs -> IP -> metadata lookup -> fix | list_resources -> view_logs(i-api) -> query_metadata(10.0.4.5) -> update_security_group(sg-db, allow 5432) |
-| hard | Recover checkout path by fixing failing upstream i-web2 | Multi-hop + pressure: logs -> IP -> metadata -> inspect -> restart | list_resources -> view_logs(lb-main) -> query_metadata(10.0.8.22) -> describe/view i-web2 -> restart_service(i-web2) |
+| command | enum | yes | One of: list_resources, describe_resource, view_logs, query_metadata, update_security_group, restart_service, submit_solution |
+| resource_id | string | conditional | Required for most actions except list_resources and query_metadata |
+| parameters | object | conditional | Used by mutating actions (for example, security-group updates) |
 
-## Grader Quality And Determinism
+Action semantics:
+- list_resources: Enumerates available resources including decoys.
+- describe_resource: Returns structured details for one resource.
+- view_logs: Returns logs for one resource.
+- query_metadata: Resolves infrastructure metadata (for example, IP address to resource ID).
+- update_security_group: Appends a rule (requires parameters.port and parameters.action where action is allow/deny).
+- restart_service: Restarts one instance/service by ID.
+- submit_solution: Declares the episode solved (or not solved).
 
-- Deterministic state: no RNG in task initialization.
-- Deterministic transitions: action handling is rule-based and reproducible.
-- Deterministic rewards: shaped by explicit achievement checkpoints and safety penalties.
-- Deterministic success: inferred from resolved incident state, not ad hoc heuristics.
+## Observation And State Space
 
-Score behavior:
+Observation model: CloudObservation
 
-- Step reward is clipped to [-1.0, 1.0].
-- Per-task inference score is clamped to strict open interval (0, 1) for validator compatibility.
-- Lower-step successful solutions naturally score higher due to per-step action cost.
+| Field | Description |
+| --- | --- |
+| output | Main command output |
+| error | Error string for failed commands |
+| system_health_status | CRITICAL, DEGRADED, HEALTHY |
+| done | Episode terminal flag |
+| reward | Step reward |
+| metadata | Diagnostics such as task, step_count, resolved, achievements |
 
-## Environment Design Details
+Hidden state model: CloudState
 
-- Runtime health states: CRITICAL, DEGRADED, HEALTHY.
-- Episode limit: 20 environment steps.
-- Hard-task drift: if unresolved after step 8, lb-external is marked DOWN.
-- Action cost: -0.01 on every step to encourage efficient plans.
+| Field | Description |
+| --- | --- |
+| task_difficulty | easy, medium, hard |
+| resources | Full resource graph including logs/rules |
+| step_count | Current step counter |
+| is_resolved | Whether root cause has been fixed |
 
-### Action Space (CloudAction)
+## Reward Design
 
-| Field | Type | Required | Notes |
-| --- | --- | --- | --- |
-| command | enum | yes | list_resources, describe_resource, view_logs, query_metadata, update_security_group, restart_service, submit_solution |
-| resource_id | string | conditional | Required for most commands except list_resources and query_metadata |
-| parameters | object | conditional | Required for query_metadata.ip_address and update_security_group.port/action |
+Reward shaping is sparse-but-guided:
+- discovery rewards for correct investigative steps
+- larger terminal rewards for correct remediation
+- penalties for unsafe or premature operations
+- fixed action cost per step (efficiency pressure)
+- timeout terminal condition after max steps
 
-Security-group mutation semantics:
+Per-step reward is clipped to [-1.0, 1.0].
+Inference task score is adjusted to remain strictly within (0.0, 1.0) for Phase-2 validator compatibility.
 
-- update_security_group requires both port and action.
-- action must be allow or deny.
-- Invalid actions are rejected and do not mutate state.
+## Detailed Task Playbooks
 
-### Observation and State Models
+### Easy Task
 
-Observation (CloudObservation): output, error, system_health_status, reward, done, metadata.
+Incident:
+- Web traffic blocked by security group.
 
-State (CloudState): task_difficulty, resources, step_count, is_resolved.
+Objective:
+- Open port 80 on sg-web.
 
-## Inference Contract And Compliance
+Typical successful sequence:
+1. list_resources
+2. describe_resource(sg-web) for context (+0.2)
+3. update_security_group(sg-web, port=80, action=allow) (+0.8, done)
 
-Mandatory environment variables:
+Expected score:
+- ~0.97 for full playbook with efficient triage
+- ~0.79 if agent skips the optional read step
 
-- API_BASE_URL
-- MODEL_NAME
-- HF_TOKEN
+### Medium Task
 
-Mandatory inference requirements:
+Incident:
+- API cannot reach DB due to blocked port 5432.
 
-- inference.py is at repository root.
-- Uses OpenAI client for all LLM calls.
-- Emits strict stdout contract only:
+Objective:
+- Confirm root cause from logs, then open port 5432 on sg-db.
 
-```text
-[START] task=<task_name> env=<benchmark> model=<model_name>
-[STEP] step=<n> action=<action_json_or_str> reward=<0.00> done=<true|false> error=<msg|null>
-[END] success=<true|false> steps=<n> score=<score> rewards=<r1,r2,...,rn>
+Typical successful sequence:
+1. list_resources
+2. view_logs(i-api) to identify DB timeout (+0.2)
+3. query_metadata(ip_address=10.0.4.5) to resolve DB target (+0.2)
+4. update_security_group(sg-db, port=5432, action=allow) (+0.6, done if logs and metadata lookup were completed)
+
+Guardrail:
+- Applying the SG change before log triage + metadata lookup gives a penalty (-0.1) and does not close the incident.
+
+Expected score:
+- ~0.97 with full investigative path (logs -> metadata lookup -> remediation)
+- below ~0.90 when metadata dependency is skipped
+
+## Determinism And Grader Transparency
+
+- Deterministic reset and transitions: no randomization is used in task generation or transition logic.
+- Transparent grading signals: observation metadata includes achievements, resolution status, termination reason, and reward breakdown events.
+- Reproducibility helper:
+
+```bash
+..\\.venv\\Scripts\\python scripts/reproducibility_check.py
 ```
 
-## Reference Baseline Behavior
+### Hard Task
 
-Recent deterministic policy-style run with current environment dynamics:
+Incident:
+- Checkout path degraded due to upstream timeout to an IP-only target that must be resolved first.
 
-| Task | Steps | Score | Outcome |
-| --- | --- | --- | --- |
-| easy | 2 | 0.780 | success=true |
-| medium | 4 | 0.960 | success=true |
-| hard | 5 | 0.999 | success=true |
+Objective:
+- Trace LB errors to the correct target, resolve resource identity via metadata, and restart i-web2 only after diagnosis.
 
-## Project Layout And Spec Files
+Typical successful sequence:
+1. list_resources
+2. view_logs(lb-main) to identify failing upstream IP (+0.2)
+3. query_metadata(ip_address=<failing_ip>) to resolve target ID (+0.2)
+4. describe_resource(i-web2) or view_logs(i-web2) (+0.2)
+5. restart_service(i-web2) (+0.8, done when all investigation achievements exist)
 
-Required OpenEnv files are present:
+Guardrails:
+- Restarting i-web2 before investigation: penalty (-0.1), no resolution.
+- Restarting healthy i-web1: penalty (-0.2).
+- Premature submit_solution in hard mode: penalty (-0.1), episode continues.
+- If unresolved after step 8 in hard mode, lb-external also fails (cascading failure), increasing pressure and noise.
 
-- openenv.yaml
-- env.py
-- models.py
-- inference.py
-- server/app.py
-- server/cloud_devops_env_environment.py
+Expected score:
+- near 1.0 after score clamping for strong trajectories (can exceed 1.0 raw before clamp)
+
+## API Endpoints
+
+Core runtime:
+- GET /health
+- POST /reset
+- POST /step
+- GET /state
+- GET /schema
+- WS /ws
+
+Web UI runtime:
+- GET /web
+- POST /web/reset
+- POST /web/step
+- GET /web/state
+- GET /web/metadata
+
+## Inference Contract
+
+inference.py requirements:
+- uses OpenAI client
+- reads API_BASE_URL, MODEL_NAME, HF_TOKEN
+- emits strict logs: [START], [STEP], [END]
 
-## Validation And Reproducibility
+Current defaults in code:
+- MODEL_NAME default: google/gemma-4-26B-A4B-it
+- MAX_STEPS (in inference loop): 15
+- success flag is derived from environment resolution state
+
+## Baselines
+
+### Deterministic task baselines
+
+| Task | Typical baseline score |
+| --- | --- |
+| easy | 0.78 |
+| medium | 0.96 |
+| hard | 0.999 |
+
+### LLM policy comparison
+
+| Model | Easy | Medium | Summary |
+| --- | --- | --- | --- |
+| gemma-3-27b-it | 0.2 | 0.2 | Underperformed on this environment |
+| gemma-4-31b-it | 1.0 | 1.0 | Perfect on both easy and medium |
+
+## Local Setup And Validation
 
 From repository root:
 
 ```bash
-# OpenEnv manifest and schema checks
+# Structure + manifest validation
 ..\\.venv\\Scripts\\openenv validate
 
-# Full submission checks including inference contract
-bash scripts/pre_submit_validate.sh --ping-url https://<your-space>.hf.space
+# Determinism/reproducibility smoke test
+..\\.venv\\Scripts\\python scripts/reproducibility_check.py
+
+# Submission-oriented local checks (without live inference)
+bash scripts/pre_submit_validate.sh --skip-inference
+
+# Build local image
+docker build -t cloud-devops-env:phase1 -f Dockerfile .
+```
 
-# Official 3-step baseline validator
-bash scripts/validate-submission.sh https://<your-space>.hf.space .
+Optional local server:
 
-# Docker smoke test
-docker build -t cloud-devops-env:latest -f Dockerfile .
-docker run --rm -p 8000:8000 cloud-devops-env:latest
+```bash
+uvicorn server.app:app --host 0.0.0.0 --port 8000
 ```
 
-## Hugging Face Space Deployment Checklist
+## Hugging Face Space Deployment
 
-1. Keep front matter intact, including openenv tag and app_port 8000.
-2. Push latest main branch to the Space repository.
-3. Configure API_BASE_URL, MODEL_NAME, HF_TOKEN in Space settings.
-4. Wait for build completion and verify endpoints:
-   - GET /health -> 200
-   - POST /reset -> 200
-5. Run inference.py once in Space logs to confirm strict START/STEP/END output.
+1. Keep this front matter block intact (includes mandatory openenv tag).
+2. Push to Space (Docker SDK).
+3. Configure secrets/variables:
+   - HF_TOKEN
+   - API_BASE_URL (for example https://router.huggingface.co/v1)
+   - MODEL_NAME
+4. Wait for build completion.
+5. Verify:
+   - GET /health returns 200
+   - POST /reset returns 200
 
 Reference:
-
 - https://huggingface.co/docs/hub/spaces-config-reference

WEB_README.md

@@ -1,172 +1,99 @@
-# Cloud DevOps RLEnv Web Playbook
+# Cloud DevOps RLEnv
 
-Use this page as your operator guide when stepping through incidents in /web.
+Use this page as the operator playbook in the /web interface.
 
-## Mission
-
-You are the on-call SRE for a production outage.
-
-Your loop is always:
+## Core Loop
 
 1. Triage quickly.
-2. Confirm root cause.
-3. Apply the minimum safe fix.
-4. Verify health and close.
-
-## What Makes This Environment Non-Trivial
-
-- Large decoy inventory: many resources are intentionally irrelevant.
-- Ambiguous telemetry: medium/hard logs expose failing IPs, not only direct IDs.
-- Action cost: every step incurs a small penalty, so efficient plans outperform exploratory spam.
-- Hard-mode drift: delayed remediation causes additional system degradation.
-
-## Command Reference
-
-### list_resources
-
-- Purpose: enumerate all available entities.
-- Use first in most runs.
-- Required fields: command only.
-
-### describe_resource
-
-- Purpose: inspect one resource configuration or current state.
-- Required fields: resource_id.
-
-### view_logs
-
-- Purpose: read operational telemetry for one resource.
-- Required fields: resource_id.
-
-### query_metadata
-
-- Purpose: resolve metadata such as ip_address -> resource_id.
-- Required fields: parameters.ip_address.
-- Typical medium/hard bridge action before mutation.
-
-### update_security_group
+2. Build evidence from logs and metadata.
+3. Apply the minimum safe remediation.
+4. Verify resolution and stop.
 
-- Purpose: mutate network policy.
-- Required fields: resource_id, parameters.port, parameters.action.
-- Valid action values: allow, deny.
+Why this matters: each action has a cost (`-0.01`), so shorter correct trajectories score higher.
 
-### restart_service
+## Command Cheat Sheet
 
-- Purpose: restart a target service/instance.
-- Required fields: resource_id.
-- Use only after root-cause confirmation.
+- `list_resources`
+: Start here to map real targets vs decoys.
 
-### submit_solution
+- `describe_resource(resource_id)`
+: Inspect config/state details.
 
-- Purpose: attempt to close incident.
-- Hard mode may continue with penalty if unresolved.
+- `view_logs(resource_id)`
+: Primary source of root-cause evidence.
 
-## JSON Action Examples
+- `query_metadata(parameters={"ip_address": "..."})`
+: Resolve IP-only clues to resource IDs (mandatory multi-hop step in medium/hard).
 
-```json
-{"command":"list_resources"}
-```
+- `update_security_group(resource_id, parameters)`
+: Requires `parameters.port` and `parameters.action` where action is `allow` or `deny`.
 
-```json
-{"command":"view_logs","resource_id":"i-api"}
-```
+- `restart_service(resource_id)`
+: Use only after causal evidence confirms target.
 
-```json
-{"command":"query_metadata","parameters":{"ip_address":"10.0.4.5"}}
-```
-
-```json
-{"command":"update_security_group","resource_id":"sg-db","parameters":{"port":5432,"action":"allow"}}
-```
-
-```json
-{"command":"restart_service","resource_id":"i-web2"}
-```
+- `submit_solution`
+: Useful for explicit closure checks; unresolved hard submissions are penalized and continue.
 
 ## Task Playbooks
 
-### Easy: Web Access Recovery
+### Easy
 
 Objective:
-
-- Open port 80 on sg-web.
+- Restore web access by allowing port `80` on `sg-web`.
 
 Strong path:
+1. `list_resources`
+2. `update_security_group("sg-web", {"port": 80, "action": "allow"})`
 
-1. list_resources
-2. update_security_group(sg-web, port=80, action=allow)
-
-Optional safer path (slightly lower efficiency):
+Typical outcome:
+- 2 steps, score around `0.78`
 
-1. list_resources
-2. describe_resource(sg-web)
-3. update_security_group(sg-web, port=80, action=allow)
-
-### Medium: API to DB Connectivity
+### Medium
 
 Objective:
-
-- Restore DB access by opening 5432 on sg-db, but only after diagnosis.
+- Restore API to DB connectivity by allowing port `5432` on `sg-db`.
 
 Strong path:
-
-1. list_resources
-2. view_logs(i-api)
-3. query_metadata(ip_address=10.0.4.5)
-4. update_security_group(sg-db, port=5432, action=allow)
+1. `list_resources`
+2. `view_logs("i-api")`
+3. `query_metadata({"ip_address": "10.0.4.5"})`
+4. `update_security_group("sg-db", {"port": 5432, "action": "allow"})`
 
 Guardrail:
+- Applying SG changes before logs + metadata lookup is penalized and does not resolve the incident.
 
-- Mutating sg-db before logs + metadata lookup is penalized and does not resolve.
-
-### Hard: Upstream Failure Under Drift
+### Hard
 
 Objective:
-
-- Find failing upstream from lb-main logs, resolve target identity, inspect, then restart i-web2.
+- Recover checkout flow by identifying the failing upstream and restarting `i-web2` safely.
 
 Strong path:
-
-1. list_resources
-2. view_logs(lb-main)
-3. query_metadata(ip_address=10.0.8.22)
-4. describe_resource(i-web2) or view_logs(i-web2)
-5. restart_service(i-web2)
+1. `list_resources`
+2. `view_logs("lb-main")`
+3. `query_metadata({"ip_address": "10.0.8.22"})`
+4. `describe_resource("i-web2")` or `view_logs("i-web2")`
+5. `restart_service("i-web2")`
 
 Guardrails:
+- Restarting `i-web1` is penalized.
+- Restarting `i-web2` without investigation is penalized.
+- If unresolved after step 8, `lb-external` also fails (cascading failure).
 
-- Restarting i-web1 is penalized.
-- Restarting i-web2 before investigation is penalized and blocked from resolution.
-- If unresolved after step 8, lb-external also fails (cascading failure).
-
-## Rewards, Termination, And Score Semantics
+## Reading Metadata
 
-- Every action includes a small cost.
-- Discovery and correct remediation provide shaped positive rewards.
-- Unsafe or premature actions apply penalties.
-- Episode terminates on resolution or max step limit.
-- Submission score is clamped to a strict open interval (0,1).
-- success reflects actual incident resolution state.
+Watch these response fields each step:
 
-## Observation Fields To Monitor
-
-- output: command result text
-- error: command-level error string
-- system_health_status: CRITICAL, DEGRADED, HEALTHY
-- reward: step reward
-- done: episode termination flag
-- metadata: task, step_count, resolved, achievements, action_cost
+- `system_health_status`: `CRITICAL` / `DEGRADED` / `HEALTHY`
+- `done`: episode ended or still running
+- `reward`: immediate signal after action cost and shaping
+- `metadata.resolved`: authoritative success flag
+- `metadata.termination_reason`: why episode ended
+- `metadata.reward_breakdown`: transparent reward events for grader/debug inspection
 
 ## Submission Contract Reminder
 
-inference.py must:
+`inference.py` must:
 
 - use OpenAI client
-- read API_BASE_URL, MODEL_NAME, HF_TOKEN
-- emit strict stdout lines with START, STEP, END markers
-
-Practical strategy:
-
-- Inspect first, mutate second.
-- Use query_metadata whenever logs give only IP clues.
-- Prefer minimal actions for higher score.
+- read `API_BASE_URL`, `MODEL_NAME`, `HF_TOKEN`
+- emit strict stdout markers: `[START]`, `[STEP]`, `[END]`

scripts/reproducibility_check.py

@@ -0,0 +1,125 @@
+#!/usr/bin/env python
+"""Deterministic reproducibility smoke test for Cloud DevOps RLEnv."""
+
+from __future__ import annotations
+
+import asyncio
+import json
+import sys
+from pathlib import Path
+from typing import Any
+
+REPO_ROOT = Path(__file__).resolve().parent.parent
+if str(REPO_ROOT) not in sys.path:
+    sys.path.insert(0, str(REPO_ROOT))
+
+from env import CloudDevOpsEnv
+from models import CloudAction
+
+SCORE_MIN = 0.001
+SCORE_MAX = 0.999
+
+# Fixed trajectories that should always resolve the incidents.
+POLICY_BY_TASK: dict[str, list[dict[str, Any]]] = {
+    "easy": [
+        {"command": "list_resources"},
+        {
+            "command": "update_security_group",
+            "resource_id": "sg-web",
+            "parameters": {"port": 80, "action": "allow"},
+        },
+    ],
+    "medium": [
+        {"command": "list_resources"},
+        {"command": "view_logs", "resource_id": "i-api"},
+        {
+            "command": "query_metadata",
+            "parameters": {"ip_address": "10.0.4.5"},
+        },
+        {
+            "command": "update_security_group",
+            "resource_id": "sg-db",
+            "parameters": {"port": 5432, "action": "allow"},
+        },
+    ],
+    "hard": [
+        {"command": "list_resources"},
+        {"command": "view_logs", "resource_id": "lb-main"},
+        {
+            "command": "query_metadata",
+            "parameters": {"ip_address": "10.0.8.22"},
+        },
+        {"command": "describe_resource", "resource_id": "i-web2"},
+        {"command": "restart_service", "resource_id": "i-web2"},
+    ],
+}
+
+
+async def run_policy(task_name: str) -> dict[str, Any]:
+    env = CloudDevOpsEnv(task_name=task_name)
+    await env.reset()
+
+    trajectory: list[dict[str, Any]] = []
+    rewards: list[float] = []
+    last = None
+
+    try:
+        for index, raw_action in enumerate(POLICY_BY_TASK[task_name], start=1):
+            action = CloudAction(**raw_action)
+            result = await env.step(action)
+            rewards.append(float(result.reward))
+            trajectory.append(
+                {
+                    "step": index,
+                    "command": action.command,
+                    "resource_id": action.resource_id,
+                    "reward": round(float(result.reward), 4),
+                    "done": bool(result.done),
+                    "error": result.observation.error,
+                    "status": result.observation.system_health_status,
+                    "resolved": bool(result.info.get("resolved", False)),
+                }
+            )
+            last = result
+            if result.done:
+                break
+
+        if last is None:
+            raise RuntimeError(f"Task {task_name} produced no steps")
+
+        score = max(SCORE_MIN, min(sum(rewards), SCORE_MAX))
+        return {
+            "task": task_name,
+            "resolved": bool(last.info.get("resolved", False)),
+            "steps": len(trajectory),
+            "score": round(score, 3),
+            "trajectory": trajectory,
+        }
+    finally:
+        await env.close()
+
+
+async def main() -> None:
+    summary: dict[str, Any] = {}
+
+    for task_name in ("easy", "medium", "hard"):
+        run_1 = await run_policy(task_name)
+        run_2 = await run_policy(task_name)
+
+        if run_1["trajectory"] != run_2["trajectory"]:
+            raise SystemExit(f"Determinism check failed for task={task_name}: trajectories differ")
+        if not run_1["resolved"]:
+            raise SystemExit(f"Policy failed to resolve task={task_name}")
+
+        summary[task_name] = {
+            "steps": run_1["steps"],
+            "score": run_1["score"],
+            "resolved": run_1["resolved"],
+        }
+
+    print("Deterministic reproducibility check passed")
+    print(json.dumps(summary, indent=2, sort_keys=True))
+
+
+if __name__ == "__main__":
+    asyncio.run(main())

server/cloud_devops_env_environment.py

@@ -217,6 +217,20 @@ class CloudDevopsEnvironment(Environment):
         self._achievements.add(achievement)
         return points
 
+    def _task_objective(self) -> str:
+        objectives = {
+            "easy": "Restore web access by allowing port 80 on sg-web.",
+            "medium": (
+                "Restore API to DB connectivity by reading i-api logs, resolving DB IP via "
+                "query_metadata, then allowing port 5432 on sg-db."
+            ),
+            "hard": (
+                "Recover checkout path by tracing lb-main upstream IP, resolving it with "
+                "query_metadata, inspecting i-web2, and restarting i-web2 safely."
+            ),
+        }
+        return objectives[self.task_name]
+
     def reset(self) -> CloudObservation:  # type: ignore[override]
         """Reset the environment to the initial state for the selected task."""
         self._achievements.clear()
@@ -242,6 +256,11 @@ class CloudDevopsEnvironment(Environment):
                 "resolved": False,
                 "task": self.task_name,
                 "total_resources": len(self._state_data.resources),
+                "objective": self._task_objective(),
+                "deterministic": True,
+                "max_steps": self.MAX_STEPS,
+                "action_cost": self.ACTION_COST,
+                "hard_cascade_trigger_step": 8,
             },
             echoed_message="Cloud Devops Env environment ready!",
             message_length=0,
@@ -257,9 +276,20 @@ class CloudDevopsEnvironment(Environment):
 
         state.step_count += 1
         reward = -self.ACTION_COST
+        reward_breakdown: list[dict[str, object]] = [
+            {"event": "action_cost", "delta": -self.ACTION_COST}
+        ]
         done = False
         output = ""
         error = None
+        termination_reason = "in_progress"
+
+        def add_reward(delta: float, event: str) -> None:
+            nonlocal reward
+            if abs(delta) < 1e-12:
+                return
+            reward += delta
+            reward_breakdown.append({"event": event, "delta": round(float(delta), 4)})
 
         try:
             if action.command == "list_resources":
@@ -276,11 +306,14 @@ class CloudDevopsEnvironment(Environment):
                 output = str(state.resources[action.resource_id])
 
                 if self.task_name == "easy" and action.resource_id == "sg-web":
-                    reward += self._reward_once("read_sg", 0.2)
+                    add_reward(self._reward_once("read_sg", 0.2), "inspect_web_sg")
                 elif self.task_name == "medium" and action.resource_id == "sg-db":
-                    reward += self._reward_once("read_sg", 0.2)
+                    add_reward(self._reward_once("read_sg", 0.2), "inspect_db_sg")
                 elif self.task_name == "hard" and action.resource_id == "i-web2":
-                    reward += self._reward_once("inspect_target", 0.2)
+                    add_reward(
+                        self._reward_once("inspect_target", 0.2),
+                        "inspect_target_instance",
+                    )
 
             elif action.command == "view_logs":
                 if not action.resource_id:
@@ -293,11 +326,14 @@ class CloudDevopsEnvironment(Environment):
                 output = str(res.get("logs", "No logs available for this resource."))
 
                 if self.task_name == "medium" and action.resource_id == "i-api":
-                    reward += self._reward_once("read_logs", 0.2)
+                    add_reward(self._reward_once("read_logs", 0.2), "inspect_api_logs")
                 elif self.task_name == "hard" and action.resource_id == "lb-main":
-                    reward += self._reward_once("inspect_lb", 0.2)
+                    add_reward(self._reward_once("inspect_lb", 0.2), "inspect_lb_logs")
                 elif self.task_name == "hard" and action.resource_id == "i-web2":
-                    reward += self._reward_once("inspect_target", 0.2)
+                    add_reward(
+                        self._reward_once("inspect_target", 0.2),
+                        "inspect_target_logs",
+                    )
 
             elif action.command == "query_metadata":
                 ip_address = None
@@ -314,9 +350,15 @@ class CloudDevopsEnvironment(Environment):
 
                 output = f"Metadata lookup: ip_address={ip_address} resource_id={resource_id}"
                 if self.task_name == "medium" and str(ip_address) == "10.0.4.5":
-                    reward += self._reward_once("lookup_db_target", 0.2)
+                    add_reward(
+                        self._reward_once("lookup_db_target", 0.2),
+                        "resolve_db_ip_dependency",
+                    )
                 elif self.task_name == "hard" and str(ip_address) == "10.0.8.22":
-                    reward += self._reward_once("lookup_upstream_target", 0.2)
+                    add_reward(
+                        self._reward_once("lookup_upstream_target", 0.2),
+                        "resolve_upstream_ip_dependency",
+                    )
 
             elif action.command == "update_security_group":
                 if not action.resource_id:
@@ -350,8 +392,9 @@ class CloudDevopsEnvironment(Environment):
                     and rule_action == "allow"
                 ):
                     state.is_resolved = True
-                    reward += 0.8
+                    add_reward(0.8, "resolve_easy_web_ingress")
                     done = True
+                    termination_reason = "resolved_easy"
                     output += "\nSUCCESS: Web server is now accessible!"
                 elif (
                     self.task_name == "medium"
@@ -365,17 +408,18 @@ class CloudDevopsEnvironment(Environment):
                     )
                     if investigated:
                         state.is_resolved = True
-                        reward += 0.6
+                        add_reward(0.6, "resolve_medium_db_connectivity")
                         done = True
+                        termination_reason = "resolved_medium"
                         output += "\nSUCCESS: Database connection restored!"
                     else:
-                        reward -= 0.1
+                        add_reward(-0.1, "unsafe_change_without_triage")
                         output += (
                             "\nWARNING: Change applied without incident triage. "
                             "Inspect API logs and resolve DB IP via query_metadata before closing the incident."
                         )
                 elif rule_action == "deny":
-                    reward -= 0.1
+                    add_reward(-0.1, "deny_rule_during_incident")
                     output += "\nWARNING: Deny rule applied during outage remediation."
 
             elif action.command == "restart_service":
@@ -399,17 +443,18 @@ class CloudDevopsEnvironment(Environment):
                                 "logs"
                             ] = "INFO: Restart successful. Memory cleared."
                             state.is_resolved = True
-                            reward += 0.8
+                            add_reward(0.8, "resolve_hard_upstream_recovery")
                             done = True
+                            termination_reason = "resolved_hard"
                             output += "\nSUCCESS: OutOfMemory loop broken. System stable."
                         else:
-                            reward -= 0.1
+                            add_reward(-0.1, "restart_without_root_cause")
                             output += (
                                 "\nWARNING: Restart denied by change policy. "
                                 "Find failing upstream IP from lb-main, resolve it with query_metadata, and inspect i-web2 first."
                             )
                     elif action.resource_id == "i-web1":
-                        reward -= 0.2
+                        add_reward(-0.2, "restart_healthy_node")
                         output += (
                             "\nWARNING: You restarted a healthy production server! "
                             "Users dropped."
@@ -418,18 +463,20 @@ class CloudDevopsEnvironment(Environment):
             elif action.command == "submit_solution":
                 if state.is_resolved:
                     done = True
+                    termination_reason = "resolved_submit_solution"
                     output = "Solution verified. System is HEALTHY."
                 else:
                     if self.task_name == "hard":
                         # In hard mode, unresolved submission should not abort the run.
                         done = False
-                        reward -= 0.1
+                        add_reward(-0.1, "premature_submit_hard")
                         output = (
                             "Solution incorrect. Incident is still CRITICAL. "
                             "Continue triage and remediation before submitting."
                         )
                     else:
                         done = True
+                        termination_reason = "incorrect_submit"
                         output = "Solution incorrect. System is still CRITICAL."
 
             else:
@@ -440,16 +487,26 @@ class CloudDevopsEnvironment(Environment):
             output = f"Command Failed: {error}"
 
         cascade_penalty, cascade_msg = self._apply_cascading_failure()
-        reward += cascade_penalty
+        add_reward(cascade_penalty, "cascading_failure_penalty")
         if cascade_msg:
             output = f"{output}{cascade_msg}" if output else cascade_msg.strip()
 
         if state.step_count >= self.MAX_STEPS and not done:
             done = True
+            termination_reason = "max_steps_timeout"
             timeout_suffix = "\nTIMEOUT: Max steps reached."
             output = f"{output}{timeout_suffix}" if output else timeout_suffix.strip()
 
+        raw_reward = reward
         reward = max(-1.0, min(1.0, reward))
+        if reward != raw_reward:
+            reward_breakdown.append(
+                {
+                    "event": "reward_clip",
+                    "delta": round(float(reward - raw_reward), 4),
+                }
+            )
+
         lb_external = state.resources.get("lb-external", {})
         if state.is_resolved:
             status = "HEALTHY"
@@ -464,6 +521,11 @@ class CloudDevopsEnvironment(Environment):
             "achievements": sorted(self._achievements),
             "total_resources": len(state.resources),
             "action_cost": self.ACTION_COST,
+            "objective": self._task_objective(),
+            "deterministic": True,
+            "max_steps": self.MAX_STEPS,
+            "termination_reason": termination_reason if done else "in_progress",
+            "reward_breakdown": reward_breakdown,
         }
 
         return CloudObservation(