GeMri98/hvac-twin-agent-v5

HVAC twin agent: one 3B LoRA that runs a thermal twin as a tool and answers from NCC Section J

Adapters: adapter_v5_1 (twin tool use only), adapter_v6 (twin tool use + grounded NCC answers; recommended). Also in this repo: rag_index/ (retrieval index), rag_train.jsonl, rag_test.jsonl, qa_*_v51.jsonl.

What it does

• Twin tool use: the model writes <call>{"tool":"simulate","cooling_pct":N}</call>; a 2R2C office-zone thermal twin runs; the model answers from the results.
• NCC answers: from retrieved NCC 2022 Vol 1 Section J passages (J1, J2, J6, J9), with [S#] citations, or "Not found in the provided sources."
• Checker (in the pipeline, not the weights): every answer sentence is checked against the retrieved passages and every number against the data or twin results; failures are redrafted once, then abstained.

Results (small, single-run; read the limits)

Test	Result
Twin tool use, seen question types, strict grading (v5.1)	93.3% normal day, 89.7% heatwave
Twin tool use, UNSEEN question type ("full cooling")	about 20% - no real generalisation
v6 keeps the <call> protocol (first turn matches gold call, 16 held-out rows)	14/16 (same as v5.1)
NCC held-out rows, grounded + cited (12): plain Qwen / v5.1 / v6	9 / 3 / 10
20 hand-posed NCC questions (16 answerable, 4 not answerable), with checker	12 answered, 4 abstained; 4/4 unanswerable refused; 0 clearly wrong on review
Twin numbers in the agent demo	matched a direct run of the twin exactly

Limits (important)

• v6 is not better than plain Qwen at reading the NCC; it is better at citing consistently and refusing, and it keeps the twin skill. Reference answers were written by plain Qwen, so it cannot exceed its teacher.
• 20 hand questions, drafted by an AI and spot-checked by the author against ncc.abcb.gov.au (3 of 12 answers checked). One run. Not a benchmark.
• The checker verifies faithfulness to the retrieved text, not relevance: two vague answers passed.
• It cannot decide whether a building scenario complies with a clause.
• Index mixes in some state variation text (e.g. NSW). Assume nothing about Victoria-specific rules.
• Synthetic twin and scenarios; not validated on a real building. Not for compliance or design decisions.

Data and attribution

NCC text: (c) Commonwealth of Australia and States and Territories, published by the Australian Building Codes Board, CC BY 4.0 (ncc.abcb.gov.au). The NABERS guide PDF is not redistributed in this repo.

v7 fine-tuning study (start: v6)

Rejection-sampling SFT, DPO and both, from v6's own graded samples. Strict % correct, 95% bootstrap CIs; * = paired difference vs v6 excludes 0.

=== A mixed normal days: STRICT % correct with 95% bootstrap CI;  * = paired difference vs v6 excludes 0 ===
           arm   n           correct tool_calls_ok         vs v6 (paired)
base (untuned) 300 31.3 [26.0, 36.7]          46.7 -57.0 [-63.0, -51.3] *
          v5.1 300 90.0 [86.3, 93.3]          92.0      +1.7 [+0.0, +3.7]
    v6 (start) 300 88.3 [84.7, 92.0]          90.0                    NaN
        v7 RFT 300 88.7 [85.0, 92.0]          92.7      +0.3 [+0.0, +1.0]
        v7 DPO 300 88.3 [84.7, 92.0]          90.7      +0.0 [+0.0, +0.0]
    v7 RFT+DPO 300 88.7 [85.0, 92.0]          92.7      +0.3 [+0.0, +1.0]


=== B heatwave+spike: STRICT % correct with 95% bootstrap CI;  * = paired difference vs v6 excludes 0 ===
           arm   n           correct tool_calls_ok         vs v6 (paired)
base (untuned) 200 33.5 [27.0, 40.0]          47.0 -53.0 [-60.0, -46.5] *
          v5.1 200 88.5 [84.0, 93.0]          93.5      +2.0 [-0.5, +4.5]
    v6 (start) 200 86.5 [82.0, 91.0]          91.0                    NaN
        v7 RFT 200 87.0 [82.0, 91.5]          95.0      +0.5 [-1.5, +2.5]
        v7 DPO 200 87.0 [82.5, 91.5]          93.0      +0.5 [+0.0, +1.5]
    v7 RFT+DPO 200 87.0 [82.0, 91.5]          95.0      +0.5 [-1.5, +2.5]


=== C held-out type `full`: STRICT % correct with 95% bootstrap CI;  * = paired difference vs v6 excludes 0 ===
           arm   n           correct tool_calls_ok         vs v6 (paired)
base (untuned) 200    5.0 [2.0, 8.5]           4.0 -19.0 [-24.5, -13.5] *
          v5.1 200 27.0 [21.0, 33.5]          44.0      +3.0 [-0.5, +6.5]
    v6 (start) 200 24.0 [18.5, 30.0]          26.5                    NaN
        v7 RFT 200 21.5 [16.0, 27.0]          41.5      -2.5 [-6.0, +0.5]
        v7 DPO 200 24.0 [18.5, 30.0]          30.5      +0.0 [+0.0, +0.0]
    v7 RFT+DPO 200 21.0 [15.5, 26.5]          41.5      -3.0 [-6.0, +0.0]

Write-up of what moved and what did not: see v7_results.json. Adapters: adapter_v7_rft, adapter_v7_dpo, adapter_v7_rft_dpo.