darkhorse0811/Affine-5ERU9B8yMyJCMX4BLwJaKGVE9NRc34y8gcNXrDo5bc9hVEsd

Affine: Comprehensive LLM/Validator Guide (llms.txt)

Overview

Affine is an incentivized RL system built on Bittensor Subnet 120. Validators continuously generate challenges across multiple environments, query miners’ hosted LLMs via Chutes, evaluate responses, and publish signed results to Cloudflare R2. A winners-take-all scoring rule uses ε-Pareto dominance over environment subsets to set on-chain weights.

Repository map (high level)

• affine/__init__.py: CLI, core models, validator/runner, R2 storage, signer client, scoring, miners/query, Prometheus metrics
• affine/envs/: task generators and evaluators
  • sat.py: k-SAT generation and evaluation
  • abd.py: program abduction (infer stdin to match output)
  • ded.py: program deduction (write code to pass test cases)
• affine/utils/: runtime utilities
  • executor.py: safe Python program runner for ABD/DED
  • dataset.py: buffered Hugging Face dataset fetcher
• Packaging/config: pyproject.toml, Dockerfile, docker-compose.yml, docker-compose.local.yml, prometheus.yml
• Ops/content: build_and_push.sh (if present), site/ (static viewer), notebooks/

CLI entrypoint

Install via uv and run the af CLI (see README for full steps).

• af -vv validate: run the validator loop (Prometheus metrics on AFFINE_METRICS_PORT)
• af -vv runner: run a continuous challenge+evaluate+sink loop (off-chain)
• af weights: compute weights once and print summary table
• af pull <uid> --model_path <dir>: download a miner’s model from Hugging Face
• af push [--model_path <dir> | --existing-repo <user/repo>] [--revision <sha>]:
  • Upload artifacts to HF (unless using --existing-repo)
  • Make repo public
  • Deploy Chute with generated config
  • Commit (model, revision, chute_id) on-chain
  • Warm up the chute until it’s hot
• af signer: start a lightweight HTTP signer service (used by validator)

Environments

• SAT (affine/envs/sat.py)

  • Generates random k-SAT formula over x1..xn; prompt asks for a satisfying assignment or UNSAT
  • Evaluation parses xN=True/False/1/0 pairs and checks every clause is satisfied

• ABD (Program Abduction, affine/envs/abd.py)

  • Uses HF dataset satpalsr/rl-python samples: Python program, example input/output
  • LLM is prompted to produce a fresh valid stdin wrapped in <INPUT>..</INPUT> tags for the given program so that its output matches the example output
  • Input is validated heuristically (line counts vs input() calls, loop shape)
  • Execution via ProgramExecutor; evaluation re-runs program with extracted input and checks output equivalence (whitespace/line tolerant)

• DED (Program Deduction, affine/envs/ded.py)

  • Prompt asks the model to produce a complete Python solution (fenced) that reads from STDIN and prints answers only
  • Verification pulls test_cases (stdin/stdout or function_call) from the sample, executes program per case, normalizes outputs, and scores 1.0 only if all pass

Querying miners (Chutes)

• Endpoint: https://{slug}.chutes.ai/v1/chat/completions
• Auth: Authorization: Bearer ${CHUTES_API_KEY}
• Payload: { model, messages: [{ role: "user", content: prompt }] }
• Retries/backoff configurable; response content extracted from choices[0].message.content
• Gated HF models are skipped (checked via Hugging Face API and optional revision presence)

Miner discovery

• Reads Bittensor metagraph for netuid 120 and revealed commitments containing { model, revision, chute_id }
• Filters out gated models and non-affine/* families (except base UID 0)
• Deduplicates by keeping earliest block per model

Results pipeline and storage (Cloudflare R2)

• Windowing
  • AFFINE_WINDOW (default 20) defines shard window based on block numbers
  • A shard key is affine/results/{WINDOW_START_BLOCK}-{HOTKEY}.json
• Index
  • affine/index.json contains a JSON array of shard keys
  • When a shard is first written, the index is updated (deduplicated and sorted)
• Sink
  • Results are signed (via signer service or local wallet) and appended to the shard; new shard triggers index update
• Local cache
  • Shards are downloaded once and stored under AFFINE_CACHE_DIR (default ~/.cache/affine/blocks), with .modified timestamp files
  • dataset(tail) streams Result objects from cached JSONL, validating signatures

R2/S3 client

• Uses aiobotocore with endpoint https://{R2_BUCKET_ID}.r2.cloudflarestorage.com
• Bucket: R2_FOLDER (default affine)
• Keys: INDEX_KEY=affine/index.json, RESULT_PREFIX=affine/results/

Scoring and weight setting

• Periodic cadence: validator waits for blocks where block % TEMPO == 0 (TEMPO=100)
• Data ingestion: last TAIL=10_000 blocks streamed via dataset(tail)
• Accumulation per miner per env: counts and success rates (accuracy)
• Eligibility: require ≥90% of per-env max sample counts
• ε-Pareto dominance
  • Not-worse threshold uses Z_NOT_WORSE and EPS_FLOOR based on standard error of difference
  • Better-on-some-env threshold uses EPS_WIN (floor) and optional Z_WIN
  • Global dominance counts over full env set; canonical best used for ties/fallbacks
• Combinatoric subset winners
  • For all non-empty env subsets S, award K_s to the ε-Pareto winner on S
  • K_1 = scale, K_s = C(N, s-1)*K_{s-1}
  • Normalize scores over eligibles to produce weights; if none eligible, assign 1.0 to canonical best
• On-chain set_weights
  • Delegated to the signer HTTP service (/set_weights) and confirmed by checking last_update
  • Fallback to local submission only if signer is unreachable

Key hyperparameters (defaults)

• NETUID=120, TAIL=10_000, ALPHA=0.9
• EPS_FLOOR=0.002, Z_NOT_WORSE=0.84, EPS_WIN=0.0015, Z_WIN=0.0

Signer service

• Start with af -v signer (listens on ${SIGNER_HOST}:${SIGNER_PORT})
• Endpoints
  • GET /healthz → { ok: true }
  • POST /sign → { signatures: [hex...], hotkey } for provided string payloads
  • POST /set_weights → triggers on-chain set_weights with confirmation
• Used by validator via ${SIGNER_URL}; includes DNS logging + request/response logging

Prometheus metrics (port/address configurable)

• Counters/Gauges
  • qcount{model}: number of LLM queries
  • score{uid,env}: per-miner per-env accuracy
  • rank{uid,env}: per-env rank among eligibles
  • weight{uid}: current weight
  • lastset: time of last successful weight set
  • nresults: processed result count
  • maxenv{env}: best accuracy per env among active miners
  • cache: local cache size (bytes)
• Exporter binds at ${AFFINE_METRICS_ADDR}:${AFFINE_METRICS_PORT}

Program execution sandbox (ABD/DED)

• ProgramExecutor limits: wallclock, CPU, memory, and output size
• Strips code fences; if program defines solve() and produces no output, auto-injects if __name__ == "__main__": runner
• Cleans up temp files; kills entire process group on timeout/truncation

Buffered dataset (Hugging Face)

• BufferedDataset fetches random windows from https://datasets-server.huggingface.co/rows with retries and exponential backoff
• Internal buffer filled concurrently and served via get(); used by ABD/DED

Configuration (env vars)

• Bittensor/Subtensor
  • SUBTENSOR_ENDPOINT (default finney), SUBTENSOR_FALLBACK (default wss://lite.sub.latent.to:443)
  • BT_WALLET_COLD, BT_WALLET_HOT
• Chutes/Hugging Face
  • CHUTES_API_KEY (required for queries/deploy), CHUTE_USER
  • HF_USER, HF_TOKEN
• R2/S3
  • R2_BUCKET_ID (account subdomain), R2_FOLDER (bucket root folder), R2_WRITE_ACCESS_KEY_ID, R2_WRITE_SECRET_ACCESS_KEY
  • AFFINE_WINDOW (shard size), AFFINE_CACHE_DIR
• Networking/Concurrency
  • AFFINE_METRICS_ADDR, AFFINE_METRICS_PORT
  • AFFINE_HTTP_CONCURRENCY (default 16), AFFINE_UPLOAD_CONCURRENCY (default 2)
• Signer
  • SIGNER_HOST, SIGNER_PORT, SIGNER_URL (e.g., http://signer:8080)
  • SIGNER_RETRIES, SIGNER_RETRY_DELAY

Docker Compose (production and local override)

• Services
  • validator: af -vv validate, metrics on 8000 (host 8001), depends on signer
  • runner: af -vv runner, metrics on 8000 (host 8002)
  • signer: exposes 8080; mounts wallet dir read-only
  • prometheus (9090) and grafana (host 8000) for telemetry
  • watchtower auto-updates images
• Local build
  • Use with override: docker compose -f docker-compose.yml -f docker-compose.local.yml up --build

SDK usage

Example from README:

import affine as af
af.trace(); af.debug(); af.info()
miners = await af.get_miners(); miner = await af.get_miners(5)
chal = await af.SAT.generate()
chals = await af.ABDUCTION().many(10); chals = await af.DEDUCTION().many(10)
response = await af.query(chal.prompt, model=miner.model)
evaluation = chal.evaluate(response)
print(evaluation.score)
async for res in af.rollouts(100):
    print(res)

Static site (R2 index viewer)

• Index key: affine/index.json (JSON array of shard keys)
• Endpoint template: https://{R2_BUCKET_ID}.r2.cloudflarestorage.com/{R2_FOLDER}/{OBJECT_KEY}
• Requires S3 SigV4 signing in the browser (region auto, service s3, x-amz-content-sha256=UNSIGNED-PAYLOAD)
• Flow: fetch index → for each key fetch shard → render list/download
• Prefer presigned URLs or read-only keys for public deployments

Notes and best practices

• Always override the example/default R2 credentials with your own via .env
• Keep HF repos private during upload; visibility is set to public right before deploy
• The validator requires a running signer service; do not expose wallet keys in validator containers
• For ABD/DED, ensure models return only the requested content (stdin or fenced python) to avoid grading penalties

Quick commands

# Validate (local):
af -vv validate

# Runner (off-chain ingestion + sink):
af -vv runner

# Pull miner model:
af -vvv pull <uid> --model_path ./my_model

# Push your model (deploy chute and commit on-chain):
af -vvv push --model_path ./my_model