Tiny Recursive Model for Solana Q&A

TRM-Solana | A 3.5M parameter recursive reasoning model trained on Solana blockchain Q&A

Model Description

This is a Tiny Recursive Model (TRM) fine-tuned on Solana blockchain development Q&A data. Unlike massive language models requiring billions of parameters, TRM achieves strong performance through recursive reasoning with just 3.5 million parameters.

Key Features

🔬 Tiny Architecture: Only 3.5M parameters (~1/1000th the size of GPT-3)
🧠 Recursive Reasoning: Iteratively refines answers through multiple reasoning cycles
⚡ Efficient: Runs on consumer hardware (CPU/MPS/small GPUs)
🎯 Specialized: Trained specifically on Solana blockchain development
📚 Well-documented: Based on peer-reviewed research

Architecture

Model: TinyRecursiveReasoningModel (TRM)
├── Layers (L): 1 transformer layer
├── High-level cycles (H): 2 reasoning iterations
├── Low-level cycles (L): 2 refinement iterations
├── Parameters: ~3.5M
├── Vocabulary: 258 tokens (byte-level)
├── Max sequence length: 512 tokens
└── Embedding dim: Variable (based on architecture)

Intended Use

Primary Use Cases

✅ Solana Development Q&A: Answer questions about Solana blockchain, smart contracts, and development ✅ Educational Tool: Learning resource for Solana developers ✅ Code Assistance: Understanding Solana program architecture and best practices ✅ Research: Studying recursive reasoning in small models

Out of Scope

❌ General-purpose chat or conversation ❌ Real-time transaction analysis ❌ Production smart contract auditing (use professional auditors) ❌ Non-Solana blockchain questions

Training Details

Training Data

Dataset: Custom Solana Q&A corpus
Size: 8,970 question-answer pairs
Split: 90% training (8,073 examples) / 10% test (897 examples)
Topics: Solana architecture, smart contracts, transactions, accounts, programs, security
Format: Instruction-input-output tuples
Encoding: Byte-level tokenization (UTF-8)

Training Procedure

Framework: PyTorch 2.8+
Hardware: Apple Silicon (M1/M2/M3) with MPS acceleration
Epochs: 1,000 - 5,000 (varies by run)
Batch Size: 64 (global batch size)
Learning Rate: 1e-4
Optimizer: AdamW (CPU-compatible fallback)
Weight Decay: 0.5
EMA: Enabled (rate: 0.999)
Gradient Clipping: Standard
Training Time: ~2-4 hours on Apple Silicon

Hyperparameters

Architecture:
  L_layers: 1
  H_cycles: 2
  L_cycles: 2

Training:
  global_batch_size: 64
  lr: 1e-4
  puzzle_emb_lr: 1e-4
  weight_decay: 0.5
  ema: true
  ema_rate: 0.999

Data:
  max_seq_len: 512
  vocab_size: 258
  encoding: byte-level (UTF-8)

Performance

Evaluation Metrics

Metric	Value	Notes
Training Loss	~2.3	Final epoch
Test Loss	~2.5	Held-out set
Parameters	3.5M	Extremely lightweight
Inference Speed	Fast	CPU-compatible
Memory Usage	<1GB	During inference

Comparison

Model	Parameters	Solana Q&A	Hardware Needed
GPT-3	175B	Good	Expensive
LLaMA-7B	7B	Good	GPU required
TRM-Solana	3.5M	Specialized	CPU/MPS

How to Use

Installation

# Install dependencies
pip install torch transformers huggingface_hub

# Or clone the full repo
git clone https://github.com/AlexiaJM/TinyRecursiveModels
cd TinyRecursiveModels
pip install -r requirements.txt

Download Model

from huggingface_hub import hf_hub_download
import torch

# Download checkpoint
checkpoint_path = hf_hub_download(
    repo_id="ordlibrary/trm-solana-v1",
    filename="model.pt"
)

# Load checkpoint
checkpoint = torch.load(checkpoint_path, map_location='cpu')
model_state = checkpoint['model_state_dict']
config = checkpoint['config']

print(f"Model trained for {checkpoint['epoch']} epochs")
print(f"Training loss: {checkpoint['train_loss']:.4f}")
print(f"Test loss: {checkpoint['test_loss']:.4f}")

Inference (Requires TRM Code)

# Note: You need the TRM model code from the repository
from models.recursive_reasoning.trm import TinyRecursiveReasoningModel_ACTV1

# Initialize model with config
model = TinyRecursiveReasoningModel_ACTV1(**config['arch'])

# Load weights
model.load_state_dict(model_state)
model.eval()

# Encode question (byte-level)
def encode_text(text, max_len=512):
    bytes_arr = np.frombuffer(text.encode('utf-8'), dtype=np.uint8)
    tokens = (bytes_arr + 2).astype(np.uint8)  # Shift for PAD/EOS

    # Pad sequence
    seq = np.zeros(max_len, dtype=np.uint8)
    seq[:len(tokens)] = tokens[:max_len-1]
    seq[min(len(tokens), max_len-1)] = 1  # EOS token

    return torch.tensor(seq).unsqueeze(0)

# Inference
question = "What is a Program Derived Address (PDA) in Solana?"
input_tensor = encode_text(question)

with torch.no_grad():
    output = model(input_tensor)
    # Decode output bytes back to text
    # (implementation depends on your decoding strategy)

Example Questions

The model can answer questions like:

"What is a Program Derived Address (PDA) in Solana?"
"How do Solana transactions differ from Ethereum?"
"What is the purpose of the System Program?"
"Explain Solana's account model"
"How does rent work in Solana?"
"What are cross-program invocations (CPI)?"

Limitations and Biases

Limitations

Specialized Domain: Only trained on Solana-related content
Small Model: Limited capacity compared to large language models
Byte-level Encoding: May struggle with very long responses
Training Data Cutoff: Knowledge limited to training data timeframe
No Real-time Updates: Does not know about post-training Solana changes

Potential Biases

Documentation Bias: Reflects common patterns in Solana documentation
English Only: Trained exclusively on English Q&A pairs
Developer-focused: Biased toward technical development questions
Format Bias: Optimized for Q&A format, not conversation

Risks and Mitigations

Risk	Mitigation
Outdated information	Always verify with official Solana docs
Security advice	Never rely solely on model for security audits
Code generation	Review and test all generated code
General blockchain questions	Model specializes in Solana only

Ethical Considerations

Transparency: Training data and methodology fully documented
Open Source: Model weights and code freely available
Educational Purpose: Designed for learning, not production deployment
Verification: Always cross-reference model outputs with official sources

Citation

If you use this model in your research, please cite:

@misc{jolicoeurmartineau2025morerecursivereasoningtiny,
      title={Less is More: Recursive Reasoning with Tiny Networks},
      author={Alexia Jolicoeur-Martineau},
      year={2025},
      eprint={2510.04871},
      archivePrefix={arXiv},
      primaryClass={cs.LG},
      url={https://arxiv.org/abs/2510.04871},
}

Model Card Authors

Model: Trained and fine-tuned by OrdLibrary Architecture: Based on TRM by Alexia Jolicoeur-Martineau Dataset: Custom Solana Q&A corpus

Additional Resources

📄 Paper: Less is More: Recursive Reasoning with Tiny Networks
💻 Code: TinyRecursiveModels GitHub
🌐 Solana Docs: docs.solana.com
🤗 Model: huggingface.co/ordlibrary/trm-solana-v1

License

MIT License - See repository for full details

Acknowledgments

Alexia Jolicoeur-Martineau for the TRM architecture
Solana Foundation for comprehensive documentation
HuggingFace for hosting infrastructure
Community contributors to Solana Q&A data

Built with ❤️ for the Solana developer community

Report Issues • Get Help

Downloads last month: -; Downloads are not tracked for this model. How to track