Qwen2.5-72B-Instruct-PyQGIS (LoRA)

Repository: icewaterdun/Qwen2.5-72B-Instruct-PyQGIS Base model: Qwen2.5-72B-Instruct Adapter type: LoRA (PEFT) Checkpoint used: checkpoint-600 (early-stop selection)

Model Summary

This repository contains a LoRA adapter that tailors Qwen2.5-72B-Instruct to PyQGIS/QGIS workflows (e.g., geoprocessing, vector/raster operations, layer styling, and common Geo/OGR tasks). The adapter focuses on instruction-following in a GIS setting, emphasizing practical coding patterns and on-tool terminology.

• Why LoRA? Lower VRAM footprint, faster iteration, and easy plug-and-play with the base model.
• Best checkpoint: checkpoint-600. Later checkpoints (800+) showed a slight regression (on the same eval split), suggesting mild overfitting.
• Training config: See the uploaded YAML in this repo for exact hyperparameters and data settings.

Evaluation

Evaluated on the same held-out set (503 items). We report cross-entropy loss and the corresponding perplexity (lower is better). Perplexity is computed from the reported loss via: [ \text{PPL} = \exp(\text{loss}) ] assuming the loss is the mean token-level cross-entropy in nats (Hugging Face Transformers’ default).

Model / Checkpoint	eval_loss	Perplexity (PPL)
Baseline (no LoRA)	2.476	11.894
ckpt-400	0.885	2.423
ckpt-600 (selected)	0.865	2.375
ckpt-800	0.898	2.455

Takeaway: The LoRA adapter substantially improves the base model on this PyQGIS-focused eval set (PPL ↓ from ~11.9 → ~2.38). Among LoRA checkpoints, 600 is best (lowest loss/PPL), while 800 begins to drift upward.

How perplexity is computed (quick note)

• If your framework reports cross-entropy loss in nats (the usual case in 🤗 Transformers), PPL = exp(loss).
• If the loss were measured in bits instead, it would be PPL = 2^(loss).
• The reported PPL is the exponentiated average token-level loss over the evaluation set.

Data & Training

• Data domain: PyQGIS/QGIS instructions and code-centric prompts (see dataset_dir and datasets referenced in the training YAML).
• Template: qwen chat template.
• Sequence lengths: cutoff length was 1024 tokens for both prompt and response packing in eval/training runs shown here.
• Precision: bf16 for evaluation; training precision and optimizer settings are in the YAML.
• Adapter: LoRA; rank/alpha/targets as specified in the uploaded training YAML.

For exact hyperparameters (optimizer, scheduler, LoRA rank/alpha/dropout, target modules, warmup, batch sizes, etc.), consult the training YAML included in this repository.

Intended Use

• Assisting with PyQGIS scripting and QGIS-related problem solving.
• Generating and explaining PyQGIS code snippets for common GIS tasks.

Limitations

• The adapter focuses on PyQGIS-specific tasks; general world knowledge or unrelated domains remain the responsibility of the base model.
• Hallucinations are still possible—especially for niche plugins, exotic projections, or very new QGIS APIs.
• Not evaluated for safety-critical, legal, or medical use.

License

• Inherits the license and usage terms of Qwen2.5-72B-Instruct for inference.
• The LoRA weights here are released for research and non-commercial evaluation; please review the base model’s license before any production use.

Citation

If you use this adapter, please cite the base model and this repository.

@misc{Qwen2.5-72B-Instruct-PyQGIS,
  title  = {Qwen2.5-72B-Instruct-PyQGIS (LoRA)},
  author = {icewaterdun},
  year   = {2025},
  note   = {LoRA adapter for PyQGIS tasks on top of Qwen2.5-72B-Instruct}
}