Model Card for FLAN-T5 Climate Action QLoRA

This is a QLoRA-finetuned version of FLAN-T5 specifically trained for climate action content analysis and generation. The model is optimized for processing and analyzing text related to climate change, sustainability, and environmental policies.

Model Details

Model Description

Developed by: Kshitiz Khanal
Shared by: kshitizkhanal7
Model type: Instruction-tuned Language Model with QLoRA fine-tuning
Language(s): English
License: Apache 2.0
Finetuned from model: google/flan-t5-base

Model Sources

Repository: https://huggingface.co/kshitizkhanal7/flan-t5-climate-qlora
Training Data: FineWeb dataset (climate action filtered)

Uses

Direct Use

The model is designed for:

Analyzing climate policies and initiatives
Summarizing climate action documents
Answering questions about climate change and environmental policies
Evaluating sustainability measures
Processing climate-related research and reports

Downstream Use

The model can be integrated into:

Climate policy analysis tools
Environmental reporting systems
Sustainability assessment frameworks
Climate research applications
Educational tools about climate change

Out-of-Scope Use

The model should not be used for:

Critical policy decisions without human oversight
Generation of climate misinformation
Technical climate science research without expert validation
Commercial deployment without proper testing
Medical or legal advice

Bias, Risks, and Limitations

Limited to climate-related content analysis
May not perform well on general domain tasks
Potential biases from web-based training data
Should not be the sole source for critical decisions
Performance varies on technical climate science topics

Recommendations

Always verify model outputs with authoritative sources
Use human expert oversight for critical applications
Consider the model as a supplementary tool, not a replacement for expert knowledge
Regular evaluation of outputs for potential biases
Use in conjunction with other data sources for comprehensive analysis

Training Details

Training Data

Source: FineWeb dataset filtered for climate content
Selection criteria: Climate-related keywords and quality metrics
Processing: Instruction-style formatting with climate focus

Training Procedure

Preprocessing

Text cleaning and normalization
Instruction templates for climate context
Maximum input length: 512 tokens
Maximum output length: 128 tokens

Training Hyperparameters

Training regime: QLoRA 4-bit fine-tuning
Epochs: 3
Learning rate: 2e-4
Batch size: 4
Gradient accumulation steps: 4
LoRA rank: 16
LoRA alpha: 32
Target modules: Query and Value matrices
LoRA dropout: 0.05

Environmental Impact

Hardware Type: Single GPU
Hours used: ~4 hours
Cloud Provider: Local
Carbon Emitted: Minimal due to QLoRA efficiency

Technical Specifications

Model Architecture and Objective

Base architecture: FLAN-T5
Objective: Climate-specific text analysis
QLoRA adaptation for efficient fine-tuning
4-bit quantization for reduced memory usage

Compute Infrastructure

Python 3.8+
PyTorch
Transformers library
bitsandbytes for quantization
PEFT for LoRA implementation

Hardware

Minimum requirements:

16GB GPU memory for inference
24GB GPU memory recommended for training
CPU inference possible but slower

Citation

If you use this model, please cite:

@misc{khanal2024climate,
  title={FLAN-T5 Climate Action QLoRA},
  author={Khanal, Kshitiz},
  year={2024},
  publisher={HuggingFace},
  howpublished={\url{https://huggingface.co/kshitizkhanal7/flan-t5-climate-qlora}}
}