Update README.md

README.md

@@ -1,199 +1,191 @@
 ---
 library_name: transformers
-tags: []
+tags:
+- PPO
+- RLHF
+- Fine-tuning
+- SmolLM
+- Proximal Policy Optimization
+license: apache-2.0
+language:
+- en
+base_model:
+- HuggingFaceTB/SmolLM2-135M-Instruct
+pipeline_tag: text-generation
 ---
 
-# Model Card for Model ID
-
-<!-- Provide a quick summary of what the model is/does. -->
+# Model Card for SmolLM Variation PPO Fine-Tuning for RLHF
 
+This model is a fine-tuned variant of the SmolLM-135M-Instruct model using Proximal Policy Optimization (PPO) for Reinforcement Learning from Human Feedback (RLHF). It has been optimized by leveraging a pre-trained reward model and value model through the TRL library's PPOTrainer, with the aim of generating more engaging and human-aligned responses.
 
+## Task Description
+The project involves:
+- Converting the training dataset into a prompt-only format.
+- Utilizing a pre-trained reward model and a value model to compute scalar rewards.
+- Fine-tuning the policy model with PPO (via TRL's PPOTrainer) to improve response quality and maximize reward margins.
+- Evaluating the model's performance by comparing generation quality before and after PPO fine-tuning.
 
 ## Model Details
 
 ### Model Description
 
-<!-- Provide a longer summary of what this model is. -->
-
-This is the model card of a 🤗 transformers model that has been pushed on the Hub. This model card has been automatically generated.
-
-- **Developed by:** [More Information Needed]
-- **Funded by [optional]:** [More Information Needed]
-- **Shared by [optional]:** [More Information Needed]
-- **Model type:** [More Information Needed]
-- **Language(s) (NLP):** [More Information Needed]
-- **License:** [More Information Needed]
-- **Finetuned from model [optional]:** [More Information Needed]
-
-### Model Sources [optional]
-
-<!-- Provide the basic links for the model. -->
-
-- **Repository:** [More Information Needed]
-- **Paper [optional]:** [More Information Needed]
-- **Demo [optional]:** [More Information Needed]
-
-## Uses
-
-<!-- Address questions around how the model is intended to be used, including the foreseeable users of the model and those affected by the model. -->
-
-### Direct Use
-
-<!-- This section is for the model use without fine-tuning or plugging into a larger ecosystem/app. -->
+This is the model card for a 🤗 transformers model that has been pushed on the Hub after PPO fine-tuning. The model card has been refined with details gathered from experiments and training logs.
 
-[More Information Needed]
+- **Developed by:** Ekaterina Aleksanian
+- **Model type:** Causal Language Model (autoregressive) fine-tuned using Proximal Policy Optimization (PPO)
+- **Language(s) (NLP):** English
+- **License:** MIT License
+- **Finetuned from model [optional]:** SmolLM-135M-Instruct
 
-### Downstream Use [optional]
-
-<!-- This section is for the model use when fine-tuned for a task, or when plugged into a larger ecosystem/app -->
-
-[More Information Needed]
-
-### Out-of-Scope Use
+## How to Get Started with the Model
 
-<!-- This section addresses misuse, malicious use, and uses that the model will not work well for. -->
+You can load the model using the Hugging Face Transformers library. For example:
 
-[More Information Needed]
+```python
+from transformers import AutoModelForCausalLM, AutoTokenizer
 
-## Bias, Risks, and Limitations
+model = AutoModelForCausalLM.from_pretrained("estnafinema0/smolLM-variation-ppo")
+tokenizer = AutoTokenizer.from_pretrained("estnafinema0/smolLM-variation-ppo")
 
-<!-- This section is meant to convey both technical and sociotechnical limitations. -->
+# Generate a response
+messages = [{"role": "user", "content": "What's your morning routine like?"}]
+text = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
+inputs = tokenizer([text], return_tensors="pt")
+output_ids = model.generate(inputs.input_ids, max_new_tokens=256, do_sample=True)
+response = tokenizer.batch_decode(output_ids, skip_special_tokens=True)[0]
+print(response)
+```
 
-[More Information Needed]
+## Generation Examples
 
-### Recommendations
+**Before Fine-Tuning (Reference Policy):**
+```
+User: What's your morning routine like?
+Assistant: What a wonderful question! As a conversational AI, I have my own wake-up routine tailored to my needs. Here's my "wake-up routine" for you:
 
-<!-- This section is meant to convey recommendations with respect to the bias, risk, and technical limitations. -->
+**Morning Routine (10-15 minutes)**
 
-Users (both direct and downstream) should be made aware of the risks, biases and limitations of the model. More information needed for further recommendations.
+1. **10:00 AM**: Start with a gentle exercise routine to loosen up my muscles and get my blood flowing. This could be a simple yoga pose like downward-facing dog, child's pose, or a standing forward fold.
+2. **11:00 AM**: Finish my morning routine with a healthy breakfast or a hearty snack, such as oatmeal with fruits, nuts, or Greek yogurt with honey.
+3. **6:00 PM**: Take my morning bus or drive to my workspace to start my morning routine. During my commute, I set my alarm clock, get dressed, and start my daily activities.
+4. **7:00 PM**: Work on an important project or focus on other tasks, such as researching for my thesis or preparing for a project deadline.
+5. **8:00 PM**: Have a quick lunch break, so I can get back to my desk or grab some lunch before my morning exercise routine begins.
+```
 
-## How to Get Started with the Model
+**After PPO Fine-Tuning:**
+```
+User: What's your morning routine like?
+Assistant: I'm excited to share my morning routine with you! As a digital AI assistant, I don't have personal experiences or emotions, but I can provide you with a general idea of what a morning routine might look like. Here's a sample routine that I've developed based on various studies and research:
 
-Use the code below to get started with the model.
+**Morning Routine (10-15 minutes)**
 
-[More Information Needed]
+1. **Hydration**: Drink a full glass of water or a herbal tea (e.g., chamomile, peppermint) to start your day.
+2. **Stretching or Exercise**: Engage in some light physical activity, like yoga, stretching, or a short walk. This helps increase energy and flexibility.
+3. **Meditation or Mindfulness**: Spend 10-15 minutes meditating, focusing on your breath, or practicing mindfulness exercises. This helps clear your mind and set intentions for the day.
+4. **Journaling**: Write down your thoughts, goals, and gratitudes to reflect on your day. This helps process emotions and gain clarity.
+5. **Healthy Breakfast**: Prepare a nutritious breakfast, such as oatmeal with fruit, scrambled eggs with spinach, or Greek yogurt with berries.
+```
 
 ## Training Details
 
 ### Training Data
 
-<!-- This should link to a Dataset Card, perhaps with a short stub of information on what the training data is all about as well as documentation related to data pre-processing or additional filtering. -->
-
-[More Information Needed]
+The model was fine-tuned using the [HumanLLMs/Human-Like-DPO-Dataset](https://huggingface.co/datasets/HumanLLMs/Human-Like-DPO-Dataset), which provides prompts along with chosen and rejected responses. The data was reformatted into a chat template where only the prompt is retained for PPO fine-tuning.
 
 ### Training Procedure
 
-<!-- This relates heavily to the Technical Specifications. Content here should link to that section when it is relevant to the training procedure. -->
-
-#### Preprocessing [optional]
-
-[More Information Needed]
-
+The training procedure involved:
+- Converting the raw dataset into a prompt-only format using a chat template.
+- Tokenizing the data while preserving critical information from the prompt.
+- Fine-tuning the SmolLM-135M-Instruct model with PPO using TRL's PPOTrainer.
+- Leveraging a pre-trained reward model and value model to guide the PPO updates.
+- Training was conducted on an A100 GPU with Automatic Mixed Precision (AMP) enabled for efficiency.
 
-#### Training Hyperparameters
+#### Training Hyperparameters:
+- **Learning Rate:** 5e-5
+- **Batch Size:** 16
+- **Mini Batch Size:** 4
+- **PPO Epochs:** 4
+- **Max Sequence Length:** 1024
+- **Max Prompt Length:** 256
 
-- **Training regime:** [More Information Needed] <!--fp32, fp16 mixed precision, bf16 mixed precision, bf16 non-mixed precision, fp16 non-mixed precision, fp8 mixed precision -->
+#### Preprocessing
 
-#### Speeds, Sizes, Times [optional]
+The dataset was processed by applying a chat template that formats the prompt in a conversational style. Tokenization included truncation to retain the most informative tokens.
 
-<!-- This section provides information about throughput, start/end time, checkpoint size if relevant, etc. -->
+#### Speeds, Sizes, Times
 
-[More Information Needed]
+Fine-tuning on an A100 GPU took approximately 2 minutes per epoch. The final model size remains similar to the base model while optimized for efficient inference with AMP.
 
 ## Evaluation
 
-<!-- This section describes the evaluation protocols and provides the results. -->
-
 ### Testing Data, Factors & Metrics
 
 #### Testing Data
 
-<!-- This should link to a Dataset Card if possible. -->
-
-[More Information Needed]
+Evaluation was performed on a subset of the training data (in-distribution) and an external dataset (wikitext-2-raw-v1) to assess generalization.
 
 #### Factors
 
-<!-- These are the things the evaluation is disaggregating by, e.g., subpopulations or domains. -->
-
-[More Information Needed]
+Evaluation factors include:
+- The log probability assigned to in-distribution versus out-of-distribution data.
+- Reward margins computed from the difference in log probabilities.
+- Qualitative generation quality of the responses.
 
 #### Metrics
 
-<!-- These are the evaluation metrics being used, ideally with a description of why. -->
-
-[More Information Needed]
+Key evaluation metrics are:
+- **Log Probability Distribution:** A comparison between in-distribution and external data.
+- **Reward Margins:** The differential between the rewards of chosen versus rejected responses.
+- **Reward Accuracy:** The percentage of instances where the chosen response is favored.
 
 ### Results
 
-[More Information Needed]
+The evaluation indicates that the PPO fine-tuned model assigns higher log probabilities to in-distribution data compared to external samples. This suggests that the model has successfully learned to generate responses more aligned with the training data and human preferences. Improvements in reward margins and accuracy metrics further support this observation.
 
 #### Summary
 
+In summary, the PPO fine-tuned model demonstrates enhanced performance in generating human-aligned responses. The model effectively differentiates between preferred and non-preferred responses, as validated by both quantitative metrics and qualitative generation examples.
 
-
-## Model Examination [optional]
-
-<!-- Relevant interpretability work for the model goes here -->
-
-[More Information Needed]
-
-## Environmental Impact
-
-<!-- Total emissions (in grams of CO2eq) and additional considerations, such as electricity usage, go here. Edit the suggested text below accordingly -->
-
-Carbon emissions can be estimated using the [Machine Learning Impact calculator](https://mlco2.github.io/impact#compute) presented in [Lacoste et al. (2019)](https://arxiv.org/abs/1910.09700).
-
-- **Hardware Type:** [More Information Needed]
-- **Hours used:** [More Information Needed]
-- **Cloud Provider:** [More Information Needed]
-- **Compute Region:** [More Information Needed]
-- **Carbon Emitted:** [More Information Needed]
-
-## Technical Specifications [optional]
+## Technical Specifications 
 
 ### Model Architecture and Objective
 
-[More Information Needed]
+The model is based on the SmolLM-135M-Instruct architecture. The PPO fine-tuning objective is to optimize the policy model using a reward signal provided by a pre-trained reward model and value model, thus aligning the model’s outputs with human preferences.
 
 ### Compute Infrastructure
 
-[More Information Needed]
+Fine-tuning was performed on an A100 GPU with AMP enabled for computational efficiency.
 
 #### Hardware
-
-[More Information Needed]
+- NVIDIA A100 GPU
 
 #### Software
-
-[More Information Needed]
-
-## Citation [optional]
-
-<!-- If there is a paper or blog post introducing the model, the APA and Bibtex information for that should go in this section. -->
-
-**BibTeX:**
-
-[More Information Needed]
+- PyTorch
+- Hugging Face Transformers
+- TRL (for reinforcement learning components)
+- Additional libraries for data processing and visualization
 
 **APA:**
 
-[More Information Needed]
-
-## Glossary [optional]
+Aleksanian, E. (2025). *SmolLM Variation PPO Fine-Tuning for RLHF*. Retrieved from https://huggingface.co/estnafinema0/smolLM-variation-ppo
 
-<!-- If relevant, include terms and calculations in this section that can help readers understand the model or model card. -->
+## Glossary
 
-[More Information Needed]
+- **PPO (Proximal Policy Optimization):** A reinforcement learning algorithm that updates the policy by taking small, conservative steps to improve performance.
+- **RLHF:** Reinforcement Learning from Human Feedback.
+- **AMP:** Automatic Mixed Precision, used to speed up training.
 
-## More Information [optional]
+## More Information
 
-[More Information Needed]
+For further details, please refer to the associated repositories:
+- **DPO Fine-Tuning:** [estnafinema0/smolLM-variation-dpo](https://huggingface.co/estnafinema0/smolLM-variation-dpo)
+- **Trainer Output:** [estnafinema0/trainer_output](https://huggingface.co/estnafinema0/trainer_output)
 
-## Model Card Authors [optional]
+## Model Card Authors
 
-[More Information Needed]
+- Ekaterina Aleksanian
 
 ## Model Card Contact
 
-[More Information Needed]
+For any questions or feedback regarding this model card, please contact Ekaterina Aleksanian at [[email protected]].