YAML Metadata
Warning:
The pipeline tag "text2text-generation" is not in the official list: text-classification, token-classification, table-question-answering, question-answering, zero-shot-classification, translation, summarization, feature-extraction, text-generation, fill-mask, sentence-similarity, text-to-speech, text-to-audio, automatic-speech-recognition, audio-to-audio, audio-classification, audio-text-to-text, voice-activity-detection, depth-estimation, image-classification, object-detection, image-segmentation, text-to-image, image-to-text, image-to-image, image-to-video, unconditional-image-generation, video-classification, reinforcement-learning, robotics, tabular-classification, tabular-regression, tabular-to-text, table-to-text, multiple-choice, text-ranking, text-retrieval, time-series-forecasting, text-to-video, image-text-to-text, visual-question-answering, document-question-answering, zero-shot-image-classification, graph-ml, mask-generation, zero-shot-object-detection, text-to-3d, image-to-3d, image-feature-extraction, video-text-to-text, keypoint-detection, visual-document-retrieval, any-to-any, video-to-video, other
Chadgpt Llama2 13b
Colab Example
https://colab.research.google.com/drive/1esMSQUSPyQtOY_3DedyQFKBlTrE9A2vM?usp=sharing
Install Prerequisite
!pip install -q git+https://github.com/huggingface/peft.git
!pip install transformers
!pip install -U accelerate
!pip install accelerate
!pip install bitsandbytes # Instal bits and bytes for inference of the model
Login Using Huggingface Token
# You need a huggingface token that can access llama2
!huggingface-cli login
Download Model
import torch
from peft import PeftModel, PeftConfig
from transformers import AutoModelForCausalLM, AutoTokenizer
peft_model_id = "danjie/Chadgpt-Llama2-13b"
config = PeftConfig.from_pretrained(peft_model_id)
model = AutoModelForCausalLM.from_pretrained(config.base_model_name_or_path, return_dict=True, load_in_8bit=True, device_map='auto')
tokenizer = AutoTokenizer.from_pretrained(config.base_model_name_or_path)
# Load the Lora model
model = PeftModel.from_pretrained(model, peft_model_id)
Inference
def talk_with_llm(tweet: str) -> str:
# Encode and move tensor into cuda if applicable.
encoded_input = tokenizer(tweet, return_tensors='pt')
encoded_input = {k: v.to("cuda") for k, v in encoded_input.items()}
output = model.generate(**encoded_input, max_new_tokens=64)
response = tokenizer.decode(output[0], skip_special_tokens=True)
return response
talk_with_llm("<User> Your sentence \n<Assistant>")
- Downloads last month
- -
Model tree for Danjie/Chadgpt-Llama2-13b
Base model
meta-llama/Llama-2-13b-chat-hf