Update README.md (#4)

- Update README.md (03bfb25f6f4e6d22ff255bbe129defb8102911d3)


Co-authored-by: Yao Zhao <[email protected]>

README.md

@@ -3,12 +3,9 @@ license: mit
 language:
 - en
 base_model:
-- inclusionAI/Ring-flash-linear-2.0
+- inclusionAI/Ring-mini-linear-2.0
 pipeline_tag: text-generation
 ---
-
-
-
 # Quantized Ring-Linear-2.0
 
 ## Introduction
@@ -34,14 +31,22 @@ To enable deployment of [Ring-Linear-2.0](https://github.com/inclusionAI/Ring-V2
 
 #### Environment Preparation
 
-Since the Pull Request (PR) has not been submitted to the vLLM community at this stage, please prepare the environment by following the steps below:
+Since the Pull Request (PR) has not been submitted to the vLLM community at this stage, please prepare the environment by following the steps below.
+
+First, create a Conda environment with Python 3.10 and CUDA 12.8:
 ```shell
-pip install torch==2.7.0 torchvision==0.22.0 
+conda create -n vllm python=3.10
+conda activate vllm
 ```
 
-Then you should install our vLLM wheel package:
+Next, install our vLLM wheel package:
 ```shell
-pip install https://media.githubusercontent.com/media/inclusionAI/Ring-V2/refs/heads/main/hybrid_linear/whls/vllm-0.8.5%2Bcuda12_8_gcc10_2_1-cp310-cp310-linux_x86_64.whl --no-deps --force-reinstall
+pip install https://media.githubusercontent.com/media/zheyishine/vllm_whl/refs/heads/main/vllm-0.8.5.post2.dev28%2Bgd327eed71.cu128-cp310-cp310-linux_x86_64.whl --force-reinstall
+```
+
+Finally, install compatible versions of transformers after vLLM is installed:
+```shell
+pip install transformers==4.51.1 
 ```
 
 #### Offline Inference
@@ -50,35 +55,39 @@ pip install https://media.githubusercontent.com/media/inclusionAI/Ring-V2/refs/h
 from transformers import AutoTokenizer
 from vllm import LLM, SamplingParams
 
-tokenizer = AutoTokenizer.from_pretrained("inclusionAI/Ring-mini-linear-2.0-GPTQ-int4")
-
-sampling_params = SamplingParams(temperature=0.6, top_p=1.0, max_tokens=16384)
-
-
-llm = LLM(model="inclusionAI/Ring-flash-linear-2.0-GPTQ-int4", dtype='auto', enable_prefix_caching=False, max_num_seqs=128)
-
-
-prompt = "Give me a short introduction to large language models."
-messages = [
-    {"role": "user", "content": prompt}
-]
-
-text = tokenizer.apply_chat_template(
-    messages,
-    tokenize=False,
-    add_generation_prompt=True
-)
-outputs = llm.generate([text], sampling_params)
+if __name__ == '__main__':
+    tokenizer = AutoTokenizer.from_pretrained("inclusionAI/Ring-flash-linear-2.0-GPTQ-int4")
+    
+    sampling_params = SamplingParams(temperature=0.6, top_p=1.0, max_tokens=16384)
+
+    # use `max_num_seqs=1` without concurrency
+    llm = LLM(model="inclusionAI/Ring-flash-linear-2.0-GPTQ-int4", dtype='auto', enable_prefix_caching=False, max_num_seqs=128)
+    
+    
+    prompt = "Give me a short introduction to large language models."
+    messages = [
+        {"role": "user", "content": prompt}
+    ]
+    
+    text = tokenizer.apply_chat_template(
+        messages,
+        tokenize=False,
+        add_generation_prompt=True
+    )
+    outputs = llm.generate([text], sampling_params)
+    for output in outputs:
+        print(output.outputs[0].text)
 ```
 
 #### Online Inference
 ```shell
-vllm serve inclusionAI/Ring-mini-linear-2.0-GPTQ-int4 \
+vllm serve inclusionAI/Ring-flash-linear-2.0-GPTQ-int4 \
               --tensor-parallel-size 2 \
               --pipeline-parallel-size 1 \
               --gpu-memory-utilization 0.90 \
-              --max-num-seqs 512 \
+              --max-num-seqs 128 \
               --no-enable-prefix-caching
+              --api-key your-api-key
 ```
 
 
@@ -114,7 +123,4 @@ This code repository is licensed under [the MIT License](https://github.com/incl
 
 ## Citation
 
-If you find our work helpful, feel free to give us a cite.
-
-
-
+If you find our work helpful, feel free to give us a cite.