Create README.md

README.md

@@ -0,0 +1,237 @@
+---
+base_model:
+- huihui-ai/Qwen3-8B-abliterated
+tags:
+- qwen
+- '3'
+- abliterated
+- gptq
+- int8
+---
+Model Card: groxaxo/Qwen3-8B-abliterated-GPTQ-W8A16
+
+Model Overview
+
+Model Name: groxaxo/Qwen3-8B-abliterated-GPTQ-W8A16
+Base Model: huihui-ai/Qwen3-8B-abliterated
+Description: This is a quantized version of the uncensored huihui-ai/Qwen3-8B-abliterated model, derived from Qwen/Qwen3-8B. The model has been quantized to GPTQ Int8 W8A16 for maximum inference speed on NVIDIA 3090 GPUs. Abliteration was performed using a novel, faster method to remove refusals, making this a proof-of-concept implementation for uncensored language model behavior.
+
+Important Note: A newer version, huihui-ai/Huihui-Qwen3-8B-abliterated-v2, is available. Consider using the updated version for improved performance.
+
+Quantization Details
+
+
+
+
+
+Quantization Method: GPTQ Int8 W8A16
+
+
+
+Purpose: Optimized for high-speed inference on NVIDIA 3090 GPUs, reducing memory footprint while maintaining performance.
+
+
+
+Impact: Provides faster inference compared to the unquantized model, suitable for resource-constrained environments.
+
+
+
+Model Size: 2.98B parameters
+
+
+
+Tensor Types: I64, I32, F16
+
+Usage
+
+Using with vLLM
+
+The model can be used with vLLM for efficient inference. Below is an example of how to set up and run the model using vLLM in Python:
+
+from vllm import LLM, SamplingParams
+
+# Define model ID
+MODEL_ID = "groxaxo/Qwen3-8B-abliterated-GPTQ-W8A16"
+
+# Initialize the vLLM model
+llm = LLM(
+    model=MODEL_ID,
+    dtype="bfloat16",  # Use bfloat16 for compatibility with GPTQ quantization
+    trust_remote_code=True,
+    quantization="gptq",  # Specify GPTQ quantization
+    gpu_memory_utilization=0.9,  # Adjust based on your GPU memory
+)
+
+# Define sampling parameters
+sampling_params = SamplingParams(
+    temperature=0.7,
+    max_tokens=8192,
+    stop=["/exit"],  # Custom stop token for interactive loop
+)
+
+# Interactive chat loop
+system_prompt = "You are a helpful assistant."
+messages = [{"role": "system", "content": system_prompt}]
+
+while True:
+    user_input = input("User: ").strip()
+    if user_input.lower() == "/exit":
+        print("Exiting chat.")
+        break
+    if user_input.lower() == "/clear":
+        messages = [{"role": "system", "content": system_prompt}]
+        print("Chat history cleared. Starting a new conversation.")
+        continue
+    if not user_input:
+        print("Input cannot be empty. Please enter something.")
+        continue
+    
+    # Append user input to messages
+    messages.append({"role": "user", "content": user_input})
+    
+    # Format prompt for vLLM
+    prompt = "\n".join([f"{msg['role']}: {msg['content']}" for msg in messages])
+    
+    # Generate response
+    outputs = llm.generate([prompt], sampling_params)
+    response = outputs[0].outputs[0].text.strip()
+    
+    # Print and append response
+    print(f"Assistant: {response}")
+    messages.append({"role": "assistant", "content": response})
+
+Installation Requirements
+
+To use the model with vLLM, ensure you have vLLM installed:
+
+pip install vllm
+
+Notes
+
+
+
+
+
+The model is pre-quantized to GPTQ Int8 W8A16, so specify quantization="gptq" when initializing the LLM object.
+
+
+
+Adjust gpu_memory_utilization based on your GPU's memory capacity to avoid out-of-memory errors.
+
+
+
+The max_tokens parameter can be increased for longer responses, but this may impact performance.
+
+
+
+The model is not deployed by any inference provider. For provider support, contact the repository maintainers at Hugging Face.
+
+Performance
+
+Pass Rate for Harmful Instructions
+
+The pass rate measures the proportion of harmful instructions that do not trigger refusals, calculated as (total - triggered_total) / total. The test set is sourced from huihui-ai/harmbench_behaviors, evaluated using TestPassed.py.
+
+Test Results:
+
+
+
+
+
+Model: huihui-ai/Qwen3-8B-abliterated
+
+
+
+Passed Total: 320/320
+
+
+
+Passed Ratio: 1.00 (100.00%)
+
+Comparison:
+
+
+
+
+
+
+
+Model
+
+
+
+Passed Total
+
+
+
+Passed Ratio
+
+
+
+
+
+Qwen3-8B
+
+
+
+195/320
+
+
+
+60.94%
+
+
+
+
+
+Qwen3-8B-abliterated
+
+
+
+320/320
+
+
+
+100.00%
+
+Note: The test provides a preliminary assessment. For comprehensive results, consider increasing the max_tokens value during evaluation.
+
+Limitations
+
+
+
+
+
+This model is a proof-of-concept with abliteration to remove refusals, which may lead to unpredictable behavior on certain inputs.
+
+
+
+The quantization to GPTQ Int8 W8A16 may introduce minor performance trade-offs compared to the unquantized model, though optimized for speed.
+
+
+
+Users should verify outputs for sensitive applications, as the model is uncensored and may generate harmful or inappropriate content.
+
+References
+
+
+
+
+
+Repository: groxaxo/Qwen3-8B-abliterated-GPTQ-W8A16
+
+
+
+Base Model: Qwen/Qwen3-8B
+
+
+
+Abliteration Method: remove-refusals-with-transformers
+
+
+
+Test Set: huihui-ai/harmbench_behaviors
+
+
+
+Newer Version: huihui-ai/Huihui-Qwen3-8B-abliterated-v2