This is the [inceptionai/jais-family-13b](https://huggingface.co/inceptionai/jais-family-13b) model converted to [OpenVINO](https://docs.openvino.ai/2025/index.html) 
with INT4 weight compression.

## Download the model

- Install huggingface-hub

```sh
pip install huggingface-hub[cli]
```

- Download the model

```sh
huggingface-cli download helenai/jais-family-13b-ov-int4-sym --local-dir helenai/jais-family-13b-ov-int4-sym
```

## Run inference

- Install/upgrade OpenVINO GenAI nightly (this is the only requirement for inference - no need to install transformers or PyTorch)

```sh
pip install --pre --upgrade openvino-genai openvino openvino-tokenizers --extra-index-url https://storage.openvinotoolkit.org/simple/wheels/pre-release
```

- Download a sample inference script (curl -O works on Windows Command Prompt and most Linux terminals). Note that this is not a chat/instruct model; this inference script 
  is only for testing model outputs. The model is not finetuned for answering questions, and the inference script will not remember history.

```sh
curl -O https://raw.githubusercontent.com/helena-intel/snippets/refs/heads/main/llm_chat/python/llm_test.py
```

- Run the script with the path to the model and the device as parameters. Change GPU to CPU to run on CPU. NPU is not yet supported for this model

```sh
python llm_test.py jais-family-13b-ov-int4-sym GPU
```

Check out [OpenVINO GenAI documentation](https://docs.openvino.ai/2025/openvino-workflow-generative/inference-with-genai.html) for more information.

## Model compression parameters

```
openvino_version         : 2025.2.0-18660-3ceeeb52d64

advanced_parameters      : {'statistics_path': None, 'awq_params': {'subset_size': 32, 'percent_to_apply': 0.002, 'alpha_min': 0.0, 'alpha_max': 1.0, 'steps': 100}, 'scale_estimation_params': {'subset_size': 64, 'initial_steps': 5, 'scale_steps': 5, 'weight_penalty': -1.0}, 'gptq_params': {'damp_percent': 0.1, 'block_size': 128, 'subset_size': 128}, 'lora_correction_params': {'adapter_rank': 8, 'num_iterations': 3, 'apply_regularization': True, 'subset_size': 128, 'use_int8_adapters': True}}
all_layers               : False
awq                      : False
backup_mode              : int8_asym
gptq                     : False
group_size               : -1
ignored_scope            : []
lora_correction          : False
mode                     : int4_sym
ratio                    : 1.0
scale_estimation         : False
sensitivity_metric       : weight_quantization_error

optimum_intel_version    : 1.22.0
optimum_version          : 1.24.0
pytorch_version          : 2.5.1+cpu
transformers_version     : 4.48.3
```