Instructions to use IFM/K2-Horizon-7B with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use IFM/K2-Horizon-7B with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="IFM/K2-Horizon-7B", trust_remote_code=True)
messages = [
    {"role": "user", "content": "Who are you?"},
]
pipe(messages)

# Load model directly
from transformers import AutoModelForCausalLM
model = AutoModelForCausalLM.from_pretrained("IFM/K2-Horizon-7B", trust_remote_code=True, device_map="auto")

Inference
Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use IFM/K2-Horizon-7B with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "IFM/K2-Horizon-7B"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "IFM/K2-Horizon-7B",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/IFM/K2-Horizon-7B

SGLang

How to use IFM/K2-Horizon-7B with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "IFM/K2-Horizon-7B" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "IFM/K2-Horizon-7B",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "IFM/K2-Horizon-7B" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "IFM/K2-Horizon-7B",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Docker Model Runner
How to use IFM/K2-Horizon-7B with Docker Model Runner:
```
docker model run hf.co/IFM/K2-Horizon-7B
```

I wish you delivered what you say you delivered

by NezTheNaughty - opened 27 days ago

Discussion

NezTheNaughty

27 days ago

•

edited 27 days ago

This is hot garbage.

jan1k

26 days ago

•

edited 26 days ago

I wish you delivered what you say you delivered
This is hot garbage.

Could you expand with few words on the hot garbage and missing delivery? What are your findings?

hunterhector

Institute of Foundation Models org 25 days ago

Hi, my friends,

I am from the IFM team. Could you let me know what specifics are you looking for?

Btw, our team do find that some of our artifacts are not finished uploading yet (e.g., we mess up on some ckpt uploading due to simple index confusion). Here is our rough follow-up plan:

code will be ready soon, some cleanup on branches will be needed.
checkpoints are a bit messy, they are there as git references but we need to provide a table explaining it better
data are halfway uploaded.

The full paper is probably gonna take longer for us to polish.

NezTheNaughty

24 days ago

•

edited 24 days ago

Excellent. Good to know! Thank you.
Literal garbage was showing up in the results, and I wasn't the only one. Benchmarks were not even coming close in most areas.
If you can straighten this out to what you have claimed, this will be an amazing piece of work.

AdityaMishra1

19 days ago

this is some hard core benchmaxxing

koiinthepond

13 days ago

•

edited 13 days ago

It is currently very hard to get this model actually running locally. I gave up trying to figure out how to compile the llama.cpp fork etc. and just used another model which benchmarked 2x worse than K2-Horizon-7B while having more parameters. It would be nice if the setup process was elaborated on more because I would love to use this model.

inikishev

9 days ago

•

edited 9 days ago

Could you please make a commit with this model architecture into llama.cpp.
I am getting Failed to load model: The installed llama.cpp does not recognise this GGUF's model architecture ('k2-horizon'). The file is valid. If the model is newer than this llama.cpp build, updating llama.cpp may add support for it.

inikishev

9 days ago

hunterhector

Institute of Foundation Models org 9 days ago

Got it my friends. We have llama.cpp ports but haven't merged them yet. Will follow up. @mvillmow-mbzuai @lmlmcat

mvillmow-mbzuai

Institute of Foundation Models org 7 days ago

@inikishev Can you explain more what your issue is with the llama.cpp fork? We are working with the llama.cpp team to get the code integrated here: https://github.com/ggml-org/llama.cpp/discussions/28308

I have also put up a PR against lemonade-server.ai here: https://github.com/lemonade-sdk/lemonade/pull/3530

It shows how to build the fork and install it with lemonade-server. The commands from the PR are:

git clone --branch model/K2Horizon --single-branch https://github.com/MBZUAI-IFM/llama.cpp.git llama.cpp-ifm
cmake -S llama.cpp-ifm -B llama.cpp-ifm/build -DGGML_METAL=ON -DCMAKE_BUILD_TYPE=Release
cmake --build llama.cpp-ifm/build --config Release --target llama-server -j

If you don't have a Mac, then you will need to set the correct GGML_ flags instead of GGML_METAL=ON from the documentation found here: https://github.com/ggml-org/llama.cpp/blob/master/docs/build.md#blas-build.

Upload images, audio, and videos by dragging in the text input, pasting, or clicking here.

Tap or paste here to upload images

· Sign up or log in to comment