Instructions to use thinkingmachines/Inkling-Small-NVFP4 with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use thinkingmachines/Inkling-Small-NVFP4 with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("image-text-to-text", model="thinkingmachines/Inkling-Small-NVFP4")
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"},
            {"type": "text", "text": "What animal is on the candy?"}
        ]
    },
]
pipe(text=messages)

# Load model directly
from transformers import AutoProcessor, AutoModelForMultimodalLM

processor = AutoProcessor.from_pretrained("thinkingmachines/Inkling-Small-NVFP4")
model = AutoModelForMultimodalLM.from_pretrained("thinkingmachines/Inkling-Small-NVFP4", device_map="auto")
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"},
            {"type": "text", "text": "What animal is on the candy?"}
        ]
    },
]
inputs = processor.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=40)
print(processor.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use thinkingmachines/Inkling-Small-NVFP4 with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "thinkingmachines/Inkling-Small-NVFP4"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "thinkingmachines/Inkling-Small-NVFP4",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Use Docker

docker model run hf.co/thinkingmachines/Inkling-Small-NVFP4

SGLang

How to use thinkingmachines/Inkling-Small-NVFP4 with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "thinkingmachines/Inkling-Small-NVFP4" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "thinkingmachines/Inkling-Small-NVFP4",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "thinkingmachines/Inkling-Small-NVFP4" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "thinkingmachines/Inkling-Small-NVFP4",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Docker Model Runner
How to use thinkingmachines/Inkling-Small-NVFP4 with Docker Model Runner:
```
docker model run hf.co/thinkingmachines/Inkling-Small-NVFP4
```

VRAM Usage?

by jtvino - opened about 7 hours ago

Discussion

jtvino

about 7 hours ago

Hi
I am trying to run this model on 2 H200 GPUs; however it looks like it keeps running out of VRAM?

vllm serve thinkingmachines/Inkling-Small-NVFP4
            --served-model-name thinkingmachines/Inkling-Small
            --reasoning-parser inkling
            --tool-call-parser inkling
            --enable-auto-tool-choice
            --tensor-parallel-size 2
            --max-model-len 200000
            --trust-remote-code
            --enable-chunked-prefill
            --enable-prefix-caching
            --kernel-config.enable_flashinfer_autotune=False
            --disable-custom-all-reduce

DEBUG 08-05 15:27:06 [plugins/__init__.py:44] No plugins for group vllm.platform_plugins found.
DEBUG 08-05 15:27:06 [platforms/__init__.py:36] Checking if TPU platform is available.
DEBUG 08-05 15:27:06 [platforms/__init__.py:55] TPU platform is not available because: No module named 'libtpu'
DEBUG 08-05 15:27:06 [platforms/__init__.py:61] Checking if CUDA platform is available.
DEBUG 08-05 15:27:06 [platforms/__init__.py:84] Confirmed CUDA platform is available.
DEBUG 08-05 15:27:06 [platforms/__init__.py:112] Checking if ROCm platform is available.
DEBUG 08-05 15:27:06 [platforms/__init__.py:126] ROCm platform is not available because: No module named 'amdsmi'
DEBUG 08-05 15:27:06 [platforms/__init__.py:133] Checking if XPU platform is available.
DEBUG 08-05 15:27:06 [platforms/__init__.py:164] Checking if CPU platform is available.
DEBUG 08-05 15:27:06 [platforms/__init__.py:61] Checking if CUDA platform is available.
DEBUG 08-05 15:27:06 [platforms/__init__.py:84] Confirmed CUDA platform is available.
DEBUG 08-05 15:27:06 [platforms/__init__.py:245] Automatically detected platform cuda.
DEBUG 08-05 15:27:18 [entrypoints/.../utils/api_utils.py:166] Setting VLLM_WORKER_MULTIPROC_METHOD to 'spawn'
DEBUG 08-05 15:27:18 [plugins/__init__.py:52] Available plugins for group vllm.general_plugins:
DEBUG 08-05 15:27:18 [plugins/__init__.py:54] - lora_filesystem_resolver -> vllm.plugins.lora_resolvers.filesystem_resolver:register_filesystem_resolver
DEBUG 08-05 15:27:18 [plugins/__init__.py:54] - lora_hf_hub_resolver -> vllm.plugins.lora_resolvers.hf_hub_resolver:register_hf_hub_resolver
DEBUG 08-05 15:27:18 [plugins/__init__.py:57] All plugins in this group will be loaded. Set `VLLM_PLUGINS` to control which plugins to load.
(APIServer pid=467) INFO 08-05 15:27:18 [entrypoints/.../utils/api_utils.py:345] 
(APIServer pid=467) INFO 08-05 15:27:18 [entrypoints/.../utils/api_utils.py:345]        █     █     █▄   ▄█
(APIServer pid=467) INFO 08-05 15:27:18 [entrypoints/.../utils/api_utils.py:345]  ▄▄ ▄█ █     █     █ ▀▄▀ █  version 0.26.1rc1.dev306+gcb8104839
(APIServer pid=467) INFO 08-05 15:27:18 [entrypoints/.../utils/api_utils.py:345]   █▄█▀ █     █     █     █  model   thinkingmachines/Inkling-Small-NVFP4
(APIServer pid=467) INFO 08-05 15:27:18 [entrypoints/.../utils/api_utils.py:345]    ▀▀  ▀▀▀▀▀ ▀▀▀▀▀ ▀     ▀
(APIServer pid=467) INFO 08-05 15:27:18 [entrypoints/.../utils/api_utils.py:345] 
(APIServer pid=467) INFO 08-05 15:27:18 [entrypoints/.../utils/api_utils.py:273] non-default args: {'model_tag': 'thinkingmachines/Inkling-Small-NVFP4', 'enable_auto_tool_choice': True, 'tool_call_parser': 'inkling', 'model': 'thinkingmachines/Inkling-Small-NVFP4', 'trust_remote_code': True, 'max_model_len': 200000, 'served_model_name': ['thinkingmachines/Inkling-Small'], 'reasoning_parser': 'inkling', 'tensor_parallel_size': 2, 'disable_custom_all_reduce': True, 'gpu_memory_utilization': 0.85, 'enable_prefix_caching': True, 'enable_chunked_prefill': True, 'kernel_config': KernelConfig(ir_op_priority=IrOpPriorityConfig(rms_norm=[], fused_add_rms_norm=[]), enable_flashinfer_autotune=False, enable_cutedsl_warmup=True, enable_jit_warmup=True, enable_bf16x3_router_gemm=False, moe_backend='auto', linear_backend='auto')}
(APIServer pid=467) WARNING 08-05 15:27:18 [envs.py:2120] Unknown vLLM environment variable detected: VLLM_MULTI_SERVE_PORT_8000_TCP
(APIServer pid=467) WARNING 08-05 15:27:18 [envs.py:2120] Unknown vLLM environment variable detected: VLLM_MULTI_SERVE_SERVICE_PORT
(APIServer pid=467) WARNING 08-05 15:27:18 [envs.py:2120] Unknown vLLM environment variable detected: VLLM_MULTI_SERVE_PORT
(APIServer pid=467) WARNING 08-05 15:27:18 [envs.py:2120] Unknown vLLM environment variable detected: VLLM_MULTI_SERVE_PORT_8000_TCP_ADDR
(APIServer pid=467) WARNING 08-05 15:27:18 [envs.py:2120] Unknown vLLM environment variable detected: VLLM_MULTI_SERVE_PORT_8000_TCP_PORT
(APIServer pid=467) WARNING 08-05 15:27:18 [envs.py:2120] Unknown vLLM environment variable detected: VLLM_MULTI_SERVE_PORT_8000_TCP_PROTO
(APIServer pid=467) WARNING 08-05 15:27:18 [envs.py:2120] Unknown vLLM environment variable detected: VLLM_MULTI_SERVE_SERVICE_HOST
(APIServer pid=467) DEBUG 08-05 15:27:19 [model_executor/models/registry.py:990] Loaded model info for class vllm.models.inkling.InklingForConditionalGeneration from cache
(APIServer pid=467) DEBUG 08-05 15:27:19 [logging_utils/log_time.py:29] Registry inspect model class: Elapsed time 0.0039657 secs
(APIServer pid=467) INFO 08-05 15:27:19 [config/model.py:645] Resolved architecture: InklingForConditionalGeneration
(APIServer pid=467) INFO 08-05 15:27:19 [config/model.py:663] Defaulting to tokenizer_mode='inkling' for InklingForConditionalGeneration
(APIServer pid=467) INFO 08-05 15:27:19 [config/model.py:1883] Using max model len 200000
(APIServer pid=467) DEBUG 08-05 15:27:20 [utils/import_utils.py:73] Loading module triton_kernels from /usr/local/lib/python3.12/dist-packages/vllm/third_party/triton_kernels/__init__.py.
(APIServer pid=467) DEBUG 08-05 15:27:20 [utils/nccl.py:35] Found nccl from library libnccl.so.2
(APIServer pid=467) DEBUG 08-05 15:27:24 [tokenizers/registry.py:83] Loading CachedHfTokenizer for tokenizer_mode='inkling'
(APIServer pid=467) DEBUG 08-05 15:27:26 [compilation/decorators.py:221] Inferred dynamic dimensions for forward method of <class 'vllm.model_executor.models.qwen3_dflash.DFlashQwen3Model'>: ['input_ids', 'positions', 'input_embeds']
(APIServer pid=467) DEBUG 08-05 15:27:26 [compilation/decorators.py:221] Inferred dynamic dimensions for forward method of <class 'vllm.model_executor.models.deepseek_v2.DeepseekV2Model'>: ['input_ids', 'positions', 'intermediate_tensors', 'inputs_embeds']
(APIServer pid=467) DEBUG 08-05 15:27:26 [compilation/decorators.py:221] Inferred dynamic dimensions for forward method of <class 'vllm.model_executor.models.deepseek_mtp.DeepSeekMTP'>: ['input_ids', 'positions', 'hidden_states', 'intermediate_tensors', 'inputs_embeds']
(APIServer pid=467) DEBUG 08-05 15:27:26 [config/model.py:1948] Generative models support chunked prefill.
(APIServer pid=467) DEBUG 08-05 15:27:26 [config/model.py:2003] Generative models support prefix caching.
(APIServer pid=467) WARNING 08-05 15:27:26 [utils/torch_utils.py:355] WARNING: Unknown kv_cache_quant_algo 'none' in model config. Supported values: ['fp8', 'nvfp4']. Falling back to 'auto'.
(APIServer pid=467) DEBUG 08-05 15:27:26 [config/parallel.py:953] Defaulting to use mp for distributed inference
(APIServer pid=467) DEBUG 08-05 15:27:26 [engine/arg_utils.py:2775] Defaulting max_num_batched_tokens to 8192 for OPENAI_API_SERVER usage context.
(APIServer pid=467) DEBUG 08-05 15:27:26 [engine/arg_utils.py:2785] Defaulting max_num_seqs to 1024 for OPENAI_API_SERVER usage context.
(APIServer pid=467) INFO 08-05 15:27:26 [config/scheduler.py:242] Chunked prefill is enabled with max_num_batched_tokens=8192.
(APIServer pid=467) WARNING 08-05 15:27:26 [model_executor/.../quantization/modelopt.py:1019] Detected ModelOpt NVFP4 checkpoint (quant_algo=NVFP4). Please note that the format is experimental and could change in future.
(APIServer pid=467) INFO 08-05 15:27:26 [config/vllm.py:1231] Auto-enabling VLLM_USE_BREAKABLE_CUDAGRAPH=1. Set VLLM_USE_BREAKABLE_CUDAGRAPH=0 to opt out.
(APIServer pid=467) DEBUG 08-05 15:27:26 [config/kernel.py:288] Setting platform-specific IR op priority defaults: IrOpPriorityConfig(rms_norm=['vllm_c', 'native'], fused_add_rms_norm=['vllm_c', 'native']), user-defined: IrOpPriorityConfig(rms_norm=[], fused_add_rms_norm=[])
(APIServer pid=467) INFO 08-05 15:27:26 [config/kernel.py:306] Final IR op priority after setting platform defaults: IrOpPriorityConfig(rms_norm=['vllm_c', 'native'], fused_add_rms_norm=['vllm_c', 'native'])
(APIServer pid=467) WARNING 08-05 15:27:26 [config/vllm.py:2267] Model Runner V2 does not yet support the thinking_token_budget request parameter. Set VLLM_USE_V2_MODEL_RUNNER=0 if this is required.
(APIServer pid=467) DEBUG 08-05 15:27:26 [config/vllm.py:1984] Max num batched tokens below allreduce-rms fusion threshold, allreduce-rms fusion will be enabled for all num_tokens.
(APIServer pid=467) INFO 08-05 15:27:26 [config/compilation.py:329] Enabled custom fusions: norm_quant, act_quant, allreduce_rms
(APIServer pid=467) DEBUG 08-05 15:27:26 [plugins/__init__.py:44] No plugins for group vllm.stat_logger_plugins found.
(APIServer pid=467) DEBUG 08-05 15:27:26 [renderers/registry.py:59] Loading InklingRenderer for renderer_mode='inkling'
(APIServer pid=467) DEBUG 08-05 15:27:26 [utils/torch_utils.py:167] OMP_NUM_THREADS is not set; defaulting Torch threads to 1.
DEBUG 08-05 15:27:28 [plugins/__init__.py:44] No plugins for group vllm.platform_plugins found.
DEBUG 08-05 15:27:28 [platforms/__init__.py:36] Checking if TPU platform is available.
DEBUG 08-05 15:27:28 [platforms/__init__.py:55] TPU platform is not available because: No module named 'libtpu'
DEBUG 08-05 15:27:28 [platforms/__init__.py:61] Checking if CUDA platform is available.
DEBUG 08-05 15:27:28 [platforms/__init__.py:84] Confirmed CUDA platform is available.
DEBUG 08-05 15:27:28 [platforms/__init__.py:112] Checking if ROCm platform is available.
DEBUG 08-05 15:27:28 [platforms/__init__.py:126] ROCm platform is not available because: No module named 'amdsmi'
DEBUG 08-05 15:27:28 [platforms/__init__.py:133] Checking if XPU platform is available.
DEBUG 08-05 15:27:28 [platforms/__init__.py:164] Checking if CPU platform is available.
DEBUG 08-05 15:27:28 [platforms/__init__.py:61] Checking if CUDA platform is available.
DEBUG 08-05 15:27:28 [platforms/__init__.py:84] Confirmed CUDA platform is available.
DEBUG 08-05 15:27:28 [platforms/__init__.py:245] Automatically detected platform cuda.
DEBUG 08-05 15:27:33 [utils/import_utils.py:73] Loading module triton_kernels from /usr/local/lib/python3.12/dist-packages/vllm/third_party/triton_kernels/__init__.py.
DEBUG 08-05 15:27:33 [utils/nccl.py:35] Found nccl from library libnccl.so.2
(EngineCore pid=770) DEBUG 08-05 15:27:34 [v1/engine/core.py:1252] Waiting for init message from front-end.
(APIServer pid=467) DEBUG 08-05 15:27:34 [v1/engine/utils.py:1341] HELLO from local core engine process 0.
(EngineCore pid=770) DEBUG 08-05 15:27:34 [v1/engine/core.py:1263] Received init message: EngineHandshakeMetadata(addresses=EngineZmqAddresses(inputs=['ipc:///tmp/424a4421-bd29-43eb-808a-88172bd80c8c'], outputs=['ipc:///tmp/aed3b494-d4ca-45d4-99d0-bea12685b322'], coordinator_input=None, coordinator_output=None, frontend_stats_publish_address=None), parallel_config={})
(EngineCore pid=770) DEBUG 08-05 15:27:34 [v1/engine/core.py:1056] Has DP Coordinator: False, stats publish address: None
(EngineCore pid=770) DEBUG 08-05 15:27:34 [plugins/__init__.py:52] Available plugins for group vllm.general_plugins:
(EngineCore pid=770) DEBUG 08-05 15:27:34 [plugins/__init__.py:54] - lora_filesystem_resolver -> vllm.plugins.lora_resolvers.filesystem_resolver:register_filesystem_resolver
(EngineCore pid=770) DEBUG 08-05 15:27:34 [plugins/__init__.py:54] - lora_hf_hub_resolver -> vllm.plugins.lora_resolvers.hf_hub_resolver:register_hf_hub_resolver
(EngineCore pid=770) DEBUG 08-05 15:27:34 [plugins/__init__.py:57] All plugins in this group will be loaded. Set `VLLM_PLUGINS` to control which plugins to load.
(EngineCore pid=770) INFO 08-05 15:27:34 [v1/engine/core.py:121] Initializing a V1 LLM engine (v0.26.1rc1.dev306+gcb8104839) with config: model='thinkingmachines/Inkling-Small-NVFP4', speculative_config=None, tokenizer='thinkingmachines/Inkling-Small-NVFP4', skip_tokenizer_init=False, tokenizer_mode=inkling, revision=None, tokenizer_revision=None, trust_remote_code=True, dtype=torch.bfloat16, max_seq_len=200000, download_dir=None, load_format=auto, tensor_parallel_size=2, pipeline_parallel_size=1, data_parallel_size=1, decode_context_parallel_size=1, dcp_comm_backend=ag_rs, disable_custom_all_reduce=True, quantization=modelopt_fp4, quantization_config=None, enforce_eager=False, enable_return_routed_experts=False, kv_cache_dtype=auto, device_config=cuda, structured_outputs_config=StructuredOutputsConfig(backend='auto', disable_any_whitespace=False, disable_additional_properties=False, reasoning_parser='inkling', reasoning_parser_plugin='', enable_in_reasoning=False), observability_config=ObservabilityConfig(show_hidden_metrics_for_version=None, otlp_traces_endpoint=None, collect_detailed_traces=None, kv_cache_metrics=False, kv_cache_metrics_sample=0.01, cudagraph_metrics=False, enable_layerwise_nvtx_tracing=False, enable_mfu_metrics=False, enable_mm_processor_stats=False, enable_logging_iteration_details=False, jit_monitor_mode='warn', jit_monitor_verbose=False), seed=0, served_model_name=thinkingmachines/Inkling-Small, enable_prefix_caching=True, enable_chunked_prefill=True, pooler_config=None, compilation_config={'mode': <CompilationMode.NONE: 0>, 'debug_dump_path': None, 'cache_dir': '', 'compile_cache_save_format': 'binary', 'backend': 'inductor', 'custom_ops': ['all'], 'ir_enable_torch_wrap': False, 'splitting_ops': [], 'compile_mm_encoder': False, 'cudagraph_mm_encoder': False, 'encoder_cudagraph_token_budgets': [], 'encoder_cudagraph_max_vision_items_per_batch': 0, 'encoder_cudagraph_max_frames_per_batch': None, 'compile_sizes': [], 'compile_ranges_endpoints': [8192], 'inductor_compile_config': {'enable_auto_functionalized_v2': False, 'combo_kernels': True, 'benchmark_combo_kernel': True}, 'inductor_passes': {}, 'cudagraph_mode': <CUDAGraphMode.FULL_AND_PIECEWISE: (2, 1)>, 'cudagraph_num_of_warmups': 1, 'cudagraph_capture_sizes': [1, 2, 4, 8, 16, 24, 32, 40, 48, 56, 64, 72, 80, 88, 96, 104, 112, 120, 128, 136, 144, 152, 160, 168, 176, 184, 192, 200, 208, 216, 224, 232, 240, 248, 256, 272, 288, 304, 320, 336, 352, 368, 384, 400, 416, 432, 448, 464, 480, 496, 512], 'cudagraph_copy_inputs': False, 'cudagraph_specialize_lora': True, 'use_inductor_graph_partition': False, 'pass_config': {'fuse_norm_quant': True, 'fuse_act_quant': True, 'fuse_attn_quant': False, 'enable_sp': False, 'fuse_gemm_comms': False, 'fuse_allreduce_rms': True, 'enable_qk_norm_rope_fusion': False, 'fuse_rope_kvcache_cat_mla': False, 'fuse_act_padding': False, 'fuse_qk_norm_rope_kvcache': False}, 'max_cudagraph_capture_size': 512, 'dynamic_shapes_config': {'type': <DynamicShapesType.BACKED: 'backed'>, 'evaluate_guards': False, 'assume_32_bit_indexing': False}, 'local_cache_dir': None, 'fast_moe_cold_start': False, 'static_all_moe_layers': []}, kernel_config=KernelConfig(ir_op_priority=IrOpPriorityConfig(rms_norm=['vllm_c', 'native'], fused_add_rms_norm=['vllm_c', 'native']), enable_flashinfer_autotune=False, enable_cutedsl_warmup=True, enable_jit_warmup=True, enable_bf16x3_router_gemm=False, moe_backend='auto', linear_backend='auto')
(EngineCore pid=770) WARNING 08-05 15:27:34 [v1/executor/multiproc_executor.py:1081] Reducing Torch parallelism from 128 threads to 1 to avoid unnecessary CPU contention. Set OMP_NUM_THREADS in the external environment to tune this value as needed.
(EngineCore pid=770) INFO 08-05 15:27:34 [v1/executor/multiproc_executor.py:140] DP group leader: node_rank=0, node_rank_within_dp=0, master_addr=127.0.0.1, mq_connect_ip=10.130.1.190 (local), world_size=2, local_world_size=2
(EngineCore pid=770) DEBUG 08-05 15:27:34 [distributed/device_communicators/shm_broadcast.py:502] Binding to ipc:///tmp/0649717d-01d8-4049-a940-23b56a37ad2d
(EngineCore pid=770) DEBUG 08-05 15:27:34 [distributed/device_communicators/shm_broadcast.py:555] vLLM message queue communication handle: Handle(local_reader_ranks=[0, 1], buffer_handle=(2, 16777216, 10, 'psm_f81cdfac'), local_subscribe_addr='ipc:///tmp/0649717d-01d8-4049-a940-23b56a37ad2d', local_notify_addr='ipc:///tmp/39a78bbb-a8f0-4f9d-bc1a-0f6d70ad66ea', remote_subscribe_addr=None, remote_addr_ipv6=False)
DEBUG 08-05 15:27:35 [plugins/__init__.py:44] No plugins for group vllm.platform_plugins found.
DEBUG 08-05 15:27:35 [platforms/__init__.py:36] Checking if TPU platform is available.
DEBUG 08-05 15:27:35 [platforms/__init__.py:55] TPU platform is not available because: No module named 'libtpu'
DEBUG 08-05 15:27:35 [platforms/__init__.py:61] Checking if CUDA platform is available.
DEBUG 08-05 15:27:35 [platforms/__init__.py:84] Confirmed CUDA platform is available.
DEBUG 08-05 15:27:35 [plugins/__init__.py:44] No plugins for group vllm.platform_plugins found.
DEBUG 08-05 15:27:35 [platforms/__init__.py:36] Checking if TPU platform is available.
DEBUG 08-05 15:27:35 [platforms/__init__.py:55] TPU platform is not available because: No module named 'libtpu'
DEBUG 08-05 15:27:35 [platforms/__init__.py:61] Checking if CUDA platform is available.
DEBUG 08-05 15:27:35 [platforms/__init__.py:84] Confirmed CUDA platform is available.
DEBUG 08-05 15:27:35 [platforms/__init__.py:112] Checking if ROCm platform is available.
DEBUG 08-05 15:27:35 [platforms/__init__.py:126] ROCm platform is not available because: No module named 'amdsmi'
DEBUG 08-05 15:27:35 [platforms/__init__.py:133] Checking if XPU platform is available.
DEBUG 08-05 15:27:35 [platforms/__init__.py:164] Checking if CPU platform is available.
DEBUG 08-05 15:27:35 [platforms/__init__.py:61] Checking if CUDA platform is available.
DEBUG 08-05 15:27:35 [platforms/__init__.py:84] Confirmed CUDA platform is available.
DEBUG 08-05 15:27:35 [platforms/__init__.py:112] Checking if ROCm platform is available.
DEBUG 08-05 15:27:35 [platforms/__init__.py:126] ROCm platform is not available because: No module named 'amdsmi'
DEBUG 08-05 15:27:35 [platforms/__init__.py:133] Checking if XPU platform is available.
DEBUG 08-05 15:27:35 [platforms/__init__.py:164] Checking if CPU platform is available.
DEBUG 08-05 15:27:35 [platforms/__init__.py:61] Checking if CUDA platform is available.
DEBUG 08-05 15:27:35 [platforms/__init__.py:84] Confirmed CUDA platform is available.
DEBUG 08-05 15:27:35 [platforms/__init__.py:245] Automatically detected platform cuda.
DEBUG 08-05 15:27:35 [platforms/__init__.py:245] Automatically detected platform cuda.
DEBUG 08-05 15:27:40 [utils/import_utils.py:73] Loading module triton_kernels from /usr/local/lib/python3.12/dist-packages/vllm/third_party/triton_kernels/__init__.py.
DEBUG 08-05 15:27:40 [utils/import_utils.py:73] Loading module triton_kernels from /usr/local/lib/python3.12/dist-packages/vllm/third_party/triton_kernels/__init__.py.
DEBUG 08-05 15:27:40 [utils/nccl.py:35] Found nccl from library libnccl.so.2
DEBUG 08-05 15:27:40 [utils/nccl.py:35] Found nccl from library libnccl.so.2
DEBUG 08-05 15:27:41 [plugins/__init__.py:52] Available plugins for group vllm.general_plugins:
DEBUG 08-05 15:27:41 [plugins/__init__.py:54] - lora_filesystem_resolver -> vllm.plugins.lora_resolvers.filesystem_resolver:register_filesystem_resolver
DEBUG 08-05 15:27:41 [plugins/__init__.py:54] - lora_hf_hub_resolver -> vllm.plugins.lora_resolvers.hf_hub_resolver:register_hf_hub_resolver
DEBUG 08-05 15:27:41 [plugins/__init__.py:57] All plugins in this group will be loaded. Set `VLLM_PLUGINS` to control which plugins to load.
DEBUG 08-05 15:27:42 [plugins/__init__.py:52] Available plugins for group vllm.general_plugins:
DEBUG 08-05 15:27:42 [plugins/__init__.py:54] - lora_filesystem_resolver -> vllm.plugins.lora_resolvers.filesystem_resolver:register_filesystem_resolver
DEBUG 08-05 15:27:42 [plugins/__init__.py:54] - lora_hf_hub_resolver -> vllm.plugins.lora_resolvers.hf_hub_resolver:register_hf_hub_resolver
DEBUG 08-05 15:27:42 [plugins/__init__.py:57] All plugins in this group will be loaded. Set `VLLM_PLUGINS` to control which plugins to load.
DEBUG 08-05 15:27:42 [tokenizers/registry.py:83] Loading CachedHfTokenizer for tokenizer_mode='inkling'
DEBUG 08-05 15:27:42 [tokenizers/registry.py:83] Loading CachedHfTokenizer for tokenizer_mode='inkling'
DEBUG 08-05 15:27:44 [config/kernel.py:85] Setting IR op priority for rms_norm to ['vllm_c', 'native']
DEBUG 08-05 15:27:44 [ir/op.py:422] Priority for vllm.ir.rms_norm set to ['vllm_c', 'native']
DEBUG 08-05 15:27:44 [config/kernel.py:85] Setting IR op priority for fused_add_rms_norm to ['vllm_c', 'native']
DEBUG 08-05 15:27:44 [ir/op.py:422] Priority for vllm.ir.fused_add_rms_norm set to ['vllm_c', 'native']
(Worker pid=916) DEBUG 08-05 15:27:44 [distributed/parallel_state.py:1596] world_size=2 rank=0 local_rank=0 distributed_init_method=tcp://127.0.0.1:33065 backend=nccl
(Worker pid=916) INFO 08-05 15:27:44 [distributed/parallel_state.py:1640] world_size=2 rank=0 local_rank=0 distributed_init_method=tcp://127.0.0.1:33065 backend=nccl
DEBUG 08-05 15:27:44 [config/kernel.py:85] Setting IR op priority for rms_norm to ['vllm_c', 'native']
DEBUG 08-05 15:27:44 [ir/op.py:422] Priority for vllm.ir.rms_norm set to ['vllm_c', 'native']
DEBUG 08-05 15:27:44 [config/kernel.py:85] Setting IR op priority for fused_add_rms_norm to ['vllm_c', 'native']
DEBUG 08-05 15:27:44 [ir/op.py:422] Priority for vllm.ir.fused_add_rms_norm set to ['vllm_c', 'native']
(Worker pid=917) DEBUG 08-05 15:27:44 [distributed/parallel_state.py:1596] world_size=2 rank=1 local_rank=1 distributed_init_method=tcp://127.0.0.1:33065 backend=nccl
(Worker pid=917) INFO 08-05 15:27:44 [distributed/parallel_state.py:1640] world_size=2 rank=1 local_rank=1 distributed_init_method=tcp://127.0.0.1:33065 backend=nccl
(Worker pid=917) DEBUG 08-05 15:27:44 [distributed/parallel_state.py:1722] Detected 1 nodes in the distributed environment
(Worker pid=916) DEBUG 08-05 15:27:44 [distributed/parallel_state.py:1722] Detected 1 nodes in the distributed environment
(Worker pid=916) INFO 08-05 15:27:44 [distributed/device_communicators/pynccl.py:113] vLLM is using nccl==2.30.7
(APIServer pid=467) DEBUG 08-05 15:27:44 [v1/engine/utils.py:1243] Waiting for 1 local, 0 remote core engine proc(s) to start.
thinkingmachines-inkling-small-h200-vllm-deploy-69577f8978z5sg5:916:916 [0] NCCL INFO ncclOsDlopen(libnccl-env.so) failed: libnccl-env.so: cannot open shared object file: No such file or directory
thinkingmachines-inkling-small-h200-vllm-deploy-69577f8978z5sg5:916:916 [0] NCCL INFO ENV/Plugin: Could not find: libnccl-env.so
thinkingmachines-inkling-small-h200-vllm-deploy-69577f8978z5sg5:916:916 [0] NCCL INFO Bootstrap: Using eth0:10.130.1.190<0>
thinkingmachines-inkling-small-h200-vllm-deploy-69577f8978z5sg5:916:1117 [0] NCCL INFO Socket handshake magic 0x564ab9f2fc4b9d6c (built-in default)
thinkingmachines-inkling-small-h200-vllm-deploy-69577f8978z5sg5:916:916 [0] NCCL INFO cudaDriverVersion 13000
thinkingmachines-inkling-small-h200-vllm-deploy-69577f8978z5sg5:916:916 [0] NCCL INFO NCCL version 2.30.7+cuda13.3
thinkingmachines-inkling-small-h200-vllm-deploy-69577f8978z5sg5:916:916 [0] NCCL INFO NCCL git version HEAD a5187fa2e
thinkingmachines-inkling-small-h200-vllm-deploy-69577f8978z5sg5:916:916 [0] NCCL INFO ncclOsDlopen(libnccl-net.so) failed: libnccl-net.so: cannot open shared object file: No such file or directory
thinkingmachines-inkling-small-h200-vllm-deploy-69577f8978z5sg5:916:916 [0] NCCL INFO NET/Plugin: Could not find: libnccl-net.so
thinkingmachines-inkling-small-h200-vllm-deploy-69577f8978z5sg5:916:916 [0] NCCL INFO NET/IB : No device found.
thinkingmachines-inkling-small-h200-vllm-deploy-69577f8978z5sg5:916:916 [0] NCCL INFO NET/IB : Using [RO]; OOB eth0:10.130.1.190<0>
thinkingmachines-inkling-small-h200-vllm-deploy-69577f8978z5sg5:916:916 [0] NCCL INFO Failed to initialize NET plugin IB
thinkingmachines-inkling-small-h200-vllm-deploy-69577f8978z5sg5:916:916 [0] NCCL INFO NET/Socket : Using [0]eth0:10.130.1.190<0>
thinkingmachines-inkling-small-h200-vllm-deploy-69577f8978z5sg5:916:916 [0] NCCL INFO Initialized NET plugin Socket
thinkingmachines-inkling-small-h200-vllm-deploy-69577f8978z5sg5:916:916 [0] NCCL INFO Assigned NET plugin Socket to comm
thinkingmachines-inkling-small-h200-vllm-deploy-69577f8978z5sg5:916:916 [0] NCCL INFO ncclOsDlopen(libnccl-rma.so) failed: libnccl-rma.so: cannot open shared object file: No such file or directory
thinkingmachines-inkling-small-h200-vllm-deploy-69577f8978z5sg5:916:916 [0] NCCL INFO RMA/Plugin: Could not find: libnccl-rma.so
thinkingmachines-inkling-small-h200-vllm-deploy-69577f8978z5sg5:916:916 [0] NCCL INFO RMA/Plugin: Failed to load external plugin libnccl-rma.so, dlHandle: (nil), ncclRma: (nil)
thinkingmachines-inkling-small-h200-vllm-deploy-69577f8978z5sg5:916:916 [0] NCCL INFO RMA/Plugin: Failed to initialize any plugin
thinkingmachines-inkling-small-h200-vllm-deploy-69577f8978z5sg5:916:916 [0] NCCL INFO ncclOsDlopen(libnccl-gin.so) failed: libnccl-gin.so: cannot open shared object file: No such file or directory
thinkingmachines-inkling-small-h200-vllm-deploy-69577f8978z5sg5:916:916 [0] NCCL INFO GIN/Plugin: Could not find: libnccl-gin.so
thinkingmachines-inkling-small-h200-vllm-deploy-69577f8978z5sg5:916:916 [0] NCCL INFO GIN/Plugin: Failed to load external plugin libnccl-gin.so, dlHandle: (nil), ncclGin: (nil)
thinkingmachines-inkling-small-h200-vllm-deploy-69577f8978z5sg5:916:916 [0] NCCL INFO GIN/Plugin: Failed to initialize any GIN plugin
thinkingmachines-inkling-small-h200-vllm-deploy-69577f8978z5sg5:916:916 [0] NCCL INFO Using network Socket
thinkingmachines-inkling-small-h200-vllm-deploy-69577f8978z5sg5:916:916 [0] NCCL INFO Loaded NVML from libnvidia-ml.so.1
thinkingmachines-inkling-small-h200-vllm-deploy-69577f8978z5sg5:916:916 [0] NCCL INFO [Rank 0] ncclCommInitRank comm 0x17245930 rank 0 nranks 2 cudaDev 0 nvmlDev 0 busId a1000 commId 0xb01e43a8871f0913 - Init START
thinkingmachines-inkling-small-h200-vllm-deploy-69577f8978z5sg5:916:916 [0] NCCL INFO RAS client listening socket at ::1<28028>
thinkingmachines-inkling-small-h200-vllm-deploy-69577f8978z5sg5:916:916 [0] NCCL INFO Bootstrap timings total 0.001047 (create 0.000038, send 0.000103, recv 0.000266, ring 0.000139, delay 0.000001)
thinkingmachines-inkling-small-h200-vllm-deploy-69577f8978zthinkingmachines-inkling-small-h200-vllm-deploy-69577f8978z5sg5:917:917 [1] NCCL INFO ncclOsDlopen(libnccl-env.so) failed: libnccl-env.so: cannot open shared object file: No such file or directory
thinkingmachines-inkling-small-h200-vllm-deploy-69577f8978z5sg5:917:917 [1] NCCL INFO ENV/Plugin: Could not find: libnccl-env.so
thinkingmachines-inkling-small-h200-vllm-deploy-69577f8978z5sg5:917:917 [1] NCCL INFO cudaDriverVersion 13000
thinkingmachines-inkling-small-h200-vllm-deploy-69577f8978z5sg5:917:917 [1] NCCL INFO Bootstrap: Using eth0:10.130.1.190<0>
thinkingmachines-inkling-small-h200-vllm-deploy-69577f8978z5sg5:917:917 [1] NCCL INFO NCCL version 2.30.7+cuda13.3
thinkingmachines-inkling-small-h200-vllm-deploy-69577f8978z5sg5:917:917 [1] NCCL INFO NCCL git version HEAD a5187fa2e
thinkingmachines-inkling-small-h200-vllm-deploy-69577f8978z5sg5:917:917 [1] NCCL INFO ncclOsDlopen(libnccl-net.so) failed: libnccl-net.so: cannot open shared object file: No such file or directory
thinkingmachines-inkling-small-h200-vllm-deploy-69577f8978z5sg5:917:917 [1] NCCL INFO NET/Plugin: Could not find: libnccl-net.so
thinkingmachines-inkling-small-h200-vllm-deploy-69577f8978z5sg5:917:917 [1] NCCL INFO NET/IB : No device found.
thinkingmachines-inkling-small-h200-vllm-deploy-69577f8978z5sg5:917:917 [1] NCCL INFO NET/IB : Using [RO]; OOB eth0:10.130.1.190<0>
thinkingmachines-inkling-small-h200-vllm-deploy-69577f8978z5sg5:917:917 [1] NCCL INFO Failed to initialize NET plugin IB
thinkingmachines-inkling-small-h200-vllm-deploy-69577f8978z5sg5:917:917 [1] NCCL INFO NET/Socket : Using [0]eth0:10.130.1.190<0>
thinkingmachines-inkling-small-h200-vllm-deploy-69577f8978z5sg5:917:917 [1] NCCL INFO Initialized NET plugin Socket
thinkingmachines-inkling-small-h200-vllm-deploy-69577f8978z5sg5:917:917 [1] NCCL INFO Assigned NET plugin Socket to comm
thinkingmachines-inkling-small-h200-vllm-deploy-69577f8978z5sg5:917:917 [1] NCCL INFO ncclOsDlopen(libnccl-rma.so) failed: libnccl-rma.so: cannot open shared object file: No such file or directory
thinkingmachines-inkling-small-h200-vllm-deploy-69577f8978z5sg5:917:917 [1] NCCL INFO RMA/Plugin: Could not find: libnccl-rma.so
thinkingmachines-inkling-small-h200-vllm-deploy-69577f8978z5sg5:917:917 [1] NCCL INFO RMA/Plugin: Failed to load external plugin libnccl-rma.so, dlHandle: (nil), ncclRma: (nil)
thinkingmachines-inkling-small-h200-vllm-deploy-69577f8978z5sg5:917:917 [1] NCCL INFO RMA/Plugin: Failed to initialize any plugin
thinkingmachines-inkling-small-h200-vllm-deploy-69577f8978z5sg5:917:917 [1] NCCL INFO ncclOsDlopen(libnccl-gin.so) failed: libnccl-gin.so: cannot open shared object file: No such file or directory
thinkingmachines-inkling-small-h200-vllm-deploy-69577f8978z5sg5:917:917 [1] NCCL INFO GIN/Plugin: Could not find: libnccl-gin.so
thinkingmachines-inkling-small-h200-vllm-deploy-69577f8978z5sg5:917:917 [1] NCCL INFO GIN/Plugin: Failed to load external plugin libnccl-gin.so, dlHandle: (nil), ncclGin: (nil)
thinkingmachines-inkling-small-h200-vllm-deploy-69577f8978z5sg5:917:917 [1] NCCL INFO GIN/Plugin: Failed to initialize any GIN plugin
thinkingmachines-inkling-small-h200-vllm-deploy-69577f8978z5sg5:917:917 [1] NCCL INFO Using network Socket
thinkingmachines-inkling-small-h200-vllm-deploy-69577f8978z5sg5:917:917 [1] NCCL INFO Loaded NVML from libnvidia-ml.so.1
thinkingmachines-inkling-small-h200-vllm-deploy-69577f8978z5sg5:917:917 [1] NCCL INFO [Rank 1] ncclCommInitRank comm 0x1961ec10 rank 1 nranks 2 cudaDev 1 nvmlDev 1 busId d1000 commId 0xb01e43a8871f0913 - Init START
thinkingmachines-inkling-small-h200-vllm-deploy-69577f8978z5sg5:917:917 [1] NCCL INFO Socket handshake magic 0x564ab9f2fc4b9d6c (built-in default)
thinkingmachines-inkling-small-h200-vllm-deploy-69577f8978z5sg5:917:917 [1] NCCL INFO RAS client listening socket at ::1<28028>
thinkingmachines-inkling-small-h200-vllm-deploy-69577f8978z5sg5:917:917 [1] NCCL INFO Bootstrap timings total 0.142605 (create 0.000039, send 0.000088, recv 0.141837, ring 0.000030, delay 0.000000)
thinkingmachines-inkling-small-h200-vllm-deploy-69577f8978z55sg5:916:916 [0] NCCL INFO NCCL_P2P_DISABLE set by environment to 1
thinkingmachines-inkling-small-h200-vllm-deploy-69577f8978z5sg5:916:916 [0] NCCL INFO NCCL_IGNORE_CPU_AFFINITY set by environment to 1.
thinkingmachines-inkling-small-h200-vllm-deploy-69577f8978z5sg5:916:916 [0] NCCL INFO ncclTopoGetCpuAffinity: Affinity for GPU 0 is 64-127,192-255. (GPU affinity = 64-127,192-255).
thinkingmachines-inkling-small-h200-vllm-deploy-69577f8978z5sg5:916:916 [0] NCCL INFO NVLS multicast support is not available on dev 0
thinkingmachines-inkling-small-h200-vllm-deploy-69577f8978z5sg5:916:916 [0] NCCL INFO Rank 0: 1 Net devices
thinkingmachines-inkling-small-h200-vllm-deploy-69577f8978z5sg5:916:916 [0] NCCL INFO Rank 0: 0 CollNet devices
thinkingmachines-inkling-small-h200-vllm-deploy-69577f8978z5sg5:916:916 [0] NCCL INFO Local Net device counts across ranks: min 1 max 1
thinkingmachines-inkling-small-h200-vllm-deploy-69577f8978z5sg5:916:916 [0] NCCL INFO Local CollNet device counts across ranks: min 0 max 0
thinkingmachines-inkling-small-h200-vllm-deploy-69577f8978z5sg5:916:916 [0] NCCL INFO comm 0x17245930 rank 0 nRanks 2 nNodes 1 localRanks 2 localRank 0 MNNVL 0
thinkingmachines-inkling-small-h200-vllm-deploy-69577f8978z5sg5:916:916 [0] NCCL INFO Channel 00/04 : 0 1
thinkingmachines-inkling-small-h200-vllm-deploy-69577f8978z5sg5:916:916 [0] NCCL INFO Channel 01/04 : 0 1
thinkingmachines-inkling-small-h200-vllm-deploy-69577f8978z5sg5:916:916 [0] NCCL INFO Channel 02/04 : 0 1
thinkingmachines-inkling-small-h200-vllm-deploy-69577f8978z5sg5:916:916 [0] NCCL INFO Channel 03/04 : 0 1
thinkingmachines-inkling-small-h200-vllm-deploy-69577f8978z5sg5:916:916 [0] NCCL INFO Trees [0] 1/-1/-1->0->-1 [1] -1/-1/-1->0->1 [2] 1/-1/-1->0->-1 [3] -1/-1/-1->0->1
thinkingmachines-inkling-small-h200-vllm-deploy-69577f8978z5sg5:916:916 [0] NCCL INFO P2P Chunksize set to 131072
thinkingmachines-inkling-small-h200-vllm-deploy-69577f8978z5sg5:916:916 [0] NCCL INFO ncclOsDlopen(libnccl-profiler.so) failed: libnccl-profiler.so: cannot open shared object file: No such file or directory
thinkingmachines-inkling-small-h200-vllm-deploy-69577f8978z5sg5:916:916 [0] NCCL INFO PROFILER/Plugin: Could not find: libnccl-profiler.so
thinkingmachines-inkling-small-h200-vllm-deploy-69577f8978z5sg5:916:916 [0] NCCL INFO Check P2P Type isAllDirectP2p 0 directMode 0 isAllCudaP2p 1
thinkingmachines-inkling-small-h200-vllm-deploy-69577f8978z5sg5:916:1121 [0] NCCL INFO [Proxy Service] Device 0 CPU core 220
thinkingmachines-inkling-small-h200-vllm-deploy-69577f8978z5sg5:916:1121 [0] NCCL INFO proxy listening socket at 10.130.1.190<39107>
thinkingmachines-inkling-small-h200-vllm-deploy-69577f8978z5sg5:916:1122 [0] NCCL INFO [Proxy Service UDS] Device 0 CPU core 76
thinkingmachines-inkling-small-h200-vllm-deploy-69577f8978z5sg5:916:916 [0] NCCL INFO ncclOsDlopen(libnccl-tuner.so) failed: libnccl-tuner.so: cannot open shared object file: No such file or directory
thinkingmachines-inkling-small-h200-vllm-deploy-69577f8978z5sg5:916:916 [0] NCCL INFO TUNER/Plugin: Could not find: libnccl-tuner.so
thinkingmachines-inkling-small-h200-vllm-deploy-69577f8978z5sg5:916:916 [0] NCCL INFO threadThresholds 8/8/64 | 16/8/64 | 512 | 512
thinkingmachines-inkling-small-h200-vllm-deploy-69577f8978z5sg5:916:916 [0] NCCL INFO 4 coll channels, 4 collnet channels, 0 nvls channels, 4 p2p channels, 2 p2p channels per peer
thinkingmachines-inkling-small-h200-vllm-deploy-69577f8978z5sg5:916:916 [0] NCCL INFO CC Off, workFifoBytes 1048576
thinkingmachines-inkling-small-h200-vllm-deploy-69577f8978z5sg5:916:916 [0] NCCL INFO ncclCommInitRank comm 0x17245930 rank 0 nranks 2 cudaDev 0 nvmlDev 0 busId a1000 commId 0xb01e43a8871f0913 - Init COMPLETE
thinkingmachines-inkling-small-h200-vllm-deploy-69577f8978z5sg5:916:916 [0] NCCL INFO Init timings - ncclCommInitRank: rank 0 nranks 2 total 0.10 (kernels 0.09, alloc 0.00, bootstrap 0.00, allgathers 0.00, topo 0.01, graphs 0.00, connections 0.00, rest 0.00)
thinkingmachines-inkling-small-h200-vllm-deploy-69577f8978z5sg5:916:916 [0] NCCL INFO Symmetric VA size=14(Worker pid=917) WARNING 08-05 15:27:44 [distributed/device_communicators/symm_mem.py:107] SymmMemCommunicator: symmetric memory multicast operations are not supported.
(Worker pid=916) WARNING 08-05 15:27:44 [distributed/device_communicators/symm_mem.py:107] SymmMemCommunicator: symmetric memory multicast operations are not supported.
(Worker pid=916) INFO 08-05 15:27:44 [distributed/device_communicators/cuda_communicator.py:266] Using ['PYNCCL'] all-reduce backends (in dispatch order) for group 'tp:0' out of potential backends: ['NCCL_SYMM_MEM', 'QUICK_REDUCE', 'FLASHINFER', 'AITER_CUSTOM', 'CUSTOM', 'SYMM_MEM', 'PYNCCL'].
(Worker pid=916) DEBUG 08-05 15:27:44 [distributed/device_communicators/shm_broadcast.py:502] Binding to ipc:///tmp/6236dad2-1642-4143-aa62-b7dbbf82befb
(Worker pid=916) DEBUG 08-05 15:27:44 [distributed/device_communicators/shm_broadcast.py:555] vLLM message queue communication handle: Handle(local_reader_ranks=[1], buffer_handle=(1, 4194304, 6, 'psm_172b091f'), local_subscribe_addr='ipc:///tmp/6236dad2-1642-4143-aa62-b7dbbf82befb', local_notify_addr='ipc:///tmp/d897c86c-6fa8-49f5-afe2-44f86787e380', remote_subscribe_addr=None, remote_addr_ipv6=False)
(Worker pid=917) DEBUG 08-05 15:27:44 [distributed/device_communicators/shm_broadcast.py:579] Connecting to ipc:///tmp/6236dad2-1642-4143-aa62-b7dbbf82befb
sg5:917:917 [1] NCCL INFO NCCL_P2P_DISABLE set by environment to 1
thinkingmachines-inkling-small-h200-vllm-deploy-69577f8978z5sg5:917:917 [1] NCCL INFO NCCL_IGNORE_CPU_AFFINITY set by environment to 1.
thinkingmachines-inkling-small-h200-vllm-deploy-69577f8978z5sg5:917:917 [1] NCCL INFO ncclTopoGetCpuAffinity: Affinity for GPU 1 is 64-127,192-255. (GPU affinity = 64-127,192-255).
thinkingmachines-inkling-small-h200-vllm-deploy-69577f8978z5sg5:917:917 [1] NCCL INFO NVLS multicast support is not available on dev 1
thinkingmachines-inkling-small-h200-vllm-deploy-69577f8978z5sg5:917:917 [1] NCCL INFO Rank 1: 1 Net devices
thinkingmachines-inkling-small-h200-vllm-deploy-69577f8978z5sg5:917:917 [1] NCCL INFO Rank 1: 0 CollNet devices
thinkingmachines-inkling-small-h200-vllm-deploy-69577f8978z5sg5:917:917 [1] NCCL INFO comm 0x1961ec10 rank 1 nRanks 2 nNodes 1 localRanks 2 localRank 1 MNNVL 0
thinkingmachines-inkling-small-h200-vllm-deploy-69577f8978z5sg5:917:917 [1] NCCL INFO Trees [0] -1/-1/-1->1->0 [1] 0/-1/-1->1->-1 [2] -1/-1/-1->1->0 [3] 0/-1/-1->1->-1
thinkingmachines-inkling-small-h200-vllm-deploy-69577f8978z5sg5:917:917 [1] NCCL INFO P2P Chunksize set to 131072
thinkingmachines-inkling-small-h200-vllm-deploy-69577f8978z5sg5:917:917 [1] NCCL INFO ncclOsDlopen(libnccl-profiler.so) failed: libnccl-profiler.so: cannot open shared object file: No such file or directory
thinkingmachines-inkling-small-h200-vllm-deploy-69577f8978z5sg5:917:917 [1] NCCL INFO PROFILER/Plugin: Could not find: libnccl-profiler.so
thinkingmachines-inkling-small-h200-vllm-deploy-69577f8978z5sg5:917:917 [1] NCCL INFO Check P2P Type isAllDirectP2p 0 directMode 0 isAllCudaP2p 1
thinkingmachines-inkling-small-h200-vllm-deploy-69577f8978z5sg5:917:1124 [0] NCCL INFO [Proxy Service UDS] Device 1 CPU core 220
thinkingmachines-inkling-small-h200-vllm-deploy-69577f8978z5sg5:917:1123 [0] NCCL INFO [Proxy Service] Device 1 CPU core 115
thinkingmachines-inkling-small-h200-vllm-deploy-69577f8978z5sg5:917:1123 [1] NCCL INFO proxy listening socket at 10.130.1.190<39423>
thinkingmachines-inkling-small-h200-vllm-deploy-69577f8978z5sg5:917:917 [1] NCCL INFO ncclOsDlopen(libnccl-tuner.so) failed: libnccl-tuner.so: cannot open shared object file: No such file or directory
thinkingmachines-inkling-small-h200-vllm-deploy-69577f8978z5sg5:917:917 [1] NCCL INFO TUNER/Plugin: Could not find: libnccl-tuner.so
thinkingmachines-inkling-small-h200-vllm-deploy-69577f8978z5sg5:917:917 [1] NCCL INFO threadThresholds 8/8/64 | 16/8/64 | 512 | 512
thinkingmachines-inkling-small-h200-vllm-deploy-69577f8978z5sg5:917:917 [1] NCCL INFO 4 coll channels, 4 collnet channels, 0 nvls channels, 4 p2p channels, 2 p2p channels per peer
thinkingmachines-inkling-small-h200-vllm-deploy-69577f8978z5sg5:917:917 [1] NCCL INFO ncclCommInitRank comm 0x1961ec10 rank 1 nranks 2 cudaDev 1 nvmlDev 1 busId d1000 commId 0xb01e43a8871f0913 - Init COMPLETE
thinkingmachines-inkling-small-h200-vllm-deploy-69577f8978z5sg5:917:917 [1] NCCL INFO Init timings - ncclCommInitRank: rank 1 nranks 2 total 0.25 (kernels 0.09, alloc 0.00, bootstrap 0.14, allgathers 0.00, topo 0.01, graphs 0.00, connections 0.00, rest 0.00)
thinkingmachines-inkling-small-h200-vllm-deploy-69577f8978z5sg5:917:917 [1] NCCL INFO Symmetric VA size=140GB
thinkingmachines-inkling-small-h200-vllm-deploy-69577f8978z5sg5:917:917 [1] NCCL INFO Channel 00 : 1[1] -> 0[0] via SHM/direct
thinkingmachines-inkling-small-h200-vllm-deploy-69577f8978z5sg5:917:917 [1] NCCL INFO Channel 01 : 1[1] -> 0[0] via SHM/direct
thinkingmachines-inkling-small-h200-vllm-deploy-69577f8978z5sg5:917:917 [1] NCCL INFO Channel 02 : 1[1] -> 0[0] via SHM/direct
thinkingmachines-inkling-small-h200-vllm-deploy-69577f8978z5sg5:917:917 [1] NCCL INFO Channel 03 : 1[1] -> 0[0] via SHM/direct
thinkingmachines-inkling-small-h200-vllm-deploy-69577f8978z5sg5:917:917 [1] NCCL INFO Connected all rings, use ring PXN 0 GDR 1
thinkingmachines-inkling-small-h200-vllm-deploy-69577f8978z5sg5:917:917 [1] NCCL INFO Initialized NET plugin Socket
thinkingmachines-inkling-small-h200-vllm-deploy-69577f8978z50GB
thinkingmachines-inkling-small-h200-vllm-deploy-69577f8978z5sg5:916:916 [0] NCCL INFO Channel 00 : 0[0] -> 1[1] via SHM/direct
thinkingmachines-inkling-small-h200-vllm-deploy-69577f8978z5sg5:916:916 [0] NCCL INFO Channel 01 : 0[0] -> 1[1] via SHM/direct
thinkingmachines-inkling-small-h200-vllm-deploy-69577f8978z5sg5:916:916 [0] NCCL INFO Channel 02 : 0[0] -> 1[1] via SHM/direct
thinkingmachines-inkling-small-h200-vllm-deploy-69577f8978z5sg5:916:916 [0] NCCL INFO Channel 03 : 0[0] -> 1[1] via SHM/direct
thinkingmachines-inkling-small-h200-vllm-deploy-69577f8978z5sg5:916:916 [0] NCCL INFO Connected all rings, use ring PXN 0 GDR 1
thinkingmachines-inkling-small-h200-vllm-deploy-69577f8978z5sg5:916:916 [0] NCCL INFO Initialized NET plugin Socket
thinkingmachines-inkling-small-h200-vllm-deploy-69577f8978z5sg5:916:916 [0] NCCL INFO Assigned NET plugin Socket to comm
thinkingmachines-inkling-small-h200-vllm-deploy-69577f8978z5sg5:916:916 [0] NCCL INFO RMA/Plugin: Failed to initialize any plugin
thinkingmachines-inkling-small-h200-vllm-deploy-69577f8978z5sg5:916:916 [0] NCCL INFO GIN/Plugin: Failed to initialize any GIN plugin
thinkingmachines-inkling-small-h200-vllm-deploy-69577f8978z5sg5:916:916 [0] NCCL INFO Using network Socket
thinkingmachines-inkling-small-h200-vllm-deploy-69577f8978z5sg5:916:916 [0] NCCL INFO [Rank 0] ncclCommInitRank comm 0x1f658970 rank 0 nranks 2 cudaDev 0 nvmlDev 0 busId a1000 commId 0x5c3ff7f5395fd00c - Init START
thinkingmachines-inkling-small-h200-vllm-deploy-69577f8978z5sg5:916:916 [0] NCCL INFO Bootstrap timings total 0.000365 (create 0.000023, send 0.000070, recv 0.000145, ring 0.000006, delay 0.000000)
thinkingmachines-inkling-small-h200-vllm-deploy-69577f8978z5sg5:916:916 [0] NCCL INFO ncclTopoGetCpuAffinity: Affinity for GPU 0 is 64-127,192-255. (GPU affinity = 64-127,192-255).
thinkingmachines-inkling-small-h200-vllm-deploy-69577f8978z5sg5:916:916 [0] NCCL INFO NVLS multicast support is not available on dev 0
thinkingmachines-inkling-small-h200-vllm-deploy-69577f8978z5sg5:916:916 [0] NCCL INFO Rank 0: 1 Net devices
thinkingmachines-inkling-small-h200-vllm-deploy-69577f8978z5sg5:916:916 [0] NCCL INFO Rank 0: 0 CollNet devices
thinkingmachines-inkling-small-h200-vllm-deploy-69577f8978z5sg5:916:916 [0] NCCL INFO Local Net device counts across ranks: min 1 max 1
thinkingmachines-inkling-small-h200-vllm-deploy-69577f8978z5sg5:916:916 [0] NCCL INFO Local CollNet device counts across ranks: min 0 max 0
thinkingmachines-inkling-small-h200-vllm-deploy-69577f8978z5sg5:916:916 [0] NCCL INFO comm 0x1f658970 rank 0 nRanks 2 nNodes 1 localRanks 2 localRank 0 MNNVL 0
thinkingmachines-inkling-small-h200-vllm-deploy-69577f8978z5sg5:916:916 [0] NCCL INFO Channel 00/04 : 0 1
thinkingmachines-inkling-small-h200-vllm-deploy-69577f8978z5sg5:916:916 [0] NCCL INFO Channel 01/04 : 0 1
thinkingmachines-inkling-small-h200-vllm-deploy-69577f8978z5sg5:916:916 [0] NCCL INFO Channel 02/04 : 0 1
thinkingmachines-inkling-small-h200-vllm-deploy-69577f8978z5sg5:916:916 [0] NCCL INFO Channel 03/04 : 0 1
thinkingmachines-inkling-small-h200-vllm-deploy-69577f8978z5sg5:916:916 [0] NCCL INFO Trees [0] 1/-1/-1->0->-1 [1] -1/-1/-1->0->1 [2] 1/-1/-1->0->-1 [3] -1/-1/-1->0->1
thinkingmachines-inkling-small-h200-vllm-deploy-69577f8978z5sg5:916:916 [0] NCCL INFO P2P Chunksize set to 131072
thinkingmachines-inkling-small-h200-vllm-deploy-69577f8978z5sg5:916:916 [0] NCCL INFO Check P2P Type isAllDirectP2p 0 directMode 0 isAllCudaP2p 1
thinkingmachines-inkling-small-h200-vllm-deploy-69577f8978z5sg5:916:1180 [0] NCCL INFO [Proxy Service] Device 0 CPU core 230
thinkingmachines-inkling-small-h200-vllm-deploy-69577f8978z5sg5:916:1180 [0] NCCL INFO proxy listening socket at 10.130.1.190<38975>
thinkingmachines-inkling-small-h200-vllm-deploy-69577f8978z5sg5:916:1183 [0] NCCL INFO [Proxy Service UDS] Device 0 CPU core 201
thinkingmachines-inkling-small-h200-vllm-deploy-69577f8978z5sg5:916:916 [0] NCCL INFO threadThresholds 8/8/64 | 16/8/64 | 512 | 512
thinkingmachines-inkling-small-h200-vllm-deploy-69577f8978z5sg5:916:916 [0] NCCL INFO 4 coll(Worker pid=916) INFO 08-05 15:27:45 [distributed/device_communicators/cuda_communicator.py:266] Using ['PYNCCL'] all-reduce backends (in dispatch order) for group 'ep:0' out of potential backends: ['NCCL_SYMM_MEM', 'QUICK_REDUCE', 'FLASHINFER', 'AITER_CUSTOM', 'CUSTOM', 'SYMM_MEM', 'PYNCCL'].
(Worker pid=916) INFO 08-05 15:27:45 [distributed/parallel_state.py:1977] rank 0 in world size 2 is assigned as DP rank 0, PP rank 0, PCP rank 0, TP rank 0, EP rank 0, EPLB rank N/A
(Worker pid=916) INFO 08-05 15:27:45 [v1/worker/gpu_worker.py:385] Using V2 Model Runner
(Worker pid=916) DEBUG 08-05 15:27:45 [v1/worker/gpu_worker.py:397] worker init memory snapshot: torch_peak=0.0GiB, torch_allocated=0.0GiB, free_memory=138.73GiB, total_memory=139.8GiB, cuda_memory=1.07GiB, torch_memory=0.0GiB, non_torch_memory=1.07GiB, timestamp=1785943665.4141102, auto_measure=True
(Worker pid=916) DEBUG 08-05 15:27:45 [v1/worker/gpu_worker.py:398] worker requested memory: 118.83GiB
(Worker pid=917) DEBUG 08-05 15:27:45 [v1/worker/gpu_worker.py:397] worker init memory snapshot: torch_peak=0.0GiB, torch_allocated=0.0GiB, free_memory=138.73GiB, total_memory=139.8GiB, cuda_memory=1.07GiB, torch_memory=0.0GiB, non_torch_memory=1.07GiB, timestamp=1785943665.4269004, auto_measure=True
(Worker pid=917) DEBUG 08-05 15:27:45 [v1/worker/gpu_worker.py:398] worker requested memory: 118.83GiB
(Worker_TP0 pid=916) INFO 08-05 15:27:45 [v1/worker/gpu/model_runner.py:326] Loading model from scratch...
(Worker_TP0 pid=916) DEBUG 08-05 15:27:46 [model_executor/.../fused_moe/layer.py:68] FusedMoEParallelConfig = FusedMoEParallelConfig(tp_size=2, pcp_size=1, dp_size=1, ep_size=1, tp_rank=0, pcp_rank=0, dp_rank=0, ep_rank=0, sp_size=1, use_ep=False, all2all_backend='allgather_reducescatter', enable_eplb=False)
(Worker_TP0 pid=916) DEBUG 08-05 15:27:46 [compilation/decorators.py:221] Inferred dynamic dimensions for forward method of <class 'vllm.model_executor.models.commandr.CohereModel'>: ['input_ids', 'positions', 'intermediate_tensors', 'inputs_embeds']
(Worker_TP1 pid=917) DEBUG 08-05 15:27:46 [model_executor/.../fused_moe/layer.py:68] FusedMoEParallelConfig = FusedMoEParallelConfig(tp_size=2, pcp_size=1, dp_size=1, ep_size=1, tp_rank=1, pcp_rank=0, dp_rank=0, ep_rank=0, sp_size=1, use_ep=False, all2all_backend='allgather_reducescatter', enable_eplb=False)
...
(Worker_TP0 pid=916) DEBUG 08-05 15:27:49 [model_executor/models/utils.py:282] Loaded weight model.layers.7.attn.k_norm.weight with shape torch.Size([128])
(Worker_TP0 pid=916) DEBUG 08-05 15:27:49 [model_executor/models/utils.py:282] Loaded weight model.layers.7.attn.k_sconv.weight with shape torch.Size([512, 1, 4])
(Worker_TP1 pid=917) DEBUG 08-05 15:27:50 [model_executor/models/utils.py:282] Loaded weight model.layers.39.attn.k_sconv.weight with shape torch.Size([512, 1, 4])
(Worker_TP0 pid=916) DEBUG 08-05 15:27:50 [model_executor/models/utils.py:282] Loaded weight model.layers.7.mlp.gate.global_scale with shape torch.Size([1])
(Worker_TP0 pid=916) DEBUG 08-05 15:27:50 [model_executor/models/utils.py:282] Loaded weight model.layers.9.attn_sconv.weight with shape torch.Size([2048, 1, 4])
(Worker_TP1 pid=917) DEBUG 08-05 15:27:50 [model_executor/models/utils.py:282] Loaded weight model.layers.39.mlp.gate.bias with shape torch.Size([256])
(Worker_TP1 pid=917) DEBUG 08-05 15:27:50 [model_executor/models/utils.py:282] Loaded weight model.layers.4.attn.k_norm.weight with shape torch.Size([128])
(Worker_TP1 pid=917) DEBUG 08-05 15:27:50 [model_executor/models/utils.py:282] Loaded weight model.layers.40.attn.v_sconv.weight with shape torch.Size([512, 1, 4])
 channels, 4 collnet channels, 0 nvls channels, 4 p2p channels, 2 p2p channels per peer
thinkingmachines-inkling-small-h200-vllm-deploy-69577f8978z5sg5:916:916 [0] NCCL INFO CC Off, workFifoBytes 1048576
thinkingmachines-inkling-small-h200-vllm-deploy-69577f8978z5sg5:916:916 [0] NCCL INFO ncclCommInitRank comm 0x1f658970 rank 0 nranks 2 cudaDev 0 nvmlDev 0 busId a1000 commId 0x5c3ff7f5395fd00c - Init COMPLETE
thinkingmachines-inkling-small-h200-vllm-deploy-69577f8978z5sg5:916:916 [0] NCCL INFO Init timings - ncclCommInitRank: rank 0 nranks 2 total 0.01 (kernels 0.00, alloc 0.00, bootstrap 0.00, allgathers 0.00, topo 0.00, graphs 0.00, connections 0.00, rest 0.00)
thinkingmachines-inkling-small-h200-vllm-deploy-69577f8978z5sg5:916:916 [0] NCCL INFO Symmetric VA size=140GB
thinkingmachines-inkling-small-h200-vllm-deploy-69577f8978z5sg5:916:916 [0] NCCL INFO Channel 00 : 0[0] -> 1[1] via SHM/direct
thinkingmachines-inkling-small-h200-vllm-deploy-69577f8978z5sg5:916:916 [0] NCCL INFO Channel 01 : 0[0] -> 1[1] via SHM/direct
thinkingmachines-inkling-small-h200-vllm-deploy-69577f8978z5sg5:916:916 [0] NCCL INFO Channel 02 : 0[0] -> 1[1] via SHM/direct
thinkingmachines-inkling-small-h200-vllm-deploy-69577f8978z5sg5:916:916 [0] NCCL INFO Channel 03 : 0[0] -> 1[1] via SHM/direct
thinkingmachines-inkling-small-h200-vllm-deploy-69577f8978z5sg5:916:916 [0] NCCL INFO Connected all rings, use ring PXN 0 GDR 1
[rank0]:[W805 15:27:50.651590942 CUDACachingAllocator.cpp:3933] memory allocation failed with OOM on device 0 while trying to allocate 20971520 bytes (free: 9502720, total: 150109880320).
[rank0]:[W805 15:27:50.652804488 CUDACachingAllocator.cpp:3933] memory allocation failed with OOM on device 0 while trying to allocate 20971520 bytes (free: 9502720, total: 150109880320).
[rank0]:[W805 15:27:50.653856615 CUDACachingAllocator.cpp:3933] memory allocation failed with OOM on device 0 while trying to allocate 20971520 bytes (free: 9502720, total: 150109880320).
[rank0]:[W805 15:27:50.654884212 CUDACachingAllocator.cpp:3933] memory allocation failed with OOM on device 0 while trying to allocate 20971520 bytes (free: 9502720, total: 150109880320).
[rank0]:[W805 15:27:50.655913239 CUDACachingAllocator.cpp:3933] memory allocation failed with OOM on device 0 while trying to allocate 20971520 bytes (free: 9502720, total: 150109880320).
[rank0]:[W805 15:27:50.657039706 CUDACachingAllocator.cpp:3933] memory allocation failed with OOM on device 0 while trying to allocate 20971520 bytes (free: 9502720, total: 150109880320).
[rank0]:[W805 15:27:50.658054933 CUDACachingAllocator.cpp:3933] memory allocation failed with OOM on device 0 while trying to allocate 20971520 bytes (free: 9502720, total: 150109880320).
[rank0]:[W805 15:27:50.659086670 CUDACachingAllocator.cpp:3933] memory allocation failed with OOM on device 0 while trying to allocate 20971520 bytes (free: 9502720, total: 150109880320).
[rank0]:[W805 15:27:50.660081537 CUDACachingAllocator.cpp:3933] memory allocation failed with OOM on device 0 while trying to allocate 20971520 bytes (free: 9502720, total: 150109880320).
[rank0]:[W805 15:27:50.661083304 CUDACachingAllocator.cpp:3933] memory allocation failed with OOM on device 0 while trying to allocate 20971520 bytes (free: 9502720, total: 150109880320).
[rank0]:[W805 15:27:50.662108681 CUDACachingAllocator.cpp:3933] memory allocation failed with OOM on device 0 while trying to allocate 20971520 bytes (free: 9502720, total: 150109880320).
[rank0]:[W805 15:27:50.663127598 CUDACachingAllocator.cpp:3933] memory allocation failed with OOM on device 0 while trying to allocate 20971520 bytes (free: 9502720, total: 150109880320).
[rank0]:[W805 15:27:50.664148955 CUDACachingAllocator.cpp:3933] memory allocation failed with OOM on device 0 while trying to allocate 20971520 bytes (free: 9502720, total: 150109880320).
[rank0]:[W805 15:27:50.665179332 CUDACachingAllocator.cpp:3933] memory allocation failed with OOM on device 0 while trying to allocate 20971520 bytes (free: 9502720, total: 150109880320).
[rank0]:[W805 15:27:50.666200629 CUDACachingAllocator.cpp:3933] memory allocation failed with OOM on device 0 while trying to allocate 20971520 bytes (free: 9502720, total: 150109880320).
[rank0]:[W805 15:27:50.667217606 CUDACachingAllocator.cpp:3933] memory allocation failed with OOM on device 0 while trying to allocate 20971520 bytes (free: 9502720, total: 150109880320).
(Worker_TP1 pid=917) DEBUG 08-05 15:27:50 [model_executor/layers/linear.py:1012] Loaded shard 1 with shape torch.Size([1024, 4096]) into model.layers.6.attn.qkvr.weight
[rank0]:[W805 15:27:50.668220333 CUDACachingAllocator.cpp:3933] memory allocation failed with OOM on device 0 while trying to allocate 20971520 bytes (free: 9502720, total: 150109880320).
[rank0]:[W805 15:27:50.669208870 CUDACachingAllocator.cpp:3933] memory allocation failed with OOM on device 0 while trying to allocate 20971520 bytes (free: 9502720, total: 150109880320).
[rank0]:[W805 15:27:50.670221587 CUDACachingAllocator.cpp:3933] memory allocation failed with OOM on device 0 while trying to allocate 20971520 bytes (free: 9502720, total: 150109880320).
[rank0]:[W805 15:27:50.671221284 CUDACachingAllocator.cpp:3933] memory allocation failed with OOM on device 0 while trying to allocate 20971520 bytes (free: 9502720, total: 150109880320).
(Worker_TP1 pid=917) DEBUG 08-05 15:27:50 [model_executor/models/utils.py:282] Loaded weight model.layers.6.attn.wo_ud.weight with shape torch.Size([4096, 2048])
[rank0]:[W805 15:27:50.672235131 CUDACachingAllocator.cpp:3933] memory allocation failed with OOM on device 0 while trying to allocate 20971520 bytes (free: 9502720, total: 150109880320).
(Worker_TP1 pid=917) DEBUG 08-05 15:27:50 [model_executor/models/utils.py:282] Loaded weight model.layers.6.mlp_norm.weight with shape torch.Size([4096])
(Worker_TP1 pid=917) DEBUG 08-05 15:27:50 [model_executor/models/utils.py:282] Loaded weight model.layers.7.attn.k_norm.weight with shape torch.Size([128])
(Worker_TP1 pid=917) DEBUG 08-05 15:27:50 [model_executor/models/utils.py:282] Loaded weight model.layers.7.attn.k_sconv.weight with shape torch.Size([512, 1, 4])
[rank0]:[W805 15:27:50.673279748 CUDACachingAllocator.cpp:3933] memory allocation failed with OOM on device 0 while trying to allocate 20971520 bytes (free: 9502720, total: 150109880320).
[rank0]:[W805 15:27:50.674318785 CUDACachingAllocator.cpp:3933] memory allocation failed with OOM on device 0 while trying to allocate 20971520 bytes (free: 9502720, total: 150109880320).
(Worker_TP0 pid=916) 
Loading safetensors checkpoint shards:  20% Completed | 2/10 [00:03<00:13,  1.64s/it]
(Worker_TP1 pid=917) DEBUG 08-05 15:27:50 [model_executor/models/utils.py:282] Loaded weight model.layers.7.mlp.gate.global_scale with shape torch.Size([1])
(Worker_TP1 pid=917) DEBUG 08-05 15:27:50 [model_executor/models/utils.py:282] Loaded weight model.layers.9.attn_sconv.weight with shape torch.Size([2048, 1, 4])
(Worker_TP0 pid=916) DEBUG 08-05 15:27:50 [model_executor/models/utils.py:282] Loaded weight model.layers.1.attn.v_sconv.weight with shape torch.Size([512, 1, 4])
(Worker_TP0 pid=916) DEBUG 08-05 15:27:50 [model_executor/models/utils.py:282] Loaded weight model.layers.1.mlp.global_scale with shape torch.Size([1])
sg5:917:917 [1] NCCL INFO Assigned NET plugin Socket to comm
thinkingmachines-inkling-small-h200-vllm-deploy-69577f8978z5sg5:917:917 [1] NCCL INFO RMA/Plugin: Failed to initialize any plugin
thinkingmachines-inkling-small-h200-vllm-deploy-69577f8978z5sg5:917:917 [1] NCCL INFO GIN/Plugin: Failed to initialize any GIN plugin
thinkingmachines-inkling-small-h200-vllm-deploy-69577f8978z5sg5:917:917 [1] NCCL INFO Using network Socket
thinkingmachines-inkling-small-h200-vllm-deploy-69577f8978z5sg5:917:917 [1] NCCL INFO [Rank 1] ncclCommInitRank comm 0x1694f800 rank 1 nranks 2 cudaDev 1 nvmlDev 1 busId d1000 commId 0x5c3ff7f5395fd00c - Init START
thinkingmachines-inkling-small-h200-vllm-deploy-69577f8978z5sg5:917:917 [1] NCCL INFO Bootstrap timings total 0.000363 (create 0.000026, send 0.000066, recv 0.000187, ring 0.000007, delay 0.000000)
thinkingmachines-inkling-small-h200-vllm-deploy-69577f8978z5sg5:917:917 [1] NCCL INFO ncclTopoGetCpuAffinity: Affinity for GPU 1 is 64-127,192-255. (GPU affinity = 64-127,192-255).
thinkingmachines-inkling-small-h200-vllm-deploy-69577f8978z5sg5:917:917 [1] NCCL INFO NVLS multicast support is not available on dev 1
thinkingmachines-inkling-small-h200-vllm-deploy-69577f8978z5sg5:917:917 [1] NCCL INFO Rank 1: 1 Net devices
thinkingmachines-inkling-small-h200-vllm-deploy-69577f8978z5sg5:917:917 [1] NCCL INFO Rank 1: 0 CollNet devices
thinkingmachines-inkling-small-h200-vllm-deploy-69577f8978z5sg5:917:917 [1] NCCL INFO comm 0x1694f800 rank 1 nRanks 2 nNodes 1 localRanks 2 localRank 1 MNNVL 0
thinkingmachines-inkling-small-h200-vllm-deploy-69577f8978z5sg5:917:917 [1] NCCL INFO Trees [0] -1/-1/-1->1->0 [1] 0/-1/-1->1->-1 [2] -1/-1/-1->1->0 [3] 0/-1/-1->1->-1
thinkingmachines-inkling-small-h200-vllm-deploy-69577f8978z5sg5:917:917 [1] NCCL INFO P2P Chunksize set to 131072
thinkingmachines-inkling-small-h200-vllm-deploy-69577f8978z5sg5:917:917 [1] NCCL INFO Check P2P Type isAllDirectP2p 0 directMode 0 isAllCudaP2p 1
thinkingmachines-inkling-small-h200-vllm-deploy-69577f8978z5sg5:917:1182 [0] NCCL INFO [Proxy Service UDS] Device 1 CPU core 94
thinkingmachines-inkling-small-h200-vllm-deploy-69577f8978z5sg5:917:1181 [0] NCCL INFO [Proxy Service] Device 1 CPU core 230
thinkingmachines-inkling-small-h200-vllm-deploy-69577f8978z5sg5:917:1181 [1] NCCL INFO proxy listening socket at 10.130.1.190<46167>
thinkingmachines-inkling-small-h200-vllm-deploy-69577f8978z5sg5:917:917 [1] NCCL INFO threadThresholds 8/8/64 | 16/8/64 | 512 | 512
thinkingmachines-inkling-small-h200-vllm-deploy-69577f8978z5sg5:917:917 [1] NCCL INFO 4 coll channels, 4 collnet channels, 0 nvls channels, 4 p2p channels, 2 p2p channels per peer
thinkingmachines-inkling-small-h200-vllm-deploy-69577f8978z5sg5:917:917 [1] NCCL INFO ncclCommInitRank comm 0x1694f800 rank 1 nranks 2 cudaDev 1 nvmlDev 1 busId d1000 commId 0x5c3ff7f5395fd00c - Init COMPLETE
thinkingmachines-inkling-small-h200-vllm-deploy-69577f8978z5sg5:917:917 [1] NCCL INFO Init timings - ncclCommInitRank: rank 1 nranks 2 total 0.01 (kernels 0.00, alloc 0.00, bootstrap 0.00, allgathers 0.00, topo 0.00, graphs 0.00, connections 0.00, rest 0.00)
thinkingmachines-inkling-small-h200-vllm-deploy-69577f8978z5sg5:917:917 [1] NCCL INFO Symmetric VA size=140GB
thinkingmachines-inkling-small-h200-vllm-deploy-69577f8978z5sg5:917:917 [1] NCCL INFO Channel 00 : 1[1] -> 0[0] via SHM/direct
thinkingmachines-inkling-small-h200-vllm-deploy-69577f8978z5sg5:917:917 [1] NCCL INFO Channel 01 : 1[1] -> 0[0] via SHM/direct
thinkingmachines-inkling-small-h200-vllm-deploy-69577f8978z5sg5:917:917 [1] NCCL INFO Channel 02 : 1[1] -> 0[0] via SHM/direct
thinkingmachines-inkling-small-h200-vllm-deploy-69577f8978z5sg5:917:917 [1] NCCL INFO Channel 03 : 1[1] -> 0[0] via SHM/direct
thinkingmachines-inkling-small-h200-vllm-deploy-69577f8978z5sg5:917:917 [1] NCCL INFO Connected all rings, use ring PXN 0 GDR 1
[rank1]:[W805 15:27:50.850742647 CUDACachingAllocator.cpp:3933] memory allocation failed with OOM on device 1 while trying to allocate 20971520 bytes (free: 9502720, total: 150109880320).
[rank1]:[W805 15:27:50.851908024 CUDACachingAllocator.cpp:3933] memory allocation failed with OOM on device 1 while trying to allocate 20971520 bytes (free: 9502720, total: 150109880320).
[rank1]:[W805 15:27:50.852918241 CUDACachingAllocator.cpp:3933] memory allocation failed with OOM on device 1 while trying to allocate 20971520 bytes (free: 9502720, total: 150109880320).
(Worker_TP0 pid=916) DEBUG 08-05 15:27:50 [model_executor/models/utils.py:282] Loaded weight model.layers.11.attn_norm.weight with shape torch.Size([4096])
[rank1]:[W805 15:27:50.853893978 CUDACachingAllocator.cpp:3933] memory allocation failed with OOM on device 1 while trying to allocate 20971520 bytes (free: 9502720, total: 150109880320).
(Worker_TP0 pid=916) DEBUG 08-05 15:27:50 [model_executor/models/utils.py:282] Loaded weight model.layers.12.attn.q_norm.weight with shape torch.Size([128])
(Worker_TP0 pid=916) DEBUG 08-05 15:27:50 [model_executor/layers/linear.py:1012] Loaded shard 3 with shape torch.Size([512, 4096]) into model.layers.12.attn.qkvr.weight
[rank1]:[W805 15:27:50.854866665 CUDACachingAllocator.cpp:3933] memory allocation failed with OOM on device 1 while trying to allocate 20971520 bytes (free: 9502720, total: 150109880320).
[rank0]:[W805 15:27:50.855542863 CUDACachingAllocator.cpp:3933] memory allocation failed with OOM on device 0 while trying to allocate 20971520 bytes (free: 9502720, total: 150109880320).
[rank1]:[W805 15:27:50.855957562 CUDACachingAllocator.cpp:3933] memory allocation failed with OOM on device 1 while trying to allocate 20971520 bytes (free: 9502720, total: 150109880320).

kurt-tm

Thinking Machines Lab org about 7 hours ago

per guidance here, can you please open an issue with vLLM? https://huggingface.co/thinkingmachines/Inkling-Small/discussions/9#6a739eb7cd0fcbd96f5aa87b

SuperbEmphasis

about 5 hours ago

There is some overhead with these models. I believe the max gpu utilization is 0.9 (90%).

You can also use --enforce-eager to sacrifice some throughput for vram.

Make sure that you arent running anything else. With this not running make sure that nvidia-smi is showing the cards free.

Do can also set the kvcache dtype to fp8.

I also tried using all four of my H100 GPUs (320GB of vram) at 80% (0.8) and I was able to get 3 streams at 256k context.... which is pretty low. So you might need 4 to run it.

Upload images, audio, and videos by dragging in the text input, pasting, or clicking here.

Tap or paste here to upload images

· Sign up or log in to comment