2025-01-20T17:44:40,182 [DEBUG] main org.pytorch.serve.util.ConfigManager - xpu-smi not available or failed: Cannot run program "xpu-smi": error=2, No such file or directory 2025-01-20T17:44:40,182 [DEBUG] main org.pytorch.serve.util.ConfigManager - xpu-smi not available or failed: Cannot run program "xpu-smi": error=2, No such file or directory 2025-01-20T17:44:40,193 [WARN ] main org.pytorch.serve.util.ConfigManager - Your torchserve instance can access any URL to load models. When deploying to production, make sure to limit the set of allowed_urls in config.properties 2025-01-20T17:44:40,193 [WARN ] main org.pytorch.serve.util.ConfigManager - Your torchserve instance can access any URL to load models. When deploying to production, make sure to limit the set of allowed_urls in config.properties 2025-01-20T17:44:40,220 [INFO ] main org.pytorch.serve.servingsdk.impl.PluginsManager - Initializing plugins manager... 2025-01-20T17:44:40,220 [INFO ] main org.pytorch.serve.servingsdk.impl.PluginsManager - Initializing plugins manager... 2025-01-20T17:44:40,345 [INFO ] main org.pytorch.serve.metrics.configuration.MetricConfiguration - Successfully loaded metrics configuration from /usr/local/lib/python3.11/dist-packages/ts/configs/metrics.yaml 2025-01-20T17:44:40,345 [INFO ] main org.pytorch.serve.metrics.configuration.MetricConfiguration - Successfully loaded metrics configuration from /usr/local/lib/python3.11/dist-packages/ts/configs/metrics.yaml 2025-01-20T17:44:40,472 [INFO ] main org.pytorch.serve.ModelServer - Torchserve version: 0.12.0 TS Home: /usr/local/lib/python3.11/dist-packages Current directory: /content Temp directory: /tmp Metrics config path: /usr/local/lib/python3.11/dist-packages/ts/configs/metrics.yaml Number of GPUs: 1 Number of CPUs: 2 Max heap size: 3246 M Python executable: /usr/bin/python3 Config file: config.properties Inference address: http://127.0.0.1:8083 Management address: http://127.0.0.1:8084 Metrics address: http://127.0.0.1:8085 Model Store: /content/model_store Initial Models: titan=titan.mar Log dir: /content/logs Metrics dir: /content/logs Netty threads: 0 Netty client threads: 0 Default workers per model: 1 Blacklist Regex: N/A Maximum Response Size: 6553500 Maximum Request Size: 6553500 Limit Maximum Image Pixels: true Prefer direct buffer: false Allowed Urls: [file://.*|http(s)?://.*] Custom python dependency for model allowed: false Enable metrics API: true Metrics mode: LOG Disable system metrics: false Workflow Store: /content/model_store CPP log config: N/A Model config: N/A System metrics command: default Model API enabled: false 2025-01-20T17:44:40,472 [INFO ] main org.pytorch.serve.ModelServer - Torchserve version: 0.12.0 TS Home: /usr/local/lib/python3.11/dist-packages Current directory: /content Temp directory: /tmp Metrics config path: /usr/local/lib/python3.11/dist-packages/ts/configs/metrics.yaml Number of GPUs: 1 Number of CPUs: 2 Max heap size: 3246 M Python executable: /usr/bin/python3 Config file: config.properties Inference address: http://127.0.0.1:8083 Management address: http://127.0.0.1:8084 Metrics address: http://127.0.0.1:8085 Model Store: /content/model_store Initial Models: titan=titan.mar Log dir: /content/logs Metrics dir: /content/logs Netty threads: 0 Netty client threads: 0 Default workers per model: 1 Blacklist Regex: N/A Maximum Response Size: 6553500 Maximum Request Size: 6553500 Limit Maximum Image Pixels: true Prefer direct buffer: false Allowed Urls: [file://.*|http(s)?://.*] Custom python dependency for model allowed: false Enable metrics API: true Metrics mode: LOG Disable system metrics: false Workflow Store: /content/model_store CPP log config: N/A Model config: N/A System metrics command: default Model API enabled: false 2025-01-20T17:44:40,496 [INFO ] main org.pytorch.serve.servingsdk.impl.PluginsManager - Loading snapshot serializer plugin... 2025-01-20T17:44:40,496 [INFO ] main org.pytorch.serve.servingsdk.impl.PluginsManager - Loading snapshot serializer plugin... 2025-01-20T17:44:40,528 [INFO ] main org.pytorch.serve.ModelServer - Loading initial models: titan.mar 2025-01-20T17:44:40,528 [INFO ] main org.pytorch.serve.ModelServer - Loading initial models: titan.mar 2025-01-20T17:45:14,031 [DEBUG] main org.pytorch.serve.wlm.ModelVersionedRefs - Adding new version 1.0 for model titan 2025-01-20T17:45:14,031 [DEBUG] main org.pytorch.serve.wlm.ModelVersionedRefs - Adding new version 1.0 for model titan 2025-01-20T17:45:14,037 [DEBUG] main org.pytorch.serve.wlm.ModelVersionedRefs - Setting default version to 1.0 for model titan 2025-01-20T17:45:14,037 [DEBUG] main org.pytorch.serve.wlm.ModelVersionedRefs - Setting default version to 1.0 for model titan 2025-01-20T17:45:14,037 [INFO ] main org.pytorch.serve.wlm.ModelManager - Model titan loaded. 2025-01-20T17:45:14,037 [INFO ] main org.pytorch.serve.wlm.ModelManager - Model titan loaded. 2025-01-20T17:45:14,037 [DEBUG] main org.pytorch.serve.wlm.ModelManager - updateModel: titan, count: 1 2025-01-20T17:45:14,037 [DEBUG] main org.pytorch.serve.wlm.ModelManager - updateModel: titan, count: 1 2025-01-20T17:45:14,056 [INFO ] main org.pytorch.serve.ModelServer - Initialize Inference server with: EpollServerSocketChannel. 2025-01-20T17:45:14,056 [INFO ] main org.pytorch.serve.ModelServer - Initialize Inference server with: EpollServerSocketChannel. 2025-01-20T17:45:14,071 [DEBUG] W-9000-titan_1.0 org.pytorch.serve.wlm.WorkerLifeCycle - Worker cmdline: [/usr/bin/python3, /usr/local/lib/python3.11/dist-packages/ts/model_service_worker.py, --sock-type, unix, --sock-name, /tmp/.ts.sock.9000, --metrics-config, /usr/local/lib/python3.11/dist-packages/ts/configs/metrics.yaml] 2025-01-20T17:45:14,071 [DEBUG] W-9000-titan_1.0 org.pytorch.serve.wlm.WorkerLifeCycle - Worker cmdline: [/usr/bin/python3, /usr/local/lib/python3.11/dist-packages/ts/model_service_worker.py, --sock-type, unix, --sock-name, /tmp/.ts.sock.9000, --metrics-config, /usr/local/lib/python3.11/dist-packages/ts/configs/metrics.yaml] 2025-01-20T17:45:14,192 [INFO ] main org.pytorch.serve.ModelServer - Inference API bind to: http://127.0.0.1:8083 2025-01-20T17:45:14,192 [INFO ] main org.pytorch.serve.ModelServer - Inference API bind to: http://127.0.0.1:8083 2025-01-20T17:45:14,194 [INFO ] main org.pytorch.serve.ModelServer - Initialize Management server with: EpollServerSocketChannel. 2025-01-20T17:45:14,194 [INFO ] main org.pytorch.serve.ModelServer - Initialize Management server with: EpollServerSocketChannel. 2025-01-20T17:45:14,198 [INFO ] main org.pytorch.serve.ModelServer - Management API bind to: http://127.0.0.1:8084 2025-01-20T17:45:14,198 [INFO ] main org.pytorch.serve.ModelServer - Management API bind to: http://127.0.0.1:8084 2025-01-20T17:45:14,199 [INFO ] main org.pytorch.serve.ModelServer - Initialize Metrics server with: EpollServerSocketChannel. 2025-01-20T17:45:14,199 [INFO ] main org.pytorch.serve.ModelServer - Initialize Metrics server with: EpollServerSocketChannel. 2025-01-20T17:45:14,200 [INFO ] main org.pytorch.serve.ModelServer - Metrics API bind to: http://127.0.0.1:8085 2025-01-20T17:45:14,200 [INFO ] main org.pytorch.serve.ModelServer - Metrics API bind to: http://127.0.0.1:8085 2025-01-20T17:45:14,940 [WARN ] pool-3-thread-1 org.pytorch.serve.metrics.MetricCollector - worker pid is not available yet. 2025-01-20T17:45:14,940 [WARN ] pool-3-thread-1 org.pytorch.serve.metrics.MetricCollector - worker pid is not available yet. 2025-01-20T17:45:15,116 [ERROR] Thread-1 org.pytorch.serve.metrics.MetricCollector - Traceback (most recent call last): File "/usr/local/lib/python3.11/dist-packages/ts/metrics/metric_collector.py", line 11, in from ts.metrics import system_metrics File "/usr/local/lib/python3.11/dist-packages/ts/__init__.py", line 10, in from . import version File "/usr/local/lib/python3.11/dist-packages/ts/version.py", line 5, in from pathlib import Path File "/usr/local/lib/python3.11/dist-packages/pathlib.py", line 10, in from collections import Sequence ImportError: cannot import name 'Sequence' from 'collections' (/usr/lib/python3.11/collections/__init__.py) 2025-01-20T17:45:15,116 [ERROR] Thread-1 org.pytorch.serve.metrics.MetricCollector - Traceback (most recent call last): File "/usr/local/lib/python3.11/dist-packages/ts/metrics/metric_collector.py", line 11, in from ts.metrics import system_metrics File "/usr/local/lib/python3.11/dist-packages/ts/__init__.py", line 10, in from . import version File "/usr/local/lib/python3.11/dist-packages/ts/version.py", line 5, in from pathlib import Path File "/usr/local/lib/python3.11/dist-packages/pathlib.py", line 10, in from collections import Sequence ImportError: cannot import name 'Sequence' from 'collections' (/usr/lib/python3.11/collections/__init__.py) 2025-01-20T17:45:16,984 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - s_name_part0=/tmp/.ts.sock, s_name_part1=9000, pid=5985 2025-01-20T17:45:16,987 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - Listening on port: /tmp/.ts.sock.9000 2025-01-20T17:45:17,000 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - Successfully loaded /usr/local/lib/python3.11/dist-packages/ts/configs/metrics.yaml. 2025-01-20T17:45:17,002 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - [PID]5985 2025-01-20T17:45:17,006 [DEBUG] W-9000-titan_1.0 org.pytorch.serve.wlm.WorkerThread - W-9000-titan_1.0 State change null -> WORKER_STARTED 2025-01-20T17:45:17,005 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - Torch worker started. 2025-01-20T17:45:17,006 [DEBUG] W-9000-titan_1.0 org.pytorch.serve.wlm.WorkerThread - W-9000-titan_1.0 State change null -> WORKER_STARTED 2025-01-20T17:45:17,010 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - Python runtime: 3.11.11 2025-01-20T17:45:17,016 [INFO ] W-9000-titan_1.0 org.pytorch.serve.wlm.WorkerThread - Connecting to: /tmp/.ts.sock.9000 2025-01-20T17:45:17,016 [INFO ] W-9000-titan_1.0 org.pytorch.serve.wlm.WorkerThread - Connecting to: /tmp/.ts.sock.9000 2025-01-20T17:45:17,032 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - Connection accepted: /tmp/.ts.sock.9000. 2025-01-20T17:45:17,046 [DEBUG] W-9000-titan_1.0 org.pytorch.serve.wlm.WorkerThread - Flushing req.cmd LOAD repeats 1 to backend at: 1737395117046 2025-01-20T17:45:17,046 [DEBUG] W-9000-titan_1.0 org.pytorch.serve.wlm.WorkerThread - Flushing req.cmd LOAD repeats 1 to backend at: 1737395117046 2025-01-20T17:45:17,049 [INFO ] W-9000-titan_1.0 org.pytorch.serve.wlm.WorkerThread - Looping backend response at: 1737395117049 2025-01-20T17:45:17,049 [INFO ] W-9000-titan_1.0 org.pytorch.serve.wlm.WorkerThread - Looping backend response at: 1737395117049 2025-01-20T17:45:17,101 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - model_name: titan, batchSize: 1 2025-01-20T17:45:17,106 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - Backend worker process died. 2025-01-20T17:45:17,106 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - Traceback (most recent call last): 2025-01-20T17:45:17,107 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - File "/usr/local/lib/python3.11/dist-packages/ts/model_service_worker.py", line 301, in 2025-01-20T17:45:17,107 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - worker.run_server() 2025-01-20T17:45:17,107 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - File "/usr/local/lib/python3.11/dist-packages/ts/model_service_worker.py", line 268, in run_server 2025-01-20T17:45:17,107 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - self.handle_connection(cl_socket) 2025-01-20T17:45:17,108 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - File "/usr/local/lib/python3.11/dist-packages/ts/model_service_worker.py", line 196, in handle_connection 2025-01-20T17:45:17,108 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - service, result, code = self.load_model(msg) 2025-01-20T17:45:17,108 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - ^^^^^^^^^^^^^^^^^^^^ 2025-01-20T17:45:17,108 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - File "/usr/local/lib/python3.11/dist-packages/ts/model_service_worker.py", line 133, in load_model 2025-01-20T17:45:17,108 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - service = model_loader.load( 2025-01-20T17:45:17,109 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - ^^^^^^^^^^^^^^^^^^ 2025-01-20T17:45:17,109 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - File "/usr/local/lib/python3.11/dist-packages/ts/model_loader.py", line 108, in load 2025-01-20T17:45:17,109 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - module, function_name = self._load_handler_file(handler) 2025-01-20T17:45:17,109 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-01-20T17:45:17,109 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - File "/usr/local/lib/python3.11/dist-packages/ts/model_loader.py", line 153, in _load_handler_file 2025-01-20T17:45:17,110 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - module = importlib.import_module(module_name) 2025-01-20T17:45:17,110 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-01-20T17:45:17,110 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - File "/usr/lib/python3.11/importlib/__init__.py", line 126, in import_module 2025-01-20T17:45:17,110 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - return _bootstrap._gcd_import(name[level:], package, level) 2025-01-20T17:45:17,110 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-01-20T17:45:17,110 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - File "", line 1204, in _gcd_import 2025-01-20T17:45:17,111 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - File "", line 1176, in _find_and_load 2025-01-20T17:45:17,116 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - File "", line 1147, in _find_and_load_unlocked 2025-01-20T17:45:17,119 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - File "", line 690, in _load_unlocked 2025-01-20T17:45:17,120 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - File "", line 936, in exec_module 2025-01-20T17:45:17,122 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - File "", line 1074, in get_code 2025-01-20T17:45:17,127 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - File "", line 1004, in source_to_code 2025-01-20T17:45:17,127 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - File "", line 241, in _call_with_frames_removed 2025-01-20T17:45:17,128 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - File "/tmp/models/a15c525d26d44618b25a3dc1a7bfcbb5/titan_handler.py", line 86 2025-01-20T17:45:17,130 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - print(" 2025-01-20T17:45:17,132 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - ^ 2025-01-20T17:45:17,133 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - SyntaxError: unterminated string literal (detected at line 86) 2025-01-20T17:45:17,135 [INFO ] epollEventLoopGroup-5-1 org.pytorch.serve.wlm.WorkerThread - 9000 Worker disconnected. WORKER_STARTED 2025-01-20T17:45:17,135 [INFO ] epollEventLoopGroup-5-1 org.pytorch.serve.wlm.WorkerThread - 9000 Worker disconnected. WORKER_STARTED 2025-01-20T17:45:17,149 [DEBUG] W-9000-titan_1.0 org.pytorch.serve.wlm.WorkerThread - System state is : WORKER_STARTED 2025-01-20T17:45:17,149 [DEBUG] W-9000-titan_1.0 org.pytorch.serve.wlm.WorkerThread - System state is : WORKER_STARTED 2025-01-20T17:45:17,149 [DEBUG] W-9000-titan_1.0 org.pytorch.serve.wlm.WorkerThread - Backend worker monitoring thread interrupted or backend worker process died., startupTimeout:120sec java.lang.InterruptedException: null at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2056) ~[?:?] at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2133) ~[?:?] at java.util.concurrent.ArrayBlockingQueue.poll(ArrayBlockingQueue.java:432) ~[?:?] at org.pytorch.serve.wlm.WorkerThread.run(WorkerThread.java:234) [model-server.jar:?] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) [?:?] at java.util.concurrent.FutureTask.run(FutureTask.java:264) [?:?] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?] at java.lang.Thread.run(Thread.java:829) [?:?] 2025-01-20T17:45:17,149 [DEBUG] W-9000-titan_1.0 org.pytorch.serve.wlm.WorkerThread - Backend worker monitoring thread interrupted or backend worker process died., startupTimeout:120sec java.lang.InterruptedException: null at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2056) ~[?:?] at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2133) ~[?:?] at java.util.concurrent.ArrayBlockingQueue.poll(ArrayBlockingQueue.java:432) ~[?:?] at org.pytorch.serve.wlm.WorkerThread.run(WorkerThread.java:234) [model-server.jar:?] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) [?:?] at java.util.concurrent.FutureTask.run(FutureTask.java:264) [?:?] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?] at java.lang.Thread.run(Thread.java:829) [?:?] 2025-01-20T17:45:17,214 [WARN ] W-9000-titan_1.0 org.pytorch.serve.wlm.BatchAggregator - Load model failed: titan, error: Worker died. 2025-01-20T17:45:17,214 [WARN ] W-9000-titan_1.0 org.pytorch.serve.wlm.BatchAggregator - Load model failed: titan, error: Worker died. 2025-01-20T17:45:17,217 [DEBUG] W-9000-titan_1.0 org.pytorch.serve.wlm.WorkerThread - W-9000-titan_1.0 State change WORKER_STARTED -> WORKER_STOPPED 2025-01-20T17:45:17,217 [DEBUG] W-9000-titan_1.0 org.pytorch.serve.wlm.WorkerThread - W-9000-titan_1.0 State change WORKER_STARTED -> WORKER_STOPPED 2025-01-20T17:45:17,217 [INFO ] W-9000-titan_1.0 org.pytorch.serve.wlm.WorkerThread - Auto recovery start timestamp: 1737395117217 2025-01-20T17:45:17,217 [INFO ] W-9000-titan_1.0 org.pytorch.serve.wlm.WorkerThread - Auto recovery start timestamp: 1737395117217 2025-01-20T17:45:17,219 [INFO ] W-9000-titan_1.0 org.pytorch.serve.wlm.WorkerThread - Retry worker: 9000 in 1 seconds. 2025-01-20T17:45:17,219 [INFO ] W-9000-titan_1.0 org.pytorch.serve.wlm.WorkerThread - Retry worker: 9000 in 1 seconds. 2025-01-20T17:45:17,261 [INFO ] W-9000-titan_1.0-stdout org.pytorch.serve.wlm.WorkerLifeCycle - Stopped Scanner - W-9000-titan_1.0-stdout 2025-01-20T17:45:17,261 [INFO ] W-9000-titan_1.0-stderr org.pytorch.serve.wlm.WorkerLifeCycle - Stopped Scanner - W-9000-titan_1.0-stderr 2025-01-20T17:45:17,261 [INFO ] W-9000-titan_1.0-stdout org.pytorch.serve.wlm.WorkerLifeCycle - Stopped Scanner - W-9000-titan_1.0-stdout 2025-01-20T17:45:17,261 [INFO ] W-9000-titan_1.0-stderr org.pytorch.serve.wlm.WorkerLifeCycle - Stopped Scanner - W-9000-titan_1.0-stderr 2025-01-20T17:45:18,220 [DEBUG] W-9000-titan_1.0 org.pytorch.serve.wlm.WorkerLifeCycle - Worker cmdline: [/usr/bin/python3, /usr/local/lib/python3.11/dist-packages/ts/model_service_worker.py, --sock-type, unix, --sock-name, /tmp/.ts.sock.9000, --metrics-config, /usr/local/lib/python3.11/dist-packages/ts/configs/metrics.yaml] 2025-01-20T17:45:18,220 [DEBUG] W-9000-titan_1.0 org.pytorch.serve.wlm.WorkerLifeCycle - Worker cmdline: [/usr/bin/python3, /usr/local/lib/python3.11/dist-packages/ts/model_service_worker.py, --sock-type, unix, --sock-name, /tmp/.ts.sock.9000, --metrics-config, /usr/local/lib/python3.11/dist-packages/ts/configs/metrics.yaml] 2025-01-20T17:45:19,799 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - s_name_part0=/tmp/.ts.sock, s_name_part1=9000, pid=6015 2025-01-20T17:45:19,801 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - Listening on port: /tmp/.ts.sock.9000 2025-01-20T17:45:19,807 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - Successfully loaded /usr/local/lib/python3.11/dist-packages/ts/configs/metrics.yaml. 2025-01-20T17:45:19,808 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - [PID]6015 2025-01-20T17:45:19,808 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - Torch worker started. 2025-01-20T17:45:19,809 [DEBUG] W-9000-titan_1.0 org.pytorch.serve.wlm.WorkerThread - W-9000-titan_1.0 State change WORKER_STOPPED -> WORKER_STARTED 2025-01-20T17:45:19,809 [DEBUG] W-9000-titan_1.0 org.pytorch.serve.wlm.WorkerThread - W-9000-titan_1.0 State change WORKER_STOPPED -> WORKER_STARTED 2025-01-20T17:45:19,809 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - Python runtime: 3.11.11 2025-01-20T17:45:19,809 [INFO ] W-9000-titan_1.0 org.pytorch.serve.wlm.WorkerThread - Connecting to: /tmp/.ts.sock.9000 2025-01-20T17:45:19,809 [INFO ] W-9000-titan_1.0 org.pytorch.serve.wlm.WorkerThread - Connecting to: /tmp/.ts.sock.9000 2025-01-20T17:45:19,811 [DEBUG] W-9000-titan_1.0 org.pytorch.serve.wlm.WorkerThread - Flushing req.cmd LOAD repeats 1 to backend at: 1737395119811 2025-01-20T17:45:19,811 [DEBUG] W-9000-titan_1.0 org.pytorch.serve.wlm.WorkerThread - Flushing req.cmd LOAD repeats 1 to backend at: 1737395119811 2025-01-20T17:45:19,812 [INFO ] W-9000-titan_1.0 org.pytorch.serve.wlm.WorkerThread - Looping backend response at: 1737395119812 2025-01-20T17:45:19,812 [INFO ] W-9000-titan_1.0 org.pytorch.serve.wlm.WorkerThread - Looping backend response at: 1737395119812 2025-01-20T17:45:19,816 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - Connection accepted: /tmp/.ts.sock.9000. 2025-01-20T17:45:19,837 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - model_name: titan, batchSize: 1 2025-01-20T17:45:19,843 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - Backend worker process died. 2025-01-20T17:45:19,846 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - Traceback (most recent call last): 2025-01-20T17:45:19,846 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - File "/usr/local/lib/python3.11/dist-packages/ts/model_service_worker.py", line 301, in 2025-01-20T17:45:19,846 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - worker.run_server() 2025-01-20T17:45:19,846 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - File "/usr/local/lib/python3.11/dist-packages/ts/model_service_worker.py", line 268, in run_server 2025-01-20T17:45:19,847 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - self.handle_connection(cl_socket) 2025-01-20T17:45:19,847 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - File "/usr/local/lib/python3.11/dist-packages/ts/model_service_worker.py", line 196, in handle_connection 2025-01-20T17:45:19,847 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - service, result, code = self.load_model(msg) 2025-01-20T17:45:19,847 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - ^^^^^^^^^^^^^^^^^^^^ 2025-01-20T17:45:19,847 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - File "/usr/local/lib/python3.11/dist-packages/ts/model_service_worker.py", line 133, in load_model 2025-01-20T17:45:19,848 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - service = model_loader.load( 2025-01-20T17:45:19,852 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - ^^^^^^^^^^^^^^^^^^ 2025-01-20T17:45:19,852 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - File "/usr/local/lib/python3.11/dist-packages/ts/model_loader.py", line 108, in load 2025-01-20T17:45:19,852 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - module, function_name = self._load_handler_file(handler) 2025-01-20T17:45:19,852 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-01-20T17:45:19,852 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - File "/usr/local/lib/python3.11/dist-packages/ts/model_loader.py", line 153, in _load_handler_file 2025-01-20T17:45:19,856 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - module = importlib.import_module(module_name) 2025-01-20T17:45:19,856 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-01-20T17:45:19,856 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - File "/usr/lib/python3.11/importlib/__init__.py", line 126, in import_module 2025-01-20T17:45:19,856 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - return _bootstrap._gcd_import(name[level:], package, level) 2025-01-20T17:45:19,856 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-01-20T17:45:19,857 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - File "", line 1204, in _gcd_import 2025-01-20T17:45:19,857 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - File "", line 1176, in _find_and_load 2025-01-20T17:45:19,857 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - File "", line 1147, in _find_and_load_unlocked 2025-01-20T17:45:19,857 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - File "", line 690, in _load_unlocked 2025-01-20T17:45:19,857 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - File "", line 936, in exec_module 2025-01-20T17:45:19,858 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - File "", line 1074, in get_code 2025-01-20T17:45:19,858 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - File "", line 1004, in source_to_code 2025-01-20T17:45:19,859 [INFO ] epollEventLoopGroup-5-2 org.pytorch.serve.wlm.WorkerThread - 9000 Worker disconnected. WORKER_STARTED 2025-01-20T17:45:19,858 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - File "", line 241, in _call_with_frames_removed 2025-01-20T17:45:19,860 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - File "/tmp/models/a15c525d26d44618b25a3dc1a7bfcbb5/titan_handler.py", line 86 2025-01-20T17:45:19,865 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - print(" 2025-01-20T17:45:19,865 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - ^ 2025-01-20T17:45:19,865 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - SyntaxError: unterminated string literal (detected at line 86) 2025-01-20T17:45:19,859 [INFO ] epollEventLoopGroup-5-2 org.pytorch.serve.wlm.WorkerThread - 9000 Worker disconnected. WORKER_STARTED 2025-01-20T17:45:19,866 [DEBUG] W-9000-titan_1.0 org.pytorch.serve.wlm.WorkerThread - System state is : WORKER_STARTED 2025-01-20T17:45:19,866 [DEBUG] W-9000-titan_1.0 org.pytorch.serve.wlm.WorkerThread - System state is : WORKER_STARTED 2025-01-20T17:45:19,866 [DEBUG] W-9000-titan_1.0 org.pytorch.serve.wlm.WorkerThread - Backend worker monitoring thread interrupted or backend worker process died., startupTimeout:120sec java.lang.InterruptedException: null at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2056) ~[?:?] at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2133) ~[?:?] at java.util.concurrent.ArrayBlockingQueue.poll(ArrayBlockingQueue.java:432) ~[?:?] at org.pytorch.serve.wlm.WorkerThread.run(WorkerThread.java:234) [model-server.jar:?] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?] at java.lang.Thread.run(Thread.java:829) [?:?] 2025-01-20T17:45:19,866 [DEBUG] W-9000-titan_1.0 org.pytorch.serve.wlm.WorkerThread - Backend worker monitoring thread interrupted or backend worker process died., startupTimeout:120sec java.lang.InterruptedException: null at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2056) ~[?:?] at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2133) ~[?:?] at java.util.concurrent.ArrayBlockingQueue.poll(ArrayBlockingQueue.java:432) ~[?:?] at org.pytorch.serve.wlm.WorkerThread.run(WorkerThread.java:234) [model-server.jar:?] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?] at java.lang.Thread.run(Thread.java:829) [?:?] 2025-01-20T17:45:19,866 [WARN ] W-9000-titan_1.0 org.pytorch.serve.wlm.BatchAggregator - Load model failed: titan, error: Worker died. 2025-01-20T17:45:19,866 [WARN ] W-9000-titan_1.0 org.pytorch.serve.wlm.BatchAggregator - Load model failed: titan, error: Worker died. 2025-01-20T17:45:19,866 [DEBUG] W-9000-titan_1.0 org.pytorch.serve.wlm.WorkerThread - W-9000-titan_1.0 State change WORKER_STARTED -> WORKER_STOPPED 2025-01-20T17:45:19,866 [DEBUG] W-9000-titan_1.0 org.pytorch.serve.wlm.WorkerThread - W-9000-titan_1.0 State change WORKER_STARTED -> WORKER_STOPPED 2025-01-20T17:45:19,867 [WARN ] W-9000-titan_1.0 org.pytorch.serve.wlm.WorkerThread - Auto recovery failed again 2025-01-20T17:45:19,867 [WARN ] W-9000-titan_1.0 org.pytorch.serve.wlm.WorkerThread - Auto recovery failed again 2025-01-20T17:45:19,867 [INFO ] W-9000-titan_1.0 org.pytorch.serve.wlm.WorkerThread - Retry worker: 9000 in 1 seconds. 2025-01-20T17:45:19,867 [INFO ] W-9000-titan_1.0 org.pytorch.serve.wlm.WorkerThread - Retry worker: 9000 in 1 seconds. 2025-01-20T17:45:19,895 [INFO ] W-9000-titan_1.0-stdout org.pytorch.serve.wlm.WorkerLifeCycle - Stopped Scanner - W-9000-titan_1.0-stdout 2025-01-20T17:45:19,895 [INFO ] W-9000-titan_1.0-stdout org.pytorch.serve.wlm.WorkerLifeCycle - Stopped Scanner - W-9000-titan_1.0-stdout 2025-01-20T17:45:19,895 [INFO ] W-9000-titan_1.0-stderr org.pytorch.serve.wlm.WorkerLifeCycle - Stopped Scanner - W-9000-titan_1.0-stderr 2025-01-20T17:45:19,895 [INFO ] W-9000-titan_1.0-stderr org.pytorch.serve.wlm.WorkerLifeCycle - Stopped Scanner - W-9000-titan_1.0-stderr 2025-01-20T17:45:20,868 [DEBUG] W-9000-titan_1.0 org.pytorch.serve.wlm.WorkerLifeCycle - Worker cmdline: [/usr/bin/python3, /usr/local/lib/python3.11/dist-packages/ts/model_service_worker.py, --sock-type, unix, --sock-name, /tmp/.ts.sock.9000, --metrics-config, /usr/local/lib/python3.11/dist-packages/ts/configs/metrics.yaml] 2025-01-20T17:45:20,868 [DEBUG] W-9000-titan_1.0 org.pytorch.serve.wlm.WorkerLifeCycle - Worker cmdline: [/usr/bin/python3, /usr/local/lib/python3.11/dist-packages/ts/model_service_worker.py, --sock-type, unix, --sock-name, /tmp/.ts.sock.9000, --metrics-config, /usr/local/lib/python3.11/dist-packages/ts/configs/metrics.yaml] 2025-01-20T17:45:22,472 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - s_name_part0=/tmp/.ts.sock, s_name_part1=9000, pid=6033 2025-01-20T17:45:22,475 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - Listening on port: /tmp/.ts.sock.9000 2025-01-20T17:45:22,482 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - Successfully loaded /usr/local/lib/python3.11/dist-packages/ts/configs/metrics.yaml. 2025-01-20T17:45:22,483 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - [PID]6033 2025-01-20T17:45:22,483 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - Torch worker started. 2025-01-20T17:45:22,483 [DEBUG] W-9000-titan_1.0 org.pytorch.serve.wlm.WorkerThread - W-9000-titan_1.0 State change WORKER_STOPPED -> WORKER_STARTED 2025-01-20T17:45:22,483 [DEBUG] W-9000-titan_1.0 org.pytorch.serve.wlm.WorkerThread - W-9000-titan_1.0 State change WORKER_STOPPED -> WORKER_STARTED 2025-01-20T17:45:22,484 [INFO ] W-9000-titan_1.0 org.pytorch.serve.wlm.WorkerThread - Connecting to: /tmp/.ts.sock.9000 2025-01-20T17:45:22,484 [INFO ] W-9000-titan_1.0 org.pytorch.serve.wlm.WorkerThread - Connecting to: /tmp/.ts.sock.9000 2025-01-20T17:45:22,484 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - Python runtime: 3.11.11 2025-01-20T17:45:22,487 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - Connection accepted: /tmp/.ts.sock.9000. 2025-01-20T17:45:22,488 [DEBUG] W-9000-titan_1.0 org.pytorch.serve.wlm.WorkerThread - Flushing req.cmd LOAD repeats 1 to backend at: 1737395122488 2025-01-20T17:45:22,488 [DEBUG] W-9000-titan_1.0 org.pytorch.serve.wlm.WorkerThread - Flushing req.cmd LOAD repeats 1 to backend at: 1737395122488 2025-01-20T17:45:22,489 [INFO ] W-9000-titan_1.0 org.pytorch.serve.wlm.WorkerThread - Looping backend response at: 1737395122489 2025-01-20T17:45:22,489 [INFO ] W-9000-titan_1.0 org.pytorch.serve.wlm.WorkerThread - Looping backend response at: 1737395122489 2025-01-20T17:45:22,517 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - model_name: titan, batchSize: 1 2025-01-20T17:45:22,520 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - Backend worker process died. 2025-01-20T17:45:22,520 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - Traceback (most recent call last): 2025-01-20T17:45:22,520 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - File "/usr/local/lib/python3.11/dist-packages/ts/model_service_worker.py", line 301, in 2025-01-20T17:45:22,520 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - worker.run_server() 2025-01-20T17:45:22,521 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - File "/usr/local/lib/python3.11/dist-packages/ts/model_service_worker.py", line 268, in run_server 2025-01-20T17:45:22,521 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - self.handle_connection(cl_socket) 2025-01-20T17:45:22,521 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - File "/usr/local/lib/python3.11/dist-packages/ts/model_service_worker.py", line 196, in handle_connection 2025-01-20T17:45:22,521 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - service, result, code = self.load_model(msg) 2025-01-20T17:45:22,521 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - ^^^^^^^^^^^^^^^^^^^^ 2025-01-20T17:45:22,523 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - File "/usr/local/lib/python3.11/dist-packages/ts/model_service_worker.py", line 133, in load_model 2025-01-20T17:45:22,523 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - service = model_loader.load( 2025-01-20T17:45:22,523 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - ^^^^^^^^^^^^^^^^^^ 2025-01-20T17:45:22,523 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - File "/usr/local/lib/python3.11/dist-packages/ts/model_loader.py", line 108, in load 2025-01-20T17:45:22,523 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - module, function_name = self._load_handler_file(handler) 2025-01-20T17:45:22,523 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-01-20T17:45:22,523 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - File "/usr/local/lib/python3.11/dist-packages/ts/model_loader.py", line 153, in _load_handler_file 2025-01-20T17:45:22,527 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - module = importlib.import_module(module_name) 2025-01-20T17:45:22,528 [INFO ] epollEventLoopGroup-5-3 org.pytorch.serve.wlm.WorkerThread - 9000 Worker disconnected. WORKER_STARTED 2025-01-20T17:45:22,528 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-01-20T17:45:22,528 [INFO ] epollEventLoopGroup-5-3 org.pytorch.serve.wlm.WorkerThread - 9000 Worker disconnected. WORKER_STARTED 2025-01-20T17:45:22,529 [DEBUG] W-9000-titan_1.0 org.pytorch.serve.wlm.WorkerThread - System state is : WORKER_STARTED 2025-01-20T17:45:22,529 [DEBUG] W-9000-titan_1.0 org.pytorch.serve.wlm.WorkerThread - System state is : WORKER_STARTED 2025-01-20T17:45:22,529 [DEBUG] W-9000-titan_1.0 org.pytorch.serve.wlm.WorkerThread - Backend worker monitoring thread interrupted or backend worker process died., startupTimeout:120sec java.lang.InterruptedException: null at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2056) ~[?:?] at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2133) ~[?:?] at java.util.concurrent.ArrayBlockingQueue.poll(ArrayBlockingQueue.java:432) ~[?:?] at org.pytorch.serve.wlm.WorkerThread.run(WorkerThread.java:234) [model-server.jar:?] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?] at java.lang.Thread.run(Thread.java:829) [?:?] 2025-01-20T17:45:22,529 [DEBUG] W-9000-titan_1.0 org.pytorch.serve.wlm.WorkerThread - Backend worker monitoring thread interrupted or backend worker process died., startupTimeout:120sec java.lang.InterruptedException: null at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2056) ~[?:?] at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2133) ~[?:?] at java.util.concurrent.ArrayBlockingQueue.poll(ArrayBlockingQueue.java:432) ~[?:?] at org.pytorch.serve.wlm.WorkerThread.run(WorkerThread.java:234) [model-server.jar:?] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?] at java.lang.Thread.run(Thread.java:829) [?:?] 2025-01-20T17:45:22,530 [WARN ] W-9000-titan_1.0 org.pytorch.serve.wlm.BatchAggregator - Load model failed: titan, error: Worker died. 2025-01-20T17:45:22,530 [WARN ] W-9000-titan_1.0 org.pytorch.serve.wlm.BatchAggregator - Load model failed: titan, error: Worker died. 2025-01-20T17:45:22,529 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - File "/usr/lib/python3.11/importlib/__init__.py", line 126, in import_module 2025-01-20T17:45:22,532 [DEBUG] W-9000-titan_1.0 org.pytorch.serve.wlm.WorkerThread - W-9000-titan_1.0 State change WORKER_STARTED -> WORKER_STOPPED 2025-01-20T17:45:22,532 [DEBUG] W-9000-titan_1.0 org.pytorch.serve.wlm.WorkerThread - W-9000-titan_1.0 State change WORKER_STARTED -> WORKER_STOPPED 2025-01-20T17:45:22,532 [WARN ] W-9000-titan_1.0 org.pytorch.serve.wlm.WorkerThread - Auto recovery failed again 2025-01-20T17:45:22,532 [WARN ] W-9000-titan_1.0 org.pytorch.serve.wlm.WorkerThread - Auto recovery failed again 2025-01-20T17:45:22,532 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - return _bootstrap._gcd_import(name[level:], package, level) 2025-01-20T17:45:22,533 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-01-20T17:45:22,533 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - File "", line 1204, in _gcd_import 2025-01-20T17:45:22,533 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - File "", line 1176, in _find_and_load 2025-01-20T17:45:22,534 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - File "", line 1147, in _find_and_load_unlocked 2025-01-20T17:45:22,534 [INFO ] W-9000-titan_1.0 org.pytorch.serve.wlm.WorkerThread - Retry worker: 9000 in 2 seconds. 2025-01-20T17:45:22,534 [INFO ] W-9000-titan_1.0 org.pytorch.serve.wlm.WorkerThread - Retry worker: 9000 in 2 seconds. 2025-01-20T17:45:22,534 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - File "", line 690, in _load_unlocked 2025-01-20T17:45:22,534 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - File "", line 936, in exec_module 2025-01-20T17:45:22,534 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - File "", line 1074, in get_code 2025-01-20T17:45:22,534 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - File "", line 1004, in source_to_code 2025-01-20T17:45:22,534 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - File "", line 241, in _call_with_frames_removed 2025-01-20T17:45:22,538 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - File "/tmp/models/a15c525d26d44618b25a3dc1a7bfcbb5/titan_handler.py", line 86 2025-01-20T17:45:22,539 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - print(" 2025-01-20T17:45:22,540 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - ^ 2025-01-20T17:45:22,540 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - SyntaxError: unterminated string literal (detected at line 86) 2025-01-20T17:45:22,542 [INFO ] W-9000-titan_1.0-stdout org.pytorch.serve.wlm.WorkerLifeCycle - Stopped Scanner - W-9000-titan_1.0-stdout 2025-01-20T17:45:22,542 [INFO ] W-9000-titan_1.0-stdout org.pytorch.serve.wlm.WorkerLifeCycle - Stopped Scanner - W-9000-titan_1.0-stdout 2025-01-20T17:45:22,562 [INFO ] W-9000-titan_1.0-stderr org.pytorch.serve.wlm.WorkerLifeCycle - Stopped Scanner - W-9000-titan_1.0-stderr 2025-01-20T17:45:22,562 [INFO ] W-9000-titan_1.0-stderr org.pytorch.serve.wlm.WorkerLifeCycle - Stopped Scanner - W-9000-titan_1.0-stderr 2025-01-20T17:45:24,533 [DEBUG] W-9000-titan_1.0 org.pytorch.serve.wlm.WorkerLifeCycle - Worker cmdline: [/usr/bin/python3, /usr/local/lib/python3.11/dist-packages/ts/model_service_worker.py, --sock-type, unix, --sock-name, /tmp/.ts.sock.9000, --metrics-config, /usr/local/lib/python3.11/dist-packages/ts/configs/metrics.yaml] 2025-01-20T17:45:24,533 [DEBUG] W-9000-titan_1.0 org.pytorch.serve.wlm.WorkerLifeCycle - Worker cmdline: [/usr/bin/python3, /usr/local/lib/python3.11/dist-packages/ts/model_service_worker.py, --sock-type, unix, --sock-name, /tmp/.ts.sock.9000, --metrics-config, /usr/local/lib/python3.11/dist-packages/ts/configs/metrics.yaml] 2025-01-20T17:45:26,121 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - s_name_part0=/tmp/.ts.sock, s_name_part1=9000, pid=6053 2025-01-20T17:45:26,125 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - Listening on port: /tmp/.ts.sock.9000 2025-01-20T17:45:26,130 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - Successfully loaded /usr/local/lib/python3.11/dist-packages/ts/configs/metrics.yaml. 2025-01-20T17:45:26,131 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - [PID]6053 2025-01-20T17:45:26,131 [DEBUG] W-9000-titan_1.0 org.pytorch.serve.wlm.WorkerThread - W-9000-titan_1.0 State change WORKER_STOPPED -> WORKER_STARTED 2025-01-20T17:45:26,131 [DEBUG] W-9000-titan_1.0 org.pytorch.serve.wlm.WorkerThread - W-9000-titan_1.0 State change WORKER_STOPPED -> WORKER_STARTED 2025-01-20T17:45:26,132 [INFO ] W-9000-titan_1.0 org.pytorch.serve.wlm.WorkerThread - Connecting to: /tmp/.ts.sock.9000 2025-01-20T17:45:26,132 [INFO ] W-9000-titan_1.0 org.pytorch.serve.wlm.WorkerThread - Connecting to: /tmp/.ts.sock.9000 2025-01-20T17:45:26,132 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - Torch worker started. 2025-01-20T17:45:26,132 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - Python runtime: 3.11.11 2025-01-20T17:45:26,134 [DEBUG] W-9000-titan_1.0 org.pytorch.serve.wlm.WorkerThread - Flushing req.cmd LOAD repeats 1 to backend at: 1737395126134 2025-01-20T17:45:26,134 [DEBUG] W-9000-titan_1.0 org.pytorch.serve.wlm.WorkerThread - Flushing req.cmd LOAD repeats 1 to backend at: 1737395126134 2025-01-20T17:45:26,135 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - Connection accepted: /tmp/.ts.sock.9000. 2025-01-20T17:45:26,136 [INFO ] W-9000-titan_1.0 org.pytorch.serve.wlm.WorkerThread - Looping backend response at: 1737395126136 2025-01-20T17:45:26,136 [INFO ] W-9000-titan_1.0 org.pytorch.serve.wlm.WorkerThread - Looping backend response at: 1737395126136 2025-01-20T17:45:26,160 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - model_name: titan, batchSize: 1 2025-01-20T17:45:26,162 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - Backend worker process died. 2025-01-20T17:45:26,162 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - Traceback (most recent call last): 2025-01-20T17:45:26,164 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - File "/usr/local/lib/python3.11/dist-packages/ts/model_service_worker.py", line 301, in 2025-01-20T17:45:26,164 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - worker.run_server() 2025-01-20T17:45:26,164 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - File "/usr/local/lib/python3.11/dist-packages/ts/model_service_worker.py", line 268, in run_server 2025-01-20T17:45:26,164 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - self.handle_connection(cl_socket) 2025-01-20T17:45:26,164 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - File "/usr/local/lib/python3.11/dist-packages/ts/model_service_worker.py", line 196, in handle_connection 2025-01-20T17:45:26,164 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - service, result, code = self.load_model(msg) 2025-01-20T17:45:26,165 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - ^^^^^^^^^^^^^^^^^^^^ 2025-01-20T17:45:26,171 [INFO ] epollEventLoopGroup-5-4 org.pytorch.serve.wlm.WorkerThread - 9000 Worker disconnected. WORKER_STARTED 2025-01-20T17:45:26,171 [INFO ] epollEventLoopGroup-5-4 org.pytorch.serve.wlm.WorkerThread - 9000 Worker disconnected. WORKER_STARTED 2025-01-20T17:45:26,171 [DEBUG] W-9000-titan_1.0 org.pytorch.serve.wlm.WorkerThread - System state is : WORKER_STARTED 2025-01-20T17:45:26,171 [DEBUG] W-9000-titan_1.0 org.pytorch.serve.wlm.WorkerThread - System state is : WORKER_STARTED 2025-01-20T17:45:26,171 [DEBUG] W-9000-titan_1.0 org.pytorch.serve.wlm.WorkerThread - Backend worker monitoring thread interrupted or backend worker process died., startupTimeout:120sec java.lang.InterruptedException: null at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2056) ~[?:?] at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2133) ~[?:?] at java.util.concurrent.ArrayBlockingQueue.poll(ArrayBlockingQueue.java:432) ~[?:?] at org.pytorch.serve.wlm.WorkerThread.run(WorkerThread.java:234) [model-server.jar:?] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?] at java.lang.Thread.run(Thread.java:829) [?:?] 2025-01-20T17:45:26,171 [DEBUG] W-9000-titan_1.0 org.pytorch.serve.wlm.WorkerThread - Backend worker monitoring thread interrupted or backend worker process died., startupTimeout:120sec java.lang.InterruptedException: null at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2056) ~[?:?] at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2133) ~[?:?] at java.util.concurrent.ArrayBlockingQueue.poll(ArrayBlockingQueue.java:432) ~[?:?] at org.pytorch.serve.wlm.WorkerThread.run(WorkerThread.java:234) [model-server.jar:?] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?] at java.lang.Thread.run(Thread.java:829) [?:?] 2025-01-20T17:45:26,172 [WARN ] W-9000-titan_1.0 org.pytorch.serve.wlm.BatchAggregator - Load model failed: titan, error: Worker died. 2025-01-20T17:45:26,172 [WARN ] W-9000-titan_1.0 org.pytorch.serve.wlm.BatchAggregator - Load model failed: titan, error: Worker died. 2025-01-20T17:45:26,172 [DEBUG] W-9000-titan_1.0 org.pytorch.serve.wlm.WorkerThread - W-9000-titan_1.0 State change WORKER_STARTED -> WORKER_STOPPED 2025-01-20T17:45:26,172 [DEBUG] W-9000-titan_1.0 org.pytorch.serve.wlm.WorkerThread - W-9000-titan_1.0 State change WORKER_STARTED -> WORKER_STOPPED 2025-01-20T17:45:26,172 [WARN ] W-9000-titan_1.0 org.pytorch.serve.wlm.WorkerThread - Auto recovery failed again 2025-01-20T17:45:26,172 [WARN ] W-9000-titan_1.0 org.pytorch.serve.wlm.WorkerThread - Auto recovery failed again 2025-01-20T17:45:26,173 [INFO ] W-9000-titan_1.0 org.pytorch.serve.wlm.WorkerThread - Retry worker: 9000 in 3 seconds. 2025-01-20T17:45:26,173 [INFO ] W-9000-titan_1.0 org.pytorch.serve.wlm.WorkerThread - Retry worker: 9000 in 3 seconds. 2025-01-20T17:45:26,167 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - File "/usr/local/lib/python3.11/dist-packages/ts/model_service_worker.py", line 133, in load_model 2025-01-20T17:45:26,175 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - service = model_loader.load( 2025-01-20T17:45:26,175 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - ^^^^^^^^^^^^^^^^^^ 2025-01-20T17:45:26,175 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - File "/usr/local/lib/python3.11/dist-packages/ts/model_loader.py", line 108, in load 2025-01-20T17:45:26,175 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - module, function_name = self._load_handler_file(handler) 2025-01-20T17:45:26,175 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-01-20T17:45:26,175 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - File "/usr/local/lib/python3.11/dist-packages/ts/model_loader.py", line 153, in _load_handler_file 2025-01-20T17:45:26,175 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - module = importlib.import_module(module_name) 2025-01-20T17:45:26,175 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-01-20T17:45:26,175 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - File "/usr/lib/python3.11/importlib/__init__.py", line 126, in import_module 2025-01-20T17:45:26,176 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - return _bootstrap._gcd_import(name[level:], package, level) 2025-01-20T17:45:26,179 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-01-20T17:45:26,182 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - File "", line 1204, in _gcd_import 2025-01-20T17:45:26,183 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - File "", line 1176, in _find_and_load 2025-01-20T17:45:26,183 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - File "", line 1147, in _find_and_load_unlocked 2025-01-20T17:45:26,184 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - File "", line 690, in _load_unlocked 2025-01-20T17:45:26,185 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - File "", line 936, in exec_module 2025-01-20T17:45:26,186 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - File "", line 1074, in get_code 2025-01-20T17:45:26,186 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - File "", line 1004, in source_to_code 2025-01-20T17:45:26,186 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - File "", line 241, in _call_with_frames_removed 2025-01-20T17:45:26,189 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - File "/tmp/models/a15c525d26d44618b25a3dc1a7bfcbb5/titan_handler.py", line 86 2025-01-20T17:45:26,189 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - print(" 2025-01-20T17:45:26,190 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - ^ 2025-01-20T17:45:26,190 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - SyntaxError: unterminated string literal (detected at line 86) 2025-01-20T17:45:26,191 [INFO ] W-9000-titan_1.0-stdout org.pytorch.serve.wlm.WorkerLifeCycle - Stopped Scanner - W-9000-titan_1.0-stdout 2025-01-20T17:45:26,191 [INFO ] W-9000-titan_1.0-stdout org.pytorch.serve.wlm.WorkerLifeCycle - Stopped Scanner - W-9000-titan_1.0-stdout 2025-01-20T17:45:26,204 [INFO ] W-9000-titan_1.0-stderr org.pytorch.serve.wlm.WorkerLifeCycle - Stopped Scanner - W-9000-titan_1.0-stderr 2025-01-20T17:45:26,204 [INFO ] W-9000-titan_1.0-stderr org.pytorch.serve.wlm.WorkerLifeCycle - Stopped Scanner - W-9000-titan_1.0-stderr 2025-01-20T17:45:29,173 [DEBUG] W-9000-titan_1.0 org.pytorch.serve.wlm.WorkerLifeCycle - Worker cmdline: [/usr/bin/python3, /usr/local/lib/python3.11/dist-packages/ts/model_service_worker.py, --sock-type, unix, --sock-name, /tmp/.ts.sock.9000, --metrics-config, /usr/local/lib/python3.11/dist-packages/ts/configs/metrics.yaml] 2025-01-20T17:45:29,173 [DEBUG] W-9000-titan_1.0 org.pytorch.serve.wlm.WorkerLifeCycle - Worker cmdline: [/usr/bin/python3, /usr/local/lib/python3.11/dist-packages/ts/model_service_worker.py, --sock-type, unix, --sock-name, /tmp/.ts.sock.9000, --metrics-config, /usr/local/lib/python3.11/dist-packages/ts/configs/metrics.yaml] 2025-01-20T17:45:31,465 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - s_name_part0=/tmp/.ts.sock, s_name_part1=9000, pid=6079 2025-01-20T17:45:31,469 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - Listening on port: /tmp/.ts.sock.9000 2025-01-20T17:45:31,474 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - Successfully loaded /usr/local/lib/python3.11/dist-packages/ts/configs/metrics.yaml. 2025-01-20T17:45:31,474 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - [PID]6079 2025-01-20T17:45:31,475 [DEBUG] W-9000-titan_1.0 org.pytorch.serve.wlm.WorkerThread - W-9000-titan_1.0 State change WORKER_STOPPED -> WORKER_STARTED 2025-01-20T17:45:31,475 [DEBUG] W-9000-titan_1.0 org.pytorch.serve.wlm.WorkerThread - W-9000-titan_1.0 State change WORKER_STOPPED -> WORKER_STARTED 2025-01-20T17:45:31,475 [INFO ] W-9000-titan_1.0 org.pytorch.serve.wlm.WorkerThread - Connecting to: /tmp/.ts.sock.9000 2025-01-20T17:45:31,475 [INFO ] W-9000-titan_1.0 org.pytorch.serve.wlm.WorkerThread - Connecting to: /tmp/.ts.sock.9000 2025-01-20T17:45:31,476 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - Torch worker started. 2025-01-20T17:45:31,477 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - Python runtime: 3.11.11 2025-01-20T17:45:31,477 [DEBUG] W-9000-titan_1.0 org.pytorch.serve.wlm.WorkerThread - Flushing req.cmd LOAD repeats 1 to backend at: 1737395131477 2025-01-20T17:45:31,477 [DEBUG] W-9000-titan_1.0 org.pytorch.serve.wlm.WorkerThread - Flushing req.cmd LOAD repeats 1 to backend at: 1737395131477 2025-01-20T17:45:31,478 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - Connection accepted: /tmp/.ts.sock.9000. 2025-01-20T17:45:31,479 [INFO ] W-9000-titan_1.0 org.pytorch.serve.wlm.WorkerThread - Looping backend response at: 1737395131479 2025-01-20T17:45:31,479 [INFO ] W-9000-titan_1.0 org.pytorch.serve.wlm.WorkerThread - Looping backend response at: 1737395131479 2025-01-20T17:45:31,481 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - model_name: titan, batchSize: 1 2025-01-20T17:45:31,484 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - Backend worker process died. 2025-01-20T17:45:31,484 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - Traceback (most recent call last): 2025-01-20T17:45:31,484 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - File "/usr/local/lib/python3.11/dist-packages/ts/model_service_worker.py", line 301, in 2025-01-20T17:45:31,484 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - worker.run_server() 2025-01-20T17:45:31,484 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - File "/usr/local/lib/python3.11/dist-packages/ts/model_service_worker.py", line 268, in run_server 2025-01-20T17:45:31,484 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - self.handle_connection(cl_socket) 2025-01-20T17:45:31,485 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - File "/usr/local/lib/python3.11/dist-packages/ts/model_service_worker.py", line 196, in handle_connection 2025-01-20T17:45:31,485 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - service, result, code = self.load_model(msg) 2025-01-20T17:45:31,485 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - ^^^^^^^^^^^^^^^^^^^^ 2025-01-20T17:45:31,485 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - File "/usr/local/lib/python3.11/dist-packages/ts/model_service_worker.py", line 133, in load_model 2025-01-20T17:45:31,485 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - service = model_loader.load( 2025-01-20T17:45:31,485 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - ^^^^^^^^^^^^^^^^^^ 2025-01-20T17:45:31,485 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - File "/usr/local/lib/python3.11/dist-packages/ts/model_loader.py", line 108, in load 2025-01-20T17:45:31,485 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - module, function_name = self._load_handler_file(handler) 2025-01-20T17:45:31,485 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-01-20T17:45:31,486 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - File "/usr/local/lib/python3.11/dist-packages/ts/model_loader.py", line 153, in _load_handler_file 2025-01-20T17:45:31,486 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - module = importlib.import_module(module_name) 2025-01-20T17:45:31,486 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-01-20T17:45:31,486 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - File "/usr/lib/python3.11/importlib/__init__.py", line 126, in import_module 2025-01-20T17:45:31,486 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - return _bootstrap._gcd_import(name[level:], package, level) 2025-01-20T17:45:31,486 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-01-20T17:45:31,486 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - File "", line 1204, in _gcd_import 2025-01-20T17:45:31,486 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - File "", line 1176, in _find_and_load 2025-01-20T17:45:31,486 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - File "", line 1147, in _find_and_load_unlocked 2025-01-20T17:45:31,490 [INFO ] epollEventLoopGroup-5-1 org.pytorch.serve.wlm.WorkerThread - 9000 Worker disconnected. WORKER_STARTED 2025-01-20T17:45:31,490 [INFO ] epollEventLoopGroup-5-1 org.pytorch.serve.wlm.WorkerThread - 9000 Worker disconnected. WORKER_STARTED 2025-01-20T17:45:31,490 [DEBUG] W-9000-titan_1.0 org.pytorch.serve.wlm.WorkerThread - System state is : WORKER_STARTED 2025-01-20T17:45:31,490 [DEBUG] W-9000-titan_1.0 org.pytorch.serve.wlm.WorkerThread - System state is : WORKER_STARTED 2025-01-20T17:45:31,487 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - File "", line 690, in _load_unlocked 2025-01-20T17:45:31,491 [DEBUG] W-9000-titan_1.0 org.pytorch.serve.wlm.WorkerThread - Backend worker monitoring thread interrupted or backend worker process died., startupTimeout:120sec java.lang.InterruptedException: null at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2056) ~[?:?] at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2133) ~[?:?] at java.util.concurrent.ArrayBlockingQueue.poll(ArrayBlockingQueue.java:432) ~[?:?] at org.pytorch.serve.wlm.WorkerThread.run(WorkerThread.java:234) [model-server.jar:?] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?] at java.lang.Thread.run(Thread.java:829) [?:?] 2025-01-20T17:45:31,491 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - File "", line 936, in exec_module 2025-01-20T17:45:31,491 [DEBUG] W-9000-titan_1.0 org.pytorch.serve.wlm.WorkerThread - Backend worker monitoring thread interrupted or backend worker process died., startupTimeout:120sec java.lang.InterruptedException: null at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2056) ~[?:?] at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2133) ~[?:?] at java.util.concurrent.ArrayBlockingQueue.poll(ArrayBlockingQueue.java:432) ~[?:?] at org.pytorch.serve.wlm.WorkerThread.run(WorkerThread.java:234) [model-server.jar:?] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?] at java.lang.Thread.run(Thread.java:829) [?:?] 2025-01-20T17:45:31,494 [WARN ] W-9000-titan_1.0 org.pytorch.serve.wlm.BatchAggregator - Load model failed: titan, error: Worker died. 2025-01-20T17:45:31,494 [WARN ] W-9000-titan_1.0 org.pytorch.serve.wlm.BatchAggregator - Load model failed: titan, error: Worker died. 2025-01-20T17:45:31,494 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - File "", line 1074, in get_code 2025-01-20T17:45:31,495 [DEBUG] W-9000-titan_1.0 org.pytorch.serve.wlm.WorkerThread - W-9000-titan_1.0 State change WORKER_STARTED -> WORKER_STOPPED 2025-01-20T17:45:31,495 [DEBUG] W-9000-titan_1.0 org.pytorch.serve.wlm.WorkerThread - W-9000-titan_1.0 State change WORKER_STARTED -> WORKER_STOPPED 2025-01-20T17:45:31,496 [WARN ] W-9000-titan_1.0 org.pytorch.serve.wlm.WorkerThread - Auto recovery failed again 2025-01-20T17:45:31,495 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - File "", line 1004, in source_to_code 2025-01-20T17:45:31,496 [WARN ] W-9000-titan_1.0 org.pytorch.serve.wlm.WorkerThread - Auto recovery failed again 2025-01-20T17:45:31,496 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - File "", line 241, in _call_with_frames_removed 2025-01-20T17:45:31,496 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - File "/tmp/models/a15c525d26d44618b25a3dc1a7bfcbb5/titan_handler.py", line 86 2025-01-20T17:45:31,496 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - print(" 2025-01-20T17:45:31,497 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - ^ 2025-01-20T17:45:31,497 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - SyntaxError: unterminated string literal (detected at line 86) 2025-01-20T17:45:31,497 [INFO ] W-9000-titan_1.0 org.pytorch.serve.wlm.WorkerThread - Retry worker: 9000 in 5 seconds. 2025-01-20T17:45:31,497 [INFO ] W-9000-titan_1.0 org.pytorch.serve.wlm.WorkerThread - Retry worker: 9000 in 5 seconds. 2025-01-20T17:45:31,525 [INFO ] W-9000-titan_1.0-stderr org.pytorch.serve.wlm.WorkerLifeCycle - Stopped Scanner - W-9000-titan_1.0-stderr 2025-01-20T17:45:31,526 [INFO ] W-9000-titan_1.0-stdout org.pytorch.serve.wlm.WorkerLifeCycle - Stopped Scanner - W-9000-titan_1.0-stdout 2025-01-20T17:45:31,526 [INFO ] W-9000-titan_1.0-stdout org.pytorch.serve.wlm.WorkerLifeCycle - Stopped Scanner - W-9000-titan_1.0-stdout 2025-01-20T17:45:31,525 [INFO ] W-9000-titan_1.0-stderr org.pytorch.serve.wlm.WorkerLifeCycle - Stopped Scanner - W-9000-titan_1.0-stderr 2025-01-20T17:45:36,498 [DEBUG] W-9000-titan_1.0 org.pytorch.serve.wlm.WorkerLifeCycle - Worker cmdline: [/usr/bin/python3, /usr/local/lib/python3.11/dist-packages/ts/model_service_worker.py, --sock-type, unix, --sock-name, /tmp/.ts.sock.9000, --metrics-config, /usr/local/lib/python3.11/dist-packages/ts/configs/metrics.yaml] 2025-01-20T17:45:36,498 [DEBUG] W-9000-titan_1.0 org.pytorch.serve.wlm.WorkerLifeCycle - Worker cmdline: [/usr/bin/python3, /usr/local/lib/python3.11/dist-packages/ts/model_service_worker.py, --sock-type, unix, --sock-name, /tmp/.ts.sock.9000, --metrics-config, /usr/local/lib/python3.11/dist-packages/ts/configs/metrics.yaml] 2025-01-20T17:45:38,144 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - s_name_part0=/tmp/.ts.sock, s_name_part1=9000, pid=6114 2025-01-20T17:45:38,146 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - Listening on port: /tmp/.ts.sock.9000 2025-01-20T17:45:38,153 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - Successfully loaded /usr/local/lib/python3.11/dist-packages/ts/configs/metrics.yaml. 2025-01-20T17:45:38,153 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - [PID]6114 2025-01-20T17:45:38,154 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - Torch worker started. 2025-01-20T17:45:38,154 [DEBUG] W-9000-titan_1.0 org.pytorch.serve.wlm.WorkerThread - W-9000-titan_1.0 State change WORKER_STOPPED -> WORKER_STARTED 2025-01-20T17:45:38,154 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - Python runtime: 3.11.11 2025-01-20T17:45:38,154 [DEBUG] W-9000-titan_1.0 org.pytorch.serve.wlm.WorkerThread - W-9000-titan_1.0 State change WORKER_STOPPED -> WORKER_STARTED 2025-01-20T17:45:38,154 [INFO ] W-9000-titan_1.0 org.pytorch.serve.wlm.WorkerThread - Connecting to: /tmp/.ts.sock.9000 2025-01-20T17:45:38,154 [INFO ] W-9000-titan_1.0 org.pytorch.serve.wlm.WorkerThread - Connecting to: /tmp/.ts.sock.9000 2025-01-20T17:45:38,155 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - Connection accepted: /tmp/.ts.sock.9000. 2025-01-20T17:45:38,156 [DEBUG] W-9000-titan_1.0 org.pytorch.serve.wlm.WorkerThread - Flushing req.cmd LOAD repeats 1 to backend at: 1737395138156 2025-01-20T17:45:38,156 [DEBUG] W-9000-titan_1.0 org.pytorch.serve.wlm.WorkerThread - Flushing req.cmd LOAD repeats 1 to backend at: 1737395138156 2025-01-20T17:45:38,157 [INFO ] W-9000-titan_1.0 org.pytorch.serve.wlm.WorkerThread - Looping backend response at: 1737395138157 2025-01-20T17:45:38,157 [INFO ] W-9000-titan_1.0 org.pytorch.serve.wlm.WorkerThread - Looping backend response at: 1737395138157 2025-01-20T17:45:38,161 [INFO ] epollEventLoopGroup-5-2 org.pytorch.serve.wlm.WorkerThread - 9000 Worker disconnected. WORKER_STARTED 2025-01-20T17:45:38,161 [INFO ] epollEventLoopGroup-5-2 org.pytorch.serve.wlm.WorkerThread - 9000 Worker disconnected. WORKER_STARTED 2025-01-20T17:45:38,161 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - model_name: titan, batchSize: 1 2025-01-20T17:45:38,161 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - Backend worker process died. 2025-01-20T17:45:38,162 [DEBUG] W-9000-titan_1.0 org.pytorch.serve.wlm.WorkerThread - System state is : WORKER_STARTED 2025-01-20T17:45:38,162 [DEBUG] W-9000-titan_1.0 org.pytorch.serve.wlm.WorkerThread - System state is : WORKER_STARTED 2025-01-20T17:45:38,163 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - Traceback (most recent call last): 2025-01-20T17:45:38,163 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - File "/usr/local/lib/python3.11/dist-packages/ts/model_service_worker.py", line 301, in 2025-01-20T17:45:38,163 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - worker.run_server() 2025-01-20T17:45:38,163 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - File "/usr/local/lib/python3.11/dist-packages/ts/model_service_worker.py", line 268, in run_server 2025-01-20T17:45:38,163 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - self.handle_connection(cl_socket) 2025-01-20T17:45:38,163 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - File "/usr/local/lib/python3.11/dist-packages/ts/model_service_worker.py", line 196, in handle_connection 2025-01-20T17:45:38,164 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - service, result, code = self.load_model(msg) 2025-01-20T17:45:38,164 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - ^^^^^^^^^^^^^^^^^^^^ 2025-01-20T17:45:38,164 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - File "/usr/local/lib/python3.11/dist-packages/ts/model_service_worker.py", line 133, in load_model 2025-01-20T17:45:38,164 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - service = model_loader.load( 2025-01-20T17:45:38,164 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - ^^^^^^^^^^^^^^^^^^ 2025-01-20T17:45:38,164 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - File "/usr/local/lib/python3.11/dist-packages/ts/model_loader.py", line 108, in load 2025-01-20T17:45:38,164 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - module, function_name = self._load_handler_file(handler) 2025-01-20T17:45:38,164 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-01-20T17:45:38,164 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - File "/usr/local/lib/python3.11/dist-packages/ts/model_loader.py", line 153, in _load_handler_file 2025-01-20T17:45:38,164 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - module = importlib.import_module(module_name) 2025-01-20T17:45:38,164 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-01-20T17:45:38,162 [DEBUG] W-9000-titan_1.0 org.pytorch.serve.wlm.WorkerThread - Backend worker monitoring thread interrupted or backend worker process died., startupTimeout:120sec java.lang.InterruptedException: null at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2056) ~[?:?] at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2133) ~[?:?] at java.util.concurrent.ArrayBlockingQueue.poll(ArrayBlockingQueue.java:432) ~[?:?] at org.pytorch.serve.wlm.WorkerThread.run(WorkerThread.java:234) [model-server.jar:?] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?] at java.lang.Thread.run(Thread.java:829) [?:?] 2025-01-20T17:45:38,162 [DEBUG] W-9000-titan_1.0 org.pytorch.serve.wlm.WorkerThread - Backend worker monitoring thread interrupted or backend worker process died., startupTimeout:120sec java.lang.InterruptedException: null at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2056) ~[?:?] at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2133) ~[?:?] at java.util.concurrent.ArrayBlockingQueue.poll(ArrayBlockingQueue.java:432) ~[?:?] at org.pytorch.serve.wlm.WorkerThread.run(WorkerThread.java:234) [model-server.jar:?] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?] at java.lang.Thread.run(Thread.java:829) [?:?] 2025-01-20T17:45:38,165 [WARN ] W-9000-titan_1.0 org.pytorch.serve.wlm.BatchAggregator - Load model failed: titan, error: Worker died. 2025-01-20T17:45:38,165 [WARN ] W-9000-titan_1.0 org.pytorch.serve.wlm.BatchAggregator - Load model failed: titan, error: Worker died. 2025-01-20T17:45:38,165 [DEBUG] W-9000-titan_1.0 org.pytorch.serve.wlm.WorkerThread - W-9000-titan_1.0 State change WORKER_STARTED -> WORKER_STOPPED 2025-01-20T17:45:38,165 [DEBUG] W-9000-titan_1.0 org.pytorch.serve.wlm.WorkerThread - W-9000-titan_1.0 State change WORKER_STARTED -> WORKER_STOPPED 2025-01-20T17:45:38,166 [WARN ] W-9000-titan_1.0 org.pytorch.serve.wlm.WorkerThread - Auto recovery failed again 2025-01-20T17:45:38,166 [WARN ] W-9000-titan_1.0 org.pytorch.serve.wlm.WorkerThread - Auto recovery failed again 2025-01-20T17:45:38,166 [INFO ] W-9000-titan_1.0 org.pytorch.serve.wlm.WorkerThread - Retry worker: 9000 in 8 seconds. 2025-01-20T17:45:38,166 [INFO ] W-9000-titan_1.0 org.pytorch.serve.wlm.WorkerThread - Retry worker: 9000 in 8 seconds. 2025-01-20T17:45:38,165 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - File "/usr/lib/python3.11/importlib/__init__.py", line 126, in import_module 2025-01-20T17:45:38,167 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - return _bootstrap._gcd_import(name[level:], package, level) 2025-01-20T17:45:38,167 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-01-20T17:45:38,168 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - File "", line 1204, in _gcd_import 2025-01-20T17:45:38,169 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - File "", line 1176, in _find_and_load 2025-01-20T17:45:38,169 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - File "", line 1147, in _find_and_load_unlocked 2025-01-20T17:45:38,169 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - File "", line 690, in _load_unlocked 2025-01-20T17:45:38,169 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - File "", line 936, in exec_module 2025-01-20T17:45:38,169 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - File "", line 1074, in get_code 2025-01-20T17:45:38,169 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - File "", line 1004, in source_to_code 2025-01-20T17:45:38,169 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - File "", line 241, in _call_with_frames_removed 2025-01-20T17:45:38,169 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - File "/tmp/models/a15c525d26d44618b25a3dc1a7bfcbb5/titan_handler.py", line 86 2025-01-20T17:45:38,170 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - print(" 2025-01-20T17:45:38,170 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - ^ 2025-01-20T17:45:38,170 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - SyntaxError: unterminated string literal (detected at line 86) 2025-01-20T17:45:38,173 [INFO ] W-9000-titan_1.0-stdout org.pytorch.serve.wlm.WorkerLifeCycle - Stopped Scanner - W-9000-titan_1.0-stdout 2025-01-20T17:45:38,173 [INFO ] W-9000-titan_1.0-stdout org.pytorch.serve.wlm.WorkerLifeCycle - Stopped Scanner - W-9000-titan_1.0-stdout 2025-01-20T17:45:38,197 [INFO ] W-9000-titan_1.0-stderr org.pytorch.serve.wlm.WorkerLifeCycle - Stopped Scanner - W-9000-titan_1.0-stderr 2025-01-20T17:45:38,197 [INFO ] W-9000-titan_1.0-stderr org.pytorch.serve.wlm.WorkerLifeCycle - Stopped Scanner - W-9000-titan_1.0-stderr 2025-01-20T17:45:46,167 [DEBUG] W-9000-titan_1.0 org.pytorch.serve.wlm.WorkerLifeCycle - Worker cmdline: [/usr/bin/python3, /usr/local/lib/python3.11/dist-packages/ts/model_service_worker.py, --sock-type, unix, --sock-name, /tmp/.ts.sock.9000, --metrics-config, /usr/local/lib/python3.11/dist-packages/ts/configs/metrics.yaml] 2025-01-20T17:45:46,167 [DEBUG] W-9000-titan_1.0 org.pytorch.serve.wlm.WorkerLifeCycle - Worker cmdline: [/usr/bin/python3, /usr/local/lib/python3.11/dist-packages/ts/model_service_worker.py, --sock-type, unix, --sock-name, /tmp/.ts.sock.9000, --metrics-config, /usr/local/lib/python3.11/dist-packages/ts/configs/metrics.yaml] 2025-01-20T17:45:47,744 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - s_name_part0=/tmp/.ts.sock, s_name_part1=9000, pid=6163 2025-01-20T17:45:47,744 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - Listening on port: /tmp/.ts.sock.9000 2025-01-20T17:45:47,752 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - Successfully loaded /usr/local/lib/python3.11/dist-packages/ts/configs/metrics.yaml. 2025-01-20T17:45:47,752 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - [PID]6163 2025-01-20T17:45:47,752 [DEBUG] W-9000-titan_1.0 org.pytorch.serve.wlm.WorkerThread - W-9000-titan_1.0 State change WORKER_STOPPED -> WORKER_STARTED 2025-01-20T17:45:47,752 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - Torch worker started. 2025-01-20T17:45:47,752 [DEBUG] W-9000-titan_1.0 org.pytorch.serve.wlm.WorkerThread - W-9000-titan_1.0 State change WORKER_STOPPED -> WORKER_STARTED 2025-01-20T17:45:47,753 [INFO ] W-9000-titan_1.0 org.pytorch.serve.wlm.WorkerThread - Connecting to: /tmp/.ts.sock.9000 2025-01-20T17:45:47,753 [INFO ] W-9000-titan_1.0 org.pytorch.serve.wlm.WorkerThread - Connecting to: /tmp/.ts.sock.9000 2025-01-20T17:45:47,754 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - Python runtime: 3.11.11 2025-01-20T17:45:47,757 [DEBUG] W-9000-titan_1.0 org.pytorch.serve.wlm.WorkerThread - Flushing req.cmd LOAD repeats 1 to backend at: 1737395147757 2025-01-20T17:45:47,757 [DEBUG] W-9000-titan_1.0 org.pytorch.serve.wlm.WorkerThread - Flushing req.cmd LOAD repeats 1 to backend at: 1737395147757 2025-01-20T17:45:47,758 [INFO ] W-9000-titan_1.0 org.pytorch.serve.wlm.WorkerThread - Looping backend response at: 1737395147757 2025-01-20T17:45:47,758 [INFO ] W-9000-titan_1.0 org.pytorch.serve.wlm.WorkerThread - Looping backend response at: 1737395147757 2025-01-20T17:45:47,759 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - Connection accepted: /tmp/.ts.sock.9000. 2025-01-20T17:45:47,761 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - model_name: titan, batchSize: 1 2025-01-20T17:45:47,763 [INFO ] epollEventLoopGroup-5-3 org.pytorch.serve.wlm.WorkerThread - 9000 Worker disconnected. WORKER_STARTED 2025-01-20T17:45:47,763 [INFO ] epollEventLoopGroup-5-3 org.pytorch.serve.wlm.WorkerThread - 9000 Worker disconnected. WORKER_STARTED 2025-01-20T17:45:47,764 [DEBUG] W-9000-titan_1.0 org.pytorch.serve.wlm.WorkerThread - System state is : WORKER_STARTED 2025-01-20T17:45:47,764 [DEBUG] W-9000-titan_1.0 org.pytorch.serve.wlm.WorkerThread - System state is : WORKER_STARTED 2025-01-20T17:45:47,764 [DEBUG] W-9000-titan_1.0 org.pytorch.serve.wlm.WorkerThread - Backend worker monitoring thread interrupted or backend worker process died., startupTimeout:120sec java.lang.InterruptedException: null at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2056) ~[?:?] at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2133) ~[?:?] at java.util.concurrent.ArrayBlockingQueue.poll(ArrayBlockingQueue.java:432) ~[?:?] at org.pytorch.serve.wlm.WorkerThread.run(WorkerThread.java:234) [model-server.jar:?] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?] at java.lang.Thread.run(Thread.java:829) [?:?] 2025-01-20T17:45:47,764 [DEBUG] W-9000-titan_1.0 org.pytorch.serve.wlm.WorkerThread - Backend worker monitoring thread interrupted or backend worker process died., startupTimeout:120sec java.lang.InterruptedException: null at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2056) ~[?:?] at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2133) ~[?:?] at java.util.concurrent.ArrayBlockingQueue.poll(ArrayBlockingQueue.java:432) ~[?:?] at org.pytorch.serve.wlm.WorkerThread.run(WorkerThread.java:234) [model-server.jar:?] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?] at java.lang.Thread.run(Thread.java:829) [?:?] 2025-01-20T17:45:47,765 [WARN ] W-9000-titan_1.0 org.pytorch.serve.wlm.BatchAggregator - Load model failed: titan, error: Worker died. 2025-01-20T17:45:47,765 [WARN ] W-9000-titan_1.0 org.pytorch.serve.wlm.BatchAggregator - Load model failed: titan, error: Worker died. 2025-01-20T17:45:47,766 [DEBUG] W-9000-titan_1.0 org.pytorch.serve.wlm.WorkerThread - W-9000-titan_1.0 State change WORKER_STARTED -> WORKER_STOPPED 2025-01-20T17:45:47,763 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - Backend worker process died. 2025-01-20T17:45:47,766 [DEBUG] W-9000-titan_1.0 org.pytorch.serve.wlm.WorkerThread - W-9000-titan_1.0 State change WORKER_STARTED -> WORKER_STOPPED 2025-01-20T17:45:47,766 [WARN ] W-9000-titan_1.0 org.pytorch.serve.wlm.WorkerThread - Auto recovery failed again 2025-01-20T17:45:47,766 [WARN ] W-9000-titan_1.0 org.pytorch.serve.wlm.WorkerThread - Auto recovery failed again 2025-01-20T17:45:47,767 [INFO ] W-9000-titan_1.0 org.pytorch.serve.wlm.WorkerThread - Retry worker: 9000 in 13 seconds. 2025-01-20T17:45:47,766 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - Traceback (most recent call last): 2025-01-20T17:45:47,768 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - File "/usr/local/lib/python3.11/dist-packages/ts/model_service_worker.py", line 301, in 2025-01-20T17:45:47,768 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - worker.run_server() 2025-01-20T17:45:47,768 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - File "/usr/local/lib/python3.11/dist-packages/ts/model_service_worker.py", line 268, in run_server 2025-01-20T17:45:47,768 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - self.handle_connection(cl_socket) 2025-01-20T17:45:47,768 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - File "/usr/local/lib/python3.11/dist-packages/ts/model_service_worker.py", line 196, in handle_connection 2025-01-20T17:45:47,768 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - service, result, code = self.load_model(msg) 2025-01-20T17:45:47,767 [INFO ] W-9000-titan_1.0 org.pytorch.serve.wlm.WorkerThread - Retry worker: 9000 in 13 seconds. 2025-01-20T17:45:47,768 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - ^^^^^^^^^^^^^^^^^^^^ 2025-01-20T17:45:47,768 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - File "/usr/local/lib/python3.11/dist-packages/ts/model_service_worker.py", line 133, in load_model 2025-01-20T17:45:47,768 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - service = model_loader.load( 2025-01-20T17:45:47,769 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - ^^^^^^^^^^^^^^^^^^ 2025-01-20T17:45:47,769 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - File "/usr/local/lib/python3.11/dist-packages/ts/model_loader.py", line 108, in load 2025-01-20T17:45:47,769 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - module, function_name = self._load_handler_file(handler) 2025-01-20T17:45:47,769 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-01-20T17:45:47,770 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - File "/usr/local/lib/python3.11/dist-packages/ts/model_loader.py", line 153, in _load_handler_file 2025-01-20T17:45:47,770 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - module = importlib.import_module(module_name) 2025-01-20T17:45:47,772 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-01-20T17:45:47,775 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - File "/usr/lib/python3.11/importlib/__init__.py", line 126, in import_module 2025-01-20T17:45:47,775 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - return _bootstrap._gcd_import(name[level:], package, level) 2025-01-20T17:45:47,776 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-01-20T17:45:47,780 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - File "", line 1204, in _gcd_import 2025-01-20T17:45:47,780 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - File "", line 1176, in _find_and_load 2025-01-20T17:45:47,780 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - File "", line 1147, in _find_and_load_unlocked 2025-01-20T17:45:47,781 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - File "", line 690, in _load_unlocked 2025-01-20T17:45:47,781 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - File "", line 936, in exec_module 2025-01-20T17:45:47,782 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - File "", line 1074, in get_code 2025-01-20T17:45:47,782 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - File "", line 1004, in source_to_code 2025-01-20T17:45:47,783 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - File "", line 241, in _call_with_frames_removed 2025-01-20T17:45:47,783 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - File "/tmp/models/a15c525d26d44618b25a3dc1a7bfcbb5/titan_handler.py", line 86 2025-01-20T17:45:47,783 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - print(" 2025-01-20T17:45:47,784 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - ^ 2025-01-20T17:45:47,785 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - SyntaxError: unterminated string literal (detected at line 86) 2025-01-20T17:45:47,788 [INFO ] W-9000-titan_1.0-stdout org.pytorch.serve.wlm.WorkerLifeCycle - Stopped Scanner - W-9000-titan_1.0-stdout 2025-01-20T17:45:47,788 [INFO ] W-9000-titan_1.0-stdout org.pytorch.serve.wlm.WorkerLifeCycle - Stopped Scanner - W-9000-titan_1.0-stdout 2025-01-20T17:45:47,806 [INFO ] W-9000-titan_1.0-stderr org.pytorch.serve.wlm.WorkerLifeCycle - Stopped Scanner - W-9000-titan_1.0-stderr 2025-01-20T17:45:47,806 [INFO ] W-9000-titan_1.0-stderr org.pytorch.serve.wlm.WorkerLifeCycle - Stopped Scanner - W-9000-titan_1.0-stderr 2025-01-20T17:46:00,768 [DEBUG] W-9000-titan_1.0 org.pytorch.serve.wlm.WorkerLifeCycle - Worker cmdline: [/usr/bin/python3, /usr/local/lib/python3.11/dist-packages/ts/model_service_worker.py, --sock-type, unix, --sock-name, /tmp/.ts.sock.9000, --metrics-config, /usr/local/lib/python3.11/dist-packages/ts/configs/metrics.yaml] 2025-01-20T17:46:00,768 [DEBUG] W-9000-titan_1.0 org.pytorch.serve.wlm.WorkerLifeCycle - Worker cmdline: [/usr/bin/python3, /usr/local/lib/python3.11/dist-packages/ts/model_service_worker.py, --sock-type, unix, --sock-name, /tmp/.ts.sock.9000, --metrics-config, /usr/local/lib/python3.11/dist-packages/ts/configs/metrics.yaml] 2025-01-20T17:46:02,338 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - s_name_part0=/tmp/.ts.sock, s_name_part1=9000, pid=6230 2025-01-20T17:46:02,339 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - Listening on port: /tmp/.ts.sock.9000 2025-01-20T17:46:02,346 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - Successfully loaded /usr/local/lib/python3.11/dist-packages/ts/configs/metrics.yaml. 2025-01-20T17:46:02,346 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - [PID]6230 2025-01-20T17:46:02,347 [DEBUG] W-9000-titan_1.0 org.pytorch.serve.wlm.WorkerThread - W-9000-titan_1.0 State change WORKER_STOPPED -> WORKER_STARTED 2025-01-20T17:46:02,347 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - Torch worker started. 2025-01-20T17:46:02,347 [DEBUG] W-9000-titan_1.0 org.pytorch.serve.wlm.WorkerThread - W-9000-titan_1.0 State change WORKER_STOPPED -> WORKER_STARTED 2025-01-20T17:46:02,347 [INFO ] W-9000-titan_1.0 org.pytorch.serve.wlm.WorkerThread - Connecting to: /tmp/.ts.sock.9000 2025-01-20T17:46:02,347 [INFO ] W-9000-titan_1.0 org.pytorch.serve.wlm.WorkerThread - Connecting to: /tmp/.ts.sock.9000 2025-01-20T17:46:02,348 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - Python runtime: 3.11.11 2025-01-20T17:46:02,349 [DEBUG] W-9000-titan_1.0 org.pytorch.serve.wlm.WorkerThread - Flushing req.cmd LOAD repeats 1 to backend at: 1737395162348 2025-01-20T17:46:02,349 [DEBUG] W-9000-titan_1.0 org.pytorch.serve.wlm.WorkerThread - Flushing req.cmd LOAD repeats 1 to backend at: 1737395162348 2025-01-20T17:46:02,349 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - Connection accepted: /tmp/.ts.sock.9000. 2025-01-20T17:46:02,349 [INFO ] W-9000-titan_1.0 org.pytorch.serve.wlm.WorkerThread - Looping backend response at: 1737395162349 2025-01-20T17:46:02,349 [INFO ] W-9000-titan_1.0 org.pytorch.serve.wlm.WorkerThread - Looping backend response at: 1737395162349 2025-01-20T17:46:02,350 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - model_name: titan, batchSize: 1 2025-01-20T17:46:02,352 [INFO ] epollEventLoopGroup-5-4 org.pytorch.serve.wlm.WorkerThread - 9000 Worker disconnected. WORKER_STARTED 2025-01-20T17:46:02,352 [INFO ] epollEventLoopGroup-5-4 org.pytorch.serve.wlm.WorkerThread - 9000 Worker disconnected. WORKER_STARTED 2025-01-20T17:46:02,353 [DEBUG] W-9000-titan_1.0 org.pytorch.serve.wlm.WorkerThread - System state is : WORKER_STARTED 2025-01-20T17:46:02,353 [DEBUG] W-9000-titan_1.0 org.pytorch.serve.wlm.WorkerThread - System state is : WORKER_STARTED 2025-01-20T17:46:02,353 [DEBUG] W-9000-titan_1.0 org.pytorch.serve.wlm.WorkerThread - Backend worker monitoring thread interrupted or backend worker process died., startupTimeout:120sec java.lang.InterruptedException: null at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2056) ~[?:?] at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2133) ~[?:?] at java.util.concurrent.ArrayBlockingQueue.poll(ArrayBlockingQueue.java:432) ~[?:?] at org.pytorch.serve.wlm.WorkerThread.run(WorkerThread.java:234) [model-server.jar:?] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?] at java.lang.Thread.run(Thread.java:829) [?:?] 2025-01-20T17:46:02,353 [DEBUG] W-9000-titan_1.0 org.pytorch.serve.wlm.WorkerThread - Backend worker monitoring thread interrupted or backend worker process died., startupTimeout:120sec java.lang.InterruptedException: null at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2056) ~[?:?] at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2133) ~[?:?] at java.util.concurrent.ArrayBlockingQueue.poll(ArrayBlockingQueue.java:432) ~[?:?] at org.pytorch.serve.wlm.WorkerThread.run(WorkerThread.java:234) [model-server.jar:?] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?] at java.lang.Thread.run(Thread.java:829) [?:?] 2025-01-20T17:46:02,354 [WARN ] W-9000-titan_1.0 org.pytorch.serve.wlm.BatchAggregator - Load model failed: titan, error: Worker died. 2025-01-20T17:46:02,354 [WARN ] W-9000-titan_1.0 org.pytorch.serve.wlm.BatchAggregator - Load model failed: titan, error: Worker died. 2025-01-20T17:46:02,354 [DEBUG] W-9000-titan_1.0 org.pytorch.serve.wlm.WorkerThread - W-9000-titan_1.0 State change WORKER_STARTED -> WORKER_STOPPED 2025-01-20T17:46:02,354 [DEBUG] W-9000-titan_1.0 org.pytorch.serve.wlm.WorkerThread - W-9000-titan_1.0 State change WORKER_STARTED -> WORKER_STOPPED 2025-01-20T17:46:02,354 [WARN ] W-9000-titan_1.0 org.pytorch.serve.wlm.WorkerThread - Auto recovery failed again 2025-01-20T17:46:02,354 [WARN ] W-9000-titan_1.0 org.pytorch.serve.wlm.WorkerThread - Auto recovery failed again 2025-01-20T17:46:02,354 [INFO ] W-9000-titan_1.0 org.pytorch.serve.wlm.WorkerThread - Retry worker: 9000 in 21 seconds. 2025-01-20T17:46:02,354 [INFO ] W-9000-titan_1.0 org.pytorch.serve.wlm.WorkerThread - Retry worker: 9000 in 21 seconds. 2025-01-20T17:46:02,352 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - Backend worker process died. 2025-01-20T17:46:02,356 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - Traceback (most recent call last): 2025-01-20T17:46:02,356 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - File "/usr/local/lib/python3.11/dist-packages/ts/model_service_worker.py", line 301, in 2025-01-20T17:46:02,356 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - worker.run_server() 2025-01-20T17:46:02,356 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - File "/usr/local/lib/python3.11/dist-packages/ts/model_service_worker.py", line 268, in run_server 2025-01-20T17:46:02,356 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - self.handle_connection(cl_socket) 2025-01-20T17:46:02,356 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - File "/usr/local/lib/python3.11/dist-packages/ts/model_service_worker.py", line 196, in handle_connection 2025-01-20T17:46:02,356 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - service, result, code = self.load_model(msg) 2025-01-20T17:46:02,356 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - ^^^^^^^^^^^^^^^^^^^^ 2025-01-20T17:46:02,356 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - File "/usr/local/lib/python3.11/dist-packages/ts/model_service_worker.py", line 133, in load_model 2025-01-20T17:46:02,356 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - service = model_loader.load( 2025-01-20T17:46:02,356 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - ^^^^^^^^^^^^^^^^^^ 2025-01-20T17:46:02,356 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - File "/usr/local/lib/python3.11/dist-packages/ts/model_loader.py", line 108, in load 2025-01-20T17:46:02,356 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - module, function_name = self._load_handler_file(handler) 2025-01-20T17:46:02,356 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-01-20T17:46:02,356 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - File "/usr/local/lib/python3.11/dist-packages/ts/model_loader.py", line 153, in _load_handler_file 2025-01-20T17:46:02,356 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - module = importlib.import_module(module_name) 2025-01-20T17:46:02,357 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-01-20T17:46:02,358 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - File "/usr/lib/python3.11/importlib/__init__.py", line 126, in import_module 2025-01-20T17:46:02,366 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - return _bootstrap._gcd_import(name[level:], package, level) 2025-01-20T17:46:02,367 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-01-20T17:46:02,367 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - File "", line 1204, in _gcd_import 2025-01-20T17:46:02,368 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - File "", line 1176, in _find_and_load 2025-01-20T17:46:02,376 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - File "", line 1147, in _find_and_load_unlocked 2025-01-20T17:46:02,376 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - File "", line 690, in _load_unlocked 2025-01-20T17:46:02,376 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - File "", line 936, in exec_module 2025-01-20T17:46:02,376 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - File "", line 1074, in get_code 2025-01-20T17:46:02,377 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - File "", line 1004, in source_to_code 2025-01-20T17:46:02,378 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - File "", line 241, in _call_with_frames_removed 2025-01-20T17:46:02,379 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - File "/tmp/models/a15c525d26d44618b25a3dc1a7bfcbb5/titan_handler.py", line 86 2025-01-20T17:46:02,379 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - print(" 2025-01-20T17:46:02,379 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - ^ 2025-01-20T17:46:02,379 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - SyntaxError: unterminated string literal (detected at line 86) 2025-01-20T17:46:02,380 [INFO ] W-9000-titan_1.0-stdout org.pytorch.serve.wlm.WorkerLifeCycle - Stopped Scanner - W-9000-titan_1.0-stdout 2025-01-20T17:46:02,380 [INFO ] W-9000-titan_1.0-stdout org.pytorch.serve.wlm.WorkerLifeCycle - Stopped Scanner - W-9000-titan_1.0-stdout 2025-01-20T17:46:02,388 [INFO ] W-9000-titan_1.0-stderr org.pytorch.serve.wlm.WorkerLifeCycle - Stopped Scanner - W-9000-titan_1.0-stderr 2025-01-20T17:46:02,388 [INFO ] W-9000-titan_1.0-stderr org.pytorch.serve.wlm.WorkerLifeCycle - Stopped Scanner - W-9000-titan_1.0-stderr 2025-01-20T17:46:14,951 [ERROR] Thread-10 org.pytorch.serve.metrics.MetricCollector - Traceback (most recent call last): File "/usr/local/lib/python3.11/dist-packages/ts/metrics/metric_collector.py", line 11, in from ts.metrics import system_metrics File "/usr/local/lib/python3.11/dist-packages/ts/__init__.py", line 10, in from . import version File "/usr/local/lib/python3.11/dist-packages/ts/version.py", line 5, in from pathlib import Path File "/usr/local/lib/python3.11/dist-packages/pathlib.py", line 10, in from collections import Sequence ImportError: cannot import name 'Sequence' from 'collections' (/usr/lib/python3.11/collections/__init__.py) 2025-01-20T17:46:14,951 [ERROR] Thread-10 org.pytorch.serve.metrics.MetricCollector - Traceback (most recent call last): File "/usr/local/lib/python3.11/dist-packages/ts/metrics/metric_collector.py", line 11, in from ts.metrics import system_metrics File "/usr/local/lib/python3.11/dist-packages/ts/__init__.py", line 10, in from . import version File "/usr/local/lib/python3.11/dist-packages/ts/version.py", line 5, in from pathlib import Path File "/usr/local/lib/python3.11/dist-packages/pathlib.py", line 10, in from collections import Sequence ImportError: cannot import name 'Sequence' from 'collections' (/usr/lib/python3.11/collections/__init__.py) 2025-01-20T17:46:23,355 [DEBUG] W-9000-titan_1.0 org.pytorch.serve.wlm.WorkerLifeCycle - Worker cmdline: [/usr/bin/python3, /usr/local/lib/python3.11/dist-packages/ts/model_service_worker.py, --sock-type, unix, --sock-name, /tmp/.ts.sock.9000, --metrics-config, /usr/local/lib/python3.11/dist-packages/ts/configs/metrics.yaml] 2025-01-20T17:46:23,355 [DEBUG] W-9000-titan_1.0 org.pytorch.serve.wlm.WorkerLifeCycle - Worker cmdline: [/usr/bin/python3, /usr/local/lib/python3.11/dist-packages/ts/model_service_worker.py, --sock-type, unix, --sock-name, /tmp/.ts.sock.9000, --metrics-config, /usr/local/lib/python3.11/dist-packages/ts/configs/metrics.yaml] 2025-01-20T17:46:24,990 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - s_name_part0=/tmp/.ts.sock, s_name_part1=9000, pid=6331 2025-01-20T17:46:24,992 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - Listening on port: /tmp/.ts.sock.9000 2025-01-20T17:46:25,005 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - Successfully loaded /usr/local/lib/python3.11/dist-packages/ts/configs/metrics.yaml. 2025-01-20T17:46:25,006 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - [PID]6331 2025-01-20T17:46:25,006 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - Torch worker started. 2025-01-20T17:46:25,006 [DEBUG] W-9000-titan_1.0 org.pytorch.serve.wlm.WorkerThread - W-9000-titan_1.0 State change WORKER_STOPPED -> WORKER_STARTED 2025-01-20T17:46:25,006 [DEBUG] W-9000-titan_1.0 org.pytorch.serve.wlm.WorkerThread - W-9000-titan_1.0 State change WORKER_STOPPED -> WORKER_STARTED 2025-01-20T17:46:25,007 [INFO ] W-9000-titan_1.0 org.pytorch.serve.wlm.WorkerThread - Connecting to: /tmp/.ts.sock.9000 2025-01-20T17:46:25,007 [INFO ] W-9000-titan_1.0 org.pytorch.serve.wlm.WorkerThread - Connecting to: /tmp/.ts.sock.9000 2025-01-20T17:46:25,007 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - Python runtime: 3.11.11 2025-01-20T17:46:25,010 [DEBUG] W-9000-titan_1.0 org.pytorch.serve.wlm.WorkerThread - Flushing req.cmd LOAD repeats 1 to backend at: 1737395185010 2025-01-20T17:46:25,010 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - Connection accepted: /tmp/.ts.sock.9000. 2025-01-20T17:46:25,010 [DEBUG] W-9000-titan_1.0 org.pytorch.serve.wlm.WorkerThread - Flushing req.cmd LOAD repeats 1 to backend at: 1737395185010 2025-01-20T17:46:25,010 [INFO ] W-9000-titan_1.0 org.pytorch.serve.wlm.WorkerThread - Looping backend response at: 1737395185010 2025-01-20T17:46:25,010 [INFO ] W-9000-titan_1.0 org.pytorch.serve.wlm.WorkerThread - Looping backend response at: 1737395185010 2025-01-20T17:46:25,012 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - model_name: titan, batchSize: 1 2025-01-20T17:46:25,017 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - Backend worker process died. 2025-01-20T17:46:25,017 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - Traceback (most recent call last): 2025-01-20T17:46:25,017 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - File "/usr/local/lib/python3.11/dist-packages/ts/model_service_worker.py", line 301, in 2025-01-20T17:46:25,018 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - worker.run_server() 2025-01-20T17:46:25,019 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - File "/usr/local/lib/python3.11/dist-packages/ts/model_service_worker.py", line 268, in run_server 2025-01-20T17:46:25,019 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - self.handle_connection(cl_socket) 2025-01-20T17:46:25,019 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - File "/usr/local/lib/python3.11/dist-packages/ts/model_service_worker.py", line 196, in handle_connection 2025-01-20T17:46:25,020 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - service, result, code = self.load_model(msg) 2025-01-20T17:46:25,020 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - ^^^^^^^^^^^^^^^^^^^^ 2025-01-20T17:46:25,020 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - File "/usr/local/lib/python3.11/dist-packages/ts/model_service_worker.py", line 133, in load_model 2025-01-20T17:46:25,021 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - service = model_loader.load( 2025-01-20T17:46:25,022 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - ^^^^^^^^^^^^^^^^^^ 2025-01-20T17:46:25,022 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - File "/usr/local/lib/python3.11/dist-packages/ts/model_loader.py", line 108, in load 2025-01-20T17:46:25,023 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - module, function_name = self._load_handler_file(handler) 2025-01-20T17:46:25,024 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-01-20T17:46:25,024 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - File "/usr/local/lib/python3.11/dist-packages/ts/model_loader.py", line 153, in _load_handler_file 2025-01-20T17:46:25,026 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - module = importlib.import_module(module_name) 2025-01-20T17:46:25,026 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-01-20T17:46:25,027 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - File "/usr/lib/python3.11/importlib/__init__.py", line 126, in import_module 2025-01-20T17:46:25,027 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - return _bootstrap._gcd_import(name[level:], package, level) 2025-01-20T17:46:25,027 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-01-20T17:46:25,028 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - File "", line 1204, in _gcd_import 2025-01-20T17:46:25,028 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - File "", line 1176, in _find_and_load 2025-01-20T17:46:25,029 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - File "", line 1147, in _find_and_load_unlocked 2025-01-20T17:46:25,030 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - File "", line 690, in _load_unlocked 2025-01-20T17:46:25,030 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - File "", line 936, in exec_module 2025-01-20T17:46:25,031 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - File "", line 1074, in get_code 2025-01-20T17:46:25,032 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - File "", line 1004, in source_to_code 2025-01-20T17:46:25,033 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - File "", line 241, in _call_with_frames_removed 2025-01-20T17:46:25,033 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - File "/tmp/models/a15c525d26d44618b25a3dc1a7bfcbb5/titan_handler.py", line 86 2025-01-20T17:46:25,036 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - print(" 2025-01-20T17:46:25,037 [INFO ] epollEventLoopGroup-5-1 org.pytorch.serve.wlm.WorkerThread - 9000 Worker disconnected. WORKER_STARTED 2025-01-20T17:46:25,036 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - ^ 2025-01-20T17:46:25,037 [INFO ] epollEventLoopGroup-5-1 org.pytorch.serve.wlm.WorkerThread - 9000 Worker disconnected. WORKER_STARTED 2025-01-20T17:46:25,039 [DEBUG] W-9000-titan_1.0 org.pytorch.serve.wlm.WorkerThread - System state is : WORKER_STARTED 2025-01-20T17:46:25,039 [DEBUG] W-9000-titan_1.0 org.pytorch.serve.wlm.WorkerThread - System state is : WORKER_STARTED 2025-01-20T17:46:25,039 [DEBUG] W-9000-titan_1.0 org.pytorch.serve.wlm.WorkerThread - Backend worker monitoring thread interrupted or backend worker process died., startupTimeout:120sec java.lang.InterruptedException: null at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2056) ~[?:?] at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2133) ~[?:?] at java.util.concurrent.ArrayBlockingQueue.poll(ArrayBlockingQueue.java:432) ~[?:?] at org.pytorch.serve.wlm.WorkerThread.run(WorkerThread.java:234) [model-server.jar:?] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?] at java.lang.Thread.run(Thread.java:829) [?:?] 2025-01-20T17:46:25,039 [DEBUG] W-9000-titan_1.0 org.pytorch.serve.wlm.WorkerThread - Backend worker monitoring thread interrupted or backend worker process died., startupTimeout:120sec java.lang.InterruptedException: null at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2056) ~[?:?] at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2133) ~[?:?] at java.util.concurrent.ArrayBlockingQueue.poll(ArrayBlockingQueue.java:432) ~[?:?] at org.pytorch.serve.wlm.WorkerThread.run(WorkerThread.java:234) [model-server.jar:?] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?] at java.lang.Thread.run(Thread.java:829) [?:?] 2025-01-20T17:46:25,039 [WARN ] W-9000-titan_1.0 org.pytorch.serve.wlm.BatchAggregator - Load model failed: titan, error: Worker died. 2025-01-20T17:46:25,039 [WARN ] W-9000-titan_1.0 org.pytorch.serve.wlm.BatchAggregator - Load model failed: titan, error: Worker died. 2025-01-20T17:46:25,039 [DEBUG] W-9000-titan_1.0 org.pytorch.serve.wlm.WorkerThread - W-9000-titan_1.0 State change WORKER_STARTED -> WORKER_STOPPED 2025-01-20T17:46:25,039 [DEBUG] W-9000-titan_1.0 org.pytorch.serve.wlm.WorkerThread - W-9000-titan_1.0 State change WORKER_STARTED -> WORKER_STOPPED 2025-01-20T17:46:25,039 [WARN ] W-9000-titan_1.0 org.pytorch.serve.wlm.WorkerThread - Auto recovery failed again 2025-01-20T17:46:25,039 [WARN ] W-9000-titan_1.0 org.pytorch.serve.wlm.WorkerThread - Auto recovery failed again 2025-01-20T17:46:25,040 [INFO ] W-9000-titan_1.0 org.pytorch.serve.wlm.WorkerThread - Retry worker: 9000 in 34 seconds. 2025-01-20T17:46:25,040 [INFO ] W-9000-titan_1.0 org.pytorch.serve.wlm.WorkerThread - Retry worker: 9000 in 34 seconds. 2025-01-20T17:46:25,038 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - SyntaxError: unterminated string literal (detected at line 86) 2025-01-20T17:46:25,053 [INFO ] W-9000-titan_1.0-stdout org.pytorch.serve.wlm.WorkerLifeCycle - Stopped Scanner - W-9000-titan_1.0-stdout 2025-01-20T17:46:25,053 [INFO ] W-9000-titan_1.0-stdout org.pytorch.serve.wlm.WorkerLifeCycle - Stopped Scanner - W-9000-titan_1.0-stdout 2025-01-20T17:46:25,077 [INFO ] W-9000-titan_1.0-stderr org.pytorch.serve.wlm.WorkerLifeCycle - Stopped Scanner - W-9000-titan_1.0-stderr 2025-01-20T17:46:25,077 [INFO ] W-9000-titan_1.0-stderr org.pytorch.serve.wlm.WorkerLifeCycle - Stopped Scanner - W-9000-titan_1.0-stderr 2025-01-20T17:48:13,199 [INFO ] main org.pytorch.serve.servingsdk.impl.PluginsManager - Loading snapshot serializer plugin... 2025-01-20T17:48:13,199 [INFO ] main org.pytorch.serve.servingsdk.impl.PluginsManager - Loading snapshot serializer plugin... 2025-01-20T17:48:13,316 [DEBUG] main org.pytorch.serve.util.ConfigManager - xpu-smi not available or failed: Cannot run program "xpu-smi": error=2, No such file or directory 2025-01-20T17:48:13,316 [DEBUG] main org.pytorch.serve.util.ConfigManager - xpu-smi not available or failed: Cannot run program "xpu-smi": error=2, No such file or directory 2025-01-20T17:48:13,320 [WARN ] main org.pytorch.serve.util.ConfigManager - Your torchserve instance can access any URL to load models. When deploying to production, make sure to limit the set of allowed_urls in config.properties 2025-01-20T17:48:13,320 [WARN ] main org.pytorch.serve.util.ConfigManager - Your torchserve instance can access any URL to load models. When deploying to production, make sure to limit the set of allowed_urls in config.properties 2025-01-20T17:48:13,353 [INFO ] main org.pytorch.serve.servingsdk.impl.PluginsManager - Initializing plugins manager... 2025-01-20T17:48:13,353 [INFO ] main org.pytorch.serve.servingsdk.impl.PluginsManager - Initializing plugins manager... 2025-01-20T17:48:13,529 [INFO ] main org.pytorch.serve.metrics.configuration.MetricConfiguration - Successfully loaded metrics configuration from /usr/local/lib/python3.11/dist-packages/ts/configs/metrics.yaml 2025-01-20T17:48:13,529 [INFO ] main org.pytorch.serve.metrics.configuration.MetricConfiguration - Successfully loaded metrics configuration from /usr/local/lib/python3.11/dist-packages/ts/configs/metrics.yaml 2025-01-20T17:48:13,661 [INFO ] main org.pytorch.serve.ModelServer - Torchserve version: 0.12.0 TS Home: /usr/local/lib/python3.11/dist-packages Current directory: /content Temp directory: /tmp Metrics config path: /usr/local/lib/python3.11/dist-packages/ts/configs/metrics.yaml Number of GPUs: 1 Number of CPUs: 2 Max heap size: 3246 M Python executable: /usr/bin/python3 Config file: logs/config/20250120174654396-shutdown.cfg Inference address: http://127.0.0.1:8083 Management address: http://127.0.0.1:8084 Metrics address: http://127.0.0.1:8085 Model Store: /content/model_store Initial Models: titan=titan.mar Log dir: /content/logs Metrics dir: /content/logs Netty threads: 0 Netty client threads: 0 Default workers per model: 1 Blacklist Regex: N/A Maximum Response Size: 6553500 Maximum Request Size: 6553500 Limit Maximum Image Pixels: true Prefer direct buffer: false Allowed Urls: [file://.*|http(s)?://.*] Custom python dependency for model allowed: false Enable metrics API: true Metrics mode: LOG Disable system metrics: false Workflow Store: /content/model_store CPP log config: N/A Model config: N/A System metrics command: default Model API enabled: false 2025-01-20T17:48:13,661 [INFO ] main org.pytorch.serve.ModelServer - Torchserve version: 0.12.0 TS Home: /usr/local/lib/python3.11/dist-packages Current directory: /content Temp directory: /tmp Metrics config path: /usr/local/lib/python3.11/dist-packages/ts/configs/metrics.yaml Number of GPUs: 1 Number of CPUs: 2 Max heap size: 3246 M Python executable: /usr/bin/python3 Config file: logs/config/20250120174654396-shutdown.cfg Inference address: http://127.0.0.1:8083 Management address: http://127.0.0.1:8084 Metrics address: http://127.0.0.1:8085 Model Store: /content/model_store Initial Models: titan=titan.mar Log dir: /content/logs Metrics dir: /content/logs Netty threads: 0 Netty client threads: 0 Default workers per model: 1 Blacklist Regex: N/A Maximum Response Size: 6553500 Maximum Request Size: 6553500 Limit Maximum Image Pixels: true Prefer direct buffer: false Allowed Urls: [file://.*|http(s)?://.*] Custom python dependency for model allowed: false Enable metrics API: true Metrics mode: LOG Disable system metrics: false Workflow Store: /content/model_store CPP log config: N/A Model config: N/A System metrics command: default Model API enabled: false 2025-01-20T17:48:13,693 [INFO ] main org.pytorch.serve.snapshot.SnapshotManager - Started restoring models from snapshot { "name": "20250120174654396-shutdown.cfg", "modelCount": 1, "created": 1737395214396, "models": { "titan": { "1.0": { "defaultVersion": true, "marName": "titan.mar", "minWorkers": 1, "maxWorkers": 1, "batchSize": 1, "maxBatchDelay": 100, "responseTimeout": 120, "startupTimeout": 120, "runtimeType": "python" } } } } 2025-01-20T17:48:13,693 [INFO ] main org.pytorch.serve.snapshot.SnapshotManager - Started restoring models from snapshot { "name": "20250120174654396-shutdown.cfg", "modelCount": 1, "created": 1737395214396, "models": { "titan": { "1.0": { "defaultVersion": true, "marName": "titan.mar", "minWorkers": 1, "maxWorkers": 1, "batchSize": 1, "maxBatchDelay": 100, "responseTimeout": 120, "startupTimeout": 120, "runtimeType": "python" } } } } 2025-01-20T17:48:13,726 [INFO ] main org.pytorch.serve.snapshot.SnapshotManager - Validating snapshot 20250120174654396-shutdown.cfg 2025-01-20T17:48:13,726 [INFO ] main org.pytorch.serve.snapshot.SnapshotManager - Validating snapshot 20250120174654396-shutdown.cfg 2025-01-20T17:48:13,732 [INFO ] main org.pytorch.serve.snapshot.SnapshotManager - Snapshot 20250120174654396-shutdown.cfg validated successfully 2025-01-20T17:48:13,732 [INFO ] main org.pytorch.serve.snapshot.SnapshotManager - Snapshot 20250120174654396-shutdown.cfg validated successfully 2025-01-20T17:48:49,665 [DEBUG] main org.pytorch.serve.wlm.ModelVersionedRefs - Adding new version 1.0 for model titan 2025-01-20T17:48:49,665 [DEBUG] main org.pytorch.serve.wlm.ModelVersionedRefs - Adding new version 1.0 for model titan 2025-01-20T17:48:49,665 [DEBUG] main org.pytorch.serve.wlm.ModelVersionedRefs - Setting default version to 1.0 for model titan 2025-01-20T17:48:49,665 [DEBUG] main org.pytorch.serve.wlm.ModelVersionedRefs - Setting default version to 1.0 for model titan 2025-01-20T17:48:49,666 [DEBUG] main org.pytorch.serve.wlm.ModelVersionedRefs - Setting default version to 1.0 for model titan 2025-01-20T17:48:49,666 [DEBUG] main org.pytorch.serve.wlm.ModelVersionedRefs - Setting default version to 1.0 for model titan 2025-01-20T17:48:49,667 [INFO ] main org.pytorch.serve.wlm.ModelManager - Model titan loaded. 2025-01-20T17:48:49,667 [INFO ] main org.pytorch.serve.wlm.ModelManager - Model titan loaded. 2025-01-20T17:48:49,667 [DEBUG] main org.pytorch.serve.wlm.ModelManager - updateModel: titan, count: 1 2025-01-20T17:48:49,667 [DEBUG] main org.pytorch.serve.wlm.ModelManager - updateModel: titan, count: 1 2025-01-20T17:48:49,685 [INFO ] main org.pytorch.serve.ModelServer - Initialize Inference server with: EpollServerSocketChannel. 2025-01-20T17:48:49,685 [INFO ] main org.pytorch.serve.ModelServer - Initialize Inference server with: EpollServerSocketChannel. 2025-01-20T17:48:49,694 [DEBUG] W-9000-titan_1.0 org.pytorch.serve.wlm.WorkerLifeCycle - Worker cmdline: [/usr/bin/python3, /usr/local/lib/python3.11/dist-packages/ts/model_service_worker.py, --sock-type, unix, --sock-name, /tmp/.ts.sock.9000, --metrics-config, /usr/local/lib/python3.11/dist-packages/ts/configs/metrics.yaml] 2025-01-20T17:48:49,694 [DEBUG] W-9000-titan_1.0 org.pytorch.serve.wlm.WorkerLifeCycle - Worker cmdline: [/usr/bin/python3, /usr/local/lib/python3.11/dist-packages/ts/model_service_worker.py, --sock-type, unix, --sock-name, /tmp/.ts.sock.9000, --metrics-config, /usr/local/lib/python3.11/dist-packages/ts/configs/metrics.yaml] 2025-01-20T17:48:49,889 [INFO ] main org.pytorch.serve.ModelServer - Inference API bind to: http://127.0.0.1:8083 2025-01-20T17:48:49,889 [INFO ] main org.pytorch.serve.ModelServer - Inference API bind to: http://127.0.0.1:8083 2025-01-20T17:48:49,890 [INFO ] main org.pytorch.serve.ModelServer - Initialize Management server with: EpollServerSocketChannel. 2025-01-20T17:48:49,890 [INFO ] main org.pytorch.serve.ModelServer - Initialize Management server with: EpollServerSocketChannel. 2025-01-20T17:48:49,899 [INFO ] main org.pytorch.serve.ModelServer - Management API bind to: http://127.0.0.1:8084 2025-01-20T17:48:49,899 [INFO ] main org.pytorch.serve.ModelServer - Management API bind to: http://127.0.0.1:8084 2025-01-20T17:48:49,900 [INFO ] main org.pytorch.serve.ModelServer - Initialize Metrics server with: EpollServerSocketChannel. 2025-01-20T17:48:49,900 [INFO ] main org.pytorch.serve.ModelServer - Initialize Metrics server with: EpollServerSocketChannel. 2025-01-20T17:48:49,901 [INFO ] main org.pytorch.serve.ModelServer - Metrics API bind to: http://127.0.0.1:8085 2025-01-20T17:48:49,901 [INFO ] main org.pytorch.serve.ModelServer - Metrics API bind to: http://127.0.0.1:8085 2025-01-20T17:48:50,609 [WARN ] pool-3-thread-1 org.pytorch.serve.metrics.MetricCollector - worker pid is not available yet. 2025-01-20T17:48:50,609 [WARN ] pool-3-thread-1 org.pytorch.serve.metrics.MetricCollector - worker pid is not available yet. 2025-01-20T17:48:50,665 [ERROR] Thread-1 org.pytorch.serve.metrics.MetricCollector - Traceback (most recent call last): File "/usr/local/lib/python3.11/dist-packages/ts/metrics/metric_collector.py", line 11, in from ts.metrics import system_metrics File "/usr/local/lib/python3.11/dist-packages/ts/__init__.py", line 10, in from . import version File "/usr/local/lib/python3.11/dist-packages/ts/version.py", line 5, in from pathlib import Path File "/usr/local/lib/python3.11/dist-packages/pathlib.py", line 10, in from collections import Sequence ImportError: cannot import name 'Sequence' from 'collections' (/usr/lib/python3.11/collections/__init__.py) 2025-01-20T17:48:50,665 [ERROR] Thread-1 org.pytorch.serve.metrics.MetricCollector - Traceback (most recent call last): File "/usr/local/lib/python3.11/dist-packages/ts/metrics/metric_collector.py", line 11, in from ts.metrics import system_metrics File "/usr/local/lib/python3.11/dist-packages/ts/__init__.py", line 10, in from . import version File "/usr/local/lib/python3.11/dist-packages/ts/version.py", line 5, in from pathlib import Path File "/usr/local/lib/python3.11/dist-packages/pathlib.py", line 10, in from collections import Sequence ImportError: cannot import name 'Sequence' from 'collections' (/usr/lib/python3.11/collections/__init__.py) 2025-01-20T17:48:52,467 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - s_name_part0=/tmp/.ts.sock, s_name_part1=9000, pid=6998 2025-01-20T17:48:52,470 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - Listening on port: /tmp/.ts.sock.9000 2025-01-20T17:48:52,482 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - Successfully loaded /usr/local/lib/python3.11/dist-packages/ts/configs/metrics.yaml. 2025-01-20T17:48:52,483 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - [PID]6998 2025-01-20T17:48:52,492 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - Torch worker started. 2025-01-20T17:48:52,492 [DEBUG] W-9000-titan_1.0 org.pytorch.serve.wlm.WorkerThread - W-9000-titan_1.0 State change null -> WORKER_STARTED 2025-01-20T17:48:52,492 [DEBUG] W-9000-titan_1.0 org.pytorch.serve.wlm.WorkerThread - W-9000-titan_1.0 State change null -> WORKER_STARTED 2025-01-20T17:48:52,494 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - Python runtime: 3.11.11 2025-01-20T17:48:52,502 [INFO ] W-9000-titan_1.0 org.pytorch.serve.wlm.WorkerThread - Connecting to: /tmp/.ts.sock.9000 2025-01-20T17:48:52,502 [INFO ] W-9000-titan_1.0 org.pytorch.serve.wlm.WorkerThread - Connecting to: /tmp/.ts.sock.9000 2025-01-20T17:48:52,530 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - Connection accepted: /tmp/.ts.sock.9000. 2025-01-20T17:48:52,537 [DEBUG] W-9000-titan_1.0 org.pytorch.serve.wlm.WorkerThread - Flushing req.cmd LOAD repeats 1 to backend at: 1737395332537 2025-01-20T17:48:52,537 [DEBUG] W-9000-titan_1.0 org.pytorch.serve.wlm.WorkerThread - Flushing req.cmd LOAD repeats 1 to backend at: 1737395332537 2025-01-20T17:48:52,547 [INFO ] W-9000-titan_1.0 org.pytorch.serve.wlm.WorkerThread - Looping backend response at: 1737395332547 2025-01-20T17:48:52,547 [INFO ] W-9000-titan_1.0 org.pytorch.serve.wlm.WorkerThread - Looping backend response at: 1737395332547 2025-01-20T17:48:52,599 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - model_name: titan, batchSize: 1 2025-01-20T17:48:52,602 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - Backend worker process died. 2025-01-20T17:48:52,603 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - Traceback (most recent call last): 2025-01-20T17:48:52,604 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - File "/usr/local/lib/python3.11/dist-packages/ts/model_service_worker.py", line 301, in 2025-01-20T17:48:52,605 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - worker.run_server() 2025-01-20T17:48:52,606 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - File "/usr/local/lib/python3.11/dist-packages/ts/model_service_worker.py", line 268, in run_server 2025-01-20T17:48:52,607 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - self.handle_connection(cl_socket) 2025-01-20T17:48:52,608 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - File "/usr/local/lib/python3.11/dist-packages/ts/model_service_worker.py", line 196, in handle_connection 2025-01-20T17:48:52,608 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - service, result, code = self.load_model(msg) 2025-01-20T17:48:52,610 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - ^^^^^^^^^^^^^^^^^^^^ 2025-01-20T17:48:52,611 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - File "/usr/local/lib/python3.11/dist-packages/ts/model_service_worker.py", line 133, in load_model 2025-01-20T17:48:52,617 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - service = model_loader.load( 2025-01-20T17:48:52,618 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - ^^^^^^^^^^^^^^^^^^ 2025-01-20T17:48:52,618 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - File "/usr/local/lib/python3.11/dist-packages/ts/model_loader.py", line 108, in load 2025-01-20T17:48:52,619 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - module, function_name = self._load_handler_file(handler) 2025-01-20T17:48:52,620 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-01-20T17:48:52,621 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - File "/usr/local/lib/python3.11/dist-packages/ts/model_loader.py", line 153, in _load_handler_file 2025-01-20T17:48:52,622 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - module = importlib.import_module(module_name) 2025-01-20T17:48:52,623 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-01-20T17:48:52,624 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - File "/usr/lib/python3.11/importlib/__init__.py", line 126, in import_module 2025-01-20T17:48:52,625 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - return _bootstrap._gcd_import(name[level:], package, level) 2025-01-20T17:48:52,628 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-01-20T17:48:52,634 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - File "", line 1204, in _gcd_import 2025-01-20T17:48:52,635 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - File "", line 1176, in _find_and_load 2025-01-20T17:48:52,636 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - File "", line 1147, in _find_and_load_unlocked 2025-01-20T17:48:52,636 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - File "", line 690, in _load_unlocked 2025-01-20T17:48:52,638 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - File "", line 936, in exec_module 2025-01-20T17:48:52,639 [INFO ] epollEventLoopGroup-5-1 org.pytorch.serve.wlm.WorkerThread - 9000 Worker disconnected. WORKER_STARTED 2025-01-20T17:48:52,638 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - File "", line 1074, in get_code 2025-01-20T17:48:52,639 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - File "", line 1004, in source_to_code 2025-01-20T17:48:52,640 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - File "", line 241, in _call_with_frames_removed 2025-01-20T17:48:52,639 [INFO ] epollEventLoopGroup-5-1 org.pytorch.serve.wlm.WorkerThread - 9000 Worker disconnected. WORKER_STARTED 2025-01-20T17:48:52,647 [DEBUG] W-9000-titan_1.0 org.pytorch.serve.wlm.WorkerThread - System state is : WORKER_STARTED 2025-01-20T17:48:52,647 [DEBUG] W-9000-titan_1.0 org.pytorch.serve.wlm.WorkerThread - System state is : WORKER_STARTED 2025-01-20T17:48:52,646 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - File "/tmp/models/e1d5abe5fc2d4242a8e27bb4f69fedad/titan_handler.py", line 86 2025-01-20T17:48:52,650 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - print(" 2025-01-20T17:48:52,652 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - ^ 2025-01-20T17:48:52,653 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - SyntaxError: unterminated string literal (detected at line 86) 2025-01-20T17:48:52,648 [DEBUG] W-9000-titan_1.0 org.pytorch.serve.wlm.WorkerThread - Backend worker monitoring thread interrupted or backend worker process died., startupTimeout:120sec java.lang.InterruptedException: null at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2056) ~[?:?] at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2133) ~[?:?] at java.util.concurrent.ArrayBlockingQueue.poll(ArrayBlockingQueue.java:432) ~[?:?] at org.pytorch.serve.wlm.WorkerThread.run(WorkerThread.java:234) [model-server.jar:?] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) [?:?] at java.util.concurrent.FutureTask.run(FutureTask.java:264) [?:?] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?] at java.lang.Thread.run(Thread.java:829) [?:?] 2025-01-20T17:48:52,648 [DEBUG] W-9000-titan_1.0 org.pytorch.serve.wlm.WorkerThread - Backend worker monitoring thread interrupted or backend worker process died., startupTimeout:120sec java.lang.InterruptedException: null at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2056) ~[?:?] at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2133) ~[?:?] at java.util.concurrent.ArrayBlockingQueue.poll(ArrayBlockingQueue.java:432) ~[?:?] at org.pytorch.serve.wlm.WorkerThread.run(WorkerThread.java:234) [model-server.jar:?] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) [?:?] at java.util.concurrent.FutureTask.run(FutureTask.java:264) [?:?] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?] at java.lang.Thread.run(Thread.java:829) [?:?] 2025-01-20T17:48:52,695 [WARN ] W-9000-titan_1.0 org.pytorch.serve.wlm.BatchAggregator - Load model failed: titan, error: Worker died. 2025-01-20T17:48:52,695 [WARN ] W-9000-titan_1.0 org.pytorch.serve.wlm.BatchAggregator - Load model failed: titan, error: Worker died. 2025-01-20T17:48:52,699 [DEBUG] W-9000-titan_1.0 org.pytorch.serve.wlm.WorkerThread - W-9000-titan_1.0 State change WORKER_STARTED -> WORKER_STOPPED 2025-01-20T17:48:52,699 [DEBUG] W-9000-titan_1.0 org.pytorch.serve.wlm.WorkerThread - W-9000-titan_1.0 State change WORKER_STARTED -> WORKER_STOPPED 2025-01-20T17:48:52,699 [INFO ] W-9000-titan_1.0 org.pytorch.serve.wlm.WorkerThread - Auto recovery start timestamp: 1737395332699 2025-01-20T17:48:52,699 [INFO ] W-9000-titan_1.0 org.pytorch.serve.wlm.WorkerThread - Auto recovery start timestamp: 1737395332699 2025-01-20T17:48:52,701 [INFO ] W-9000-titan_1.0 org.pytorch.serve.wlm.WorkerThread - Retry worker: 9000 in 1 seconds. 2025-01-20T17:48:52,701 [INFO ] W-9000-titan_1.0 org.pytorch.serve.wlm.WorkerThread - Retry worker: 9000 in 1 seconds. 2025-01-20T17:48:52,737 [INFO ] W-9000-titan_1.0-stdout org.pytorch.serve.wlm.WorkerLifeCycle - Stopped Scanner - W-9000-titan_1.0-stdout 2025-01-20T17:48:52,737 [INFO ] W-9000-titan_1.0-stdout org.pytorch.serve.wlm.WorkerLifeCycle - Stopped Scanner - W-9000-titan_1.0-stdout 2025-01-20T17:48:52,738 [INFO ] W-9000-titan_1.0-stderr org.pytorch.serve.wlm.WorkerLifeCycle - Stopped Scanner - W-9000-titan_1.0-stderr 2025-01-20T17:48:52,738 [INFO ] W-9000-titan_1.0-stderr org.pytorch.serve.wlm.WorkerLifeCycle - Stopped Scanner - W-9000-titan_1.0-stderr 2025-01-20T17:48:53,702 [DEBUG] W-9000-titan_1.0 org.pytorch.serve.wlm.WorkerLifeCycle - Worker cmdline: [/usr/bin/python3, /usr/local/lib/python3.11/dist-packages/ts/model_service_worker.py, --sock-type, unix, --sock-name, /tmp/.ts.sock.9000, --metrics-config, /usr/local/lib/python3.11/dist-packages/ts/configs/metrics.yaml] 2025-01-20T17:48:53,702 [DEBUG] W-9000-titan_1.0 org.pytorch.serve.wlm.WorkerLifeCycle - Worker cmdline: [/usr/bin/python3, /usr/local/lib/python3.11/dist-packages/ts/model_service_worker.py, --sock-type, unix, --sock-name, /tmp/.ts.sock.9000, --metrics-config, /usr/local/lib/python3.11/dist-packages/ts/configs/metrics.yaml] 2025-01-20T17:48:55,334 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - s_name_part0=/tmp/.ts.sock, s_name_part1=9000, pid=7028 2025-01-20T17:48:55,336 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - Listening on port: /tmp/.ts.sock.9000 2025-01-20T17:48:55,343 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - Successfully loaded /usr/local/lib/python3.11/dist-packages/ts/configs/metrics.yaml. 2025-01-20T17:48:55,343 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - [PID]7028 2025-01-20T17:48:55,344 [DEBUG] W-9000-titan_1.0 org.pytorch.serve.wlm.WorkerThread - W-9000-titan_1.0 State change WORKER_STOPPED -> WORKER_STARTED 2025-01-20T17:48:55,344 [DEBUG] W-9000-titan_1.0 org.pytorch.serve.wlm.WorkerThread - W-9000-titan_1.0 State change WORKER_STOPPED -> WORKER_STARTED 2025-01-20T17:48:55,344 [INFO ] W-9000-titan_1.0 org.pytorch.serve.wlm.WorkerThread - Connecting to: /tmp/.ts.sock.9000 2025-01-20T17:48:55,344 [INFO ] W-9000-titan_1.0 org.pytorch.serve.wlm.WorkerThread - Connecting to: /tmp/.ts.sock.9000 2025-01-20T17:48:55,345 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - Torch worker started. 2025-01-20T17:48:55,345 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - Python runtime: 3.11.11 2025-01-20T17:48:55,349 [DEBUG] W-9000-titan_1.0 org.pytorch.serve.wlm.WorkerThread - Flushing req.cmd LOAD repeats 1 to backend at: 1737395335349 2025-01-20T17:48:55,349 [DEBUG] W-9000-titan_1.0 org.pytorch.serve.wlm.WorkerThread - Flushing req.cmd LOAD repeats 1 to backend at: 1737395335349 2025-01-20T17:48:55,350 [INFO ] W-9000-titan_1.0 org.pytorch.serve.wlm.WorkerThread - Looping backend response at: 1737395335350 2025-01-20T17:48:55,350 [INFO ] W-9000-titan_1.0 org.pytorch.serve.wlm.WorkerThread - Looping backend response at: 1737395335350 2025-01-20T17:48:55,350 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - Connection accepted: /tmp/.ts.sock.9000. 2025-01-20T17:48:55,379 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - model_name: titan, batchSize: 1 2025-01-20T17:48:55,381 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - Backend worker process died. 2025-01-20T17:48:55,383 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - Traceback (most recent call last): 2025-01-20T17:48:55,388 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - File "/usr/local/lib/python3.11/dist-packages/ts/model_service_worker.py", line 301, in 2025-01-20T17:48:55,389 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - worker.run_server() 2025-01-20T17:48:55,390 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - File "/usr/local/lib/python3.11/dist-packages/ts/model_service_worker.py", line 268, in run_server 2025-01-20T17:48:55,402 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - self.handle_connection(cl_socket) 2025-01-20T17:48:55,405 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - File "/usr/local/lib/python3.11/dist-packages/ts/model_service_worker.py", line 196, in handle_connection 2025-01-20T17:48:55,406 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - service, result, code = self.load_model(msg) 2025-01-20T17:48:55,401 [INFO ] epollEventLoopGroup-5-2 org.pytorch.serve.wlm.WorkerThread - 9000 Worker disconnected. WORKER_STARTED 2025-01-20T17:48:55,409 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - ^^^^^^^^^^^^^^^^^^^^ 2025-01-20T17:48:55,412 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - File "/usr/local/lib/python3.11/dist-packages/ts/model_service_worker.py", line 133, in load_model 2025-01-20T17:48:55,418 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - service = model_loader.load( 2025-01-20T17:48:55,418 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - ^^^^^^^^^^^^^^^^^^ 2025-01-20T17:48:55,418 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - File "/usr/local/lib/python3.11/dist-packages/ts/model_loader.py", line 108, in load 2025-01-20T17:48:55,418 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - module, function_name = self._load_handler_file(handler) 2025-01-20T17:48:55,401 [INFO ] epollEventLoopGroup-5-2 org.pytorch.serve.wlm.WorkerThread - 9000 Worker disconnected. WORKER_STARTED 2025-01-20T17:48:55,418 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-01-20T17:48:55,421 [DEBUG] W-9000-titan_1.0 org.pytorch.serve.wlm.WorkerThread - System state is : WORKER_STARTED 2025-01-20T17:48:55,421 [DEBUG] W-9000-titan_1.0 org.pytorch.serve.wlm.WorkerThread - System state is : WORKER_STARTED 2025-01-20T17:48:55,421 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - File "/usr/local/lib/python3.11/dist-packages/ts/model_loader.py", line 153, in _load_handler_file 2025-01-20T17:48:55,421 [DEBUG] W-9000-titan_1.0 org.pytorch.serve.wlm.WorkerThread - Backend worker monitoring thread interrupted or backend worker process died., startupTimeout:120sec java.lang.InterruptedException: null at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2056) ~[?:?] at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2133) ~[?:?] at java.util.concurrent.ArrayBlockingQueue.poll(ArrayBlockingQueue.java:432) ~[?:?] at org.pytorch.serve.wlm.WorkerThread.run(WorkerThread.java:234) [model-server.jar:?] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?] at java.lang.Thread.run(Thread.java:829) [?:?] 2025-01-20T17:48:55,421 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - module = importlib.import_module(module_name) 2025-01-20T17:48:55,421 [DEBUG] W-9000-titan_1.0 org.pytorch.serve.wlm.WorkerThread - Backend worker monitoring thread interrupted or backend worker process died., startupTimeout:120sec java.lang.InterruptedException: null at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2056) ~[?:?] at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2133) ~[?:?] at java.util.concurrent.ArrayBlockingQueue.poll(ArrayBlockingQueue.java:432) ~[?:?] at org.pytorch.serve.wlm.WorkerThread.run(WorkerThread.java:234) [model-server.jar:?] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?] at java.lang.Thread.run(Thread.java:829) [?:?] 2025-01-20T17:48:55,422 [WARN ] W-9000-titan_1.0 org.pytorch.serve.wlm.BatchAggregator - Load model failed: titan, error: Worker died. 2025-01-20T17:48:55,422 [WARN ] W-9000-titan_1.0 org.pytorch.serve.wlm.BatchAggregator - Load model failed: titan, error: Worker died. 2025-01-20T17:48:55,422 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-01-20T17:48:55,422 [DEBUG] W-9000-titan_1.0 org.pytorch.serve.wlm.WorkerThread - W-9000-titan_1.0 State change WORKER_STARTED -> WORKER_STOPPED 2025-01-20T17:48:55,422 [DEBUG] W-9000-titan_1.0 org.pytorch.serve.wlm.WorkerThread - W-9000-titan_1.0 State change WORKER_STARTED -> WORKER_STOPPED 2025-01-20T17:48:55,422 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - File "/usr/lib/python3.11/importlib/__init__.py", line 126, in import_module 2025-01-20T17:48:55,422 [WARN ] W-9000-titan_1.0 org.pytorch.serve.wlm.WorkerThread - Auto recovery failed again 2025-01-20T17:48:55,422 [WARN ] W-9000-titan_1.0 org.pytorch.serve.wlm.WorkerThread - Auto recovery failed again 2025-01-20T17:48:55,422 [INFO ] W-9000-titan_1.0 org.pytorch.serve.wlm.WorkerThread - Retry worker: 9000 in 1 seconds. 2025-01-20T17:48:55,422 [INFO ] W-9000-titan_1.0 org.pytorch.serve.wlm.WorkerThread - Retry worker: 9000 in 1 seconds. 2025-01-20T17:48:55,423 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - return _bootstrap._gcd_import(name[level:], package, level) 2025-01-20T17:48:55,423 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-01-20T17:48:55,423 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - File "", line 1204, in _gcd_import 2025-01-20T17:48:55,423 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - File "", line 1176, in _find_and_load 2025-01-20T17:48:55,423 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - File "", line 1147, in _find_and_load_unlocked 2025-01-20T17:48:55,424 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - File "", line 690, in _load_unlocked 2025-01-20T17:48:55,424 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - File "", line 936, in exec_module 2025-01-20T17:48:55,424 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - File "", line 1074, in get_code 2025-01-20T17:48:55,425 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - File "", line 1004, in source_to_code 2025-01-20T17:48:55,428 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - File "", line 241, in _call_with_frames_removed 2025-01-20T17:48:55,430 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - File "/tmp/models/e1d5abe5fc2d4242a8e27bb4f69fedad/titan_handler.py", line 86 2025-01-20T17:48:55,430 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - print(" 2025-01-20T17:48:55,431 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - ^ 2025-01-20T17:48:55,431 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - SyntaxError: unterminated string literal (detected at line 86) 2025-01-20T17:48:55,431 [INFO ] W-9000-titan_1.0-stdout org.pytorch.serve.wlm.WorkerLifeCycle - Stopped Scanner - W-9000-titan_1.0-stdout 2025-01-20T17:48:55,431 [INFO ] W-9000-titan_1.0-stdout org.pytorch.serve.wlm.WorkerLifeCycle - Stopped Scanner - W-9000-titan_1.0-stdout 2025-01-20T17:48:55,474 [INFO ] W-9000-titan_1.0-stderr org.pytorch.serve.wlm.WorkerLifeCycle - Stopped Scanner - W-9000-titan_1.0-stderr 2025-01-20T17:48:55,474 [INFO ] W-9000-titan_1.0-stderr org.pytorch.serve.wlm.WorkerLifeCycle - Stopped Scanner - W-9000-titan_1.0-stderr 2025-01-20T17:48:56,423 [DEBUG] W-9000-titan_1.0 org.pytorch.serve.wlm.WorkerLifeCycle - Worker cmdline: [/usr/bin/python3, /usr/local/lib/python3.11/dist-packages/ts/model_service_worker.py, --sock-type, unix, --sock-name, /tmp/.ts.sock.9000, --metrics-config, /usr/local/lib/python3.11/dist-packages/ts/configs/metrics.yaml] 2025-01-20T17:48:56,423 [DEBUG] W-9000-titan_1.0 org.pytorch.serve.wlm.WorkerLifeCycle - Worker cmdline: [/usr/bin/python3, /usr/local/lib/python3.11/dist-packages/ts/model_service_worker.py, --sock-type, unix, --sock-name, /tmp/.ts.sock.9000, --metrics-config, /usr/local/lib/python3.11/dist-packages/ts/configs/metrics.yaml] 2025-01-20T17:48:58,009 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - s_name_part0=/tmp/.ts.sock, s_name_part1=9000, pid=7044 2025-01-20T17:48:58,013 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - Listening on port: /tmp/.ts.sock.9000 2025-01-20T17:48:58,022 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - Successfully loaded /usr/local/lib/python3.11/dist-packages/ts/configs/metrics.yaml. 2025-01-20T17:48:58,023 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - [PID]7044 2025-01-20T17:48:58,024 [DEBUG] W-9000-titan_1.0 org.pytorch.serve.wlm.WorkerThread - W-9000-titan_1.0 State change WORKER_STOPPED -> WORKER_STARTED 2025-01-20T17:48:58,024 [DEBUG] W-9000-titan_1.0 org.pytorch.serve.wlm.WorkerThread - W-9000-titan_1.0 State change WORKER_STOPPED -> WORKER_STARTED 2025-01-20T17:48:58,024 [INFO ] W-9000-titan_1.0 org.pytorch.serve.wlm.WorkerThread - Connecting to: /tmp/.ts.sock.9000 2025-01-20T17:48:58,024 [INFO ] W-9000-titan_1.0 org.pytorch.serve.wlm.WorkerThread - Connecting to: /tmp/.ts.sock.9000 2025-01-20T17:48:58,025 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - Torch worker started. 2025-01-20T17:48:58,026 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - Python runtime: 3.11.11 2025-01-20T17:48:58,028 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - Connection accepted: /tmp/.ts.sock.9000. 2025-01-20T17:48:58,029 [DEBUG] W-9000-titan_1.0 org.pytorch.serve.wlm.WorkerThread - Flushing req.cmd LOAD repeats 1 to backend at: 1737395338029 2025-01-20T17:48:58,029 [DEBUG] W-9000-titan_1.0 org.pytorch.serve.wlm.WorkerThread - Flushing req.cmd LOAD repeats 1 to backend at: 1737395338029 2025-01-20T17:48:58,030 [INFO ] W-9000-titan_1.0 org.pytorch.serve.wlm.WorkerThread - Looping backend response at: 1737395338030 2025-01-20T17:48:58,030 [INFO ] W-9000-titan_1.0 org.pytorch.serve.wlm.WorkerThread - Looping backend response at: 1737395338030 2025-01-20T17:48:58,050 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - model_name: titan, batchSize: 1 2025-01-20T17:48:58,055 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - Backend worker process died. 2025-01-20T17:48:58,055 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - Traceback (most recent call last): 2025-01-20T17:48:58,056 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - File "/usr/local/lib/python3.11/dist-packages/ts/model_service_worker.py", line 301, in 2025-01-20T17:48:58,056 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - worker.run_server() 2025-01-20T17:48:58,057 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - File "/usr/local/lib/python3.11/dist-packages/ts/model_service_worker.py", line 268, in run_server 2025-01-20T17:48:58,058 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - self.handle_connection(cl_socket) 2025-01-20T17:48:58,058 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - File "/usr/local/lib/python3.11/dist-packages/ts/model_service_worker.py", line 196, in handle_connection 2025-01-20T17:48:58,059 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - service, result, code = self.load_model(msg) 2025-01-20T17:48:58,059 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - ^^^^^^^^^^^^^^^^^^^^ 2025-01-20T17:48:58,062 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - File "/usr/local/lib/python3.11/dist-packages/ts/model_service_worker.py", line 133, in load_model 2025-01-20T17:48:58,063 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - service = model_loader.load( 2025-01-20T17:48:58,064 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - ^^^^^^^^^^^^^^^^^^ 2025-01-20T17:48:58,065 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - File "/usr/local/lib/python3.11/dist-packages/ts/model_loader.py", line 108, in load 2025-01-20T17:48:58,065 [INFO ] epollEventLoopGroup-5-3 org.pytorch.serve.wlm.WorkerThread - 9000 Worker disconnected. WORKER_STARTED 2025-01-20T17:48:58,065 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - module, function_name = self._load_handler_file(handler) 2025-01-20T17:48:58,066 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-01-20T17:48:58,067 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - File "/usr/local/lib/python3.11/dist-packages/ts/model_loader.py", line 153, in _load_handler_file 2025-01-20T17:48:58,065 [INFO ] epollEventLoopGroup-5-3 org.pytorch.serve.wlm.WorkerThread - 9000 Worker disconnected. WORKER_STARTED 2025-01-20T17:48:58,068 [DEBUG] W-9000-titan_1.0 org.pytorch.serve.wlm.WorkerThread - System state is : WORKER_STARTED 2025-01-20T17:48:58,068 [DEBUG] W-9000-titan_1.0 org.pytorch.serve.wlm.WorkerThread - System state is : WORKER_STARTED 2025-01-20T17:48:58,068 [DEBUG] W-9000-titan_1.0 org.pytorch.serve.wlm.WorkerThread - Backend worker monitoring thread interrupted or backend worker process died., startupTimeout:120sec java.lang.InterruptedException: null at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2056) ~[?:?] at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2133) ~[?:?] at java.util.concurrent.ArrayBlockingQueue.poll(ArrayBlockingQueue.java:432) ~[?:?] at org.pytorch.serve.wlm.WorkerThread.run(WorkerThread.java:234) [model-server.jar:?] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?] at java.lang.Thread.run(Thread.java:829) [?:?] 2025-01-20T17:48:58,068 [DEBUG] W-9000-titan_1.0 org.pytorch.serve.wlm.WorkerThread - Backend worker monitoring thread interrupted or backend worker process died., startupTimeout:120sec java.lang.InterruptedException: null at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2056) ~[?:?] at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2133) ~[?:?] at java.util.concurrent.ArrayBlockingQueue.poll(ArrayBlockingQueue.java:432) ~[?:?] at org.pytorch.serve.wlm.WorkerThread.run(WorkerThread.java:234) [model-server.jar:?] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?] at java.lang.Thread.run(Thread.java:829) [?:?] 2025-01-20T17:48:58,068 [WARN ] W-9000-titan_1.0 org.pytorch.serve.wlm.BatchAggregator - Load model failed: titan, error: Worker died. 2025-01-20T17:48:58,068 [WARN ] W-9000-titan_1.0 org.pytorch.serve.wlm.BatchAggregator - Load model failed: titan, error: Worker died. 2025-01-20T17:48:58,069 [DEBUG] W-9000-titan_1.0 org.pytorch.serve.wlm.WorkerThread - W-9000-titan_1.0 State change WORKER_STARTED -> WORKER_STOPPED 2025-01-20T17:48:58,069 [DEBUG] W-9000-titan_1.0 org.pytorch.serve.wlm.WorkerThread - W-9000-titan_1.0 State change WORKER_STARTED -> WORKER_STOPPED 2025-01-20T17:48:58,069 [WARN ] W-9000-titan_1.0 org.pytorch.serve.wlm.WorkerThread - Auto recovery failed again 2025-01-20T17:48:58,069 [WARN ] W-9000-titan_1.0 org.pytorch.serve.wlm.WorkerThread - Auto recovery failed again 2025-01-20T17:48:58,069 [INFO ] W-9000-titan_1.0 org.pytorch.serve.wlm.WorkerThread - Retry worker: 9000 in 2 seconds. 2025-01-20T17:48:58,069 [INFO ] W-9000-titan_1.0 org.pytorch.serve.wlm.WorkerThread - Retry worker: 9000 in 2 seconds. 2025-01-20T17:48:58,074 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - module = importlib.import_module(module_name) 2025-01-20T17:48:58,074 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-01-20T17:48:58,076 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - File "/usr/lib/python3.11/importlib/__init__.py", line 126, in import_module 2025-01-20T17:48:58,077 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - return _bootstrap._gcd_import(name[level:], package, level) 2025-01-20T17:48:58,080 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-01-20T17:48:58,080 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - File "", line 1204, in _gcd_import 2025-01-20T17:48:58,081 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - File "", line 1176, in _find_and_load 2025-01-20T17:48:58,082 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - File "", line 1147, in _find_and_load_unlocked 2025-01-20T17:48:58,082 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - File "", line 690, in _load_unlocked 2025-01-20T17:48:58,082 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - File "", line 936, in exec_module 2025-01-20T17:48:58,083 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - File "", line 1074, in get_code 2025-01-20T17:48:58,083 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - File "", line 1004, in source_to_code 2025-01-20T17:48:58,083 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - File "", line 241, in _call_with_frames_removed 2025-01-20T17:48:58,084 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - File "/tmp/models/e1d5abe5fc2d4242a8e27bb4f69fedad/titan_handler.py", line 86 2025-01-20T17:48:58,084 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - print(" 2025-01-20T17:48:58,091 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - ^ 2025-01-20T17:48:58,091 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - SyntaxError: unterminated string literal (detected at line 86) 2025-01-20T17:48:58,092 [INFO ] W-9000-titan_1.0-stdout org.pytorch.serve.wlm.WorkerLifeCycle - Stopped Scanner - W-9000-titan_1.0-stdout 2025-01-20T17:48:58,092 [INFO ] W-9000-titan_1.0-stdout org.pytorch.serve.wlm.WorkerLifeCycle - Stopped Scanner - W-9000-titan_1.0-stdout 2025-01-20T17:48:58,115 [INFO ] W-9000-titan_1.0-stderr org.pytorch.serve.wlm.WorkerLifeCycle - Stopped Scanner - W-9000-titan_1.0-stderr 2025-01-20T17:48:58,115 [INFO ] W-9000-titan_1.0-stderr org.pytorch.serve.wlm.WorkerLifeCycle - Stopped Scanner - W-9000-titan_1.0-stderr 2025-01-20T17:49:00,070 [DEBUG] W-9000-titan_1.0 org.pytorch.serve.wlm.WorkerLifeCycle - Worker cmdline: [/usr/bin/python3, /usr/local/lib/python3.11/dist-packages/ts/model_service_worker.py, --sock-type, unix, --sock-name, /tmp/.ts.sock.9000, --metrics-config, /usr/local/lib/python3.11/dist-packages/ts/configs/metrics.yaml] 2025-01-20T17:49:00,070 [DEBUG] W-9000-titan_1.0 org.pytorch.serve.wlm.WorkerLifeCycle - Worker cmdline: [/usr/bin/python3, /usr/local/lib/python3.11/dist-packages/ts/model_service_worker.py, --sock-type, unix, --sock-name, /tmp/.ts.sock.9000, --metrics-config, /usr/local/lib/python3.11/dist-packages/ts/configs/metrics.yaml] 2025-01-20T17:49:01,657 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - s_name_part0=/tmp/.ts.sock, s_name_part1=9000, pid=7066 2025-01-20T17:49:01,658 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - Listening on port: /tmp/.ts.sock.9000 2025-01-20T17:49:01,666 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - Successfully loaded /usr/local/lib/python3.11/dist-packages/ts/configs/metrics.yaml. 2025-01-20T17:49:01,666 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - [PID]7066 2025-01-20T17:49:01,667 [DEBUG] W-9000-titan_1.0 org.pytorch.serve.wlm.WorkerThread - W-9000-titan_1.0 State change WORKER_STOPPED -> WORKER_STARTED 2025-01-20T17:49:01,667 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - Torch worker started. 2025-01-20T17:49:01,667 [DEBUG] W-9000-titan_1.0 org.pytorch.serve.wlm.WorkerThread - W-9000-titan_1.0 State change WORKER_STOPPED -> WORKER_STARTED 2025-01-20T17:49:01,668 [INFO ] W-9000-titan_1.0 org.pytorch.serve.wlm.WorkerThread - Connecting to: /tmp/.ts.sock.9000 2025-01-20T17:49:01,668 [INFO ] W-9000-titan_1.0 org.pytorch.serve.wlm.WorkerThread - Connecting to: /tmp/.ts.sock.9000 2025-01-20T17:49:01,668 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - Python runtime: 3.11.11 2025-01-20T17:49:01,672 [DEBUG] W-9000-titan_1.0 org.pytorch.serve.wlm.WorkerThread - Flushing req.cmd LOAD repeats 1 to backend at: 1737395341672 2025-01-20T17:49:01,672 [DEBUG] W-9000-titan_1.0 org.pytorch.serve.wlm.WorkerThread - Flushing req.cmd LOAD repeats 1 to backend at: 1737395341672 2025-01-20T17:49:01,672 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - Connection accepted: /tmp/.ts.sock.9000. 2025-01-20T17:49:01,673 [INFO ] W-9000-titan_1.0 org.pytorch.serve.wlm.WorkerThread - Looping backend response at: 1737395341673 2025-01-20T17:49:01,673 [INFO ] W-9000-titan_1.0 org.pytorch.serve.wlm.WorkerThread - Looping backend response at: 1737395341673 2025-01-20T17:49:01,693 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - model_name: titan, batchSize: 1 2025-01-20T17:49:01,695 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - Backend worker process died. 2025-01-20T17:49:01,695 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - Traceback (most recent call last): 2025-01-20T17:49:01,696 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - File "/usr/local/lib/python3.11/dist-packages/ts/model_service_worker.py", line 301, in 2025-01-20T17:49:01,696 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - worker.run_server() 2025-01-20T17:49:01,696 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - File "/usr/local/lib/python3.11/dist-packages/ts/model_service_worker.py", line 268, in run_server 2025-01-20T17:49:01,696 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - self.handle_connection(cl_socket) 2025-01-20T17:49:01,697 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - File "/usr/local/lib/python3.11/dist-packages/ts/model_service_worker.py", line 196, in handle_connection 2025-01-20T17:49:01,697 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - service, result, code = self.load_model(msg) 2025-01-20T17:49:01,697 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - ^^^^^^^^^^^^^^^^^^^^ 2025-01-20T17:49:01,697 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - File "/usr/local/lib/python3.11/dist-packages/ts/model_service_worker.py", line 133, in load_model 2025-01-20T17:49:01,697 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - service = model_loader.load( 2025-01-20T17:49:01,697 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - ^^^^^^^^^^^^^^^^^^ 2025-01-20T17:49:01,697 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - File "/usr/local/lib/python3.11/dist-packages/ts/model_loader.py", line 108, in load 2025-01-20T17:49:01,697 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - module, function_name = self._load_handler_file(handler) 2025-01-20T17:49:01,699 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-01-20T17:49:01,699 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - File "/usr/local/lib/python3.11/dist-packages/ts/model_loader.py", line 153, in _load_handler_file 2025-01-20T17:49:01,699 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - module = importlib.import_module(module_name) 2025-01-20T17:49:01,700 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-01-20T17:49:01,700 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - File "/usr/lib/python3.11/importlib/__init__.py", line 126, in import_module 2025-01-20T17:49:01,700 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - return _bootstrap._gcd_import(name[level:], package, level) 2025-01-20T17:49:01,700 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-01-20T17:49:01,700 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - File "", line 1204, in _gcd_import 2025-01-20T17:49:01,702 [INFO ] epollEventLoopGroup-5-4 org.pytorch.serve.wlm.WorkerThread - 9000 Worker disconnected. WORKER_STARTED 2025-01-20T17:49:01,702 [INFO ] epollEventLoopGroup-5-4 org.pytorch.serve.wlm.WorkerThread - 9000 Worker disconnected. WORKER_STARTED 2025-01-20T17:49:01,702 [DEBUG] W-9000-titan_1.0 org.pytorch.serve.wlm.WorkerThread - System state is : WORKER_STARTED 2025-01-20T17:49:01,702 [DEBUG] W-9000-titan_1.0 org.pytorch.serve.wlm.WorkerThread - System state is : WORKER_STARTED 2025-01-20T17:49:01,702 [DEBUG] W-9000-titan_1.0 org.pytorch.serve.wlm.WorkerThread - Backend worker monitoring thread interrupted or backend worker process died., startupTimeout:120sec java.lang.InterruptedException: null at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2056) ~[?:?] at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2133) ~[?:?] at java.util.concurrent.ArrayBlockingQueue.poll(ArrayBlockingQueue.java:432) ~[?:?] at org.pytorch.serve.wlm.WorkerThread.run(WorkerThread.java:234) [model-server.jar:?] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?] at java.lang.Thread.run(Thread.java:829) [?:?] 2025-01-20T17:49:01,702 [DEBUG] W-9000-titan_1.0 org.pytorch.serve.wlm.WorkerThread - Backend worker monitoring thread interrupted or backend worker process died., startupTimeout:120sec java.lang.InterruptedException: null at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2056) ~[?:?] at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2133) ~[?:?] at java.util.concurrent.ArrayBlockingQueue.poll(ArrayBlockingQueue.java:432) ~[?:?] at org.pytorch.serve.wlm.WorkerThread.run(WorkerThread.java:234) [model-server.jar:?] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?] at java.lang.Thread.run(Thread.java:829) [?:?] 2025-01-20T17:49:01,703 [WARN ] W-9000-titan_1.0 org.pytorch.serve.wlm.BatchAggregator - Load model failed: titan, error: Worker died. 2025-01-20T17:49:01,703 [WARN ] W-9000-titan_1.0 org.pytorch.serve.wlm.BatchAggregator - Load model failed: titan, error: Worker died. 2025-01-20T17:49:01,703 [DEBUG] W-9000-titan_1.0 org.pytorch.serve.wlm.WorkerThread - W-9000-titan_1.0 State change WORKER_STARTED -> WORKER_STOPPED 2025-01-20T17:49:01,703 [DEBUG] W-9000-titan_1.0 org.pytorch.serve.wlm.WorkerThread - W-9000-titan_1.0 State change WORKER_STARTED -> WORKER_STOPPED 2025-01-20T17:49:01,703 [WARN ] W-9000-titan_1.0 org.pytorch.serve.wlm.WorkerThread - Auto recovery failed again 2025-01-20T17:49:01,703 [WARN ] W-9000-titan_1.0 org.pytorch.serve.wlm.WorkerThread - Auto recovery failed again 2025-01-20T17:49:01,703 [INFO ] W-9000-titan_1.0 org.pytorch.serve.wlm.WorkerThread - Retry worker: 9000 in 3 seconds. 2025-01-20T17:49:01,703 [INFO ] W-9000-titan_1.0 org.pytorch.serve.wlm.WorkerThread - Retry worker: 9000 in 3 seconds. 2025-01-20T17:49:01,701 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - File "", line 1176, in _find_and_load 2025-01-20T17:49:01,707 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - File "", line 1147, in _find_and_load_unlocked 2025-01-20T17:49:01,707 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - File "", line 690, in _load_unlocked 2025-01-20T17:49:01,708 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - File "", line 936, in exec_module 2025-01-20T17:49:01,708 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - File "", line 1074, in get_code 2025-01-20T17:49:01,709 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - File "", line 1004, in source_to_code 2025-01-20T17:49:01,710 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - File "", line 241, in _call_with_frames_removed 2025-01-20T17:49:01,710 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - File "/tmp/models/e1d5abe5fc2d4242a8e27bb4f69fedad/titan_handler.py", line 86 2025-01-20T17:49:01,710 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - print(" 2025-01-20T17:49:01,710 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - ^ 2025-01-20T17:49:01,711 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - SyntaxError: unterminated string literal (detected at line 86) 2025-01-20T17:49:01,712 [INFO ] W-9000-titan_1.0-stdout org.pytorch.serve.wlm.WorkerLifeCycle - Stopped Scanner - W-9000-titan_1.0-stdout 2025-01-20T17:49:01,712 [INFO ] W-9000-titan_1.0-stdout org.pytorch.serve.wlm.WorkerLifeCycle - Stopped Scanner - W-9000-titan_1.0-stdout 2025-01-20T17:49:01,732 [INFO ] W-9000-titan_1.0-stderr org.pytorch.serve.wlm.WorkerLifeCycle - Stopped Scanner - W-9000-titan_1.0-stderr 2025-01-20T17:49:01,732 [INFO ] W-9000-titan_1.0-stderr org.pytorch.serve.wlm.WorkerLifeCycle - Stopped Scanner - W-9000-titan_1.0-stderr 2025-01-20T17:49:04,704 [DEBUG] W-9000-titan_1.0 org.pytorch.serve.wlm.WorkerLifeCycle - Worker cmdline: [/usr/bin/python3, /usr/local/lib/python3.11/dist-packages/ts/model_service_worker.py, --sock-type, unix, --sock-name, /tmp/.ts.sock.9000, --metrics-config, /usr/local/lib/python3.11/dist-packages/ts/configs/metrics.yaml] 2025-01-20T17:49:04,704 [DEBUG] W-9000-titan_1.0 org.pytorch.serve.wlm.WorkerLifeCycle - Worker cmdline: [/usr/bin/python3, /usr/local/lib/python3.11/dist-packages/ts/model_service_worker.py, --sock-type, unix, --sock-name, /tmp/.ts.sock.9000, --metrics-config, /usr/local/lib/python3.11/dist-packages/ts/configs/metrics.yaml] 2025-01-20T17:49:07,218 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - s_name_part0=/tmp/.ts.sock, s_name_part1=9000, pid=7092 2025-01-20T17:49:07,221 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - Listening on port: /tmp/.ts.sock.9000 2025-01-20T17:49:07,234 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - Successfully loaded /usr/local/lib/python3.11/dist-packages/ts/configs/metrics.yaml. 2025-01-20T17:49:07,235 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - [PID]7092 2025-01-20T17:49:07,236 [DEBUG] W-9000-titan_1.0 org.pytorch.serve.wlm.WorkerThread - W-9000-titan_1.0 State change WORKER_STOPPED -> WORKER_STARTED 2025-01-20T17:49:07,236 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - Torch worker started. 2025-01-20T17:49:07,236 [DEBUG] W-9000-titan_1.0 org.pytorch.serve.wlm.WorkerThread - W-9000-titan_1.0 State change WORKER_STOPPED -> WORKER_STARTED 2025-01-20T17:49:07,237 [INFO ] W-9000-titan_1.0 org.pytorch.serve.wlm.WorkerThread - Connecting to: /tmp/.ts.sock.9000 2025-01-20T17:49:07,237 [INFO ] W-9000-titan_1.0 org.pytorch.serve.wlm.WorkerThread - Connecting to: /tmp/.ts.sock.9000 2025-01-20T17:49:07,238 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - Python runtime: 3.11.11 2025-01-20T17:49:07,240 [DEBUG] W-9000-titan_1.0 org.pytorch.serve.wlm.WorkerThread - Flushing req.cmd LOAD repeats 1 to backend at: 1737395347240 2025-01-20T17:49:07,240 [DEBUG] W-9000-titan_1.0 org.pytorch.serve.wlm.WorkerThread - Flushing req.cmd LOAD repeats 1 to backend at: 1737395347240 2025-01-20T17:49:07,245 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - Connection accepted: /tmp/.ts.sock.9000. 2025-01-20T17:49:07,246 [INFO ] W-9000-titan_1.0 org.pytorch.serve.wlm.WorkerThread - Looping backend response at: 1737395347246 2025-01-20T17:49:07,246 [INFO ] W-9000-titan_1.0 org.pytorch.serve.wlm.WorkerThread - Looping backend response at: 1737395347246 2025-01-20T17:49:07,248 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - model_name: titan, batchSize: 1 2025-01-20T17:49:07,252 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - Backend worker process died. 2025-01-20T17:49:07,253 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - Traceback (most recent call last): 2025-01-20T17:49:07,253 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - File "/usr/local/lib/python3.11/dist-packages/ts/model_service_worker.py", line 301, in 2025-01-20T17:49:07,255 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - worker.run_server() 2025-01-20T17:49:07,255 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - File "/usr/local/lib/python3.11/dist-packages/ts/model_service_worker.py", line 268, in run_server 2025-01-20T17:49:07,256 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - self.handle_connection(cl_socket) 2025-01-20T17:49:07,257 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - File "/usr/local/lib/python3.11/dist-packages/ts/model_service_worker.py", line 196, in handle_connection 2025-01-20T17:49:07,260 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - service, result, code = self.load_model(msg) 2025-01-20T17:49:07,261 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - ^^^^^^^^^^^^^^^^^^^^ 2025-01-20T17:49:07,262 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - File "/usr/local/lib/python3.11/dist-packages/ts/model_service_worker.py", line 133, in load_model 2025-01-20T17:49:07,262 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - service = model_loader.load( 2025-01-20T17:49:07,263 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - ^^^^^^^^^^^^^^^^^^ 2025-01-20T17:49:07,264 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - File "/usr/local/lib/python3.11/dist-packages/ts/model_loader.py", line 108, in load 2025-01-20T17:49:07,265 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - module, function_name = self._load_handler_file(handler) 2025-01-20T17:49:07,265 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-01-20T17:49:07,273 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - File "/usr/local/lib/python3.11/dist-packages/ts/model_loader.py", line 153, in _load_handler_file 2025-01-20T17:49:07,278 [INFO ] epollEventLoopGroup-5-1 org.pytorch.serve.wlm.WorkerThread - 9000 Worker disconnected. WORKER_STARTED 2025-01-20T17:49:07,278 [INFO ] epollEventLoopGroup-5-1 org.pytorch.serve.wlm.WorkerThread - 9000 Worker disconnected. WORKER_STARTED 2025-01-20T17:49:07,278 [DEBUG] W-9000-titan_1.0 org.pytorch.serve.wlm.WorkerThread - System state is : WORKER_STARTED 2025-01-20T17:49:07,277 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - module = importlib.import_module(module_name) 2025-01-20T17:49:07,278 [DEBUG] W-9000-titan_1.0 org.pytorch.serve.wlm.WorkerThread - System state is : WORKER_STARTED 2025-01-20T17:49:07,278 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-01-20T17:49:07,279 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - File "/usr/lib/python3.11/importlib/__init__.py", line 126, in import_module 2025-01-20T17:49:07,278 [DEBUG] W-9000-titan_1.0 org.pytorch.serve.wlm.WorkerThread - Backend worker monitoring thread interrupted or backend worker process died., startupTimeout:120sec java.lang.InterruptedException: null at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2056) ~[?:?] at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2133) ~[?:?] at java.util.concurrent.ArrayBlockingQueue.poll(ArrayBlockingQueue.java:432) ~[?:?] at org.pytorch.serve.wlm.WorkerThread.run(WorkerThread.java:234) [model-server.jar:?] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?] at java.lang.Thread.run(Thread.java:829) [?:?] 2025-01-20T17:49:07,278 [DEBUG] W-9000-titan_1.0 org.pytorch.serve.wlm.WorkerThread - Backend worker monitoring thread interrupted or backend worker process died., startupTimeout:120sec java.lang.InterruptedException: null at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2056) ~[?:?] at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2133) ~[?:?] at java.util.concurrent.ArrayBlockingQueue.poll(ArrayBlockingQueue.java:432) ~[?:?] at org.pytorch.serve.wlm.WorkerThread.run(WorkerThread.java:234) [model-server.jar:?] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?] at java.lang.Thread.run(Thread.java:829) [?:?] 2025-01-20T17:49:07,279 [WARN ] W-9000-titan_1.0 org.pytorch.serve.wlm.BatchAggregator - Load model failed: titan, error: Worker died. 2025-01-20T17:49:07,279 [WARN ] W-9000-titan_1.0 org.pytorch.serve.wlm.BatchAggregator - Load model failed: titan, error: Worker died. 2025-01-20T17:49:07,279 [DEBUG] W-9000-titan_1.0 org.pytorch.serve.wlm.WorkerThread - W-9000-titan_1.0 State change WORKER_STARTED -> WORKER_STOPPED 2025-01-20T17:49:07,279 [DEBUG] W-9000-titan_1.0 org.pytorch.serve.wlm.WorkerThread - W-9000-titan_1.0 State change WORKER_STARTED -> WORKER_STOPPED 2025-01-20T17:49:07,279 [WARN ] W-9000-titan_1.0 org.pytorch.serve.wlm.WorkerThread - Auto recovery failed again 2025-01-20T17:49:07,279 [WARN ] W-9000-titan_1.0 org.pytorch.serve.wlm.WorkerThread - Auto recovery failed again 2025-01-20T17:49:07,280 [INFO ] W-9000-titan_1.0 org.pytorch.serve.wlm.WorkerThread - Retry worker: 9000 in 5 seconds. 2025-01-20T17:49:07,280 [INFO ] W-9000-titan_1.0 org.pytorch.serve.wlm.WorkerThread - Retry worker: 9000 in 5 seconds. 2025-01-20T17:49:07,282 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - return _bootstrap._gcd_import(name[level:], package, level) 2025-01-20T17:49:07,282 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-01-20T17:49:07,283 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - File "", line 1204, in _gcd_import 2025-01-20T17:49:07,283 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - File "", line 1176, in _find_and_load 2025-01-20T17:49:07,284 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - File "", line 1147, in _find_and_load_unlocked 2025-01-20T17:49:07,285 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - File "", line 690, in _load_unlocked 2025-01-20T17:49:07,286 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - File "", line 936, in exec_module 2025-01-20T17:49:07,288 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - File "", line 1074, in get_code 2025-01-20T17:49:07,288 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - File "", line 1004, in source_to_code 2025-01-20T17:49:07,289 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - File "", line 241, in _call_with_frames_removed 2025-01-20T17:49:07,289 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - File "/tmp/models/e1d5abe5fc2d4242a8e27bb4f69fedad/titan_handler.py", line 86 2025-01-20T17:49:07,290 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - print(" 2025-01-20T17:49:07,291 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - ^ 2025-01-20T17:49:07,293 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - SyntaxError: unterminated string literal (detected at line 86) 2025-01-20T17:49:07,294 [INFO ] W-9000-titan_1.0-stdout org.pytorch.serve.wlm.WorkerLifeCycle - Stopped Scanner - W-9000-titan_1.0-stdout 2025-01-20T17:49:07,294 [INFO ] W-9000-titan_1.0-stdout org.pytorch.serve.wlm.WorkerLifeCycle - Stopped Scanner - W-9000-titan_1.0-stdout 2025-01-20T17:49:07,328 [INFO ] W-9000-titan_1.0-stderr org.pytorch.serve.wlm.WorkerLifeCycle - Stopped Scanner - W-9000-titan_1.0-stderr 2025-01-20T17:49:07,328 [INFO ] W-9000-titan_1.0-stderr org.pytorch.serve.wlm.WorkerLifeCycle - Stopped Scanner - W-9000-titan_1.0-stderr 2025-01-20T17:49:12,280 [DEBUG] W-9000-titan_1.0 org.pytorch.serve.wlm.WorkerLifeCycle - Worker cmdline: [/usr/bin/python3, /usr/local/lib/python3.11/dist-packages/ts/model_service_worker.py, --sock-type, unix, --sock-name, /tmp/.ts.sock.9000, --metrics-config, /usr/local/lib/python3.11/dist-packages/ts/configs/metrics.yaml] 2025-01-20T17:49:12,280 [DEBUG] W-9000-titan_1.0 org.pytorch.serve.wlm.WorkerLifeCycle - Worker cmdline: [/usr/bin/python3, /usr/local/lib/python3.11/dist-packages/ts/model_service_worker.py, --sock-type, unix, --sock-name, /tmp/.ts.sock.9000, --metrics-config, /usr/local/lib/python3.11/dist-packages/ts/configs/metrics.yaml] 2025-01-20T17:49:13,860 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - s_name_part0=/tmp/.ts.sock, s_name_part1=9000, pid=7129 2025-01-20T17:49:13,862 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - Listening on port: /tmp/.ts.sock.9000 2025-01-20T17:49:13,873 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - Successfully loaded /usr/local/lib/python3.11/dist-packages/ts/configs/metrics.yaml. 2025-01-20T17:49:13,873 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - [PID]7129 2025-01-20T17:49:13,874 [DEBUG] W-9000-titan_1.0 org.pytorch.serve.wlm.WorkerThread - W-9000-titan_1.0 State change WORKER_STOPPED -> WORKER_STARTED 2025-01-20T17:49:13,874 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - Torch worker started. 2025-01-20T17:49:13,874 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - Python runtime: 3.11.11 2025-01-20T17:49:13,874 [DEBUG] W-9000-titan_1.0 org.pytorch.serve.wlm.WorkerThread - W-9000-titan_1.0 State change WORKER_STOPPED -> WORKER_STARTED 2025-01-20T17:49:13,875 [INFO ] W-9000-titan_1.0 org.pytorch.serve.wlm.WorkerThread - Connecting to: /tmp/.ts.sock.9000 2025-01-20T17:49:13,875 [INFO ] W-9000-titan_1.0 org.pytorch.serve.wlm.WorkerThread - Connecting to: /tmp/.ts.sock.9000 2025-01-20T17:49:13,876 [DEBUG] W-9000-titan_1.0 org.pytorch.serve.wlm.WorkerThread - Flushing req.cmd LOAD repeats 1 to backend at: 1737395353876 2025-01-20T17:49:13,876 [DEBUG] W-9000-titan_1.0 org.pytorch.serve.wlm.WorkerThread - Flushing req.cmd LOAD repeats 1 to backend at: 1737395353876 2025-01-20T17:49:13,877 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - Connection accepted: /tmp/.ts.sock.9000. 2025-01-20T17:49:13,878 [INFO ] W-9000-titan_1.0 org.pytorch.serve.wlm.WorkerThread - Looping backend response at: 1737395353877 2025-01-20T17:49:13,878 [INFO ] W-9000-titan_1.0 org.pytorch.serve.wlm.WorkerThread - Looping backend response at: 1737395353877 2025-01-20T17:49:13,879 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - model_name: titan, batchSize: 1 2025-01-20T17:49:13,882 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - Backend worker process died. 2025-01-20T17:49:13,882 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - Traceback (most recent call last): 2025-01-20T17:49:13,882 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - File "/usr/local/lib/python3.11/dist-packages/ts/model_service_worker.py", line 301, in 2025-01-20T17:49:13,882 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - worker.run_server() 2025-01-20T17:49:13,882 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - File "/usr/local/lib/python3.11/dist-packages/ts/model_service_worker.py", line 268, in run_server 2025-01-20T17:49:13,882 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - self.handle_connection(cl_socket) 2025-01-20T17:49:13,882 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - File "/usr/local/lib/python3.11/dist-packages/ts/model_service_worker.py", line 196, in handle_connection 2025-01-20T17:49:13,882 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - service, result, code = self.load_model(msg) 2025-01-20T17:49:13,882 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - ^^^^^^^^^^^^^^^^^^^^ 2025-01-20T17:49:13,883 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - File "/usr/local/lib/python3.11/dist-packages/ts/model_service_worker.py", line 133, in load_model 2025-01-20T17:49:13,883 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - service = model_loader.load( 2025-01-20T17:49:13,883 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - ^^^^^^^^^^^^^^^^^^ 2025-01-20T17:49:13,883 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - File "/usr/local/lib/python3.11/dist-packages/ts/model_loader.py", line 108, in load 2025-01-20T17:49:13,883 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - module, function_name = self._load_handler_file(handler) 2025-01-20T17:49:13,883 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-01-20T17:49:13,883 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - File "/usr/local/lib/python3.11/dist-packages/ts/model_loader.py", line 153, in _load_handler_file 2025-01-20T17:49:13,883 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - module = importlib.import_module(module_name) 2025-01-20T17:49:13,883 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-01-20T17:49:13,883 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - File "/usr/lib/python3.11/importlib/__init__.py", line 126, in import_module 2025-01-20T17:49:13,883 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - return _bootstrap._gcd_import(name[level:], package, level) 2025-01-20T17:49:13,884 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-01-20T17:49:13,884 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - File "", line 1204, in _gcd_import 2025-01-20T17:49:13,884 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - File "", line 1176, in _find_and_load 2025-01-20T17:49:13,884 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - File "", line 1147, in _find_and_load_unlocked 2025-01-20T17:49:13,884 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - File "", line 690, in _load_unlocked 2025-01-20T17:49:13,884 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - File "", line 936, in exec_module 2025-01-20T17:49:13,884 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - File "", line 1074, in get_code 2025-01-20T17:49:13,884 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - File "", line 1004, in source_to_code 2025-01-20T17:49:13,885 [INFO ] epollEventLoopGroup-5-2 org.pytorch.serve.wlm.WorkerThread - 9000 Worker disconnected. WORKER_STARTED 2025-01-20T17:49:13,885 [INFO ] epollEventLoopGroup-5-2 org.pytorch.serve.wlm.WorkerThread - 9000 Worker disconnected. WORKER_STARTED 2025-01-20T17:49:13,885 [DEBUG] W-9000-titan_1.0 org.pytorch.serve.wlm.WorkerThread - System state is : WORKER_STARTED 2025-01-20T17:49:13,885 [DEBUG] W-9000-titan_1.0 org.pytorch.serve.wlm.WorkerThread - System state is : WORKER_STARTED 2025-01-20T17:49:13,885 [DEBUG] W-9000-titan_1.0 org.pytorch.serve.wlm.WorkerThread - Backend worker monitoring thread interrupted or backend worker process died., startupTimeout:120sec java.lang.InterruptedException: null at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2056) ~[?:?] at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2133) ~[?:?] at java.util.concurrent.ArrayBlockingQueue.poll(ArrayBlockingQueue.java:432) ~[?:?] at org.pytorch.serve.wlm.WorkerThread.run(WorkerThread.java:234) [model-server.jar:?] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?] at java.lang.Thread.run(Thread.java:829) [?:?] 2025-01-20T17:49:13,885 [DEBUG] W-9000-titan_1.0 org.pytorch.serve.wlm.WorkerThread - Backend worker monitoring thread interrupted or backend worker process died., startupTimeout:120sec java.lang.InterruptedException: null at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2056) ~[?:?] at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2133) ~[?:?] at java.util.concurrent.ArrayBlockingQueue.poll(ArrayBlockingQueue.java:432) ~[?:?] at org.pytorch.serve.wlm.WorkerThread.run(WorkerThread.java:234) [model-server.jar:?] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?] at java.lang.Thread.run(Thread.java:829) [?:?] 2025-01-20T17:49:13,886 [WARN ] W-9000-titan_1.0 org.pytorch.serve.wlm.BatchAggregator - Load model failed: titan, error: Worker died. 2025-01-20T17:49:13,886 [WARN ] W-9000-titan_1.0 org.pytorch.serve.wlm.BatchAggregator - Load model failed: titan, error: Worker died. 2025-01-20T17:49:13,886 [DEBUG] W-9000-titan_1.0 org.pytorch.serve.wlm.WorkerThread - W-9000-titan_1.0 State change WORKER_STARTED -> WORKER_STOPPED 2025-01-20T17:49:13,886 [DEBUG] W-9000-titan_1.0 org.pytorch.serve.wlm.WorkerThread - W-9000-titan_1.0 State change WORKER_STARTED -> WORKER_STOPPED 2025-01-20T17:49:13,886 [WARN ] W-9000-titan_1.0 org.pytorch.serve.wlm.WorkerThread - Auto recovery failed again 2025-01-20T17:49:13,886 [WARN ] W-9000-titan_1.0 org.pytorch.serve.wlm.WorkerThread - Auto recovery failed again 2025-01-20T17:49:13,884 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - File "", line 241, in _call_with_frames_removed 2025-01-20T17:49:13,886 [INFO ] W-9000-titan_1.0 org.pytorch.serve.wlm.WorkerThread - Retry worker: 9000 in 8 seconds. 2025-01-20T17:49:13,887 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - File "/tmp/models/e1d5abe5fc2d4242a8e27bb4f69fedad/titan_handler.py", line 86 2025-01-20T17:49:13,890 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - print(" 2025-01-20T17:49:13,886 [INFO ] W-9000-titan_1.0 org.pytorch.serve.wlm.WorkerThread - Retry worker: 9000 in 8 seconds. 2025-01-20T17:49:13,890 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - ^ 2025-01-20T17:49:13,890 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - SyntaxError: unterminated string literal (detected at line 86) 2025-01-20T17:49:13,890 [INFO ] W-9000-titan_1.0-stdout org.pytorch.serve.wlm.WorkerLifeCycle - Stopped Scanner - W-9000-titan_1.0-stdout 2025-01-20T17:49:13,890 [INFO ] W-9000-titan_1.0-stdout org.pytorch.serve.wlm.WorkerLifeCycle - Stopped Scanner - W-9000-titan_1.0-stdout 2025-01-20T17:49:13,929 [INFO ] W-9000-titan_1.0-stderr org.pytorch.serve.wlm.WorkerLifeCycle - Stopped Scanner - W-9000-titan_1.0-stderr 2025-01-20T17:49:13,929 [INFO ] W-9000-titan_1.0-stderr org.pytorch.serve.wlm.WorkerLifeCycle - Stopped Scanner - W-9000-titan_1.0-stderr 2025-01-20T17:49:21,887 [DEBUG] W-9000-titan_1.0 org.pytorch.serve.wlm.WorkerLifeCycle - Worker cmdline: [/usr/bin/python3, /usr/local/lib/python3.11/dist-packages/ts/model_service_worker.py, --sock-type, unix, --sock-name, /tmp/.ts.sock.9000, --metrics-config, /usr/local/lib/python3.11/dist-packages/ts/configs/metrics.yaml] 2025-01-20T17:49:21,887 [DEBUG] W-9000-titan_1.0 org.pytorch.serve.wlm.WorkerLifeCycle - Worker cmdline: [/usr/bin/python3, /usr/local/lib/python3.11/dist-packages/ts/model_service_worker.py, --sock-type, unix, --sock-name, /tmp/.ts.sock.9000, --metrics-config, /usr/local/lib/python3.11/dist-packages/ts/configs/metrics.yaml] 2025-01-20T17:49:23,456 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - s_name_part0=/tmp/.ts.sock, s_name_part1=9000, pid=7176 2025-01-20T17:49:23,457 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - Listening on port: /tmp/.ts.sock.9000 2025-01-20T17:49:23,464 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - Successfully loaded /usr/local/lib/python3.11/dist-packages/ts/configs/metrics.yaml. 2025-01-20T17:49:23,465 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - [PID]7176 2025-01-20T17:49:23,465 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - Torch worker started. 2025-01-20T17:49:23,465 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - Python runtime: 3.11.11 2025-01-20T17:49:23,465 [DEBUG] W-9000-titan_1.0 org.pytorch.serve.wlm.WorkerThread - W-9000-titan_1.0 State change WORKER_STOPPED -> WORKER_STARTED 2025-01-20T17:49:23,465 [DEBUG] W-9000-titan_1.0 org.pytorch.serve.wlm.WorkerThread - W-9000-titan_1.0 State change WORKER_STOPPED -> WORKER_STARTED 2025-01-20T17:49:23,466 [INFO ] W-9000-titan_1.0 org.pytorch.serve.wlm.WorkerThread - Connecting to: /tmp/.ts.sock.9000 2025-01-20T17:49:23,466 [INFO ] W-9000-titan_1.0 org.pytorch.serve.wlm.WorkerThread - Connecting to: /tmp/.ts.sock.9000 2025-01-20T17:49:23,467 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - Connection accepted: /tmp/.ts.sock.9000. 2025-01-20T17:49:23,468 [DEBUG] W-9000-titan_1.0 org.pytorch.serve.wlm.WorkerThread - Flushing req.cmd LOAD repeats 1 to backend at: 1737395363468 2025-01-20T17:49:23,468 [DEBUG] W-9000-titan_1.0 org.pytorch.serve.wlm.WorkerThread - Flushing req.cmd LOAD repeats 1 to backend at: 1737395363468 2025-01-20T17:49:23,469 [INFO ] W-9000-titan_1.0 org.pytorch.serve.wlm.WorkerThread - Looping backend response at: 1737395363469 2025-01-20T17:49:23,469 [INFO ] W-9000-titan_1.0 org.pytorch.serve.wlm.WorkerThread - Looping backend response at: 1737395363469 2025-01-20T17:49:23,469 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - model_name: titan, batchSize: 1 2025-01-20T17:49:23,472 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - Backend worker process died. 2025-01-20T17:49:23,472 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - Traceback (most recent call last): 2025-01-20T17:49:23,472 [INFO ] epollEventLoopGroup-5-3 org.pytorch.serve.wlm.WorkerThread - 9000 Worker disconnected. WORKER_STARTED 2025-01-20T17:49:23,472 [INFO ] epollEventLoopGroup-5-3 org.pytorch.serve.wlm.WorkerThread - 9000 Worker disconnected. WORKER_STARTED 2025-01-20T17:49:23,472 [DEBUG] W-9000-titan_1.0 org.pytorch.serve.wlm.WorkerThread - System state is : WORKER_STARTED 2025-01-20T17:49:23,472 [DEBUG] W-9000-titan_1.0 org.pytorch.serve.wlm.WorkerThread - System state is : WORKER_STARTED 2025-01-20T17:49:23,472 [DEBUG] W-9000-titan_1.0 org.pytorch.serve.wlm.WorkerThread - Backend worker monitoring thread interrupted or backend worker process died., startupTimeout:120sec java.lang.InterruptedException: null at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2056) ~[?:?] at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2133) ~[?:?] at java.util.concurrent.ArrayBlockingQueue.poll(ArrayBlockingQueue.java:432) ~[?:?] at org.pytorch.serve.wlm.WorkerThread.run(WorkerThread.java:234) [model-server.jar:?] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?] at java.lang.Thread.run(Thread.java:829) [?:?] 2025-01-20T17:49:23,472 [DEBUG] W-9000-titan_1.0 org.pytorch.serve.wlm.WorkerThread - Backend worker monitoring thread interrupted or backend worker process died., startupTimeout:120sec java.lang.InterruptedException: null at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2056) ~[?:?] at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2133) ~[?:?] at java.util.concurrent.ArrayBlockingQueue.poll(ArrayBlockingQueue.java:432) ~[?:?] at org.pytorch.serve.wlm.WorkerThread.run(WorkerThread.java:234) [model-server.jar:?] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?] at java.lang.Thread.run(Thread.java:829) [?:?] 2025-01-20T17:49:23,473 [WARN ] W-9000-titan_1.0 org.pytorch.serve.wlm.BatchAggregator - Load model failed: titan, error: Worker died. 2025-01-20T17:49:23,473 [WARN ] W-9000-titan_1.0 org.pytorch.serve.wlm.BatchAggregator - Load model failed: titan, error: Worker died. 2025-01-20T17:49:23,473 [DEBUG] W-9000-titan_1.0 org.pytorch.serve.wlm.WorkerThread - W-9000-titan_1.0 State change WORKER_STARTED -> WORKER_STOPPED 2025-01-20T17:49:23,473 [DEBUG] W-9000-titan_1.0 org.pytorch.serve.wlm.WorkerThread - W-9000-titan_1.0 State change WORKER_STARTED -> WORKER_STOPPED 2025-01-20T17:49:23,474 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - File "/usr/local/lib/python3.11/dist-packages/ts/model_service_worker.py", line 301, in 2025-01-20T17:49:23,474 [WARN ] W-9000-titan_1.0 org.pytorch.serve.wlm.WorkerThread - Auto recovery failed again 2025-01-20T17:49:23,474 [WARN ] W-9000-titan_1.0 org.pytorch.serve.wlm.WorkerThread - Auto recovery failed again 2025-01-20T17:49:23,474 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - worker.run_server() 2025-01-20T17:49:23,474 [INFO ] W-9000-titan_1.0 org.pytorch.serve.wlm.WorkerThread - Retry worker: 9000 in 13 seconds. 2025-01-20T17:49:23,474 [INFO ] W-9000-titan_1.0 org.pytorch.serve.wlm.WorkerThread - Retry worker: 9000 in 13 seconds. 2025-01-20T17:49:23,474 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - File "/usr/local/lib/python3.11/dist-packages/ts/model_service_worker.py", line 268, in run_server 2025-01-20T17:49:23,475 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - self.handle_connection(cl_socket) 2025-01-20T17:49:23,475 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - File "/usr/local/lib/python3.11/dist-packages/ts/model_service_worker.py", line 196, in handle_connection 2025-01-20T17:49:23,475 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - service, result, code = self.load_model(msg) 2025-01-20T17:49:23,475 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - ^^^^^^^^^^^^^^^^^^^^ 2025-01-20T17:49:23,475 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - File "/usr/local/lib/python3.11/dist-packages/ts/model_service_worker.py", line 133, in load_model 2025-01-20T17:49:23,475 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - service = model_loader.load( 2025-01-20T17:49:23,475 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - ^^^^^^^^^^^^^^^^^^ 2025-01-20T17:49:23,476 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - File "/usr/local/lib/python3.11/dist-packages/ts/model_loader.py", line 108, in load 2025-01-20T17:49:23,476 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - module, function_name = self._load_handler_file(handler) 2025-01-20T17:49:23,476 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-01-20T17:49:23,476 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - File "/usr/local/lib/python3.11/dist-packages/ts/model_loader.py", line 153, in _load_handler_file 2025-01-20T17:49:23,476 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - module = importlib.import_module(module_name) 2025-01-20T17:49:23,476 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-01-20T17:49:23,476 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - File "/usr/lib/python3.11/importlib/__init__.py", line 126, in import_module 2025-01-20T17:49:23,476 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - return _bootstrap._gcd_import(name[level:], package, level) 2025-01-20T17:49:23,476 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-01-20T17:49:23,476 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - File "", line 1204, in _gcd_import 2025-01-20T17:49:23,477 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - File "", line 1176, in _find_and_load 2025-01-20T17:49:23,477 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - File "", line 1147, in _find_and_load_unlocked 2025-01-20T17:49:23,477 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - File "", line 690, in _load_unlocked 2025-01-20T17:49:23,477 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - File "", line 936, in exec_module 2025-01-20T17:49:23,477 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - File "", line 1074, in get_code 2025-01-20T17:49:23,477 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - File "", line 1004, in source_to_code 2025-01-20T17:49:23,477 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - File "", line 241, in _call_with_frames_removed 2025-01-20T17:49:23,477 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - File "/tmp/models/e1d5abe5fc2d4242a8e27bb4f69fedad/titan_handler.py", line 86 2025-01-20T17:49:23,477 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - print(" 2025-01-20T17:49:23,477 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - ^ 2025-01-20T17:49:23,477 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - SyntaxError: unterminated string literal (detected at line 86) 2025-01-20T17:49:23,481 [INFO ] W-9000-titan_1.0-stdout org.pytorch.serve.wlm.WorkerLifeCycle - Stopped Scanner - W-9000-titan_1.0-stdout 2025-01-20T17:49:23,481 [INFO ] W-9000-titan_1.0-stdout org.pytorch.serve.wlm.WorkerLifeCycle - Stopped Scanner - W-9000-titan_1.0-stdout 2025-01-20T17:49:23,515 [INFO ] W-9000-titan_1.0-stderr org.pytorch.serve.wlm.WorkerLifeCycle - Stopped Scanner - W-9000-titan_1.0-stderr 2025-01-20T17:49:23,515 [INFO ] W-9000-titan_1.0-stderr org.pytorch.serve.wlm.WorkerLifeCycle - Stopped Scanner - W-9000-titan_1.0-stderr 2025-01-20T17:49:36,475 [DEBUG] W-9000-titan_1.0 org.pytorch.serve.wlm.WorkerLifeCycle - Worker cmdline: [/usr/bin/python3, /usr/local/lib/python3.11/dist-packages/ts/model_service_worker.py, --sock-type, unix, --sock-name, /tmp/.ts.sock.9000, --metrics-config, /usr/local/lib/python3.11/dist-packages/ts/configs/metrics.yaml] 2025-01-20T17:49:36,475 [DEBUG] W-9000-titan_1.0 org.pytorch.serve.wlm.WorkerLifeCycle - Worker cmdline: [/usr/bin/python3, /usr/local/lib/python3.11/dist-packages/ts/model_service_worker.py, --sock-type, unix, --sock-name, /tmp/.ts.sock.9000, --metrics-config, /usr/local/lib/python3.11/dist-packages/ts/configs/metrics.yaml] 2025-01-20T17:49:38,047 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - s_name_part0=/tmp/.ts.sock, s_name_part1=9000, pid=7239 2025-01-20T17:49:38,048 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - Listening on port: /tmp/.ts.sock.9000 2025-01-20T17:49:38,055 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - Successfully loaded /usr/local/lib/python3.11/dist-packages/ts/configs/metrics.yaml. 2025-01-20T17:49:38,056 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - [PID]7239 2025-01-20T17:49:38,056 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - Torch worker started. 2025-01-20T17:49:38,056 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - Python runtime: 3.11.11 2025-01-20T17:49:38,057 [DEBUG] W-9000-titan_1.0 org.pytorch.serve.wlm.WorkerThread - W-9000-titan_1.0 State change WORKER_STOPPED -> WORKER_STARTED 2025-01-20T17:49:38,057 [DEBUG] W-9000-titan_1.0 org.pytorch.serve.wlm.WorkerThread - W-9000-titan_1.0 State change WORKER_STOPPED -> WORKER_STARTED 2025-01-20T17:49:38,057 [INFO ] W-9000-titan_1.0 org.pytorch.serve.wlm.WorkerThread - Connecting to: /tmp/.ts.sock.9000 2025-01-20T17:49:38,057 [INFO ] W-9000-titan_1.0 org.pytorch.serve.wlm.WorkerThread - Connecting to: /tmp/.ts.sock.9000 2025-01-20T17:49:38,059 [DEBUG] W-9000-titan_1.0 org.pytorch.serve.wlm.WorkerThread - Flushing req.cmd LOAD repeats 1 to backend at: 1737395378059 2025-01-20T17:49:38,059 [DEBUG] W-9000-titan_1.0 org.pytorch.serve.wlm.WorkerThread - Flushing req.cmd LOAD repeats 1 to backend at: 1737395378059 2025-01-20T17:49:38,060 [INFO ] W-9000-titan_1.0 org.pytorch.serve.wlm.WorkerThread - Looping backend response at: 1737395378060 2025-01-20T17:49:38,060 [INFO ] W-9000-titan_1.0 org.pytorch.serve.wlm.WorkerThread - Looping backend response at: 1737395378060 2025-01-20T17:49:38,062 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - Connection accepted: /tmp/.ts.sock.9000. 2025-01-20T17:49:38,062 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - model_name: titan, batchSize: 1 2025-01-20T17:49:38,065 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - Backend worker process died. 2025-01-20T17:49:38,065 [INFO ] epollEventLoopGroup-5-4 org.pytorch.serve.wlm.WorkerThread - 9000 Worker disconnected. WORKER_STARTED 2025-01-20T17:49:38,065 [INFO ] epollEventLoopGroup-5-4 org.pytorch.serve.wlm.WorkerThread - 9000 Worker disconnected. WORKER_STARTED 2025-01-20T17:49:38,066 [DEBUG] W-9000-titan_1.0 org.pytorch.serve.wlm.WorkerThread - System state is : WORKER_STARTED 2025-01-20T17:49:38,066 [DEBUG] W-9000-titan_1.0 org.pytorch.serve.wlm.WorkerThread - System state is : WORKER_STARTED 2025-01-20T17:49:38,066 [DEBUG] W-9000-titan_1.0 org.pytorch.serve.wlm.WorkerThread - Backend worker monitoring thread interrupted or backend worker process died., startupTimeout:120sec java.lang.InterruptedException: null at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2056) ~[?:?] at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2133) ~[?:?] at java.util.concurrent.ArrayBlockingQueue.poll(ArrayBlockingQueue.java:432) ~[?:?] at org.pytorch.serve.wlm.WorkerThread.run(WorkerThread.java:234) [model-server.jar:?] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?] at java.lang.Thread.run(Thread.java:829) [?:?] 2025-01-20T17:49:38,066 [DEBUG] W-9000-titan_1.0 org.pytorch.serve.wlm.WorkerThread - Backend worker monitoring thread interrupted or backend worker process died., startupTimeout:120sec java.lang.InterruptedException: null at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2056) ~[?:?] at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2133) ~[?:?] at java.util.concurrent.ArrayBlockingQueue.poll(ArrayBlockingQueue.java:432) ~[?:?] at org.pytorch.serve.wlm.WorkerThread.run(WorkerThread.java:234) [model-server.jar:?] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?] at java.lang.Thread.run(Thread.java:829) [?:?] 2025-01-20T17:49:38,067 [WARN ] W-9000-titan_1.0 org.pytorch.serve.wlm.BatchAggregator - Load model failed: titan, error: Worker died. 2025-01-20T17:49:38,067 [WARN ] W-9000-titan_1.0 org.pytorch.serve.wlm.BatchAggregator - Load model failed: titan, error: Worker died. 2025-01-20T17:49:38,069 [DEBUG] W-9000-titan_1.0 org.pytorch.serve.wlm.WorkerThread - W-9000-titan_1.0 State change WORKER_STARTED -> WORKER_STOPPED 2025-01-20T17:49:38,069 [DEBUG] W-9000-titan_1.0 org.pytorch.serve.wlm.WorkerThread - W-9000-titan_1.0 State change WORKER_STARTED -> WORKER_STOPPED 2025-01-20T17:49:38,072 [WARN ] W-9000-titan_1.0 org.pytorch.serve.wlm.WorkerThread - Auto recovery failed again 2025-01-20T17:49:38,072 [WARN ] W-9000-titan_1.0 org.pytorch.serve.wlm.WorkerThread - Auto recovery failed again 2025-01-20T17:49:38,072 [INFO ] W-9000-titan_1.0 org.pytorch.serve.wlm.WorkerThread - Retry worker: 9000 in 21 seconds. 2025-01-20T17:49:38,072 [INFO ] W-9000-titan_1.0 org.pytorch.serve.wlm.WorkerThread - Retry worker: 9000 in 21 seconds. 2025-01-20T17:49:38,065 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - Traceback (most recent call last): 2025-01-20T17:49:38,078 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - File "/usr/local/lib/python3.11/dist-packages/ts/model_service_worker.py", line 301, in 2025-01-20T17:49:38,079 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - worker.run_server() 2025-01-20T17:49:38,080 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - File "/usr/local/lib/python3.11/dist-packages/ts/model_service_worker.py", line 268, in run_server 2025-01-20T17:49:38,081 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - self.handle_connection(cl_socket) 2025-01-20T17:49:38,083 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - File "/usr/local/lib/python3.11/dist-packages/ts/model_service_worker.py", line 196, in handle_connection 2025-01-20T17:49:38,084 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - service, result, code = self.load_model(msg) 2025-01-20T17:49:38,085 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - ^^^^^^^^^^^^^^^^^^^^ 2025-01-20T17:49:38,086 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - File "/usr/local/lib/python3.11/dist-packages/ts/model_service_worker.py", line 133, in load_model 2025-01-20T17:49:38,087 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - service = model_loader.load( 2025-01-20T17:49:38,090 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - ^^^^^^^^^^^^^^^^^^ 2025-01-20T17:49:38,091 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - File "/usr/local/lib/python3.11/dist-packages/ts/model_loader.py", line 108, in load 2025-01-20T17:49:38,091 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - module, function_name = self._load_handler_file(handler) 2025-01-20T17:49:38,091 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-01-20T17:49:38,092 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - File "/usr/local/lib/python3.11/dist-packages/ts/model_loader.py", line 153, in _load_handler_file 2025-01-20T17:49:38,093 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - module = importlib.import_module(module_name) 2025-01-20T17:49:38,093 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-01-20T17:49:38,093 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - File "/usr/lib/python3.11/importlib/__init__.py", line 126, in import_module 2025-01-20T17:49:38,094 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - return _bootstrap._gcd_import(name[level:], package, level) 2025-01-20T17:49:38,094 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-01-20T17:49:38,095 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - File "", line 1204, in _gcd_import 2025-01-20T17:49:38,095 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - File "", line 1176, in _find_and_load 2025-01-20T17:49:38,095 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - File "", line 1147, in _find_and_load_unlocked 2025-01-20T17:49:38,096 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - File "", line 690, in _load_unlocked 2025-01-20T17:49:38,096 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - File "", line 936, in exec_module 2025-01-20T17:49:38,097 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - File "", line 1074, in get_code 2025-01-20T17:49:38,097 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - File "", line 1004, in source_to_code 2025-01-20T17:49:38,097 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - File "", line 241, in _call_with_frames_removed 2025-01-20T17:49:38,098 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - File "/tmp/models/e1d5abe5fc2d4242a8e27bb4f69fedad/titan_handler.py", line 86 2025-01-20T17:49:38,098 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - print(" 2025-01-20T17:49:38,099 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - ^ 2025-01-20T17:49:38,099 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - SyntaxError: unterminated string literal (detected at line 86) 2025-01-20T17:49:38,100 [INFO ] W-9000-titan_1.0-stdout org.pytorch.serve.wlm.WorkerLifeCycle - Stopped Scanner - W-9000-titan_1.0-stdout 2025-01-20T17:49:38,100 [INFO ] W-9000-titan_1.0-stdout org.pytorch.serve.wlm.WorkerLifeCycle - Stopped Scanner - W-9000-titan_1.0-stdout 2025-01-20T17:49:38,118 [INFO ] W-9000-titan_1.0-stderr org.pytorch.serve.wlm.WorkerLifeCycle - Stopped Scanner - W-9000-titan_1.0-stderr 2025-01-20T17:49:38,118 [INFO ] W-9000-titan_1.0-stderr org.pytorch.serve.wlm.WorkerLifeCycle - Stopped Scanner - W-9000-titan_1.0-stderr 2025-01-20T17:49:50,629 [ERROR] Thread-10 org.pytorch.serve.metrics.MetricCollector - Traceback (most recent call last): File "/usr/local/lib/python3.11/dist-packages/ts/metrics/metric_collector.py", line 11, in from ts.metrics import system_metrics File "/usr/local/lib/python3.11/dist-packages/ts/__init__.py", line 10, in from . import version File "/usr/local/lib/python3.11/dist-packages/ts/version.py", line 5, in from pathlib import Path File "/usr/local/lib/python3.11/dist-packages/pathlib.py", line 10, in from collections import Sequence ImportError: cannot import name 'Sequence' from 'collections' (/usr/lib/python3.11/collections/__init__.py) 2025-01-20T17:49:50,629 [ERROR] Thread-10 org.pytorch.serve.metrics.MetricCollector - Traceback (most recent call last): File "/usr/local/lib/python3.11/dist-packages/ts/metrics/metric_collector.py", line 11, in from ts.metrics import system_metrics File "/usr/local/lib/python3.11/dist-packages/ts/__init__.py", line 10, in from . import version File "/usr/local/lib/python3.11/dist-packages/ts/version.py", line 5, in from pathlib import Path File "/usr/local/lib/python3.11/dist-packages/pathlib.py", line 10, in from collections import Sequence ImportError: cannot import name 'Sequence' from 'collections' (/usr/lib/python3.11/collections/__init__.py) 2025-01-20T17:49:59,073 [DEBUG] W-9000-titan_1.0 org.pytorch.serve.wlm.WorkerLifeCycle - Worker cmdline: [/usr/bin/python3, /usr/local/lib/python3.11/dist-packages/ts/model_service_worker.py, --sock-type, unix, --sock-name, /tmp/.ts.sock.9000, --metrics-config, /usr/local/lib/python3.11/dist-packages/ts/configs/metrics.yaml] 2025-01-20T17:49:59,073 [DEBUG] W-9000-titan_1.0 org.pytorch.serve.wlm.WorkerLifeCycle - Worker cmdline: [/usr/bin/python3, /usr/local/lib/python3.11/dist-packages/ts/model_service_worker.py, --sock-type, unix, --sock-name, /tmp/.ts.sock.9000, --metrics-config, /usr/local/lib/python3.11/dist-packages/ts/configs/metrics.yaml] 2025-01-20T17:50:00,640 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - s_name_part0=/tmp/.ts.sock, s_name_part1=9000, pid=7344 2025-01-20T17:50:00,646 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - Listening on port: /tmp/.ts.sock.9000 2025-01-20T17:50:00,651 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - Successfully loaded /usr/local/lib/python3.11/dist-packages/ts/configs/metrics.yaml. 2025-01-20T17:50:00,652 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - [PID]7344 2025-01-20T17:50:00,653 [DEBUG] W-9000-titan_1.0 org.pytorch.serve.wlm.WorkerThread - W-9000-titan_1.0 State change WORKER_STOPPED -> WORKER_STARTED 2025-01-20T17:50:00,653 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - Torch worker started. 2025-01-20T17:50:00,653 [DEBUG] W-9000-titan_1.0 org.pytorch.serve.wlm.WorkerThread - W-9000-titan_1.0 State change WORKER_STOPPED -> WORKER_STARTED 2025-01-20T17:50:00,654 [INFO ] W-9000-titan_1.0 org.pytorch.serve.wlm.WorkerThread - Connecting to: /tmp/.ts.sock.9000 2025-01-20T17:50:00,654 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - Python runtime: 3.11.11 2025-01-20T17:50:00,654 [INFO ] W-9000-titan_1.0 org.pytorch.serve.wlm.WorkerThread - Connecting to: /tmp/.ts.sock.9000 2025-01-20T17:50:00,657 [DEBUG] W-9000-titan_1.0 org.pytorch.serve.wlm.WorkerThread - Flushing req.cmd LOAD repeats 1 to backend at: 1737395400657 2025-01-20T17:50:00,657 [DEBUG] W-9000-titan_1.0 org.pytorch.serve.wlm.WorkerThread - Flushing req.cmd LOAD repeats 1 to backend at: 1737395400657 2025-01-20T17:50:00,658 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - Connection accepted: /tmp/.ts.sock.9000. 2025-01-20T17:50:00,658 [INFO ] W-9000-titan_1.0 org.pytorch.serve.wlm.WorkerThread - Looping backend response at: 1737395400658 2025-01-20T17:50:00,658 [INFO ] W-9000-titan_1.0 org.pytorch.serve.wlm.WorkerThread - Looping backend response at: 1737395400658 2025-01-20T17:50:00,660 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - model_name: titan, batchSize: 1 2025-01-20T17:50:00,665 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - Backend worker process died. 2025-01-20T17:50:00,666 [INFO ] epollEventLoopGroup-5-1 org.pytorch.serve.wlm.WorkerThread - 9000 Worker disconnected. WORKER_STARTED 2025-01-20T17:50:00,666 [INFO ] epollEventLoopGroup-5-1 org.pytorch.serve.wlm.WorkerThread - 9000 Worker disconnected. WORKER_STARTED 2025-01-20T17:50:00,666 [DEBUG] W-9000-titan_1.0 org.pytorch.serve.wlm.WorkerThread - System state is : WORKER_STARTED 2025-01-20T17:50:00,666 [DEBUG] W-9000-titan_1.0 org.pytorch.serve.wlm.WorkerThread - System state is : WORKER_STARTED 2025-01-20T17:50:00,666 [DEBUG] W-9000-titan_1.0 org.pytorch.serve.wlm.WorkerThread - Backend worker monitoring thread interrupted or backend worker process died., startupTimeout:120sec java.lang.InterruptedException: null at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2056) ~[?:?] at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2133) ~[?:?] at java.util.concurrent.ArrayBlockingQueue.poll(ArrayBlockingQueue.java:432) ~[?:?] at org.pytorch.serve.wlm.WorkerThread.run(WorkerThread.java:234) [model-server.jar:?] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?] at java.lang.Thread.run(Thread.java:829) [?:?] 2025-01-20T17:50:00,666 [DEBUG] W-9000-titan_1.0 org.pytorch.serve.wlm.WorkerThread - Backend worker monitoring thread interrupted or backend worker process died., startupTimeout:120sec java.lang.InterruptedException: null at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2056) ~[?:?] at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2133) ~[?:?] at java.util.concurrent.ArrayBlockingQueue.poll(ArrayBlockingQueue.java:432) ~[?:?] at org.pytorch.serve.wlm.WorkerThread.run(WorkerThread.java:234) [model-server.jar:?] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?] at java.lang.Thread.run(Thread.java:829) [?:?] 2025-01-20T17:50:00,668 [WARN ] W-9000-titan_1.0 org.pytorch.serve.wlm.BatchAggregator - Load model failed: titan, error: Worker died. 2025-01-20T17:50:00,668 [WARN ] W-9000-titan_1.0 org.pytorch.serve.wlm.BatchAggregator - Load model failed: titan, error: Worker died. 2025-01-20T17:50:00,668 [DEBUG] W-9000-titan_1.0 org.pytorch.serve.wlm.WorkerThread - W-9000-titan_1.0 State change WORKER_STARTED -> WORKER_STOPPED 2025-01-20T17:50:00,668 [DEBUG] W-9000-titan_1.0 org.pytorch.serve.wlm.WorkerThread - W-9000-titan_1.0 State change WORKER_STARTED -> WORKER_STOPPED 2025-01-20T17:50:00,668 [WARN ] W-9000-titan_1.0 org.pytorch.serve.wlm.WorkerThread - Auto recovery failed again 2025-01-20T17:50:00,668 [WARN ] W-9000-titan_1.0 org.pytorch.serve.wlm.WorkerThread - Auto recovery failed again 2025-01-20T17:50:00,668 [INFO ] W-9000-titan_1.0 org.pytorch.serve.wlm.WorkerThread - Retry worker: 9000 in 34 seconds. 2025-01-20T17:50:00,668 [INFO ] W-9000-titan_1.0 org.pytorch.serve.wlm.WorkerThread - Retry worker: 9000 in 34 seconds. 2025-01-20T17:50:00,667 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - Traceback (most recent call last): 2025-01-20T17:50:00,669 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - File "/usr/local/lib/python3.11/dist-packages/ts/model_service_worker.py", line 301, in 2025-01-20T17:50:00,669 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - worker.run_server() 2025-01-20T17:50:00,669 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - File "/usr/local/lib/python3.11/dist-packages/ts/model_service_worker.py", line 268, in run_server 2025-01-20T17:50:00,669 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - self.handle_connection(cl_socket) 2025-01-20T17:50:00,670 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - File "/usr/local/lib/python3.11/dist-packages/ts/model_service_worker.py", line 196, in handle_connection 2025-01-20T17:50:00,671 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - service, result, code = self.load_model(msg) 2025-01-20T17:50:00,671 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - ^^^^^^^^^^^^^^^^^^^^ 2025-01-20T17:50:00,672 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - File "/usr/local/lib/python3.11/dist-packages/ts/model_service_worker.py", line 133, in load_model 2025-01-20T17:50:00,672 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - service = model_loader.load( 2025-01-20T17:50:00,672 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - ^^^^^^^^^^^^^^^^^^ 2025-01-20T17:50:00,673 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - File "/usr/local/lib/python3.11/dist-packages/ts/model_loader.py", line 108, in load 2025-01-20T17:50:00,673 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - module, function_name = self._load_handler_file(handler) 2025-01-20T17:50:00,675 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-01-20T17:50:00,675 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - File "/usr/local/lib/python3.11/dist-packages/ts/model_loader.py", line 153, in _load_handler_file 2025-01-20T17:50:00,676 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - module = importlib.import_module(module_name) 2025-01-20T17:50:00,677 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-01-20T17:50:00,677 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - File "/usr/lib/python3.11/importlib/__init__.py", line 126, in import_module 2025-01-20T17:50:00,678 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - return _bootstrap._gcd_import(name[level:], package, level) 2025-01-20T17:50:00,678 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-01-20T17:50:00,678 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - File "", line 1204, in _gcd_import 2025-01-20T17:50:00,678 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - File "", line 1176, in _find_and_load 2025-01-20T17:50:00,680 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - File "", line 1147, in _find_and_load_unlocked 2025-01-20T17:50:00,681 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - File "", line 690, in _load_unlocked 2025-01-20T17:50:00,681 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - File "", line 936, in exec_module 2025-01-20T17:50:00,682 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - File "", line 1074, in get_code 2025-01-20T17:50:00,682 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - File "", line 1004, in source_to_code 2025-01-20T17:50:00,682 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - File "", line 241, in _call_with_frames_removed 2025-01-20T17:50:00,684 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - File "/tmp/models/e1d5abe5fc2d4242a8e27bb4f69fedad/titan_handler.py", line 86 2025-01-20T17:50:00,684 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - print(" 2025-01-20T17:50:00,684 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - ^ 2025-01-20T17:50:00,684 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - SyntaxError: unterminated string literal (detected at line 86) 2025-01-20T17:50:00,685 [INFO ] W-9000-titan_1.0-stdout org.pytorch.serve.wlm.WorkerLifeCycle - Stopped Scanner - W-9000-titan_1.0-stdout 2025-01-20T17:50:00,685 [INFO ] W-9000-titan_1.0-stdout org.pytorch.serve.wlm.WorkerLifeCycle - Stopped Scanner - W-9000-titan_1.0-stdout 2025-01-20T17:50:00,713 [INFO ] W-9000-titan_1.0-stderr org.pytorch.serve.wlm.WorkerLifeCycle - Stopped Scanner - W-9000-titan_1.0-stderr 2025-01-20T17:50:00,713 [INFO ] W-9000-titan_1.0-stderr org.pytorch.serve.wlm.WorkerLifeCycle - Stopped Scanner - W-9000-titan_1.0-stderr 2025-01-20T17:55:02,003 [INFO ] main org.pytorch.serve.servingsdk.impl.PluginsManager - Loading snapshot serializer plugin... 2025-01-20T17:55:02,003 [INFO ] main org.pytorch.serve.servingsdk.impl.PluginsManager - Loading snapshot serializer plugin... 2025-01-20T17:55:02,086 [DEBUG] main org.pytorch.serve.util.ConfigManager - xpu-smi not available or failed: Cannot run program "xpu-smi": error=2, No such file or directory 2025-01-20T17:55:02,086 [DEBUG] main org.pytorch.serve.util.ConfigManager - xpu-smi not available or failed: Cannot run program "xpu-smi": error=2, No such file or directory 2025-01-20T17:55:02,089 [WARN ] main org.pytorch.serve.util.ConfigManager - Your torchserve instance can access any URL to load models. When deploying to production, make sure to limit the set of allowed_urls in config.properties 2025-01-20T17:55:02,089 [WARN ] main org.pytorch.serve.util.ConfigManager - Your torchserve instance can access any URL to load models. When deploying to production, make sure to limit the set of allowed_urls in config.properties 2025-01-20T17:55:02,115 [INFO ] main org.pytorch.serve.servingsdk.impl.PluginsManager - Initializing plugins manager... 2025-01-20T17:55:02,115 [INFO ] main org.pytorch.serve.servingsdk.impl.PluginsManager - Initializing plugins manager... 2025-01-20T17:55:02,217 [INFO ] main org.pytorch.serve.metrics.configuration.MetricConfiguration - Successfully loaded metrics configuration from /usr/local/lib/python3.11/dist-packages/ts/configs/metrics.yaml 2025-01-20T17:55:02,217 [INFO ] main org.pytorch.serve.metrics.configuration.MetricConfiguration - Successfully loaded metrics configuration from /usr/local/lib/python3.11/dist-packages/ts/configs/metrics.yaml 2025-01-20T17:55:02,316 [INFO ] main org.pytorch.serve.ModelServer - Torchserve version: 0.12.0 TS Home: /usr/local/lib/python3.11/dist-packages Current directory: /content Temp directory: /tmp Metrics config path: /usr/local/lib/python3.11/dist-packages/ts/configs/metrics.yaml Number of GPUs: 1 Number of CPUs: 2 Max heap size: 3246 M Python executable: /usr/bin/python3 Config file: logs/config/20250120175002976-shutdown.cfg Inference address: http://127.0.0.1:8083 Management address: http://127.0.0.1:8084 Metrics address: http://127.0.0.1:8085 Model Store: /content/model_store Initial Models: titan=titan.mar Log dir: /content/logs Metrics dir: /content/logs Netty threads: 0 Netty client threads: 0 Default workers per model: 1 Blacklist Regex: N/A Maximum Response Size: 6553500 Maximum Request Size: 6553500 Limit Maximum Image Pixels: true Prefer direct buffer: false Allowed Urls: [file://.*|http(s)?://.*] Custom python dependency for model allowed: false Enable metrics API: true Metrics mode: LOG Disable system metrics: false Workflow Store: /content/model_store CPP log config: N/A Model config: N/A System metrics command: default Model API enabled: false 2025-01-20T17:55:02,316 [INFO ] main org.pytorch.serve.ModelServer - Torchserve version: 0.12.0 TS Home: /usr/local/lib/python3.11/dist-packages Current directory: /content Temp directory: /tmp Metrics config path: /usr/local/lib/python3.11/dist-packages/ts/configs/metrics.yaml Number of GPUs: 1 Number of CPUs: 2 Max heap size: 3246 M Python executable: /usr/bin/python3 Config file: logs/config/20250120175002976-shutdown.cfg Inference address: http://127.0.0.1:8083 Management address: http://127.0.0.1:8084 Metrics address: http://127.0.0.1:8085 Model Store: /content/model_store Initial Models: titan=titan.mar Log dir: /content/logs Metrics dir: /content/logs Netty threads: 0 Netty client threads: 0 Default workers per model: 1 Blacklist Regex: N/A Maximum Response Size: 6553500 Maximum Request Size: 6553500 Limit Maximum Image Pixels: true Prefer direct buffer: false Allowed Urls: [file://.*|http(s)?://.*] Custom python dependency for model allowed: false Enable metrics API: true Metrics mode: LOG Disable system metrics: false Workflow Store: /content/model_store CPP log config: N/A Model config: N/A System metrics command: default Model API enabled: false 2025-01-20T17:55:02,331 [INFO ] main org.pytorch.serve.snapshot.SnapshotManager - Started restoring models from snapshot { "name": "20250120175002976-shutdown.cfg", "modelCount": 1, "created": 1737395402976, "models": { "titan": { "1.0": { "defaultVersion": true, "marName": "titan.mar", "minWorkers": 1, "maxWorkers": 1, "batchSize": 1, "maxBatchDelay": 100, "responseTimeout": 120, "startupTimeout": 120, "runtimeType": "python" } } } } 2025-01-20T17:55:02,331 [INFO ] main org.pytorch.serve.snapshot.SnapshotManager - Started restoring models from snapshot { "name": "20250120175002976-shutdown.cfg", "modelCount": 1, "created": 1737395402976, "models": { "titan": { "1.0": { "defaultVersion": true, "marName": "titan.mar", "minWorkers": 1, "maxWorkers": 1, "batchSize": 1, "maxBatchDelay": 100, "responseTimeout": 120, "startupTimeout": 120, "runtimeType": "python" } } } } 2025-01-20T17:55:02,342 [INFO ] main org.pytorch.serve.snapshot.SnapshotManager - Validating snapshot 20250120175002976-shutdown.cfg 2025-01-20T17:55:02,342 [INFO ] main org.pytorch.serve.snapshot.SnapshotManager - Validating snapshot 20250120175002976-shutdown.cfg 2025-01-20T17:55:02,344 [INFO ] main org.pytorch.serve.snapshot.SnapshotManager - Snapshot 20250120175002976-shutdown.cfg validated successfully 2025-01-20T17:55:02,344 [INFO ] main org.pytorch.serve.snapshot.SnapshotManager - Snapshot 20250120175002976-shutdown.cfg validated successfully 2025-01-20T17:55:33,574 [DEBUG] main org.pytorch.serve.wlm.ModelVersionedRefs - Adding new version 1.0 for model titan 2025-01-20T17:55:33,574 [DEBUG] main org.pytorch.serve.wlm.ModelVersionedRefs - Adding new version 1.0 for model titan 2025-01-20T17:55:33,574 [DEBUG] main org.pytorch.serve.wlm.ModelVersionedRefs - Setting default version to 1.0 for model titan 2025-01-20T17:55:33,574 [DEBUG] main org.pytorch.serve.wlm.ModelVersionedRefs - Setting default version to 1.0 for model titan 2025-01-20T17:55:33,574 [DEBUG] main org.pytorch.serve.wlm.ModelVersionedRefs - Setting default version to 1.0 for model titan 2025-01-20T17:55:33,574 [DEBUG] main org.pytorch.serve.wlm.ModelVersionedRefs - Setting default version to 1.0 for model titan 2025-01-20T17:55:33,574 [INFO ] main org.pytorch.serve.wlm.ModelManager - Model titan loaded. 2025-01-20T17:55:33,574 [INFO ] main org.pytorch.serve.wlm.ModelManager - Model titan loaded. 2025-01-20T17:55:33,575 [DEBUG] main org.pytorch.serve.wlm.ModelManager - updateModel: titan, count: 1 2025-01-20T17:55:33,575 [DEBUG] main org.pytorch.serve.wlm.ModelManager - updateModel: titan, count: 1 2025-01-20T17:55:33,600 [INFO ] main org.pytorch.serve.ModelServer - Initialize Inference server with: EpollServerSocketChannel. 2025-01-20T17:55:33,600 [INFO ] main org.pytorch.serve.ModelServer - Initialize Inference server with: EpollServerSocketChannel. 2025-01-20T17:55:33,608 [DEBUG] W-9000-titan_1.0 org.pytorch.serve.wlm.WorkerLifeCycle - Worker cmdline: [/usr/bin/python3, /usr/local/lib/python3.11/dist-packages/ts/model_service_worker.py, --sock-type, unix, --sock-name, /tmp/.ts.sock.9000, --metrics-config, /usr/local/lib/python3.11/dist-packages/ts/configs/metrics.yaml] 2025-01-20T17:55:33,608 [DEBUG] W-9000-titan_1.0 org.pytorch.serve.wlm.WorkerLifeCycle - Worker cmdline: [/usr/bin/python3, /usr/local/lib/python3.11/dist-packages/ts/model_service_worker.py, --sock-type, unix, --sock-name, /tmp/.ts.sock.9000, --metrics-config, /usr/local/lib/python3.11/dist-packages/ts/configs/metrics.yaml] 2025-01-20T17:55:33,814 [INFO ] main org.pytorch.serve.ModelServer - Inference API bind to: http://127.0.0.1:8083 2025-01-20T17:55:33,814 [INFO ] main org.pytorch.serve.ModelServer - Inference API bind to: http://127.0.0.1:8083 2025-01-20T17:55:33,814 [INFO ] main org.pytorch.serve.ModelServer - Initialize Management server with: EpollServerSocketChannel. 2025-01-20T17:55:33,814 [INFO ] main org.pytorch.serve.ModelServer - Initialize Management server with: EpollServerSocketChannel. 2025-01-20T17:55:33,823 [INFO ] main org.pytorch.serve.ModelServer - Management API bind to: http://127.0.0.1:8084 2025-01-20T17:55:33,823 [INFO ] main org.pytorch.serve.ModelServer - Management API bind to: http://127.0.0.1:8084 2025-01-20T17:55:33,824 [INFO ] main org.pytorch.serve.ModelServer - Initialize Metrics server with: EpollServerSocketChannel. 2025-01-20T17:55:33,824 [INFO ] main org.pytorch.serve.ModelServer - Initialize Metrics server with: EpollServerSocketChannel. 2025-01-20T17:55:33,825 [INFO ] main org.pytorch.serve.ModelServer - Metrics API bind to: http://127.0.0.1:8085 2025-01-20T17:55:33,825 [INFO ] main org.pytorch.serve.ModelServer - Metrics API bind to: http://127.0.0.1:8085 2025-01-20T17:55:34,562 [WARN ] pool-3-thread-1 org.pytorch.serve.metrics.MetricCollector - worker pid is not available yet. 2025-01-20T17:55:34,562 [WARN ] pool-3-thread-1 org.pytorch.serve.metrics.MetricCollector - worker pid is not available yet. 2025-01-20T17:55:34,623 [ERROR] Thread-1 org.pytorch.serve.metrics.MetricCollector - Traceback (most recent call last): File "/usr/local/lib/python3.11/dist-packages/ts/metrics/metric_collector.py", line 11, in from ts.metrics import system_metrics File "/usr/local/lib/python3.11/dist-packages/ts/__init__.py", line 10, in from . import version File "/usr/local/lib/python3.11/dist-packages/ts/version.py", line 5, in from pathlib import Path File "/usr/local/lib/python3.11/dist-packages/pathlib.py", line 10, in from collections import Sequence ImportError: cannot import name 'Sequence' from 'collections' (/usr/lib/python3.11/collections/__init__.py) 2025-01-20T17:55:34,623 [ERROR] Thread-1 org.pytorch.serve.metrics.MetricCollector - Traceback (most recent call last): File "/usr/local/lib/python3.11/dist-packages/ts/metrics/metric_collector.py", line 11, in from ts.metrics import system_metrics File "/usr/local/lib/python3.11/dist-packages/ts/__init__.py", line 10, in from . import version File "/usr/local/lib/python3.11/dist-packages/ts/version.py", line 5, in from pathlib import Path File "/usr/local/lib/python3.11/dist-packages/pathlib.py", line 10, in from collections import Sequence ImportError: cannot import name 'Sequence' from 'collections' (/usr/lib/python3.11/collections/__init__.py) 2025-01-20T17:55:36,137 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - s_name_part0=/tmp/.ts.sock, s_name_part1=9000, pid=8794 2025-01-20T17:55:36,139 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - Listening on port: /tmp/.ts.sock.9000 2025-01-20T17:55:36,149 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - Successfully loaded /usr/local/lib/python3.11/dist-packages/ts/configs/metrics.yaml. 2025-01-20T17:55:36,149 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - [PID]8794 2025-01-20T17:55:36,150 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - Torch worker started. 2025-01-20T17:55:36,150 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - Python runtime: 3.11.11 2025-01-20T17:55:36,151 [DEBUG] W-9000-titan_1.0 org.pytorch.serve.wlm.WorkerThread - W-9000-titan_1.0 State change null -> WORKER_STARTED 2025-01-20T17:55:36,151 [DEBUG] W-9000-titan_1.0 org.pytorch.serve.wlm.WorkerThread - W-9000-titan_1.0 State change null -> WORKER_STARTED 2025-01-20T17:55:36,159 [INFO ] W-9000-titan_1.0 org.pytorch.serve.wlm.WorkerThread - Connecting to: /tmp/.ts.sock.9000 2025-01-20T17:55:36,159 [INFO ] W-9000-titan_1.0 org.pytorch.serve.wlm.WorkerThread - Connecting to: /tmp/.ts.sock.9000 2025-01-20T17:55:36,171 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - Connection accepted: /tmp/.ts.sock.9000. 2025-01-20T17:55:36,178 [DEBUG] W-9000-titan_1.0 org.pytorch.serve.wlm.WorkerThread - Flushing req.cmd LOAD repeats 1 to backend at: 1737395736177 2025-01-20T17:55:36,178 [DEBUG] W-9000-titan_1.0 org.pytorch.serve.wlm.WorkerThread - Flushing req.cmd LOAD repeats 1 to backend at: 1737395736177 2025-01-20T17:55:36,182 [INFO ] W-9000-titan_1.0 org.pytorch.serve.wlm.WorkerThread - Looping backend response at: 1737395736182 2025-01-20T17:55:36,182 [INFO ] W-9000-titan_1.0 org.pytorch.serve.wlm.WorkerThread - Looping backend response at: 1737395736182 2025-01-20T17:55:36,222 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - model_name: titan, batchSize: 1 2025-01-20T17:55:36,226 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - Backend worker process died. 2025-01-20T17:55:36,226 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - Traceback (most recent call last): 2025-01-20T17:55:36,226 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - File "/usr/local/lib/python3.11/dist-packages/ts/model_service_worker.py", line 301, in 2025-01-20T17:55:36,226 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - worker.run_server() 2025-01-20T17:55:36,227 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - File "/usr/local/lib/python3.11/dist-packages/ts/model_service_worker.py", line 268, in run_server 2025-01-20T17:55:36,232 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - self.handle_connection(cl_socket) 2025-01-20T17:55:36,232 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - File "/usr/local/lib/python3.11/dist-packages/ts/model_service_worker.py", line 196, in handle_connection 2025-01-20T17:55:36,232 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - service, result, code = self.load_model(msg) 2025-01-20T17:55:36,234 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - ^^^^^^^^^^^^^^^^^^^^ 2025-01-20T17:55:36,234 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - File "/usr/local/lib/python3.11/dist-packages/ts/model_service_worker.py", line 133, in load_model 2025-01-20T17:55:36,234 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - service = model_loader.load( 2025-01-20T17:55:36,234 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - ^^^^^^^^^^^^^^^^^^ 2025-01-20T17:55:36,237 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - File "/usr/local/lib/python3.11/dist-packages/ts/model_loader.py", line 125, in load 2025-01-20T17:55:36,237 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - entry_point, initialize_fn = self._get_class_entry_point(module) 2025-01-20T17:55:36,238 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-01-20T17:55:36,238 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - File "/usr/local/lib/python3.11/dist-packages/ts/model_loader.py", line 185, in _get_class_entry_point 2025-01-20T17:55:36,238 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - model_service = model_class() 2025-01-20T17:55:36,238 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - ^^^^^^^^^^^^^ 2025-01-20T17:55:36,238 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - File "/tmp/models/17fba6ef1a9d4697a83e026d885557fd/titan_handler.py", line 7, in __init__ 2025-01-20T17:55:36,239 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - self.device = 'cuda' if torch.cuda.is_available() else 'cpu' 2025-01-20T17:55:36,239 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - ^^^^^ 2025-01-20T17:55:36,239 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - NameError: name 'torch' is not defined 2025-01-20T17:55:36,248 [INFO ] epollEventLoopGroup-5-1 org.pytorch.serve.wlm.WorkerThread - 9000 Worker disconnected. WORKER_STARTED 2025-01-20T17:55:36,248 [INFO ] epollEventLoopGroup-5-1 org.pytorch.serve.wlm.WorkerThread - 9000 Worker disconnected. WORKER_STARTED 2025-01-20T17:55:36,252 [DEBUG] W-9000-titan_1.0 org.pytorch.serve.wlm.WorkerThread - System state is : WORKER_STARTED 2025-01-20T17:55:36,252 [DEBUG] W-9000-titan_1.0 org.pytorch.serve.wlm.WorkerThread - System state is : WORKER_STARTED 2025-01-20T17:55:36,255 [DEBUG] W-9000-titan_1.0 org.pytorch.serve.wlm.WorkerThread - Backend worker monitoring thread interrupted or backend worker process died., startupTimeout:120sec java.lang.InterruptedException: null at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2056) ~[?:?] at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2133) ~[?:?] at java.util.concurrent.ArrayBlockingQueue.poll(ArrayBlockingQueue.java:432) ~[?:?] at org.pytorch.serve.wlm.WorkerThread.run(WorkerThread.java:234) [model-server.jar:?] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) [?:?] at java.util.concurrent.FutureTask.run(FutureTask.java:264) [?:?] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?] at java.lang.Thread.run(Thread.java:829) [?:?] 2025-01-20T17:55:36,255 [DEBUG] W-9000-titan_1.0 org.pytorch.serve.wlm.WorkerThread - Backend worker monitoring thread interrupted or backend worker process died., startupTimeout:120sec java.lang.InterruptedException: null at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2056) ~[?:?] at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2133) ~[?:?] at java.util.concurrent.ArrayBlockingQueue.poll(ArrayBlockingQueue.java:432) ~[?:?] at org.pytorch.serve.wlm.WorkerThread.run(WorkerThread.java:234) [model-server.jar:?] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) [?:?] at java.util.concurrent.FutureTask.run(FutureTask.java:264) [?:?] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?] at java.lang.Thread.run(Thread.java:829) [?:?] 2025-01-20T17:55:36,296 [WARN ] W-9000-titan_1.0 org.pytorch.serve.wlm.BatchAggregator - Load model failed: titan, error: Worker died. 2025-01-20T17:55:36,296 [WARN ] W-9000-titan_1.0 org.pytorch.serve.wlm.BatchAggregator - Load model failed: titan, error: Worker died. 2025-01-20T17:55:36,297 [DEBUG] W-9000-titan_1.0 org.pytorch.serve.wlm.WorkerThread - W-9000-titan_1.0 State change WORKER_STARTED -> WORKER_STOPPED 2025-01-20T17:55:36,297 [DEBUG] W-9000-titan_1.0 org.pytorch.serve.wlm.WorkerThread - W-9000-titan_1.0 State change WORKER_STARTED -> WORKER_STOPPED 2025-01-20T17:55:36,297 [INFO ] W-9000-titan_1.0 org.pytorch.serve.wlm.WorkerThread - Auto recovery start timestamp: 1737395736297 2025-01-20T17:55:36,297 [INFO ] W-9000-titan_1.0 org.pytorch.serve.wlm.WorkerThread - Auto recovery start timestamp: 1737395736297 2025-01-20T17:55:36,299 [INFO ] W-9000-titan_1.0 org.pytorch.serve.wlm.WorkerThread - Retry worker: 9000 in 1 seconds. 2025-01-20T17:55:36,299 [INFO ] W-9000-titan_1.0 org.pytorch.serve.wlm.WorkerThread - Retry worker: 9000 in 1 seconds. 2025-01-20T17:55:36,334 [INFO ] W-9000-titan_1.0-stderr org.pytorch.serve.wlm.WorkerLifeCycle - Stopped Scanner - W-9000-titan_1.0-stderr 2025-01-20T17:55:36,334 [INFO ] W-9000-titan_1.0-stderr org.pytorch.serve.wlm.WorkerLifeCycle - Stopped Scanner - W-9000-titan_1.0-stderr 2025-01-20T17:55:36,336 [INFO ] W-9000-titan_1.0-stdout org.pytorch.serve.wlm.WorkerLifeCycle - Stopped Scanner - W-9000-titan_1.0-stdout 2025-01-20T17:55:36,336 [INFO ] W-9000-titan_1.0-stdout org.pytorch.serve.wlm.WorkerLifeCycle - Stopped Scanner - W-9000-titan_1.0-stdout 2025-01-20T17:55:37,300 [DEBUG] W-9000-titan_1.0 org.pytorch.serve.wlm.WorkerLifeCycle - Worker cmdline: [/usr/bin/python3, /usr/local/lib/python3.11/dist-packages/ts/model_service_worker.py, --sock-type, unix, --sock-name, /tmp/.ts.sock.9000, --metrics-config, /usr/local/lib/python3.11/dist-packages/ts/configs/metrics.yaml] 2025-01-20T17:55:37,300 [DEBUG] W-9000-titan_1.0 org.pytorch.serve.wlm.WorkerLifeCycle - Worker cmdline: [/usr/bin/python3, /usr/local/lib/python3.11/dist-packages/ts/model_service_worker.py, --sock-type, unix, --sock-name, /tmp/.ts.sock.9000, --metrics-config, /usr/local/lib/python3.11/dist-packages/ts/configs/metrics.yaml] 2025-01-20T17:55:38,858 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - s_name_part0=/tmp/.ts.sock, s_name_part1=9000, pid=8824 2025-01-20T17:55:38,861 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - Listening on port: /tmp/.ts.sock.9000 2025-01-20T17:55:38,871 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - Successfully loaded /usr/local/lib/python3.11/dist-packages/ts/configs/metrics.yaml. 2025-01-20T17:55:38,872 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - [PID]8824 2025-01-20T17:55:38,872 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - Torch worker started. 2025-01-20T17:55:38,872 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - Python runtime: 3.11.11 2025-01-20T17:55:38,872 [DEBUG] W-9000-titan_1.0 org.pytorch.serve.wlm.WorkerThread - W-9000-titan_1.0 State change WORKER_STOPPED -> WORKER_STARTED 2025-01-20T17:55:38,872 [DEBUG] W-9000-titan_1.0 org.pytorch.serve.wlm.WorkerThread - W-9000-titan_1.0 State change WORKER_STOPPED -> WORKER_STARTED 2025-01-20T17:55:38,873 [INFO ] W-9000-titan_1.0 org.pytorch.serve.wlm.WorkerThread - Connecting to: /tmp/.ts.sock.9000 2025-01-20T17:55:38,873 [INFO ] W-9000-titan_1.0 org.pytorch.serve.wlm.WorkerThread - Connecting to: /tmp/.ts.sock.9000 2025-01-20T17:55:38,875 [DEBUG] W-9000-titan_1.0 org.pytorch.serve.wlm.WorkerThread - Flushing req.cmd LOAD repeats 1 to backend at: 1737395738875 2025-01-20T17:55:38,875 [DEBUG] W-9000-titan_1.0 org.pytorch.serve.wlm.WorkerThread - Flushing req.cmd LOAD repeats 1 to backend at: 1737395738875 2025-01-20T17:55:38,876 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - Connection accepted: /tmp/.ts.sock.9000. 2025-01-20T17:55:38,876 [INFO ] W-9000-titan_1.0 org.pytorch.serve.wlm.WorkerThread - Looping backend response at: 1737395738876 2025-01-20T17:55:38,876 [INFO ] W-9000-titan_1.0 org.pytorch.serve.wlm.WorkerThread - Looping backend response at: 1737395738876 2025-01-20T17:55:38,903 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - model_name: titan, batchSize: 1 2025-01-20T17:55:38,905 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - Backend worker process died. 2025-01-20T17:55:38,906 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - Traceback (most recent call last): 2025-01-20T17:55:38,908 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - File "/usr/local/lib/python3.11/dist-packages/ts/model_service_worker.py", line 301, in 2025-01-20T17:55:38,909 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - worker.run_server() 2025-01-20T17:55:38,910 [INFO ] epollEventLoopGroup-5-2 org.pytorch.serve.wlm.WorkerThread - 9000 Worker disconnected. WORKER_STARTED 2025-01-20T17:55:38,910 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - File "/usr/local/lib/python3.11/dist-packages/ts/model_service_worker.py", line 268, in run_server 2025-01-20T17:55:38,912 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - self.handle_connection(cl_socket) 2025-01-20T17:55:38,910 [INFO ] epollEventLoopGroup-5-2 org.pytorch.serve.wlm.WorkerThread - 9000 Worker disconnected. WORKER_STARTED 2025-01-20T17:55:38,919 [DEBUG] W-9000-titan_1.0 org.pytorch.serve.wlm.WorkerThread - System state is : WORKER_STARTED 2025-01-20T17:55:38,919 [DEBUG] W-9000-titan_1.0 org.pytorch.serve.wlm.WorkerThread - System state is : WORKER_STARTED 2025-01-20T17:55:38,919 [DEBUG] W-9000-titan_1.0 org.pytorch.serve.wlm.WorkerThread - Backend worker monitoring thread interrupted or backend worker process died., startupTimeout:120sec java.lang.InterruptedException: null at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2056) ~[?:?] at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2133) ~[?:?] at java.util.concurrent.ArrayBlockingQueue.poll(ArrayBlockingQueue.java:432) ~[?:?] at org.pytorch.serve.wlm.WorkerThread.run(WorkerThread.java:234) [model-server.jar:?] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?] at java.lang.Thread.run(Thread.java:829) [?:?] 2025-01-20T17:55:38,919 [DEBUG] W-9000-titan_1.0 org.pytorch.serve.wlm.WorkerThread - Backend worker monitoring thread interrupted or backend worker process died., startupTimeout:120sec java.lang.InterruptedException: null at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2056) ~[?:?] at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2133) ~[?:?] at java.util.concurrent.ArrayBlockingQueue.poll(ArrayBlockingQueue.java:432) ~[?:?] at org.pytorch.serve.wlm.WorkerThread.run(WorkerThread.java:234) [model-server.jar:?] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?] at java.lang.Thread.run(Thread.java:829) [?:?] 2025-01-20T17:55:38,915 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - File "/usr/local/lib/python3.11/dist-packages/ts/model_service_worker.py", line 196, in handle_connection 2025-01-20T17:55:38,921 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - service, result, code = self.load_model(msg) 2025-01-20T17:55:38,922 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - ^^^^^^^^^^^^^^^^^^^^ 2025-01-20T17:55:38,923 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - File "/usr/local/lib/python3.11/dist-packages/ts/model_service_worker.py", line 133, in load_model 2025-01-20T17:55:38,920 [WARN ] W-9000-titan_1.0 org.pytorch.serve.wlm.BatchAggregator - Load model failed: titan, error: Worker died. 2025-01-20T17:55:38,920 [WARN ] W-9000-titan_1.0 org.pytorch.serve.wlm.BatchAggregator - Load model failed: titan, error: Worker died. 2025-01-20T17:55:38,931 [DEBUG] W-9000-titan_1.0 org.pytorch.serve.wlm.WorkerThread - W-9000-titan_1.0 State change WORKER_STARTED -> WORKER_STOPPED 2025-01-20T17:55:38,931 [DEBUG] W-9000-titan_1.0 org.pytorch.serve.wlm.WorkerThread - W-9000-titan_1.0 State change WORKER_STARTED -> WORKER_STOPPED 2025-01-20T17:55:38,931 [WARN ] W-9000-titan_1.0 org.pytorch.serve.wlm.WorkerThread - Auto recovery failed again 2025-01-20T17:55:38,931 [WARN ] W-9000-titan_1.0 org.pytorch.serve.wlm.WorkerThread - Auto recovery failed again 2025-01-20T17:55:38,932 [INFO ] W-9000-titan_1.0 org.pytorch.serve.wlm.WorkerThread - Retry worker: 9000 in 1 seconds. 2025-01-20T17:55:38,932 [INFO ] W-9000-titan_1.0 org.pytorch.serve.wlm.WorkerThread - Retry worker: 9000 in 1 seconds. 2025-01-20T17:55:38,924 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - service = model_loader.load( 2025-01-20T17:55:38,934 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - ^^^^^^^^^^^^^^^^^^ 2025-01-20T17:55:38,935 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - File "/usr/local/lib/python3.11/dist-packages/ts/model_loader.py", line 125, in load 2025-01-20T17:55:38,935 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - entry_point, initialize_fn = self._get_class_entry_point(module) 2025-01-20T17:55:38,937 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-01-20T17:55:38,937 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - File "/usr/local/lib/python3.11/dist-packages/ts/model_loader.py", line 185, in _get_class_entry_point 2025-01-20T17:55:38,938 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - model_service = model_class() 2025-01-20T17:55:38,940 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - ^^^^^^^^^^^^^ 2025-01-20T17:55:38,941 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - File "/tmp/models/17fba6ef1a9d4697a83e026d885557fd/titan_handler.py", line 7, in __init__ 2025-01-20T17:55:38,942 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - self.device = 'cuda' if torch.cuda.is_available() else 'cpu' 2025-01-20T17:55:38,943 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - ^^^^^ 2025-01-20T17:55:38,943 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - NameError: name 'torch' is not defined 2025-01-20T17:55:38,944 [INFO ] W-9000-titan_1.0-stdout org.pytorch.serve.wlm.WorkerLifeCycle - Stopped Scanner - W-9000-titan_1.0-stdout 2025-01-20T17:55:38,944 [INFO ] W-9000-titan_1.0-stdout org.pytorch.serve.wlm.WorkerLifeCycle - Stopped Scanner - W-9000-titan_1.0-stdout 2025-01-20T17:55:38,962 [INFO ] W-9000-titan_1.0-stderr org.pytorch.serve.wlm.WorkerLifeCycle - Stopped Scanner - W-9000-titan_1.0-stderr 2025-01-20T17:55:38,962 [INFO ] W-9000-titan_1.0-stderr org.pytorch.serve.wlm.WorkerLifeCycle - Stopped Scanner - W-9000-titan_1.0-stderr 2025-01-20T17:55:39,933 [DEBUG] W-9000-titan_1.0 org.pytorch.serve.wlm.WorkerLifeCycle - Worker cmdline: [/usr/bin/python3, /usr/local/lib/python3.11/dist-packages/ts/model_service_worker.py, --sock-type, unix, --sock-name, /tmp/.ts.sock.9000, --metrics-config, /usr/local/lib/python3.11/dist-packages/ts/configs/metrics.yaml] 2025-01-20T17:55:39,933 [DEBUG] W-9000-titan_1.0 org.pytorch.serve.wlm.WorkerLifeCycle - Worker cmdline: [/usr/bin/python3, /usr/local/lib/python3.11/dist-packages/ts/model_service_worker.py, --sock-type, unix, --sock-name, /tmp/.ts.sock.9000, --metrics-config, /usr/local/lib/python3.11/dist-packages/ts/configs/metrics.yaml] 2025-01-20T17:55:42,269 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - s_name_part0=/tmp/.ts.sock, s_name_part1=9000, pid=8838 2025-01-20T17:55:42,274 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - Listening on port: /tmp/.ts.sock.9000 2025-01-20T17:55:42,286 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - Successfully loaded /usr/local/lib/python3.11/dist-packages/ts/configs/metrics.yaml. 2025-01-20T17:55:42,287 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - [PID]8838 2025-01-20T17:55:42,287 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - Torch worker started. 2025-01-20T17:55:42,288 [DEBUG] W-9000-titan_1.0 org.pytorch.serve.wlm.WorkerThread - W-9000-titan_1.0 State change WORKER_STOPPED -> WORKER_STARTED 2025-01-20T17:55:42,288 [DEBUG] W-9000-titan_1.0 org.pytorch.serve.wlm.WorkerThread - W-9000-titan_1.0 State change WORKER_STOPPED -> WORKER_STARTED 2025-01-20T17:55:42,288 [INFO ] W-9000-titan_1.0 org.pytorch.serve.wlm.WorkerThread - Connecting to: /tmp/.ts.sock.9000 2025-01-20T17:55:42,288 [INFO ] W-9000-titan_1.0 org.pytorch.serve.wlm.WorkerThread - Connecting to: /tmp/.ts.sock.9000 2025-01-20T17:55:42,289 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - Python runtime: 3.11.11 2025-01-20T17:55:42,291 [DEBUG] W-9000-titan_1.0 org.pytorch.serve.wlm.WorkerThread - Flushing req.cmd LOAD repeats 1 to backend at: 1737395742291 2025-01-20T17:55:42,291 [DEBUG] W-9000-titan_1.0 org.pytorch.serve.wlm.WorkerThread - Flushing req.cmd LOAD repeats 1 to backend at: 1737395742291 2025-01-20T17:55:42,292 [INFO ] W-9000-titan_1.0 org.pytorch.serve.wlm.WorkerThread - Looping backend response at: 1737395742292 2025-01-20T17:55:42,292 [INFO ] W-9000-titan_1.0 org.pytorch.serve.wlm.WorkerThread - Looping backend response at: 1737395742292 2025-01-20T17:55:42,295 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - Connection accepted: /tmp/.ts.sock.9000. 2025-01-20T17:55:42,319 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - model_name: titan, batchSize: 1 2025-01-20T17:55:42,321 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - Backend worker process died. 2025-01-20T17:55:42,322 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - Traceback (most recent call last): 2025-01-20T17:55:42,332 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - File "/usr/local/lib/python3.11/dist-packages/ts/model_service_worker.py", line 301, in 2025-01-20T17:55:42,341 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - worker.run_server() 2025-01-20T17:55:42,342 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - File "/usr/local/lib/python3.11/dist-packages/ts/model_service_worker.py", line 268, in run_server 2025-01-20T17:55:42,344 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - self.handle_connection(cl_socket) 2025-01-20T17:55:42,347 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - File "/usr/local/lib/python3.11/dist-packages/ts/model_service_worker.py", line 196, in handle_connection 2025-01-20T17:55:42,347 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - service, result, code = self.load_model(msg) 2025-01-20T17:55:42,348 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - ^^^^^^^^^^^^^^^^^^^^ 2025-01-20T17:55:42,349 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - File "/usr/local/lib/python3.11/dist-packages/ts/model_service_worker.py", line 133, in load_model 2025-01-20T17:55:42,350 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - service = model_loader.load( 2025-01-20T17:55:42,351 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - ^^^^^^^^^^^^^^^^^^ 2025-01-20T17:55:42,353 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - File "/usr/local/lib/python3.11/dist-packages/ts/model_loader.py", line 125, in load 2025-01-20T17:55:42,355 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - entry_point, initialize_fn = self._get_class_entry_point(module) 2025-01-20T17:55:42,360 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-01-20T17:55:42,361 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - File "/usr/local/lib/python3.11/dist-packages/ts/model_loader.py", line 185, in _get_class_entry_point 2025-01-20T17:55:42,362 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - model_service = model_class() 2025-01-20T17:55:42,363 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - ^^^^^^^^^^^^^ 2025-01-20T17:55:42,364 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - File "/tmp/models/17fba6ef1a9d4697a83e026d885557fd/titan_handler.py", line 7, in __init__ 2025-01-20T17:55:42,365 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - self.device = 'cuda' if torch.cuda.is_available() else 'cpu' 2025-01-20T17:55:42,366 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - ^^^^^ 2025-01-20T17:55:42,367 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - NameError: name 'torch' is not defined 2025-01-20T17:55:42,372 [INFO ] epollEventLoopGroup-5-3 org.pytorch.serve.wlm.WorkerThread - 9000 Worker disconnected. WORKER_STARTED 2025-01-20T17:55:42,372 [INFO ] epollEventLoopGroup-5-3 org.pytorch.serve.wlm.WorkerThread - 9000 Worker disconnected. WORKER_STARTED 2025-01-20T17:55:42,374 [DEBUG] W-9000-titan_1.0 org.pytorch.serve.wlm.WorkerThread - System state is : WORKER_STARTED 2025-01-20T17:55:42,374 [DEBUG] W-9000-titan_1.0 org.pytorch.serve.wlm.WorkerThread - System state is : WORKER_STARTED 2025-01-20T17:55:42,374 [DEBUG] W-9000-titan_1.0 org.pytorch.serve.wlm.WorkerThread - Backend worker monitoring thread interrupted or backend worker process died., startupTimeout:120sec java.lang.InterruptedException: null at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2056) ~[?:?] at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2133) ~[?:?] at java.util.concurrent.ArrayBlockingQueue.poll(ArrayBlockingQueue.java:432) ~[?:?] at org.pytorch.serve.wlm.WorkerThread.run(WorkerThread.java:234) [model-server.jar:?] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?] at java.lang.Thread.run(Thread.java:829) [?:?] 2025-01-20T17:55:42,374 [DEBUG] W-9000-titan_1.0 org.pytorch.serve.wlm.WorkerThread - Backend worker monitoring thread interrupted or backend worker process died., startupTimeout:120sec java.lang.InterruptedException: null at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2056) ~[?:?] at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2133) ~[?:?] at java.util.concurrent.ArrayBlockingQueue.poll(ArrayBlockingQueue.java:432) ~[?:?] at org.pytorch.serve.wlm.WorkerThread.run(WorkerThread.java:234) [model-server.jar:?] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?] at java.lang.Thread.run(Thread.java:829) [?:?] 2025-01-20T17:55:42,374 [WARN ] W-9000-titan_1.0 org.pytorch.serve.wlm.BatchAggregator - Load model failed: titan, error: Worker died. 2025-01-20T17:55:42,374 [WARN ] W-9000-titan_1.0 org.pytorch.serve.wlm.BatchAggregator - Load model failed: titan, error: Worker died. 2025-01-20T17:55:42,375 [DEBUG] W-9000-titan_1.0 org.pytorch.serve.wlm.WorkerThread - W-9000-titan_1.0 State change WORKER_STARTED -> WORKER_STOPPED 2025-01-20T17:55:42,375 [DEBUG] W-9000-titan_1.0 org.pytorch.serve.wlm.WorkerThread - W-9000-titan_1.0 State change WORKER_STARTED -> WORKER_STOPPED 2025-01-20T17:55:42,375 [WARN ] W-9000-titan_1.0 org.pytorch.serve.wlm.WorkerThread - Auto recovery failed again 2025-01-20T17:55:42,375 [WARN ] W-9000-titan_1.0 org.pytorch.serve.wlm.WorkerThread - Auto recovery failed again 2025-01-20T17:55:42,375 [INFO ] W-9000-titan_1.0 org.pytorch.serve.wlm.WorkerThread - Retry worker: 9000 in 2 seconds. 2025-01-20T17:55:42,375 [INFO ] W-9000-titan_1.0 org.pytorch.serve.wlm.WorkerThread - Retry worker: 9000 in 2 seconds. 2025-01-20T17:55:42,414 [INFO ] W-9000-titan_1.0-stderr org.pytorch.serve.wlm.WorkerLifeCycle - Stopped Scanner - W-9000-titan_1.0-stderr 2025-01-20T17:55:42,414 [INFO ] W-9000-titan_1.0-stderr org.pytorch.serve.wlm.WorkerLifeCycle - Stopped Scanner - W-9000-titan_1.0-stderr 2025-01-20T17:55:42,415 [INFO ] W-9000-titan_1.0-stdout org.pytorch.serve.wlm.WorkerLifeCycle - Stopped Scanner - W-9000-titan_1.0-stdout 2025-01-20T17:55:42,415 [INFO ] W-9000-titan_1.0-stdout org.pytorch.serve.wlm.WorkerLifeCycle - Stopped Scanner - W-9000-titan_1.0-stdout 2025-01-20T17:55:44,376 [DEBUG] W-9000-titan_1.0 org.pytorch.serve.wlm.WorkerLifeCycle - Worker cmdline: [/usr/bin/python3, /usr/local/lib/python3.11/dist-packages/ts/model_service_worker.py, --sock-type, unix, --sock-name, /tmp/.ts.sock.9000, --metrics-config, /usr/local/lib/python3.11/dist-packages/ts/configs/metrics.yaml] 2025-01-20T17:55:44,376 [DEBUG] W-9000-titan_1.0 org.pytorch.serve.wlm.WorkerLifeCycle - Worker cmdline: [/usr/bin/python3, /usr/local/lib/python3.11/dist-packages/ts/model_service_worker.py, --sock-type, unix, --sock-name, /tmp/.ts.sock.9000, --metrics-config, /usr/local/lib/python3.11/dist-packages/ts/configs/metrics.yaml] 2025-01-20T17:55:45,927 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - s_name_part0=/tmp/.ts.sock, s_name_part1=9000, pid=8868 2025-01-20T17:55:45,932 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - Listening on port: /tmp/.ts.sock.9000 2025-01-20T17:55:45,937 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - Successfully loaded /usr/local/lib/python3.11/dist-packages/ts/configs/metrics.yaml. 2025-01-20T17:55:45,938 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - [PID]8868 2025-01-20T17:55:45,938 [DEBUG] W-9000-titan_1.0 org.pytorch.serve.wlm.WorkerThread - W-9000-titan_1.0 State change WORKER_STOPPED -> WORKER_STARTED 2025-01-20T17:55:45,938 [DEBUG] W-9000-titan_1.0 org.pytorch.serve.wlm.WorkerThread - W-9000-titan_1.0 State change WORKER_STOPPED -> WORKER_STARTED 2025-01-20T17:55:45,938 [INFO ] W-9000-titan_1.0 org.pytorch.serve.wlm.WorkerThread - Connecting to: /tmp/.ts.sock.9000 2025-01-20T17:55:45,938 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - Torch worker started. 2025-01-20T17:55:45,938 [INFO ] W-9000-titan_1.0 org.pytorch.serve.wlm.WorkerThread - Connecting to: /tmp/.ts.sock.9000 2025-01-20T17:55:45,939 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - Python runtime: 3.11.11 2025-01-20T17:55:45,941 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - Connection accepted: /tmp/.ts.sock.9000. 2025-01-20T17:55:45,942 [DEBUG] W-9000-titan_1.0 org.pytorch.serve.wlm.WorkerThread - Flushing req.cmd LOAD repeats 1 to backend at: 1737395745942 2025-01-20T17:55:45,942 [DEBUG] W-9000-titan_1.0 org.pytorch.serve.wlm.WorkerThread - Flushing req.cmd LOAD repeats 1 to backend at: 1737395745942 2025-01-20T17:55:45,943 [INFO ] W-9000-titan_1.0 org.pytorch.serve.wlm.WorkerThread - Looping backend response at: 1737395745943 2025-01-20T17:55:45,943 [INFO ] W-9000-titan_1.0 org.pytorch.serve.wlm.WorkerThread - Looping backend response at: 1737395745943 2025-01-20T17:55:45,971 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - model_name: titan, batchSize: 1 2025-01-20T17:55:45,971 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - Backend worker process died. 2025-01-20T17:55:45,971 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - Traceback (most recent call last): 2025-01-20T17:55:45,971 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - File "/usr/local/lib/python3.11/dist-packages/ts/model_service_worker.py", line 301, in 2025-01-20T17:55:45,971 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - worker.run_server() 2025-01-20T17:55:45,971 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - File "/usr/local/lib/python3.11/dist-packages/ts/model_service_worker.py", line 268, in run_server 2025-01-20T17:55:45,972 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - self.handle_connection(cl_socket) 2025-01-20T17:55:45,972 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - File "/usr/local/lib/python3.11/dist-packages/ts/model_service_worker.py", line 196, in handle_connection 2025-01-20T17:55:45,972 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - service, result, code = self.load_model(msg) 2025-01-20T17:55:45,972 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - ^^^^^^^^^^^^^^^^^^^^ 2025-01-20T17:55:45,972 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - File "/usr/local/lib/python3.11/dist-packages/ts/model_service_worker.py", line 133, in load_model 2025-01-20T17:55:45,972 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - service = model_loader.load( 2025-01-20T17:55:45,972 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - ^^^^^^^^^^^^^^^^^^ 2025-01-20T17:55:45,973 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - File "/usr/local/lib/python3.11/dist-packages/ts/model_loader.py", line 125, in load 2025-01-20T17:55:45,973 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - entry_point, initialize_fn = self._get_class_entry_point(module) 2025-01-20T17:55:45,973 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-01-20T17:55:45,973 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - File "/usr/local/lib/python3.11/dist-packages/ts/model_loader.py", line 185, in _get_class_entry_point 2025-01-20T17:55:45,973 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - model_service = model_class() 2025-01-20T17:55:45,980 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - ^^^^^^^^^^^^^ 2025-01-20T17:55:45,981 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - File "/tmp/models/17fba6ef1a9d4697a83e026d885557fd/titan_handler.py", line 7, in __init__ 2025-01-20T17:55:45,982 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - self.device = 'cuda' if torch.cuda.is_available() else 'cpu' 2025-01-20T17:55:45,982 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - ^^^^^ 2025-01-20T17:55:45,983 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - NameError: name 'torch' is not defined 2025-01-20T17:55:45,984 [INFO ] epollEventLoopGroup-5-4 org.pytorch.serve.wlm.WorkerThread - 9000 Worker disconnected. WORKER_STARTED 2025-01-20T17:55:45,984 [INFO ] epollEventLoopGroup-5-4 org.pytorch.serve.wlm.WorkerThread - 9000 Worker disconnected. WORKER_STARTED 2025-01-20T17:55:45,987 [DEBUG] W-9000-titan_1.0 org.pytorch.serve.wlm.WorkerThread - System state is : WORKER_STARTED 2025-01-20T17:55:45,987 [DEBUG] W-9000-titan_1.0 org.pytorch.serve.wlm.WorkerThread - System state is : WORKER_STARTED 2025-01-20T17:55:45,988 [DEBUG] W-9000-titan_1.0 org.pytorch.serve.wlm.WorkerThread - Backend worker monitoring thread interrupted or backend worker process died., startupTimeout:120sec java.lang.InterruptedException: null at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2056) ~[?:?] at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2133) ~[?:?] at java.util.concurrent.ArrayBlockingQueue.poll(ArrayBlockingQueue.java:432) ~[?:?] at org.pytorch.serve.wlm.WorkerThread.run(WorkerThread.java:234) [model-server.jar:?] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?] at java.lang.Thread.run(Thread.java:829) [?:?] 2025-01-20T17:55:45,988 [DEBUG] W-9000-titan_1.0 org.pytorch.serve.wlm.WorkerThread - Backend worker monitoring thread interrupted or backend worker process died., startupTimeout:120sec java.lang.InterruptedException: null at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2056) ~[?:?] at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2133) ~[?:?] at java.util.concurrent.ArrayBlockingQueue.poll(ArrayBlockingQueue.java:432) ~[?:?] at org.pytorch.serve.wlm.WorkerThread.run(WorkerThread.java:234) [model-server.jar:?] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?] at java.lang.Thread.run(Thread.java:829) [?:?] 2025-01-20T17:55:45,988 [WARN ] W-9000-titan_1.0 org.pytorch.serve.wlm.BatchAggregator - Load model failed: titan, error: Worker died. 2025-01-20T17:55:45,988 [WARN ] W-9000-titan_1.0 org.pytorch.serve.wlm.BatchAggregator - Load model failed: titan, error: Worker died. 2025-01-20T17:55:45,988 [DEBUG] W-9000-titan_1.0 org.pytorch.serve.wlm.WorkerThread - W-9000-titan_1.0 State change WORKER_STARTED -> WORKER_STOPPED 2025-01-20T17:55:45,988 [DEBUG] W-9000-titan_1.0 org.pytorch.serve.wlm.WorkerThread - W-9000-titan_1.0 State change WORKER_STARTED -> WORKER_STOPPED 2025-01-20T17:55:45,988 [WARN ] W-9000-titan_1.0 org.pytorch.serve.wlm.WorkerThread - Auto recovery failed again 2025-01-20T17:55:45,988 [WARN ] W-9000-titan_1.0 org.pytorch.serve.wlm.WorkerThread - Auto recovery failed again 2025-01-20T17:55:45,989 [INFO ] W-9000-titan_1.0 org.pytorch.serve.wlm.WorkerThread - Retry worker: 9000 in 3 seconds. 2025-01-20T17:55:45,989 [INFO ] W-9000-titan_1.0 org.pytorch.serve.wlm.WorkerThread - Retry worker: 9000 in 3 seconds. 2025-01-20T17:55:46,012 [INFO ] W-9000-titan_1.0-stderr org.pytorch.serve.wlm.WorkerLifeCycle - Stopped Scanner - W-9000-titan_1.0-stderr 2025-01-20T17:55:46,012 [INFO ] W-9000-titan_1.0-stdout org.pytorch.serve.wlm.WorkerLifeCycle - Stopped Scanner - W-9000-titan_1.0-stdout 2025-01-20T17:55:46,012 [INFO ] W-9000-titan_1.0-stderr org.pytorch.serve.wlm.WorkerLifeCycle - Stopped Scanner - W-9000-titan_1.0-stderr 2025-01-20T17:55:46,012 [INFO ] W-9000-titan_1.0-stdout org.pytorch.serve.wlm.WorkerLifeCycle - Stopped Scanner - W-9000-titan_1.0-stdout 2025-01-20T17:55:48,989 [DEBUG] W-9000-titan_1.0 org.pytorch.serve.wlm.WorkerLifeCycle - Worker cmdline: [/usr/bin/python3, /usr/local/lib/python3.11/dist-packages/ts/model_service_worker.py, --sock-type, unix, --sock-name, /tmp/.ts.sock.9000, --metrics-config, /usr/local/lib/python3.11/dist-packages/ts/configs/metrics.yaml] 2025-01-20T17:55:48,989 [DEBUG] W-9000-titan_1.0 org.pytorch.serve.wlm.WorkerLifeCycle - Worker cmdline: [/usr/bin/python3, /usr/local/lib/python3.11/dist-packages/ts/model_service_worker.py, --sock-type, unix, --sock-name, /tmp/.ts.sock.9000, --metrics-config, /usr/local/lib/python3.11/dist-packages/ts/configs/metrics.yaml] 2025-01-20T17:55:50,540 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - s_name_part0=/tmp/.ts.sock, s_name_part1=9000, pid=8890 2025-01-20T17:55:50,542 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - Listening on port: /tmp/.ts.sock.9000 2025-01-20T17:55:50,549 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - Successfully loaded /usr/local/lib/python3.11/dist-packages/ts/configs/metrics.yaml. 2025-01-20T17:55:50,550 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - [PID]8890 2025-01-20T17:55:50,550 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - Torch worker started. 2025-01-20T17:55:50,550 [DEBUG] W-9000-titan_1.0 org.pytorch.serve.wlm.WorkerThread - W-9000-titan_1.0 State change WORKER_STOPPED -> WORKER_STARTED 2025-01-20T17:55:50,550 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - Python runtime: 3.11.11 2025-01-20T17:55:50,550 [DEBUG] W-9000-titan_1.0 org.pytorch.serve.wlm.WorkerThread - W-9000-titan_1.0 State change WORKER_STOPPED -> WORKER_STARTED 2025-01-20T17:55:50,550 [INFO ] W-9000-titan_1.0 org.pytorch.serve.wlm.WorkerThread - Connecting to: /tmp/.ts.sock.9000 2025-01-20T17:55:50,550 [INFO ] W-9000-titan_1.0 org.pytorch.serve.wlm.WorkerThread - Connecting to: /tmp/.ts.sock.9000 2025-01-20T17:55:50,554 [DEBUG] W-9000-titan_1.0 org.pytorch.serve.wlm.WorkerThread - Flushing req.cmd LOAD repeats 1 to backend at: 1737395750554 2025-01-20T17:55:50,554 [DEBUG] W-9000-titan_1.0 org.pytorch.serve.wlm.WorkerThread - Flushing req.cmd LOAD repeats 1 to backend at: 1737395750554 2025-01-20T17:55:50,555 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - Connection accepted: /tmp/.ts.sock.9000. 2025-01-20T17:55:50,556 [INFO ] W-9000-titan_1.0 org.pytorch.serve.wlm.WorkerThread - Looping backend response at: 1737395750556 2025-01-20T17:55:50,556 [INFO ] W-9000-titan_1.0 org.pytorch.serve.wlm.WorkerThread - Looping backend response at: 1737395750556 2025-01-20T17:55:50,558 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - model_name: titan, batchSize: 1 2025-01-20T17:55:50,559 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - Backend worker process died. 2025-01-20T17:55:50,559 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - Traceback (most recent call last): 2025-01-20T17:55:50,559 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - File "/usr/local/lib/python3.11/dist-packages/ts/model_service_worker.py", line 301, in 2025-01-20T17:55:50,560 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - worker.run_server() 2025-01-20T17:55:50,560 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - File "/usr/local/lib/python3.11/dist-packages/ts/model_service_worker.py", line 268, in run_server 2025-01-20T17:55:50,560 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - self.handle_connection(cl_socket) 2025-01-20T17:55:50,560 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - File "/usr/local/lib/python3.11/dist-packages/ts/model_service_worker.py", line 196, in handle_connection 2025-01-20T17:55:50,560 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - service, result, code = self.load_model(msg) 2025-01-20T17:55:50,560 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - ^^^^^^^^^^^^^^^^^^^^ 2025-01-20T17:55:50,560 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - File "/usr/local/lib/python3.11/dist-packages/ts/model_service_worker.py", line 133, in load_model 2025-01-20T17:55:50,560 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - service = model_loader.load( 2025-01-20T17:55:50,560 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - ^^^^^^^^^^^^^^^^^^ 2025-01-20T17:55:50,561 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - File "/usr/local/lib/python3.11/dist-packages/ts/model_loader.py", line 125, in load 2025-01-20T17:55:50,561 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - entry_point, initialize_fn = self._get_class_entry_point(module) 2025-01-20T17:55:50,561 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-01-20T17:55:50,561 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - File "/usr/local/lib/python3.11/dist-packages/ts/model_loader.py", line 185, in _get_class_entry_point 2025-01-20T17:55:50,561 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - model_service = model_class() 2025-01-20T17:55:50,561 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - ^^^^^^^^^^^^^ 2025-01-20T17:55:50,564 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - File "/tmp/models/17fba6ef1a9d4697a83e026d885557fd/titan_handler.py", line 7, in __init__ 2025-01-20T17:55:50,564 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - self.device = 'cuda' if torch.cuda.is_available() else 'cpu' 2025-01-20T17:55:50,564 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - ^^^^^ 2025-01-20T17:55:50,564 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - NameError: name 'torch' is not defined 2025-01-20T17:55:50,564 [INFO ] epollEventLoopGroup-5-1 org.pytorch.serve.wlm.WorkerThread - 9000 Worker disconnected. WORKER_STARTED 2025-01-20T17:55:50,564 [INFO ] epollEventLoopGroup-5-1 org.pytorch.serve.wlm.WorkerThread - 9000 Worker disconnected. WORKER_STARTED 2025-01-20T17:55:50,565 [DEBUG] W-9000-titan_1.0 org.pytorch.serve.wlm.WorkerThread - System state is : WORKER_STARTED 2025-01-20T17:55:50,565 [DEBUG] W-9000-titan_1.0 org.pytorch.serve.wlm.WorkerThread - System state is : WORKER_STARTED 2025-01-20T17:55:50,566 [DEBUG] W-9000-titan_1.0 org.pytorch.serve.wlm.WorkerThread - Backend worker monitoring thread interrupted or backend worker process died., startupTimeout:120sec java.lang.InterruptedException: null at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2056) ~[?:?] at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2133) ~[?:?] at java.util.concurrent.ArrayBlockingQueue.poll(ArrayBlockingQueue.java:432) ~[?:?] at org.pytorch.serve.wlm.WorkerThread.run(WorkerThread.java:234) [model-server.jar:?] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?] at java.lang.Thread.run(Thread.java:829) [?:?] 2025-01-20T17:55:50,566 [DEBUG] W-9000-titan_1.0 org.pytorch.serve.wlm.WorkerThread - Backend worker monitoring thread interrupted or backend worker process died., startupTimeout:120sec java.lang.InterruptedException: null at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2056) ~[?:?] at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2133) ~[?:?] at java.util.concurrent.ArrayBlockingQueue.poll(ArrayBlockingQueue.java:432) ~[?:?] at org.pytorch.serve.wlm.WorkerThread.run(WorkerThread.java:234) [model-server.jar:?] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?] at java.lang.Thread.run(Thread.java:829) [?:?] 2025-01-20T17:55:50,566 [WARN ] W-9000-titan_1.0 org.pytorch.serve.wlm.BatchAggregator - Load model failed: titan, error: Worker died. 2025-01-20T17:55:50,566 [WARN ] W-9000-titan_1.0 org.pytorch.serve.wlm.BatchAggregator - Load model failed: titan, error: Worker died. 2025-01-20T17:55:50,566 [DEBUG] W-9000-titan_1.0 org.pytorch.serve.wlm.WorkerThread - W-9000-titan_1.0 State change WORKER_STARTED -> WORKER_STOPPED 2025-01-20T17:55:50,566 [DEBUG] W-9000-titan_1.0 org.pytorch.serve.wlm.WorkerThread - W-9000-titan_1.0 State change WORKER_STARTED -> WORKER_STOPPED 2025-01-20T17:55:50,566 [WARN ] W-9000-titan_1.0 org.pytorch.serve.wlm.WorkerThread - Auto recovery failed again 2025-01-20T17:55:50,566 [WARN ] W-9000-titan_1.0 org.pytorch.serve.wlm.WorkerThread - Auto recovery failed again 2025-01-20T17:55:50,567 [INFO ] W-9000-titan_1.0 org.pytorch.serve.wlm.WorkerThread - Retry worker: 9000 in 5 seconds. 2025-01-20T17:55:50,567 [INFO ] W-9000-titan_1.0 org.pytorch.serve.wlm.WorkerThread - Retry worker: 9000 in 5 seconds. 2025-01-20T17:55:50,599 [INFO ] W-9000-titan_1.0-stdout org.pytorch.serve.wlm.WorkerLifeCycle - Stopped Scanner - W-9000-titan_1.0-stdout 2025-01-20T17:55:50,599 [INFO ] W-9000-titan_1.0-stdout org.pytorch.serve.wlm.WorkerLifeCycle - Stopped Scanner - W-9000-titan_1.0-stdout 2025-01-20T17:55:50,600 [INFO ] W-9000-titan_1.0-stderr org.pytorch.serve.wlm.WorkerLifeCycle - Stopped Scanner - W-9000-titan_1.0-stderr 2025-01-20T17:55:50,600 [INFO ] W-9000-titan_1.0-stderr org.pytorch.serve.wlm.WorkerLifeCycle - Stopped Scanner - W-9000-titan_1.0-stderr 2025-01-20T17:55:55,567 [DEBUG] W-9000-titan_1.0 org.pytorch.serve.wlm.WorkerLifeCycle - Worker cmdline: [/usr/bin/python3, /usr/local/lib/python3.11/dist-packages/ts/model_service_worker.py, --sock-type, unix, --sock-name, /tmp/.ts.sock.9000, --metrics-config, /usr/local/lib/python3.11/dist-packages/ts/configs/metrics.yaml] 2025-01-20T17:55:55,567 [DEBUG] W-9000-titan_1.0 org.pytorch.serve.wlm.WorkerLifeCycle - Worker cmdline: [/usr/bin/python3, /usr/local/lib/python3.11/dist-packages/ts/model_service_worker.py, --sock-type, unix, --sock-name, /tmp/.ts.sock.9000, --metrics-config, /usr/local/lib/python3.11/dist-packages/ts/configs/metrics.yaml] 2025-01-20T17:55:57,130 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - s_name_part0=/tmp/.ts.sock, s_name_part1=9000, pid=8925 2025-01-20T17:55:57,133 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - Listening on port: /tmp/.ts.sock.9000 2025-01-20T17:55:57,140 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - Successfully loaded /usr/local/lib/python3.11/dist-packages/ts/configs/metrics.yaml. 2025-01-20T17:55:57,140 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - [PID]8925 2025-01-20T17:55:57,141 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - Torch worker started. 2025-01-20T17:55:57,141 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - Python runtime: 3.11.11 2025-01-20T17:55:57,141 [DEBUG] W-9000-titan_1.0 org.pytorch.serve.wlm.WorkerThread - W-9000-titan_1.0 State change WORKER_STOPPED -> WORKER_STARTED 2025-01-20T17:55:57,141 [DEBUG] W-9000-titan_1.0 org.pytorch.serve.wlm.WorkerThread - W-9000-titan_1.0 State change WORKER_STOPPED -> WORKER_STARTED 2025-01-20T17:55:57,141 [INFO ] W-9000-titan_1.0 org.pytorch.serve.wlm.WorkerThread - Connecting to: /tmp/.ts.sock.9000 2025-01-20T17:55:57,141 [INFO ] W-9000-titan_1.0 org.pytorch.serve.wlm.WorkerThread - Connecting to: /tmp/.ts.sock.9000 2025-01-20T17:55:57,143 [DEBUG] W-9000-titan_1.0 org.pytorch.serve.wlm.WorkerThread - Flushing req.cmd LOAD repeats 1 to backend at: 1737395757142 2025-01-20T17:55:57,143 [DEBUG] W-9000-titan_1.0 org.pytorch.serve.wlm.WorkerThread - Flushing req.cmd LOAD repeats 1 to backend at: 1737395757142 2025-01-20T17:55:57,143 [INFO ] W-9000-titan_1.0 org.pytorch.serve.wlm.WorkerThread - Looping backend response at: 1737395757143 2025-01-20T17:55:57,143 [INFO ] W-9000-titan_1.0 org.pytorch.serve.wlm.WorkerThread - Looping backend response at: 1737395757143 2025-01-20T17:55:57,144 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - Connection accepted: /tmp/.ts.sock.9000. 2025-01-20T17:55:57,145 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - model_name: titan, batchSize: 1 2025-01-20T17:55:57,147 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - Backend worker process died. 2025-01-20T17:55:57,148 [INFO ] epollEventLoopGroup-5-2 org.pytorch.serve.wlm.WorkerThread - 9000 Worker disconnected. WORKER_STARTED 2025-01-20T17:55:57,147 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - Traceback (most recent call last): 2025-01-20T17:55:57,148 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - File "/usr/local/lib/python3.11/dist-packages/ts/model_service_worker.py", line 301, in 2025-01-20T17:55:57,148 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - worker.run_server() 2025-01-20T17:55:57,148 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - File "/usr/local/lib/python3.11/dist-packages/ts/model_service_worker.py", line 268, in run_server 2025-01-20T17:55:57,149 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - self.handle_connection(cl_socket) 2025-01-20T17:55:57,150 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - File "/usr/local/lib/python3.11/dist-packages/ts/model_service_worker.py", line 196, in handle_connection 2025-01-20T17:55:57,150 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - service, result, code = self.load_model(msg) 2025-01-20T17:55:57,148 [INFO ] epollEventLoopGroup-5-2 org.pytorch.serve.wlm.WorkerThread - 9000 Worker disconnected. WORKER_STARTED 2025-01-20T17:55:57,151 [DEBUG] W-9000-titan_1.0 org.pytorch.serve.wlm.WorkerThread - System state is : WORKER_STARTED 2025-01-20T17:55:57,151 [DEBUG] W-9000-titan_1.0 org.pytorch.serve.wlm.WorkerThread - System state is : WORKER_STARTED 2025-01-20T17:55:57,151 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - ^^^^^^^^^^^^^^^^^^^^ 2025-01-20T17:55:57,151 [DEBUG] W-9000-titan_1.0 org.pytorch.serve.wlm.WorkerThread - Backend worker monitoring thread interrupted or backend worker process died., startupTimeout:120sec java.lang.InterruptedException: null at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2056) ~[?:?] at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2133) ~[?:?] at java.util.concurrent.ArrayBlockingQueue.poll(ArrayBlockingQueue.java:432) ~[?:?] at org.pytorch.serve.wlm.WorkerThread.run(WorkerThread.java:234) [model-server.jar:?] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?] at java.lang.Thread.run(Thread.java:829) [?:?] 2025-01-20T17:55:57,152 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - File "/usr/local/lib/python3.11/dist-packages/ts/model_service_worker.py", line 133, in load_model 2025-01-20T17:55:57,151 [DEBUG] W-9000-titan_1.0 org.pytorch.serve.wlm.WorkerThread - Backend worker monitoring thread interrupted or backend worker process died., startupTimeout:120sec java.lang.InterruptedException: null at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2056) ~[?:?] at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2133) ~[?:?] at java.util.concurrent.ArrayBlockingQueue.poll(ArrayBlockingQueue.java:432) ~[?:?] at org.pytorch.serve.wlm.WorkerThread.run(WorkerThread.java:234) [model-server.jar:?] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?] at java.lang.Thread.run(Thread.java:829) [?:?] 2025-01-20T17:55:57,156 [WARN ] W-9000-titan_1.0 org.pytorch.serve.wlm.BatchAggregator - Load model failed: titan, error: Worker died. 2025-01-20T17:55:57,156 [WARN ] W-9000-titan_1.0 org.pytorch.serve.wlm.BatchAggregator - Load model failed: titan, error: Worker died. 2025-01-20T17:55:57,153 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - service = model_loader.load( 2025-01-20T17:55:57,156 [DEBUG] W-9000-titan_1.0 org.pytorch.serve.wlm.WorkerThread - W-9000-titan_1.0 State change WORKER_STARTED -> WORKER_STOPPED 2025-01-20T17:55:57,156 [DEBUG] W-9000-titan_1.0 org.pytorch.serve.wlm.WorkerThread - W-9000-titan_1.0 State change WORKER_STARTED -> WORKER_STOPPED 2025-01-20T17:55:57,156 [WARN ] W-9000-titan_1.0 org.pytorch.serve.wlm.WorkerThread - Auto recovery failed again 2025-01-20T17:55:57,156 [WARN ] W-9000-titan_1.0 org.pytorch.serve.wlm.WorkerThread - Auto recovery failed again 2025-01-20T17:55:57,157 [INFO ] W-9000-titan_1.0 org.pytorch.serve.wlm.WorkerThread - Retry worker: 9000 in 8 seconds. 2025-01-20T17:55:57,157 [INFO ] W-9000-titan_1.0 org.pytorch.serve.wlm.WorkerThread - Retry worker: 9000 in 8 seconds. 2025-01-20T17:55:57,157 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - ^^^^^^^^^^^^^^^^^^ 2025-01-20T17:55:57,158 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - File "/usr/local/lib/python3.11/dist-packages/ts/model_loader.py", line 125, in load 2025-01-20T17:55:57,162 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - entry_point, initialize_fn = self._get_class_entry_point(module) 2025-01-20T17:55:57,162 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-01-20T17:55:57,162 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - File "/usr/local/lib/python3.11/dist-packages/ts/model_loader.py", line 185, in _get_class_entry_point 2025-01-20T17:55:57,163 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - model_service = model_class() 2025-01-20T17:55:57,163 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - ^^^^^^^^^^^^^ 2025-01-20T17:55:57,163 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - File "/tmp/models/17fba6ef1a9d4697a83e026d885557fd/titan_handler.py", line 7, in __init__ 2025-01-20T17:55:57,163 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - self.device = 'cuda' if torch.cuda.is_available() else 'cpu' 2025-01-20T17:55:57,164 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - ^^^^^ 2025-01-20T17:55:57,166 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - NameError: name 'torch' is not defined 2025-01-20T17:55:57,167 [INFO ] W-9000-titan_1.0-stdout org.pytorch.serve.wlm.WorkerLifeCycle - Stopped Scanner - W-9000-titan_1.0-stdout 2025-01-20T17:55:57,167 [INFO ] W-9000-titan_1.0-stdout org.pytorch.serve.wlm.WorkerLifeCycle - Stopped Scanner - W-9000-titan_1.0-stdout 2025-01-20T17:55:57,187 [INFO ] W-9000-titan_1.0-stderr org.pytorch.serve.wlm.WorkerLifeCycle - Stopped Scanner - W-9000-titan_1.0-stderr 2025-01-20T17:55:57,187 [INFO ] W-9000-titan_1.0-stderr org.pytorch.serve.wlm.WorkerLifeCycle - Stopped Scanner - W-9000-titan_1.0-stderr 2025-01-20T17:56:05,158 [DEBUG] W-9000-titan_1.0 org.pytorch.serve.wlm.WorkerLifeCycle - Worker cmdline: [/usr/bin/python3, /usr/local/lib/python3.11/dist-packages/ts/model_service_worker.py, --sock-type, unix, --sock-name, /tmp/.ts.sock.9000, --metrics-config, /usr/local/lib/python3.11/dist-packages/ts/configs/metrics.yaml] 2025-01-20T17:56:05,158 [DEBUG] W-9000-titan_1.0 org.pytorch.serve.wlm.WorkerLifeCycle - Worker cmdline: [/usr/bin/python3, /usr/local/lib/python3.11/dist-packages/ts/model_service_worker.py, --sock-type, unix, --sock-name, /tmp/.ts.sock.9000, --metrics-config, /usr/local/lib/python3.11/dist-packages/ts/configs/metrics.yaml] 2025-01-20T17:56:06,875 [WARN ] W-9000-titan_1.0-stderr MODEL_LOG - Traceback (most recent call last): 2025-01-20T17:56:06,877 [WARN ] W-9000-titan_1.0-stderr MODEL_LOG - File "/usr/local/lib/python3.11/dist-packages/ts/model_service_worker.py", line 15, in 2025-01-20T17:56:06,879 [WARN ] W-9000-titan_1.0-stderr MODEL_LOG - from ts.async_service import AsyncService 2025-01-20T17:56:06,880 [WARN ] W-9000-titan_1.0-stderr MODEL_LOG - File "/usr/local/lib/python3.11/dist-packages/ts/async_service.py", line 14, in 2025-01-20T17:56:06,881 [WARN ] W-9000-titan_1.0-stderr MODEL_LOG - from ts.handler_utils.utils import create_predict_response 2025-01-20T17:56:06,882 [WARN ] W-9000-titan_1.0-stderr MODEL_LOG - File "/usr/local/lib/python3.11/dist-packages/ts/handler_utils/utils.py", line 5, in 2025-01-20T17:56:06,884 [WARN ] W-9000-titan_1.0-stderr MODEL_LOG - from ts.protocol.otf_message_handler import create_predict_response 2025-01-20T17:56:06,885 [WARN ] W-9000-titan_1.0-stderr MODEL_LOG - File "/usr/local/lib/python3.11/dist-packages/ts/protocol/otf_message_handler.py", line 14, in 2025-01-20T17:56:06,886 [WARN ] W-9000-titan_1.0-stderr MODEL_LOG - import torch 2025-01-20T17:56:06,886 [WARN ] W-9000-titan_1.0-stderr MODEL_LOG - File "/usr/local/lib/python3.11/dist-packages/torch/__init__.py", line 2486, in 2025-01-20T17:56:06,888 [WARN ] W-9000-titan_1.0-stderr MODEL_LOG - from torch import _meta_registrations 2025-01-20T17:56:06,889 [WARN ] W-9000-titan_1.0-stderr MODEL_LOG - File "/usr/local/lib/python3.11/dist-packages/torch/_meta_registrations.py", line 10, in 2025-01-20T17:56:06,892 [WARN ] W-9000-titan_1.0-stderr MODEL_LOG - from torch._decomp import ( 2025-01-20T17:56:06,894 [WARN ] W-9000-titan_1.0-stderr MODEL_LOG - File "/usr/local/lib/python3.11/dist-packages/torch/_decomp/__init__.py", line 249, in 2025-01-20T17:56:06,895 [WARN ] W-9000-titan_1.0-stderr MODEL_LOG - import torch._decomp.decompositions 2025-01-20T17:56:06,896 [WARN ] W-9000-titan_1.0-stderr MODEL_LOG - File "/usr/local/lib/python3.11/dist-packages/torch/_decomp/decompositions.py", line 15, in 2025-01-20T17:56:06,897 [WARN ] W-9000-titan_1.0-stderr MODEL_LOG - import torch._prims as prims 2025-01-20T17:56:06,898 [WARN ] W-9000-titan_1.0-stderr MODEL_LOG - File "/usr/local/lib/python3.11/dist-packages/torch/_prims/__init__.py", line 523, in 2025-01-20T17:56:06,899 [WARN ] W-9000-titan_1.0-stderr MODEL_LOG - abs = _make_elementwise_unary_prim( 2025-01-20T17:56:06,900 [WARN ] W-9000-titan_1.0-stderr MODEL_LOG - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-01-20T17:56:06,901 [WARN ] W-9000-titan_1.0-stderr MODEL_LOG - File "/usr/local/lib/python3.11/dist-packages/torch/_prims/__init__.py", line 491, in _make_elementwise_unary_prim 2025-01-20T17:56:06,902 [WARN ] W-9000-titan_1.0-stderr MODEL_LOG - return _make_prim( 2025-01-20T17:56:06,903 [WARN ] W-9000-titan_1.0-stderr MODEL_LOG - ^^^^^^^^^^^ 2025-01-20T17:56:06,903 [WARN ] W-9000-titan_1.0-stderr MODEL_LOG - File "/usr/local/lib/python3.11/dist-packages/torch/_prims/__init__.py", line 319, in _make_prim 2025-01-20T17:56:06,905 [WARN ] W-9000-titan_1.0-stderr MODEL_LOG - prim_def = torch.library.custom_op( 2025-01-20T17:56:06,906 [WARN ] W-9000-titan_1.0-stderr MODEL_LOG - ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-01-20T17:56:06,907 [WARN ] W-9000-titan_1.0-stderr MODEL_LOG - File "/usr/local/lib/python3.11/dist-packages/torch/_library/custom_ops.py", line 157, in custom_op 2025-01-20T17:56:06,908 [WARN ] W-9000-titan_1.0-stderr MODEL_LOG - return inner(fn) 2025-01-20T17:56:06,908 [WARN ] W-9000-titan_1.0-stderr MODEL_LOG - ^^^^^^^^^ 2025-01-20T17:56:06,912 [WARN ] W-9000-titan_1.0-stderr MODEL_LOG - File "/usr/local/lib/python3.11/dist-packages/torch/_library/custom_ops.py", line 138, in inner 2025-01-20T17:56:06,913 [WARN ] W-9000-titan_1.0-stderr MODEL_LOG - result = CustomOpDef(namespace, opname, schema_str, fn) 2025-01-20T17:56:06,914 [WARN ] W-9000-titan_1.0-stderr MODEL_LOG - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-01-20T17:56:06,915 [WARN ] W-9000-titan_1.0-stderr MODEL_LOG - File "/usr/local/lib/python3.11/dist-packages/torch/_library/custom_ops.py", line 186, in __init__ 2025-01-20T17:56:06,916 [WARN ] W-9000-titan_1.0-stderr MODEL_LOG - self._register_to_dispatcher() 2025-01-20T17:56:06,919 [WARN ] W-9000-titan_1.0-stderr MODEL_LOG - File "/usr/local/lib/python3.11/dist-packages/torch/_library/custom_ops.py", line 616, in _register_to_dispatcher 2025-01-20T17:56:06,920 [WARN ] W-9000-titan_1.0-stderr MODEL_LOG - lib._register_fake(self._name, fake_impl, _stacklevel=4) 2025-01-20T17:56:06,920 [WARN ] W-9000-titan_1.0-stderr MODEL_LOG - File "/usr/local/lib/python3.11/dist-packages/torch/library.py", line 163, in _register_fake 2025-01-20T17:56:06,921 [WARN ] W-9000-titan_1.0-stderr MODEL_LOG - source = torch._library.utils.get_source(_stacklevel + 1) 2025-01-20T17:56:06,921 [WARN ] W-9000-titan_1.0-stderr MODEL_LOG - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-01-20T17:56:06,923 [WARN ] W-9000-titan_1.0-stderr MODEL_LOG - File "/usr/local/lib/python3.11/dist-packages/torch/_library/utils.py", line 42, in get_source 2025-01-20T17:56:06,924 [WARN ] W-9000-titan_1.0-stderr MODEL_LOG - frame = inspect.getframeinfo(sys._getframe(stacklevel)) 2025-01-20T17:56:06,925 [WARN ] W-9000-titan_1.0-stderr MODEL_LOG - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-01-20T17:56:06,926 [WARN ] W-9000-titan_1.0-stderr MODEL_LOG - File "/usr/lib/python3.11/inspect.py", line 1688, in getframeinfo 2025-01-20T17:56:06,928 [WARN ] W-9000-titan_1.0-stderr MODEL_LOG - lines, lnum = findsource(frame) 2025-01-20T17:56:06,928 [WARN ] W-9000-titan_1.0-stderr MODEL_LOG - ^^^^^^^^^^^^^^^^^ 2025-01-20T17:56:06,930 [WARN ] W-9000-titan_1.0-stderr MODEL_LOG - File "/usr/lib/python3.11/inspect.py", line 1071, in findsource 2025-01-20T17:56:06,931 [WARN ] W-9000-titan_1.0-stderr MODEL_LOG - module = getmodule(object, file) 2025-01-20T17:56:06,942 [WARN ] W-9000-titan_1.0-stderr MODEL_LOG - ^^^^^^^^^^^^^^^^^^^^^^^ 2025-01-20T17:56:06,942 [WARN ] W-9000-titan_1.0-stderr MODEL_LOG - File "/usr/lib/python3.11/inspect.py", line 997, in getmodule 2025-01-20T17:56:06,944 [WARN ] W-9000-titan_1.0-stderr MODEL_LOG - os.path.realpath(f)] = module.__name__ 2025-01-20T17:56:06,945 [WARN ] W-9000-titan_1.0-stderr MODEL_LOG - ^^^^^^^^^^^^^^^^^^^ 2025-01-20T17:56:06,945 [WARN ] W-9000-titan_1.0-stderr MODEL_LOG - File "", line 416, in realpath 2025-01-20T17:56:06,947 [WARN ] W-9000-titan_1.0-stderr MODEL_LOG - File "", line 451, in _joinrealpath 2025-01-20T17:56:06,948 [WARN ] W-9000-titan_1.0-stderr MODEL_LOG - KeyboardInterrupt 2025-01-20T18:07:55,019 [INFO ] main org.pytorch.serve.servingsdk.impl.PluginsManager - Loading snapshot serializer plugin... 2025-01-20T18:07:55,019 [INFO ] main org.pytorch.serve.servingsdk.impl.PluginsManager - Loading snapshot serializer plugin... 2025-01-20T18:07:55,170 [DEBUG] main org.pytorch.serve.util.ConfigManager - xpu-smi not available or failed: Cannot run program "xpu-smi": error=2, No such file or directory 2025-01-20T18:07:55,170 [DEBUG] main org.pytorch.serve.util.ConfigManager - xpu-smi not available or failed: Cannot run program "xpu-smi": error=2, No such file or directory 2025-01-20T18:07:55,172 [WARN ] main org.pytorch.serve.util.ConfigManager - Your torchserve instance can access any URL to load models. When deploying to production, make sure to limit the set of allowed_urls in config.properties 2025-01-20T18:07:55,172 [WARN ] main org.pytorch.serve.util.ConfigManager - Your torchserve instance can access any URL to load models. When deploying to production, make sure to limit the set of allowed_urls in config.properties 2025-01-20T18:07:55,242 [INFO ] main org.pytorch.serve.servingsdk.impl.PluginsManager - Initializing plugins manager... 2025-01-20T18:07:55,242 [INFO ] main org.pytorch.serve.servingsdk.impl.PluginsManager - Initializing plugins manager... 2025-01-20T18:07:55,421 [INFO ] main org.pytorch.serve.metrics.configuration.MetricConfiguration - Successfully loaded metrics configuration from /usr/local/lib/python3.11/dist-packages/ts/configs/metrics.yaml 2025-01-20T18:07:55,421 [INFO ] main org.pytorch.serve.metrics.configuration.MetricConfiguration - Successfully loaded metrics configuration from /usr/local/lib/python3.11/dist-packages/ts/configs/metrics.yaml 2025-01-20T18:07:55,545 [INFO ] main org.pytorch.serve.ModelServer - Torchserve version: 0.12.0 TS Home: /usr/local/lib/python3.11/dist-packages Current directory: /content Temp directory: /tmp Metrics config path: /usr/local/lib/python3.11/dist-packages/ts/configs/metrics.yaml Number of GPUs: 1 Number of CPUs: 2 Max heap size: 3246 M Python executable: /usr/bin/python3 Config file: logs/config/20250120175606986-shutdown.cfg Inference address: http://127.0.0.1:8083 Management address: http://127.0.0.1:8084 Metrics address: http://127.0.0.1:8085 Model Store: /content/model_store Initial Models: titan=titan.mar Log dir: /content/logs Metrics dir: /content/logs Netty threads: 0 Netty client threads: 0 Default workers per model: 1 Blacklist Regex: N/A Maximum Response Size: 6553500 Maximum Request Size: 6553500 Limit Maximum Image Pixels: true Prefer direct buffer: false Allowed Urls: [file://.*|http(s)?://.*] Custom python dependency for model allowed: false Enable metrics API: true Metrics mode: LOG Disable system metrics: false Workflow Store: /content/model_store CPP log config: N/A Model config: N/A System metrics command: default Model API enabled: false 2025-01-20T18:07:55,545 [INFO ] main org.pytorch.serve.ModelServer - Torchserve version: 0.12.0 TS Home: /usr/local/lib/python3.11/dist-packages Current directory: /content Temp directory: /tmp Metrics config path: /usr/local/lib/python3.11/dist-packages/ts/configs/metrics.yaml Number of GPUs: 1 Number of CPUs: 2 Max heap size: 3246 M Python executable: /usr/bin/python3 Config file: logs/config/20250120175606986-shutdown.cfg Inference address: http://127.0.0.1:8083 Management address: http://127.0.0.1:8084 Metrics address: http://127.0.0.1:8085 Model Store: /content/model_store Initial Models: titan=titan.mar Log dir: /content/logs Metrics dir: /content/logs Netty threads: 0 Netty client threads: 0 Default workers per model: 1 Blacklist Regex: N/A Maximum Response Size: 6553500 Maximum Request Size: 6553500 Limit Maximum Image Pixels: true Prefer direct buffer: false Allowed Urls: [file://.*|http(s)?://.*] Custom python dependency for model allowed: false Enable metrics API: true Metrics mode: LOG Disable system metrics: false Workflow Store: /content/model_store CPP log config: N/A Model config: N/A System metrics command: default Model API enabled: false 2025-01-20T18:07:55,574 [INFO ] main org.pytorch.serve.snapshot.SnapshotManager - Started restoring models from snapshot { "name": "20250120175606986-shutdown.cfg", "modelCount": 1, "created": 1737395766986, "models": { "titan": { "1.0": { "defaultVersion": true, "marName": "titan.mar", "minWorkers": 1, "maxWorkers": 1, "batchSize": 1, "maxBatchDelay": 100, "responseTimeout": 120, "startupTimeout": 120, "runtimeType": "python" } } } } 2025-01-20T18:07:55,574 [INFO ] main org.pytorch.serve.snapshot.SnapshotManager - Started restoring models from snapshot { "name": "20250120175606986-shutdown.cfg", "modelCount": 1, "created": 1737395766986, "models": { "titan": { "1.0": { "defaultVersion": true, "marName": "titan.mar", "minWorkers": 1, "maxWorkers": 1, "batchSize": 1, "maxBatchDelay": 100, "responseTimeout": 120, "startupTimeout": 120, "runtimeType": "python" } } } } 2025-01-20T18:07:55,600 [INFO ] main org.pytorch.serve.snapshot.SnapshotManager - Validating snapshot 20250120175606986-shutdown.cfg 2025-01-20T18:07:55,600 [INFO ] main org.pytorch.serve.snapshot.SnapshotManager - Validating snapshot 20250120175606986-shutdown.cfg 2025-01-20T18:07:55,605 [INFO ] main org.pytorch.serve.snapshot.SnapshotManager - Snapshot 20250120175606986-shutdown.cfg validated successfully 2025-01-20T18:07:55,605 [INFO ] main org.pytorch.serve.snapshot.SnapshotManager - Snapshot 20250120175606986-shutdown.cfg validated successfully 2025-01-20T18:08:29,088 [DEBUG] main org.pytorch.serve.wlm.ModelVersionedRefs - Adding new version 1.0 for model titan 2025-01-20T18:08:29,088 [DEBUG] main org.pytorch.serve.wlm.ModelVersionedRefs - Adding new version 1.0 for model titan 2025-01-20T18:08:29,090 [DEBUG] main org.pytorch.serve.wlm.ModelVersionedRefs - Setting default version to 1.0 for model titan 2025-01-20T18:08:29,090 [DEBUG] main org.pytorch.serve.wlm.ModelVersionedRefs - Setting default version to 1.0 for model titan 2025-01-20T18:08:29,091 [DEBUG] main org.pytorch.serve.wlm.ModelVersionedRefs - Setting default version to 1.0 for model titan 2025-01-20T18:08:29,091 [DEBUG] main org.pytorch.serve.wlm.ModelVersionedRefs - Setting default version to 1.0 for model titan 2025-01-20T18:08:29,092 [INFO ] main org.pytorch.serve.wlm.ModelManager - Model titan loaded. 2025-01-20T18:08:29,092 [INFO ] main org.pytorch.serve.wlm.ModelManager - Model titan loaded. 2025-01-20T18:08:29,093 [DEBUG] main org.pytorch.serve.wlm.ModelManager - updateModel: titan, count: 1 2025-01-20T18:08:29,093 [DEBUG] main org.pytorch.serve.wlm.ModelManager - updateModel: titan, count: 1 2025-01-20T18:08:29,107 [INFO ] main org.pytorch.serve.ModelServer - Initialize Inference server with: EpollServerSocketChannel. 2025-01-20T18:08:29,107 [INFO ] main org.pytorch.serve.ModelServer - Initialize Inference server with: EpollServerSocketChannel. 2025-01-20T18:08:29,123 [DEBUG] W-9000-titan_1.0 org.pytorch.serve.wlm.WorkerLifeCycle - Worker cmdline: [/usr/bin/python3, /usr/local/lib/python3.11/dist-packages/ts/model_service_worker.py, --sock-type, unix, --sock-name, /tmp/.ts.sock.9000, --metrics-config, /usr/local/lib/python3.11/dist-packages/ts/configs/metrics.yaml] 2025-01-20T18:08:29,123 [DEBUG] W-9000-titan_1.0 org.pytorch.serve.wlm.WorkerLifeCycle - Worker cmdline: [/usr/bin/python3, /usr/local/lib/python3.11/dist-packages/ts/model_service_worker.py, --sock-type, unix, --sock-name, /tmp/.ts.sock.9000, --metrics-config, /usr/local/lib/python3.11/dist-packages/ts/configs/metrics.yaml] 2025-01-20T18:08:29,328 [INFO ] main org.pytorch.serve.ModelServer - Inference API bind to: http://127.0.0.1:8083 2025-01-20T18:08:29,328 [INFO ] main org.pytorch.serve.ModelServer - Inference API bind to: http://127.0.0.1:8083 2025-01-20T18:08:29,329 [INFO ] main org.pytorch.serve.ModelServer - Initialize Management server with: EpollServerSocketChannel. 2025-01-20T18:08:29,329 [INFO ] main org.pytorch.serve.ModelServer - Initialize Management server with: EpollServerSocketChannel. 2025-01-20T18:08:29,336 [INFO ] main org.pytorch.serve.ModelServer - Management API bind to: http://127.0.0.1:8084 2025-01-20T18:08:29,336 [INFO ] main org.pytorch.serve.ModelServer - Management API bind to: http://127.0.0.1:8084 2025-01-20T18:08:29,336 [INFO ] main org.pytorch.serve.ModelServer - Initialize Metrics server with: EpollServerSocketChannel. 2025-01-20T18:08:29,336 [INFO ] main org.pytorch.serve.ModelServer - Initialize Metrics server with: EpollServerSocketChannel. 2025-01-20T18:08:29,338 [INFO ] main org.pytorch.serve.ModelServer - Metrics API bind to: http://127.0.0.1:8085 2025-01-20T18:08:29,338 [INFO ] main org.pytorch.serve.ModelServer - Metrics API bind to: http://127.0.0.1:8085 2025-01-20T18:08:30,156 [WARN ] pool-3-thread-1 org.pytorch.serve.metrics.MetricCollector - worker pid is not available yet. 2025-01-20T18:08:30,156 [WARN ] pool-3-thread-1 org.pytorch.serve.metrics.MetricCollector - worker pid is not available yet. 2025-01-20T18:08:30,219 [ERROR] Thread-1 org.pytorch.serve.metrics.MetricCollector - Traceback (most recent call last): File "/usr/local/lib/python3.11/dist-packages/ts/metrics/metric_collector.py", line 11, in from ts.metrics import system_metrics File "/usr/local/lib/python3.11/dist-packages/ts/__init__.py", line 10, in from . import version File "/usr/local/lib/python3.11/dist-packages/ts/version.py", line 5, in from pathlib import Path File "/usr/local/lib/python3.11/dist-packages/pathlib.py", line 10, in from collections import Sequence ImportError: cannot import name 'Sequence' from 'collections' (/usr/lib/python3.11/collections/__init__.py) 2025-01-20T18:08:30,219 [ERROR] Thread-1 org.pytorch.serve.metrics.MetricCollector - Traceback (most recent call last): File "/usr/local/lib/python3.11/dist-packages/ts/metrics/metric_collector.py", line 11, in from ts.metrics import system_metrics File "/usr/local/lib/python3.11/dist-packages/ts/__init__.py", line 10, in from . import version File "/usr/local/lib/python3.11/dist-packages/ts/version.py", line 5, in from pathlib import Path File "/usr/local/lib/python3.11/dist-packages/pathlib.py", line 10, in from collections import Sequence ImportError: cannot import name 'Sequence' from 'collections' (/usr/lib/python3.11/collections/__init__.py) 2025-01-20T18:08:31,851 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - s_name_part0=/tmp/.ts.sock, s_name_part1=9000, pid=12171 2025-01-20T18:08:31,854 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - Listening on port: /tmp/.ts.sock.9000 2025-01-20T18:08:31,860 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - Successfully loaded /usr/local/lib/python3.11/dist-packages/ts/configs/metrics.yaml. 2025-01-20T18:08:31,860 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - [PID]12171 2025-01-20T18:08:31,860 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - Torch worker started. 2025-01-20T18:08:31,861 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - Python runtime: 3.11.11 2025-01-20T18:08:31,861 [DEBUG] W-9000-titan_1.0 org.pytorch.serve.wlm.WorkerThread - W-9000-titan_1.0 State change null -> WORKER_STARTED 2025-01-20T18:08:31,861 [DEBUG] W-9000-titan_1.0 org.pytorch.serve.wlm.WorkerThread - W-9000-titan_1.0 State change null -> WORKER_STARTED 2025-01-20T18:08:31,868 [INFO ] W-9000-titan_1.0 org.pytorch.serve.wlm.WorkerThread - Connecting to: /tmp/.ts.sock.9000 2025-01-20T18:08:31,868 [INFO ] W-9000-titan_1.0 org.pytorch.serve.wlm.WorkerThread - Connecting to: /tmp/.ts.sock.9000 2025-01-20T18:08:31,877 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - Connection accepted: /tmp/.ts.sock.9000. 2025-01-20T18:08:31,883 [DEBUG] W-9000-titan_1.0 org.pytorch.serve.wlm.WorkerThread - Flushing req.cmd LOAD repeats 1 to backend at: 1737396511883 2025-01-20T18:08:31,883 [DEBUG] W-9000-titan_1.0 org.pytorch.serve.wlm.WorkerThread - Flushing req.cmd LOAD repeats 1 to backend at: 1737396511883 2025-01-20T18:08:31,887 [INFO ] W-9000-titan_1.0 org.pytorch.serve.wlm.WorkerThread - Looping backend response at: 1737396511887 2025-01-20T18:08:31,887 [INFO ] W-9000-titan_1.0 org.pytorch.serve.wlm.WorkerThread - Looping backend response at: 1737396511887 2025-01-20T18:08:31,929 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - model_name: titan, batchSize: 1 2025-01-20T18:08:31,934 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - Backend worker process died. 2025-01-20T18:08:31,935 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - Traceback (most recent call last): 2025-01-20T18:08:31,935 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - File "/usr/local/lib/python3.11/dist-packages/ts/model_service_worker.py", line 301, in 2025-01-20T18:08:31,935 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - worker.run_server() 2025-01-20T18:08:31,935 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - File "/usr/local/lib/python3.11/dist-packages/ts/model_service_worker.py", line 268, in run_server 2025-01-20T18:08:31,936 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - self.handle_connection(cl_socket) 2025-01-20T18:08:31,936 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - File "/usr/local/lib/python3.11/dist-packages/ts/model_service_worker.py", line 196, in handle_connection 2025-01-20T18:08:31,936 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - service, result, code = self.load_model(msg) 2025-01-20T18:08:31,936 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - ^^^^^^^^^^^^^^^^^^^^ 2025-01-20T18:08:31,937 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - File "/usr/local/lib/python3.11/dist-packages/ts/model_service_worker.py", line 133, in load_model 2025-01-20T18:08:31,937 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - service = model_loader.load( 2025-01-20T18:08:31,937 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - ^^^^^^^^^^^^^^^^^^ 2025-01-20T18:08:31,942 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - File "/usr/local/lib/python3.11/dist-packages/ts/model_loader.py", line 125, in load 2025-01-20T18:08:31,943 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - entry_point, initialize_fn = self._get_class_entry_point(module) 2025-01-20T18:08:31,944 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-01-20T18:08:31,944 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - File "/usr/local/lib/python3.11/dist-packages/ts/model_loader.py", line 178, in _get_class_entry_point 2025-01-20T18:08:31,946 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - raise ValueError( 2025-01-20T18:08:31,947 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - ValueError: Expected only one class in custom service code or a function entry point [, , , , ] 2025-01-20T18:08:31,949 [INFO ] epollEventLoopGroup-5-1 org.pytorch.serve.wlm.WorkerThread - 9000 Worker disconnected. WORKER_STARTED 2025-01-20T18:08:31,949 [INFO ] epollEventLoopGroup-5-1 org.pytorch.serve.wlm.WorkerThread - 9000 Worker disconnected. WORKER_STARTED 2025-01-20T18:08:31,951 [DEBUG] W-9000-titan_1.0 org.pytorch.serve.wlm.WorkerThread - System state is : WORKER_STARTED 2025-01-20T18:08:31,951 [DEBUG] W-9000-titan_1.0 org.pytorch.serve.wlm.WorkerThread - System state is : WORKER_STARTED 2025-01-20T18:08:31,951 [DEBUG] W-9000-titan_1.0 org.pytorch.serve.wlm.WorkerThread - Backend worker monitoring thread interrupted or backend worker process died., startupTimeout:120sec java.lang.InterruptedException: null at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2056) ~[?:?] at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2133) ~[?:?] at java.util.concurrent.ArrayBlockingQueue.poll(ArrayBlockingQueue.java:432) ~[?:?] at org.pytorch.serve.wlm.WorkerThread.run(WorkerThread.java:234) [model-server.jar:?] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) [?:?] at java.util.concurrent.FutureTask.run(FutureTask.java:264) [?:?] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?] at java.lang.Thread.run(Thread.java:829) [?:?] 2025-01-20T18:08:31,951 [DEBUG] W-9000-titan_1.0 org.pytorch.serve.wlm.WorkerThread - Backend worker monitoring thread interrupted or backend worker process died., startupTimeout:120sec java.lang.InterruptedException: null at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2056) ~[?:?] at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2133) ~[?:?] at java.util.concurrent.ArrayBlockingQueue.poll(ArrayBlockingQueue.java:432) ~[?:?] at org.pytorch.serve.wlm.WorkerThread.run(WorkerThread.java:234) [model-server.jar:?] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) [?:?] at java.util.concurrent.FutureTask.run(FutureTask.java:264) [?:?] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?] at java.lang.Thread.run(Thread.java:829) [?:?] 2025-01-20T18:08:31,984 [WARN ] W-9000-titan_1.0 org.pytorch.serve.wlm.BatchAggregator - Load model failed: titan, error: Worker died. 2025-01-20T18:08:31,984 [WARN ] W-9000-titan_1.0 org.pytorch.serve.wlm.BatchAggregator - Load model failed: titan, error: Worker died. 2025-01-20T18:08:31,984 [DEBUG] W-9000-titan_1.0 org.pytorch.serve.wlm.WorkerThread - W-9000-titan_1.0 State change WORKER_STARTED -> WORKER_STOPPED 2025-01-20T18:08:31,984 [DEBUG] W-9000-titan_1.0 org.pytorch.serve.wlm.WorkerThread - W-9000-titan_1.0 State change WORKER_STARTED -> WORKER_STOPPED 2025-01-20T18:08:31,985 [INFO ] W-9000-titan_1.0 org.pytorch.serve.wlm.WorkerThread - Auto recovery start timestamp: 1737396511985 2025-01-20T18:08:31,985 [INFO ] W-9000-titan_1.0 org.pytorch.serve.wlm.WorkerThread - Auto recovery start timestamp: 1737396511985 2025-01-20T18:08:31,987 [INFO ] W-9000-titan_1.0 org.pytorch.serve.wlm.WorkerThread - Retry worker: 9000 in 1 seconds. 2025-01-20T18:08:31,987 [INFO ] W-9000-titan_1.0 org.pytorch.serve.wlm.WorkerThread - Retry worker: 9000 in 1 seconds. 2025-01-20T18:08:32,012 [INFO ] W-9000-titan_1.0-stderr org.pytorch.serve.wlm.WorkerLifeCycle - Stopped Scanner - W-9000-titan_1.0-stderr 2025-01-20T18:08:32,012 [INFO ] W-9000-titan_1.0-stderr org.pytorch.serve.wlm.WorkerLifeCycle - Stopped Scanner - W-9000-titan_1.0-stderr 2025-01-20T18:08:32,015 [INFO ] W-9000-titan_1.0-stdout org.pytorch.serve.wlm.WorkerLifeCycle - Stopped Scanner - W-9000-titan_1.0-stdout 2025-01-20T18:08:32,015 [INFO ] W-9000-titan_1.0-stdout org.pytorch.serve.wlm.WorkerLifeCycle - Stopped Scanner - W-9000-titan_1.0-stdout 2025-01-20T18:08:32,988 [DEBUG] W-9000-titan_1.0 org.pytorch.serve.wlm.WorkerLifeCycle - Worker cmdline: [/usr/bin/python3, /usr/local/lib/python3.11/dist-packages/ts/model_service_worker.py, --sock-type, unix, --sock-name, /tmp/.ts.sock.9000, --metrics-config, /usr/local/lib/python3.11/dist-packages/ts/configs/metrics.yaml] 2025-01-20T18:08:32,988 [DEBUG] W-9000-titan_1.0 org.pytorch.serve.wlm.WorkerLifeCycle - Worker cmdline: [/usr/bin/python3, /usr/local/lib/python3.11/dist-packages/ts/model_service_worker.py, --sock-type, unix, --sock-name, /tmp/.ts.sock.9000, --metrics-config, /usr/local/lib/python3.11/dist-packages/ts/configs/metrics.yaml] 2025-01-20T18:08:34,550 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - s_name_part0=/tmp/.ts.sock, s_name_part1=9000, pid=12197 2025-01-20T18:08:34,554 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - Listening on port: /tmp/.ts.sock.9000 2025-01-20T18:08:34,559 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - Successfully loaded /usr/local/lib/python3.11/dist-packages/ts/configs/metrics.yaml. 2025-01-20T18:08:34,560 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - [PID]12197 2025-01-20T18:08:34,560 [DEBUG] W-9000-titan_1.0 org.pytorch.serve.wlm.WorkerThread - W-9000-titan_1.0 State change WORKER_STOPPED -> WORKER_STARTED 2025-01-20T18:08:34,560 [DEBUG] W-9000-titan_1.0 org.pytorch.serve.wlm.WorkerThread - W-9000-titan_1.0 State change WORKER_STOPPED -> WORKER_STARTED 2025-01-20T18:08:34,561 [INFO ] W-9000-titan_1.0 org.pytorch.serve.wlm.WorkerThread - Connecting to: /tmp/.ts.sock.9000 2025-01-20T18:08:34,561 [INFO ] W-9000-titan_1.0 org.pytorch.serve.wlm.WorkerThread - Connecting to: /tmp/.ts.sock.9000 2025-01-20T18:08:34,561 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - Torch worker started. 2025-01-20T18:08:34,562 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - Python runtime: 3.11.11 2025-01-20T18:08:34,565 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - Connection accepted: /tmp/.ts.sock.9000. 2025-01-20T18:08:34,567 [DEBUG] W-9000-titan_1.0 org.pytorch.serve.wlm.WorkerThread - Flushing req.cmd LOAD repeats 1 to backend at: 1737396514567 2025-01-20T18:08:34,567 [DEBUG] W-9000-titan_1.0 org.pytorch.serve.wlm.WorkerThread - Flushing req.cmd LOAD repeats 1 to backend at: 1737396514567 2025-01-20T18:08:34,568 [INFO ] W-9000-titan_1.0 org.pytorch.serve.wlm.WorkerThread - Looping backend response at: 1737396514568 2025-01-20T18:08:34,568 [INFO ] W-9000-titan_1.0 org.pytorch.serve.wlm.WorkerThread - Looping backend response at: 1737396514568 2025-01-20T18:08:34,593 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - model_name: titan, batchSize: 1 2025-01-20T18:08:34,594 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - Backend worker process died. 2025-01-20T18:08:34,595 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - Traceback (most recent call last): 2025-01-20T18:08:34,596 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - File "/usr/local/lib/python3.11/dist-packages/ts/model_service_worker.py", line 301, in 2025-01-20T18:08:34,596 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - worker.run_server() 2025-01-20T18:08:34,596 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - File "/usr/local/lib/python3.11/dist-packages/ts/model_service_worker.py", line 268, in run_server 2025-01-20T18:08:34,596 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - self.handle_connection(cl_socket) 2025-01-20T18:08:34,596 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - File "/usr/local/lib/python3.11/dist-packages/ts/model_service_worker.py", line 196, in handle_connection 2025-01-20T18:08:34,598 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - service, result, code = self.load_model(msg) 2025-01-20T18:08:34,598 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - ^^^^^^^^^^^^^^^^^^^^ 2025-01-20T18:08:34,598 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - File "/usr/local/lib/python3.11/dist-packages/ts/model_service_worker.py", line 133, in load_model 2025-01-20T18:08:34,599 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - service = model_loader.load( 2025-01-20T18:08:34,599 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - ^^^^^^^^^^^^^^^^^^ 2025-01-20T18:08:34,599 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - File "/usr/local/lib/python3.11/dist-packages/ts/model_loader.py", line 125, in load 2025-01-20T18:08:34,599 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - entry_point, initialize_fn = self._get_class_entry_point(module) 2025-01-20T18:08:34,599 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-01-20T18:08:34,606 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - File "/usr/local/lib/python3.11/dist-packages/ts/model_loader.py", line 178, in _get_class_entry_point 2025-01-20T18:08:34,606 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - raise ValueError( 2025-01-20T18:08:34,607 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - ValueError: Expected only one class in custom service code or a function entry point [, , , , ] 2025-01-20T18:08:34,609 [INFO ] epollEventLoopGroup-5-2 org.pytorch.serve.wlm.WorkerThread - 9000 Worker disconnected. WORKER_STARTED 2025-01-20T18:08:34,609 [INFO ] epollEventLoopGroup-5-2 org.pytorch.serve.wlm.WorkerThread - 9000 Worker disconnected. WORKER_STARTED 2025-01-20T18:08:34,614 [DEBUG] W-9000-titan_1.0 org.pytorch.serve.wlm.WorkerThread - System state is : WORKER_STARTED 2025-01-20T18:08:34,614 [DEBUG] W-9000-titan_1.0 org.pytorch.serve.wlm.WorkerThread - System state is : WORKER_STARTED 2025-01-20T18:08:34,614 [DEBUG] W-9000-titan_1.0 org.pytorch.serve.wlm.WorkerThread - Backend worker monitoring thread interrupted or backend worker process died., startupTimeout:120sec java.lang.InterruptedException: null at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2056) ~[?:?] at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2133) ~[?:?] at java.util.concurrent.ArrayBlockingQueue.poll(ArrayBlockingQueue.java:432) ~[?:?] at org.pytorch.serve.wlm.WorkerThread.run(WorkerThread.java:234) [model-server.jar:?] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?] at java.lang.Thread.run(Thread.java:829) [?:?] 2025-01-20T18:08:34,614 [DEBUG] W-9000-titan_1.0 org.pytorch.serve.wlm.WorkerThread - Backend worker monitoring thread interrupted or backend worker process died., startupTimeout:120sec java.lang.InterruptedException: null at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2056) ~[?:?] at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2133) ~[?:?] at java.util.concurrent.ArrayBlockingQueue.poll(ArrayBlockingQueue.java:432) ~[?:?] at org.pytorch.serve.wlm.WorkerThread.run(WorkerThread.java:234) [model-server.jar:?] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?] at java.lang.Thread.run(Thread.java:829) [?:?] 2025-01-20T18:08:34,614 [WARN ] W-9000-titan_1.0 org.pytorch.serve.wlm.BatchAggregator - Load model failed: titan, error: Worker died. 2025-01-20T18:08:34,614 [WARN ] W-9000-titan_1.0 org.pytorch.serve.wlm.BatchAggregator - Load model failed: titan, error: Worker died. 2025-01-20T18:08:34,615 [DEBUG] W-9000-titan_1.0 org.pytorch.serve.wlm.WorkerThread - W-9000-titan_1.0 State change WORKER_STARTED -> WORKER_STOPPED 2025-01-20T18:08:34,615 [DEBUG] W-9000-titan_1.0 org.pytorch.serve.wlm.WorkerThread - W-9000-titan_1.0 State change WORKER_STARTED -> WORKER_STOPPED 2025-01-20T18:08:34,615 [WARN ] W-9000-titan_1.0 org.pytorch.serve.wlm.WorkerThread - Auto recovery failed again 2025-01-20T18:08:34,615 [WARN ] W-9000-titan_1.0 org.pytorch.serve.wlm.WorkerThread - Auto recovery failed again 2025-01-20T18:08:34,615 [INFO ] W-9000-titan_1.0 org.pytorch.serve.wlm.WorkerThread - Retry worker: 9000 in 1 seconds. 2025-01-20T18:08:34,615 [INFO ] W-9000-titan_1.0 org.pytorch.serve.wlm.WorkerThread - Retry worker: 9000 in 1 seconds. 2025-01-20T18:08:34,641 [INFO ] W-9000-titan_1.0-stdout org.pytorch.serve.wlm.WorkerLifeCycle - Stopped Scanner - W-9000-titan_1.0-stdout 2025-01-20T18:08:34,641 [INFO ] W-9000-titan_1.0-stdout org.pytorch.serve.wlm.WorkerLifeCycle - Stopped Scanner - W-9000-titan_1.0-stdout 2025-01-20T18:08:34,641 [INFO ] W-9000-titan_1.0-stderr org.pytorch.serve.wlm.WorkerLifeCycle - Stopped Scanner - W-9000-titan_1.0-stderr 2025-01-20T18:08:34,641 [INFO ] W-9000-titan_1.0-stderr org.pytorch.serve.wlm.WorkerLifeCycle - Stopped Scanner - W-9000-titan_1.0-stderr 2025-01-20T18:08:35,616 [DEBUG] W-9000-titan_1.0 org.pytorch.serve.wlm.WorkerLifeCycle - Worker cmdline: [/usr/bin/python3, /usr/local/lib/python3.11/dist-packages/ts/model_service_worker.py, --sock-type, unix, --sock-name, /tmp/.ts.sock.9000, --metrics-config, /usr/local/lib/python3.11/dist-packages/ts/configs/metrics.yaml] 2025-01-20T18:08:35,616 [DEBUG] W-9000-titan_1.0 org.pytorch.serve.wlm.WorkerLifeCycle - Worker cmdline: [/usr/bin/python3, /usr/local/lib/python3.11/dist-packages/ts/model_service_worker.py, --sock-type, unix, --sock-name, /tmp/.ts.sock.9000, --metrics-config, /usr/local/lib/python3.11/dist-packages/ts/configs/metrics.yaml] 2025-01-20T18:08:37,596 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - s_name_part0=/tmp/.ts.sock, s_name_part1=9000, pid=12217 2025-01-20T18:08:37,607 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - Listening on port: /tmp/.ts.sock.9000 2025-01-20T18:08:37,621 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - Successfully loaded /usr/local/lib/python3.11/dist-packages/ts/configs/metrics.yaml. 2025-01-20T18:08:37,622 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - [PID]12217 2025-01-20T18:08:37,623 [DEBUG] W-9000-titan_1.0 org.pytorch.serve.wlm.WorkerThread - W-9000-titan_1.0 State change WORKER_STOPPED -> WORKER_STARTED 2025-01-20T18:08:37,623 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - Torch worker started. 2025-01-20T18:08:37,623 [DEBUG] W-9000-titan_1.0 org.pytorch.serve.wlm.WorkerThread - W-9000-titan_1.0 State change WORKER_STOPPED -> WORKER_STARTED 2025-01-20T18:08:37,625 [INFO ] W-9000-titan_1.0 org.pytorch.serve.wlm.WorkerThread - Connecting to: /tmp/.ts.sock.9000 2025-01-20T18:08:37,625 [INFO ] W-9000-titan_1.0 org.pytorch.serve.wlm.WorkerThread - Connecting to: /tmp/.ts.sock.9000 2025-01-20T18:08:37,628 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - Python runtime: 3.11.11 2025-01-20T18:08:37,633 [DEBUG] W-9000-titan_1.0 org.pytorch.serve.wlm.WorkerThread - Flushing req.cmd LOAD repeats 1 to backend at: 1737396517633 2025-01-20T18:08:37,633 [DEBUG] W-9000-titan_1.0 org.pytorch.serve.wlm.WorkerThread - Flushing req.cmd LOAD repeats 1 to backend at: 1737396517633 2025-01-20T18:08:37,634 [INFO ] W-9000-titan_1.0 org.pytorch.serve.wlm.WorkerThread - Looping backend response at: 1737396517633 2025-01-20T18:08:37,634 [INFO ] W-9000-titan_1.0 org.pytorch.serve.wlm.WorkerThread - Looping backend response at: 1737396517633 2025-01-20T18:08:37,639 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - Connection accepted: /tmp/.ts.sock.9000. 2025-01-20T18:08:37,669 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - model_name: titan, batchSize: 1 2025-01-20T18:08:37,673 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - Backend worker process died. 2025-01-20T18:08:37,673 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - Traceback (most recent call last): 2025-01-20T18:08:37,674 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - File "/usr/local/lib/python3.11/dist-packages/ts/model_service_worker.py", line 301, in 2025-01-20T18:08:37,675 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - worker.run_server() 2025-01-20T18:08:37,676 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - File "/usr/local/lib/python3.11/dist-packages/ts/model_service_worker.py", line 268, in run_server 2025-01-20T18:08:37,676 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - self.handle_connection(cl_socket) 2025-01-20T18:08:37,680 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - File "/usr/local/lib/python3.11/dist-packages/ts/model_service_worker.py", line 196, in handle_connection 2025-01-20T18:08:37,681 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - service, result, code = self.load_model(msg) 2025-01-20T18:08:37,682 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - ^^^^^^^^^^^^^^^^^^^^ 2025-01-20T18:08:37,683 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - File "/usr/local/lib/python3.11/dist-packages/ts/model_service_worker.py", line 133, in load_model 2025-01-20T18:08:37,683 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - service = model_loader.load( 2025-01-20T18:08:37,684 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - ^^^^^^^^^^^^^^^^^^ 2025-01-20T18:08:37,688 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - File "/usr/local/lib/python3.11/dist-packages/ts/model_loader.py", line 125, in load 2025-01-20T18:08:37,689 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - entry_point, initialize_fn = self._get_class_entry_point(module) 2025-01-20T18:08:37,690 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-01-20T18:08:37,691 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - File "/usr/local/lib/python3.11/dist-packages/ts/model_loader.py", line 178, in _get_class_entry_point 2025-01-20T18:08:37,692 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - raise ValueError( 2025-01-20T18:08:37,692 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - ValueError: Expected only one class in custom service code or a function entry point [, , , , ] 2025-01-20T18:08:37,697 [INFO ] epollEventLoopGroup-5-3 org.pytorch.serve.wlm.WorkerThread - 9000 Worker disconnected. WORKER_STARTED 2025-01-20T18:08:37,697 [INFO ] epollEventLoopGroup-5-3 org.pytorch.serve.wlm.WorkerThread - 9000 Worker disconnected. WORKER_STARTED 2025-01-20T18:08:37,698 [DEBUG] W-9000-titan_1.0 org.pytorch.serve.wlm.WorkerThread - System state is : WORKER_STARTED 2025-01-20T18:08:37,698 [DEBUG] W-9000-titan_1.0 org.pytorch.serve.wlm.WorkerThread - System state is : WORKER_STARTED 2025-01-20T18:08:37,698 [DEBUG] W-9000-titan_1.0 org.pytorch.serve.wlm.WorkerThread - Backend worker monitoring thread interrupted or backend worker process died., startupTimeout:120sec java.lang.InterruptedException: null at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2056) ~[?:?] at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2133) ~[?:?] at java.util.concurrent.ArrayBlockingQueue.poll(ArrayBlockingQueue.java:432) ~[?:?] at org.pytorch.serve.wlm.WorkerThread.run(WorkerThread.java:234) [model-server.jar:?] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?] at java.lang.Thread.run(Thread.java:829) [?:?] 2025-01-20T18:08:37,698 [DEBUG] W-9000-titan_1.0 org.pytorch.serve.wlm.WorkerThread - Backend worker monitoring thread interrupted or backend worker process died., startupTimeout:120sec java.lang.InterruptedException: null at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2056) ~[?:?] at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2133) ~[?:?] at java.util.concurrent.ArrayBlockingQueue.poll(ArrayBlockingQueue.java:432) ~[?:?] at org.pytorch.serve.wlm.WorkerThread.run(WorkerThread.java:234) [model-server.jar:?] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?] at java.lang.Thread.run(Thread.java:829) [?:?] 2025-01-20T18:08:37,707 [WARN ] W-9000-titan_1.0 org.pytorch.serve.wlm.BatchAggregator - Load model failed: titan, error: Worker died. 2025-01-20T18:08:37,707 [WARN ] W-9000-titan_1.0 org.pytorch.serve.wlm.BatchAggregator - Load model failed: titan, error: Worker died. 2025-01-20T18:08:37,707 [DEBUG] W-9000-titan_1.0 org.pytorch.serve.wlm.WorkerThread - W-9000-titan_1.0 State change WORKER_STARTED -> WORKER_STOPPED 2025-01-20T18:08:37,707 [DEBUG] W-9000-titan_1.0 org.pytorch.serve.wlm.WorkerThread - W-9000-titan_1.0 State change WORKER_STARTED -> WORKER_STOPPED 2025-01-20T18:08:37,707 [WARN ] W-9000-titan_1.0 org.pytorch.serve.wlm.WorkerThread - Auto recovery failed again 2025-01-20T18:08:37,707 [WARN ] W-9000-titan_1.0 org.pytorch.serve.wlm.WorkerThread - Auto recovery failed again 2025-01-20T18:08:37,708 [INFO ] W-9000-titan_1.0 org.pytorch.serve.wlm.WorkerThread - Retry worker: 9000 in 2 seconds. 2025-01-20T18:08:37,708 [INFO ] W-9000-titan_1.0 org.pytorch.serve.wlm.WorkerThread - Retry worker: 9000 in 2 seconds. 2025-01-20T18:08:37,752 [INFO ] W-9000-titan_1.0-stderr org.pytorch.serve.wlm.WorkerLifeCycle - Stopped Scanner - W-9000-titan_1.0-stderr 2025-01-20T18:08:37,752 [INFO ] W-9000-titan_1.0-stderr org.pytorch.serve.wlm.WorkerLifeCycle - Stopped Scanner - W-9000-titan_1.0-stderr 2025-01-20T18:08:37,753 [INFO ] W-9000-titan_1.0-stdout org.pytorch.serve.wlm.WorkerLifeCycle - Stopped Scanner - W-9000-titan_1.0-stdout 2025-01-20T18:08:37,753 [INFO ] W-9000-titan_1.0-stdout org.pytorch.serve.wlm.WorkerLifeCycle - Stopped Scanner - W-9000-titan_1.0-stdout 2025-01-20T18:08:39,708 [DEBUG] W-9000-titan_1.0 org.pytorch.serve.wlm.WorkerLifeCycle - Worker cmdline: [/usr/bin/python3, /usr/local/lib/python3.11/dist-packages/ts/model_service_worker.py, --sock-type, unix, --sock-name, /tmp/.ts.sock.9000, --metrics-config, /usr/local/lib/python3.11/dist-packages/ts/configs/metrics.yaml] 2025-01-20T18:08:39,708 [DEBUG] W-9000-titan_1.0 org.pytorch.serve.wlm.WorkerLifeCycle - Worker cmdline: [/usr/bin/python3, /usr/local/lib/python3.11/dist-packages/ts/model_service_worker.py, --sock-type, unix, --sock-name, /tmp/.ts.sock.9000, --metrics-config, /usr/local/lib/python3.11/dist-packages/ts/configs/metrics.yaml] 2025-01-20T18:08:41,278 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - s_name_part0=/tmp/.ts.sock, s_name_part1=9000, pid=12239 2025-01-20T18:08:41,279 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - Listening on port: /tmp/.ts.sock.9000 2025-01-20T18:08:41,286 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - Successfully loaded /usr/local/lib/python3.11/dist-packages/ts/configs/metrics.yaml. 2025-01-20T18:08:41,286 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - [PID]12239 2025-01-20T18:08:41,287 [DEBUG] W-9000-titan_1.0 org.pytorch.serve.wlm.WorkerThread - W-9000-titan_1.0 State change WORKER_STOPPED -> WORKER_STARTED 2025-01-20T18:08:41,287 [DEBUG] W-9000-titan_1.0 org.pytorch.serve.wlm.WorkerThread - W-9000-titan_1.0 State change WORKER_STOPPED -> WORKER_STARTED 2025-01-20T18:08:41,287 [INFO ] W-9000-titan_1.0 org.pytorch.serve.wlm.WorkerThread - Connecting to: /tmp/.ts.sock.9000 2025-01-20T18:08:41,287 [INFO ] W-9000-titan_1.0 org.pytorch.serve.wlm.WorkerThread - Connecting to: /tmp/.ts.sock.9000 2025-01-20T18:08:41,287 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - Torch worker started. 2025-01-20T18:08:41,288 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - Python runtime: 3.11.11 2025-01-20T18:08:41,291 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - Connection accepted: /tmp/.ts.sock.9000. 2025-01-20T18:08:41,292 [DEBUG] W-9000-titan_1.0 org.pytorch.serve.wlm.WorkerThread - Flushing req.cmd LOAD repeats 1 to backend at: 1737396521292 2025-01-20T18:08:41,292 [DEBUG] W-9000-titan_1.0 org.pytorch.serve.wlm.WorkerThread - Flushing req.cmd LOAD repeats 1 to backend at: 1737396521292 2025-01-20T18:08:41,292 [INFO ] W-9000-titan_1.0 org.pytorch.serve.wlm.WorkerThread - Looping backend response at: 1737396521292 2025-01-20T18:08:41,292 [INFO ] W-9000-titan_1.0 org.pytorch.serve.wlm.WorkerThread - Looping backend response at: 1737396521292 2025-01-20T18:08:41,317 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - model_name: titan, batchSize: 1 2025-01-20T18:08:41,320 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - Backend worker process died. 2025-01-20T18:08:41,320 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - Traceback (most recent call last): 2025-01-20T18:08:41,320 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - File "/usr/local/lib/python3.11/dist-packages/ts/model_service_worker.py", line 301, in 2025-01-20T18:08:41,320 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - worker.run_server() 2025-01-20T18:08:41,320 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - File "/usr/local/lib/python3.11/dist-packages/ts/model_service_worker.py", line 268, in run_server 2025-01-20T18:08:41,321 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - self.handle_connection(cl_socket) 2025-01-20T18:08:41,321 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - File "/usr/local/lib/python3.11/dist-packages/ts/model_service_worker.py", line 196, in handle_connection 2025-01-20T18:08:41,321 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - service, result, code = self.load_model(msg) 2025-01-20T18:08:41,321 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - ^^^^^^^^^^^^^^^^^^^^ 2025-01-20T18:08:41,321 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - File "/usr/local/lib/python3.11/dist-packages/ts/model_service_worker.py", line 133, in load_model 2025-01-20T18:08:41,322 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - service = model_loader.load( 2025-01-20T18:08:41,322 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - ^^^^^^^^^^^^^^^^^^ 2025-01-20T18:08:41,322 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - File "/usr/local/lib/python3.11/dist-packages/ts/model_loader.py", line 125, in load 2025-01-20T18:08:41,322 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - entry_point, initialize_fn = self._get_class_entry_point(module) 2025-01-20T18:08:41,322 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-01-20T18:08:41,322 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - File "/usr/local/lib/python3.11/dist-packages/ts/model_loader.py", line 178, in _get_class_entry_point 2025-01-20T18:08:41,329 [INFO ] epollEventLoopGroup-5-4 org.pytorch.serve.wlm.WorkerThread - 9000 Worker disconnected. WORKER_STARTED 2025-01-20T18:08:41,329 [INFO ] epollEventLoopGroup-5-4 org.pytorch.serve.wlm.WorkerThread - 9000 Worker disconnected. WORKER_STARTED 2025-01-20T18:08:41,331 [DEBUG] W-9000-titan_1.0 org.pytorch.serve.wlm.WorkerThread - System state is : WORKER_STARTED 2025-01-20T18:08:41,331 [DEBUG] W-9000-titan_1.0 org.pytorch.serve.wlm.WorkerThread - System state is : WORKER_STARTED 2025-01-20T18:08:41,331 [DEBUG] W-9000-titan_1.0 org.pytorch.serve.wlm.WorkerThread - Backend worker monitoring thread interrupted or backend worker process died., startupTimeout:120sec java.lang.InterruptedException: null at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2056) ~[?:?] at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2133) ~[?:?] at java.util.concurrent.ArrayBlockingQueue.poll(ArrayBlockingQueue.java:432) ~[?:?] at org.pytorch.serve.wlm.WorkerThread.run(WorkerThread.java:234) [model-server.jar:?] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?] at java.lang.Thread.run(Thread.java:829) [?:?] 2025-01-20T18:08:41,331 [DEBUG] W-9000-titan_1.0 org.pytorch.serve.wlm.WorkerThread - Backend worker monitoring thread interrupted or backend worker process died., startupTimeout:120sec java.lang.InterruptedException: null at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2056) ~[?:?] at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2133) ~[?:?] at java.util.concurrent.ArrayBlockingQueue.poll(ArrayBlockingQueue.java:432) ~[?:?] at org.pytorch.serve.wlm.WorkerThread.run(WorkerThread.java:234) [model-server.jar:?] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?] at java.lang.Thread.run(Thread.java:829) [?:?] 2025-01-20T18:08:41,332 [WARN ] W-9000-titan_1.0 org.pytorch.serve.wlm.BatchAggregator - Load model failed: titan, error: Worker died. 2025-01-20T18:08:41,332 [WARN ] W-9000-titan_1.0 org.pytorch.serve.wlm.BatchAggregator - Load model failed: titan, error: Worker died. 2025-01-20T18:08:41,332 [DEBUG] W-9000-titan_1.0 org.pytorch.serve.wlm.WorkerThread - W-9000-titan_1.0 State change WORKER_STARTED -> WORKER_STOPPED 2025-01-20T18:08:41,332 [DEBUG] W-9000-titan_1.0 org.pytorch.serve.wlm.WorkerThread - W-9000-titan_1.0 State change WORKER_STARTED -> WORKER_STOPPED 2025-01-20T18:08:41,332 [WARN ] W-9000-titan_1.0 org.pytorch.serve.wlm.WorkerThread - Auto recovery failed again 2025-01-20T18:08:41,332 [WARN ] W-9000-titan_1.0 org.pytorch.serve.wlm.WorkerThread - Auto recovery failed again 2025-01-20T18:08:41,333 [INFO ] W-9000-titan_1.0 org.pytorch.serve.wlm.WorkerThread - Retry worker: 9000 in 3 seconds. 2025-01-20T18:08:41,333 [INFO ] W-9000-titan_1.0 org.pytorch.serve.wlm.WorkerThread - Retry worker: 9000 in 3 seconds. 2025-01-20T18:08:41,331 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - raise ValueError( 2025-01-20T18:08:41,334 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - ValueError: Expected only one class in custom service code or a function entry point [, , , , ] 2025-01-20T18:08:41,335 [INFO ] W-9000-titan_1.0-stdout org.pytorch.serve.wlm.WorkerLifeCycle - Stopped Scanner - W-9000-titan_1.0-stdout 2025-01-20T18:08:41,335 [INFO ] W-9000-titan_1.0-stdout org.pytorch.serve.wlm.WorkerLifeCycle - Stopped Scanner - W-9000-titan_1.0-stdout 2025-01-20T18:08:41,356 [INFO ] W-9000-titan_1.0-stderr org.pytorch.serve.wlm.WorkerLifeCycle - Stopped Scanner - W-9000-titan_1.0-stderr 2025-01-20T18:08:41,356 [INFO ] W-9000-titan_1.0-stderr org.pytorch.serve.wlm.WorkerLifeCycle - Stopped Scanner - W-9000-titan_1.0-stderr 2025-01-20T18:08:44,333 [DEBUG] W-9000-titan_1.0 org.pytorch.serve.wlm.WorkerLifeCycle - Worker cmdline: [/usr/bin/python3, /usr/local/lib/python3.11/dist-packages/ts/model_service_worker.py, --sock-type, unix, --sock-name, /tmp/.ts.sock.9000, --metrics-config, /usr/local/lib/python3.11/dist-packages/ts/configs/metrics.yaml] 2025-01-20T18:08:44,333 [DEBUG] W-9000-titan_1.0 org.pytorch.serve.wlm.WorkerLifeCycle - Worker cmdline: [/usr/bin/python3, /usr/local/lib/python3.11/dist-packages/ts/model_service_worker.py, --sock-type, unix, --sock-name, /tmp/.ts.sock.9000, --metrics-config, /usr/local/lib/python3.11/dist-packages/ts/configs/metrics.yaml] 2025-01-20T18:08:45,891 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - s_name_part0=/tmp/.ts.sock, s_name_part1=9000, pid=12267 2025-01-20T18:08:45,895 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - Listening on port: /tmp/.ts.sock.9000 2025-01-20T18:08:45,899 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - Successfully loaded /usr/local/lib/python3.11/dist-packages/ts/configs/metrics.yaml. 2025-01-20T18:08:45,900 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - [PID]12267 2025-01-20T18:08:45,900 [DEBUG] W-9000-titan_1.0 org.pytorch.serve.wlm.WorkerThread - W-9000-titan_1.0 State change WORKER_STOPPED -> WORKER_STARTED 2025-01-20T18:08:45,900 [DEBUG] W-9000-titan_1.0 org.pytorch.serve.wlm.WorkerThread - W-9000-titan_1.0 State change WORKER_STOPPED -> WORKER_STARTED 2025-01-20T18:08:45,900 [INFO ] W-9000-titan_1.0 org.pytorch.serve.wlm.WorkerThread - Connecting to: /tmp/.ts.sock.9000 2025-01-20T18:08:45,900 [INFO ] W-9000-titan_1.0 org.pytorch.serve.wlm.WorkerThread - Connecting to: /tmp/.ts.sock.9000 2025-01-20T18:08:45,901 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - Torch worker started. 2025-01-20T18:08:45,901 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - Python runtime: 3.11.11 2025-01-20T18:08:45,902 [DEBUG] W-9000-titan_1.0 org.pytorch.serve.wlm.WorkerThread - Flushing req.cmd LOAD repeats 1 to backend at: 1737396525902 2025-01-20T18:08:45,902 [DEBUG] W-9000-titan_1.0 org.pytorch.serve.wlm.WorkerThread - Flushing req.cmd LOAD repeats 1 to backend at: 1737396525902 2025-01-20T18:08:45,903 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - Connection accepted: /tmp/.ts.sock.9000. 2025-01-20T18:08:45,903 [INFO ] W-9000-titan_1.0 org.pytorch.serve.wlm.WorkerThread - Looping backend response at: 1737396525903 2025-01-20T18:08:45,903 [INFO ] W-9000-titan_1.0 org.pytorch.serve.wlm.WorkerThread - Looping backend response at: 1737396525903 2025-01-20T18:08:45,905 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - model_name: titan, batchSize: 1 2025-01-20T18:08:45,907 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - Backend worker process died. 2025-01-20T18:08:45,907 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - Traceback (most recent call last): 2025-01-20T18:08:45,907 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - File "/usr/local/lib/python3.11/dist-packages/ts/model_service_worker.py", line 301, in 2025-01-20T18:08:45,907 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - worker.run_server() 2025-01-20T18:08:45,908 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - File "/usr/local/lib/python3.11/dist-packages/ts/model_service_worker.py", line 268, in run_server 2025-01-20T18:08:45,908 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - self.handle_connection(cl_socket) 2025-01-20T18:08:45,908 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - File "/usr/local/lib/python3.11/dist-packages/ts/model_service_worker.py", line 196, in handle_connection 2025-01-20T18:08:45,908 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - service, result, code = self.load_model(msg) 2025-01-20T18:08:45,908 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - ^^^^^^^^^^^^^^^^^^^^ 2025-01-20T18:08:45,908 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - File "/usr/local/lib/python3.11/dist-packages/ts/model_service_worker.py", line 133, in load_model 2025-01-20T18:08:45,908 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - service = model_loader.load( 2025-01-20T18:08:45,909 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - ^^^^^^^^^^^^^^^^^^ 2025-01-20T18:08:45,909 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - File "/usr/local/lib/python3.11/dist-packages/ts/model_loader.py", line 125, in load 2025-01-20T18:08:45,909 [INFO ] epollEventLoopGroup-5-1 org.pytorch.serve.wlm.WorkerThread - 9000 Worker disconnected. WORKER_STARTED 2025-01-20T18:08:45,909 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - entry_point, initialize_fn = self._get_class_entry_point(module) 2025-01-20T18:08:45,909 [INFO ] epollEventLoopGroup-5-1 org.pytorch.serve.wlm.WorkerThread - 9000 Worker disconnected. WORKER_STARTED 2025-01-20T18:08:45,911 [DEBUG] W-9000-titan_1.0 org.pytorch.serve.wlm.WorkerThread - System state is : WORKER_STARTED 2025-01-20T18:08:45,911 [DEBUG] W-9000-titan_1.0 org.pytorch.serve.wlm.WorkerThread - System state is : WORKER_STARTED 2025-01-20T18:08:45,911 [DEBUG] W-9000-titan_1.0 org.pytorch.serve.wlm.WorkerThread - Backend worker monitoring thread interrupted or backend worker process died., startupTimeout:120sec java.lang.InterruptedException: null at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2056) ~[?:?] at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2133) ~[?:?] at java.util.concurrent.ArrayBlockingQueue.poll(ArrayBlockingQueue.java:432) ~[?:?] at org.pytorch.serve.wlm.WorkerThread.run(WorkerThread.java:234) [model-server.jar:?] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?] at java.lang.Thread.run(Thread.java:829) [?:?] 2025-01-20T18:08:45,911 [DEBUG] W-9000-titan_1.0 org.pytorch.serve.wlm.WorkerThread - Backend worker monitoring thread interrupted or backend worker process died., startupTimeout:120sec java.lang.InterruptedException: null at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2056) ~[?:?] at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2133) ~[?:?] at java.util.concurrent.ArrayBlockingQueue.poll(ArrayBlockingQueue.java:432) ~[?:?] at org.pytorch.serve.wlm.WorkerThread.run(WorkerThread.java:234) [model-server.jar:?] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?] at java.lang.Thread.run(Thread.java:829) [?:?] 2025-01-20T18:08:45,912 [WARN ] W-9000-titan_1.0 org.pytorch.serve.wlm.BatchAggregator - Load model failed: titan, error: Worker died. 2025-01-20T18:08:45,912 [WARN ] W-9000-titan_1.0 org.pytorch.serve.wlm.BatchAggregator - Load model failed: titan, error: Worker died. 2025-01-20T18:08:45,912 [DEBUG] W-9000-titan_1.0 org.pytorch.serve.wlm.WorkerThread - W-9000-titan_1.0 State change WORKER_STARTED -> WORKER_STOPPED 2025-01-20T18:08:45,912 [DEBUG] W-9000-titan_1.0 org.pytorch.serve.wlm.WorkerThread - W-9000-titan_1.0 State change WORKER_STARTED -> WORKER_STOPPED 2025-01-20T18:08:45,912 [WARN ] W-9000-titan_1.0 org.pytorch.serve.wlm.WorkerThread - Auto recovery failed again 2025-01-20T18:08:45,912 [WARN ] W-9000-titan_1.0 org.pytorch.serve.wlm.WorkerThread - Auto recovery failed again 2025-01-20T18:08:45,912 [INFO ] W-9000-titan_1.0 org.pytorch.serve.wlm.WorkerThread - Retry worker: 9000 in 5 seconds. 2025-01-20T18:08:45,912 [INFO ] W-9000-titan_1.0 org.pytorch.serve.wlm.WorkerThread - Retry worker: 9000 in 5 seconds. 2025-01-20T18:08:45,913 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-01-20T18:08:45,913 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - File "/usr/local/lib/python3.11/dist-packages/ts/model_loader.py", line 178, in _get_class_entry_point 2025-01-20T18:08:45,913 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - raise ValueError( 2025-01-20T18:08:45,913 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - ValueError: Expected only one class in custom service code or a function entry point [, , , , ] 2025-01-20T18:08:45,914 [INFO ] W-9000-titan_1.0-stdout org.pytorch.serve.wlm.WorkerLifeCycle - Stopped Scanner - W-9000-titan_1.0-stdout 2025-01-20T18:08:45,914 [INFO ] W-9000-titan_1.0-stdout org.pytorch.serve.wlm.WorkerLifeCycle - Stopped Scanner - W-9000-titan_1.0-stdout 2025-01-20T18:08:45,960 [INFO ] W-9000-titan_1.0-stderr org.pytorch.serve.wlm.WorkerLifeCycle - Stopped Scanner - W-9000-titan_1.0-stderr 2025-01-20T18:08:45,960 [INFO ] W-9000-titan_1.0-stderr org.pytorch.serve.wlm.WorkerLifeCycle - Stopped Scanner - W-9000-titan_1.0-stderr 2025-01-20T18:08:50,913 [DEBUG] W-9000-titan_1.0 org.pytorch.serve.wlm.WorkerLifeCycle - Worker cmdline: [/usr/bin/python3, /usr/local/lib/python3.11/dist-packages/ts/model_service_worker.py, --sock-type, unix, --sock-name, /tmp/.ts.sock.9000, --metrics-config, /usr/local/lib/python3.11/dist-packages/ts/configs/metrics.yaml] 2025-01-20T18:08:50,913 [DEBUG] W-9000-titan_1.0 org.pytorch.serve.wlm.WorkerLifeCycle - Worker cmdline: [/usr/bin/python3, /usr/local/lib/python3.11/dist-packages/ts/model_service_worker.py, --sock-type, unix, --sock-name, /tmp/.ts.sock.9000, --metrics-config, /usr/local/lib/python3.11/dist-packages/ts/configs/metrics.yaml] 2025-01-20T18:08:52,625 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - s_name_part0=/tmp/.ts.sock, s_name_part1=9000, pid=12298 2025-01-20T18:08:52,627 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - Listening on port: /tmp/.ts.sock.9000 2025-01-20T18:08:52,635 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - Successfully loaded /usr/local/lib/python3.11/dist-packages/ts/configs/metrics.yaml. 2025-01-20T18:08:52,635 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - [PID]12298 2025-01-20T18:08:52,636 [DEBUG] W-9000-titan_1.0 org.pytorch.serve.wlm.WorkerThread - W-9000-titan_1.0 State change WORKER_STOPPED -> WORKER_STARTED 2025-01-20T18:08:52,636 [DEBUG] W-9000-titan_1.0 org.pytorch.serve.wlm.WorkerThread - W-9000-titan_1.0 State change WORKER_STOPPED -> WORKER_STARTED 2025-01-20T18:08:52,636 [INFO ] W-9000-titan_1.0 org.pytorch.serve.wlm.WorkerThread - Connecting to: /tmp/.ts.sock.9000 2025-01-20T18:08:52,636 [INFO ] W-9000-titan_1.0 org.pytorch.serve.wlm.WorkerThread - Connecting to: /tmp/.ts.sock.9000 2025-01-20T18:08:52,637 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - Torch worker started. 2025-01-20T18:08:52,637 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - Python runtime: 3.11.11 2025-01-20T18:08:52,639 [DEBUG] W-9000-titan_1.0 org.pytorch.serve.wlm.WorkerThread - Flushing req.cmd LOAD repeats 1 to backend at: 1737396532639 2025-01-20T18:08:52,639 [DEBUG] W-9000-titan_1.0 org.pytorch.serve.wlm.WorkerThread - Flushing req.cmd LOAD repeats 1 to backend at: 1737396532639 2025-01-20T18:08:52,639 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - Connection accepted: /tmp/.ts.sock.9000. 2025-01-20T18:08:52,640 [INFO ] W-9000-titan_1.0 org.pytorch.serve.wlm.WorkerThread - Looping backend response at: 1737396532639 2025-01-20T18:08:52,640 [INFO ] W-9000-titan_1.0 org.pytorch.serve.wlm.WorkerThread - Looping backend response at: 1737396532639 2025-01-20T18:08:52,643 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - model_name: titan, batchSize: 1 2025-01-20T18:08:52,646 [INFO ] epollEventLoopGroup-5-2 org.pytorch.serve.wlm.WorkerThread - 9000 Worker disconnected. WORKER_STARTED 2025-01-20T18:08:52,645 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - Backend worker process died. 2025-01-20T18:08:52,646 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - Traceback (most recent call last): 2025-01-20T18:08:52,646 [INFO ] epollEventLoopGroup-5-2 org.pytorch.serve.wlm.WorkerThread - 9000 Worker disconnected. WORKER_STARTED 2025-01-20T18:08:52,647 [DEBUG] W-9000-titan_1.0 org.pytorch.serve.wlm.WorkerThread - System state is : WORKER_STARTED 2025-01-20T18:08:52,647 [DEBUG] W-9000-titan_1.0 org.pytorch.serve.wlm.WorkerThread - System state is : WORKER_STARTED 2025-01-20T18:08:52,647 [DEBUG] W-9000-titan_1.0 org.pytorch.serve.wlm.WorkerThread - Backend worker monitoring thread interrupted or backend worker process died., startupTimeout:120sec java.lang.InterruptedException: null at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2056) ~[?:?] at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2133) ~[?:?] at java.util.concurrent.ArrayBlockingQueue.poll(ArrayBlockingQueue.java:432) ~[?:?] at org.pytorch.serve.wlm.WorkerThread.run(WorkerThread.java:234) [model-server.jar:?] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?] at java.lang.Thread.run(Thread.java:829) [?:?] 2025-01-20T18:08:52,647 [DEBUG] W-9000-titan_1.0 org.pytorch.serve.wlm.WorkerThread - Backend worker monitoring thread interrupted or backend worker process died., startupTimeout:120sec java.lang.InterruptedException: null at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2056) ~[?:?] at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2133) ~[?:?] at java.util.concurrent.ArrayBlockingQueue.poll(ArrayBlockingQueue.java:432) ~[?:?] at org.pytorch.serve.wlm.WorkerThread.run(WorkerThread.java:234) [model-server.jar:?] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?] at java.lang.Thread.run(Thread.java:829) [?:?] 2025-01-20T18:08:52,648 [WARN ] W-9000-titan_1.0 org.pytorch.serve.wlm.BatchAggregator - Load model failed: titan, error: Worker died. 2025-01-20T18:08:52,648 [WARN ] W-9000-titan_1.0 org.pytorch.serve.wlm.BatchAggregator - Load model failed: titan, error: Worker died. 2025-01-20T18:08:52,649 [DEBUG] W-9000-titan_1.0 org.pytorch.serve.wlm.WorkerThread - W-9000-titan_1.0 State change WORKER_STARTED -> WORKER_STOPPED 2025-01-20T18:08:52,649 [DEBUG] W-9000-titan_1.0 org.pytorch.serve.wlm.WorkerThread - W-9000-titan_1.0 State change WORKER_STARTED -> WORKER_STOPPED 2025-01-20T18:08:52,650 [WARN ] W-9000-titan_1.0 org.pytorch.serve.wlm.WorkerThread - Auto recovery failed again 2025-01-20T18:08:52,650 [WARN ] W-9000-titan_1.0 org.pytorch.serve.wlm.WorkerThread - Auto recovery failed again 2025-01-20T18:08:52,650 [INFO ] W-9000-titan_1.0 org.pytorch.serve.wlm.WorkerThread - Retry worker: 9000 in 8 seconds. 2025-01-20T18:08:52,650 [INFO ] W-9000-titan_1.0 org.pytorch.serve.wlm.WorkerThread - Retry worker: 9000 in 8 seconds. 2025-01-20T18:08:52,651 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - File "/usr/local/lib/python3.11/dist-packages/ts/model_service_worker.py", line 301, in 2025-01-20T18:08:52,651 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - worker.run_server() 2025-01-20T18:08:52,652 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - File "/usr/local/lib/python3.11/dist-packages/ts/model_service_worker.py", line 268, in run_server 2025-01-20T18:08:52,653 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - self.handle_connection(cl_socket) 2025-01-20T18:08:52,653 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - File "/usr/local/lib/python3.11/dist-packages/ts/model_service_worker.py", line 196, in handle_connection 2025-01-20T18:08:52,653 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - service, result, code = self.load_model(msg) 2025-01-20T18:08:52,653 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - ^^^^^^^^^^^^^^^^^^^^ 2025-01-20T18:08:52,653 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - File "/usr/local/lib/python3.11/dist-packages/ts/model_service_worker.py", line 133, in load_model 2025-01-20T18:08:52,653 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - service = model_loader.load( 2025-01-20T18:08:52,654 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - ^^^^^^^^^^^^^^^^^^ 2025-01-20T18:08:52,654 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - File "/usr/local/lib/python3.11/dist-packages/ts/model_loader.py", line 125, in load 2025-01-20T18:08:52,654 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - entry_point, initialize_fn = self._get_class_entry_point(module) 2025-01-20T18:08:52,654 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-01-20T18:08:52,654 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - File "/usr/local/lib/python3.11/dist-packages/ts/model_loader.py", line 178, in _get_class_entry_point 2025-01-20T18:08:52,655 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - raise ValueError( 2025-01-20T18:08:52,655 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - ValueError: Expected only one class in custom service code or a function entry point [, , , , ] 2025-01-20T18:08:52,655 [INFO ] W-9000-titan_1.0-stdout org.pytorch.serve.wlm.WorkerLifeCycle - Stopped Scanner - W-9000-titan_1.0-stdout 2025-01-20T18:08:52,655 [INFO ] W-9000-titan_1.0-stdout org.pytorch.serve.wlm.WorkerLifeCycle - Stopped Scanner - W-9000-titan_1.0-stdout 2025-01-20T18:08:52,677 [INFO ] W-9000-titan_1.0-stderr org.pytorch.serve.wlm.WorkerLifeCycle - Stopped Scanner - W-9000-titan_1.0-stderr 2025-01-20T18:08:52,677 [INFO ] W-9000-titan_1.0-stderr org.pytorch.serve.wlm.WorkerLifeCycle - Stopped Scanner - W-9000-titan_1.0-stderr 2025-01-20T18:09:00,651 [DEBUG] W-9000-titan_1.0 org.pytorch.serve.wlm.WorkerLifeCycle - Worker cmdline: [/usr/bin/python3, /usr/local/lib/python3.11/dist-packages/ts/model_service_worker.py, --sock-type, unix, --sock-name, /tmp/.ts.sock.9000, --metrics-config, /usr/local/lib/python3.11/dist-packages/ts/configs/metrics.yaml] 2025-01-20T18:09:00,651 [DEBUG] W-9000-titan_1.0 org.pytorch.serve.wlm.WorkerLifeCycle - Worker cmdline: [/usr/bin/python3, /usr/local/lib/python3.11/dist-packages/ts/model_service_worker.py, --sock-type, unix, --sock-name, /tmp/.ts.sock.9000, --metrics-config, /usr/local/lib/python3.11/dist-packages/ts/configs/metrics.yaml] 2025-01-20T18:09:02,675 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - s_name_part0=/tmp/.ts.sock, s_name_part1=9000, pid=12345 2025-01-20T18:09:02,677 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - Listening on port: /tmp/.ts.sock.9000 2025-01-20T18:09:02,696 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - Successfully loaded /usr/local/lib/python3.11/dist-packages/ts/configs/metrics.yaml. 2025-01-20T18:09:02,696 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - [PID]12345 2025-01-20T18:09:02,697 [DEBUG] W-9000-titan_1.0 org.pytorch.serve.wlm.WorkerThread - W-9000-titan_1.0 State change WORKER_STOPPED -> WORKER_STARTED 2025-01-20T18:09:02,697 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - Torch worker started. 2025-01-20T18:09:02,697 [DEBUG] W-9000-titan_1.0 org.pytorch.serve.wlm.WorkerThread - W-9000-titan_1.0 State change WORKER_STOPPED -> WORKER_STARTED 2025-01-20T18:09:02,698 [INFO ] W-9000-titan_1.0 org.pytorch.serve.wlm.WorkerThread - Connecting to: /tmp/.ts.sock.9000 2025-01-20T18:09:02,698 [INFO ] W-9000-titan_1.0 org.pytorch.serve.wlm.WorkerThread - Connecting to: /tmp/.ts.sock.9000 2025-01-20T18:09:02,700 [DEBUG] W-9000-titan_1.0 org.pytorch.serve.wlm.WorkerThread - Flushing req.cmd LOAD repeats 1 to backend at: 1737396542700 2025-01-20T18:09:02,700 [DEBUG] W-9000-titan_1.0 org.pytorch.serve.wlm.WorkerThread - Flushing req.cmd LOAD repeats 1 to backend at: 1737396542700 2025-01-20T18:09:02,701 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - Python runtime: 3.11.11 2025-01-20T18:09:02,705 [INFO ] W-9000-titan_1.0 org.pytorch.serve.wlm.WorkerThread - Looping backend response at: 1737396542705 2025-01-20T18:09:02,705 [INFO ] W-9000-titan_1.0 org.pytorch.serve.wlm.WorkerThread - Looping backend response at: 1737396542705 2025-01-20T18:09:02,706 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - Connection accepted: /tmp/.ts.sock.9000. 2025-01-20T18:09:02,707 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - model_name: titan, batchSize: 1 2025-01-20T18:09:02,709 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - Backend worker process died. 2025-01-20T18:09:02,709 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - Traceback (most recent call last): 2025-01-20T18:09:02,709 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - File "/usr/local/lib/python3.11/dist-packages/ts/model_service_worker.py", line 301, in 2025-01-20T18:09:02,709 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - worker.run_server() 2025-01-20T18:09:02,710 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - File "/usr/local/lib/python3.11/dist-packages/ts/model_service_worker.py", line 268, in run_server 2025-01-20T18:09:02,710 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - self.handle_connection(cl_socket) 2025-01-20T18:09:02,710 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - File "/usr/local/lib/python3.11/dist-packages/ts/model_service_worker.py", line 196, in handle_connection 2025-01-20T18:09:02,711 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - service, result, code = self.load_model(msg) 2025-01-20T18:09:02,711 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - ^^^^^^^^^^^^^^^^^^^^ 2025-01-20T18:09:02,711 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - File "/usr/local/lib/python3.11/dist-packages/ts/model_service_worker.py", line 133, in load_model 2025-01-20T18:09:02,712 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - service = model_loader.load( 2025-01-20T18:09:02,712 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - ^^^^^^^^^^^^^^^^^^ 2025-01-20T18:09:02,712 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - File "/usr/local/lib/python3.11/dist-packages/ts/model_loader.py", line 125, in load 2025-01-20T18:09:02,712 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - entry_point, initialize_fn = self._get_class_entry_point(module) 2025-01-20T18:09:02,712 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-01-20T18:09:02,712 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - File "/usr/local/lib/python3.11/dist-packages/ts/model_loader.py", line 178, in _get_class_entry_point 2025-01-20T18:09:02,712 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - raise ValueError( 2025-01-20T18:09:02,712 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - ValueError: Expected only one class in custom service code or a function entry point [, , , , ] 2025-01-20T18:09:02,713 [INFO ] epollEventLoopGroup-5-3 org.pytorch.serve.wlm.WorkerThread - 9000 Worker disconnected. WORKER_STARTED 2025-01-20T18:09:02,713 [INFO ] epollEventLoopGroup-5-3 org.pytorch.serve.wlm.WorkerThread - 9000 Worker disconnected. WORKER_STARTED 2025-01-20T18:09:02,713 [DEBUG] W-9000-titan_1.0 org.pytorch.serve.wlm.WorkerThread - System state is : WORKER_STARTED 2025-01-20T18:09:02,713 [DEBUG] W-9000-titan_1.0 org.pytorch.serve.wlm.WorkerThread - System state is : WORKER_STARTED 2025-01-20T18:09:02,713 [DEBUG] W-9000-titan_1.0 org.pytorch.serve.wlm.WorkerThread - Backend worker monitoring thread interrupted or backend worker process died., startupTimeout:120sec java.lang.InterruptedException: null at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2056) ~[?:?] at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2133) ~[?:?] at java.util.concurrent.ArrayBlockingQueue.poll(ArrayBlockingQueue.java:432) ~[?:?] at org.pytorch.serve.wlm.WorkerThread.run(WorkerThread.java:234) [model-server.jar:?] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?] at java.lang.Thread.run(Thread.java:829) [?:?] 2025-01-20T18:09:02,713 [DEBUG] W-9000-titan_1.0 org.pytorch.serve.wlm.WorkerThread - Backend worker monitoring thread interrupted or backend worker process died., startupTimeout:120sec java.lang.InterruptedException: null at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2056) ~[?:?] at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2133) ~[?:?] at java.util.concurrent.ArrayBlockingQueue.poll(ArrayBlockingQueue.java:432) ~[?:?] at org.pytorch.serve.wlm.WorkerThread.run(WorkerThread.java:234) [model-server.jar:?] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?] at java.lang.Thread.run(Thread.java:829) [?:?] 2025-01-20T18:09:02,714 [WARN ] W-9000-titan_1.0 org.pytorch.serve.wlm.BatchAggregator - Load model failed: titan, error: Worker died. 2025-01-20T18:09:02,714 [WARN ] W-9000-titan_1.0 org.pytorch.serve.wlm.BatchAggregator - Load model failed: titan, error: Worker died. 2025-01-20T18:09:02,714 [DEBUG] W-9000-titan_1.0 org.pytorch.serve.wlm.WorkerThread - W-9000-titan_1.0 State change WORKER_STARTED -> WORKER_STOPPED 2025-01-20T18:09:02,714 [DEBUG] W-9000-titan_1.0 org.pytorch.serve.wlm.WorkerThread - W-9000-titan_1.0 State change WORKER_STARTED -> WORKER_STOPPED 2025-01-20T18:09:02,714 [WARN ] W-9000-titan_1.0 org.pytorch.serve.wlm.WorkerThread - Auto recovery failed again 2025-01-20T18:09:02,714 [WARN ] W-9000-titan_1.0 org.pytorch.serve.wlm.WorkerThread - Auto recovery failed again 2025-01-20T18:09:02,714 [INFO ] W-9000-titan_1.0 org.pytorch.serve.wlm.WorkerThread - Retry worker: 9000 in 13 seconds. 2025-01-20T18:09:02,714 [INFO ] W-9000-titan_1.0 org.pytorch.serve.wlm.WorkerThread - Retry worker: 9000 in 13 seconds. 2025-01-20T18:09:02,759 [INFO ] W-9000-titan_1.0-stderr org.pytorch.serve.wlm.WorkerLifeCycle - Stopped Scanner - W-9000-titan_1.0-stderr 2025-01-20T18:09:02,759 [INFO ] W-9000-titan_1.0-stderr org.pytorch.serve.wlm.WorkerLifeCycle - Stopped Scanner - W-9000-titan_1.0-stderr 2025-01-20T18:09:02,760 [INFO ] W-9000-titan_1.0-stdout org.pytorch.serve.wlm.WorkerLifeCycle - Stopped Scanner - W-9000-titan_1.0-stdout 2025-01-20T18:09:02,760 [INFO ] W-9000-titan_1.0-stdout org.pytorch.serve.wlm.WorkerLifeCycle - Stopped Scanner - W-9000-titan_1.0-stdout 2025-01-20T18:09:15,715 [DEBUG] W-9000-titan_1.0 org.pytorch.serve.wlm.WorkerLifeCycle - Worker cmdline: [/usr/bin/python3, /usr/local/lib/python3.11/dist-packages/ts/model_service_worker.py, --sock-type, unix, --sock-name, /tmp/.ts.sock.9000, --metrics-config, /usr/local/lib/python3.11/dist-packages/ts/configs/metrics.yaml] 2025-01-20T18:09:15,715 [DEBUG] W-9000-titan_1.0 org.pytorch.serve.wlm.WorkerLifeCycle - Worker cmdline: [/usr/bin/python3, /usr/local/lib/python3.11/dist-packages/ts/model_service_worker.py, --sock-type, unix, --sock-name, /tmp/.ts.sock.9000, --metrics-config, /usr/local/lib/python3.11/dist-packages/ts/configs/metrics.yaml] 2025-01-20T18:09:17,832 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - s_name_part0=/tmp/.ts.sock, s_name_part1=9000, pid=12416 2025-01-20T18:09:17,833 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - Listening on port: /tmp/.ts.sock.9000 2025-01-20T18:09:17,840 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - Successfully loaded /usr/local/lib/python3.11/dist-packages/ts/configs/metrics.yaml. 2025-01-20T18:09:17,841 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - [PID]12416 2025-01-20T18:09:17,841 [DEBUG] W-9000-titan_1.0 org.pytorch.serve.wlm.WorkerThread - W-9000-titan_1.0 State change WORKER_STOPPED -> WORKER_STARTED 2025-01-20T18:09:17,841 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - Torch worker started. 2025-01-20T18:09:17,842 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - Python runtime: 3.11.11 2025-01-20T18:09:17,841 [DEBUG] W-9000-titan_1.0 org.pytorch.serve.wlm.WorkerThread - W-9000-titan_1.0 State change WORKER_STOPPED -> WORKER_STARTED 2025-01-20T18:09:17,842 [INFO ] W-9000-titan_1.0 org.pytorch.serve.wlm.WorkerThread - Connecting to: /tmp/.ts.sock.9000 2025-01-20T18:09:17,842 [INFO ] W-9000-titan_1.0 org.pytorch.serve.wlm.WorkerThread - Connecting to: /tmp/.ts.sock.9000 2025-01-20T18:09:17,843 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - Connection accepted: /tmp/.ts.sock.9000. 2025-01-20T18:09:17,843 [DEBUG] W-9000-titan_1.0 org.pytorch.serve.wlm.WorkerThread - Flushing req.cmd LOAD repeats 1 to backend at: 1737396557843 2025-01-20T18:09:17,843 [DEBUG] W-9000-titan_1.0 org.pytorch.serve.wlm.WorkerThread - Flushing req.cmd LOAD repeats 1 to backend at: 1737396557843 2025-01-20T18:09:17,844 [INFO ] W-9000-titan_1.0 org.pytorch.serve.wlm.WorkerThread - Looping backend response at: 1737396557844 2025-01-20T18:09:17,844 [INFO ] W-9000-titan_1.0 org.pytorch.serve.wlm.WorkerThread - Looping backend response at: 1737396557844 2025-01-20T18:09:17,845 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - model_name: titan, batchSize: 1 2025-01-20T18:09:17,847 [INFO ] epollEventLoopGroup-5-4 org.pytorch.serve.wlm.WorkerThread - 9000 Worker disconnected. WORKER_STARTED 2025-01-20T18:09:17,847 [INFO ] epollEventLoopGroup-5-4 org.pytorch.serve.wlm.WorkerThread - 9000 Worker disconnected. WORKER_STARTED 2025-01-20T18:09:17,848 [DEBUG] W-9000-titan_1.0 org.pytorch.serve.wlm.WorkerThread - System state is : WORKER_STARTED 2025-01-20T18:09:17,848 [DEBUG] W-9000-titan_1.0 org.pytorch.serve.wlm.WorkerThread - System state is : WORKER_STARTED 2025-01-20T18:09:17,848 [DEBUG] W-9000-titan_1.0 org.pytorch.serve.wlm.WorkerThread - Backend worker monitoring thread interrupted or backend worker process died., startupTimeout:120sec java.lang.InterruptedException: null at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2056) ~[?:?] at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2133) ~[?:?] at java.util.concurrent.ArrayBlockingQueue.poll(ArrayBlockingQueue.java:432) ~[?:?] at org.pytorch.serve.wlm.WorkerThread.run(WorkerThread.java:234) [model-server.jar:?] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?] at java.lang.Thread.run(Thread.java:829) [?:?] 2025-01-20T18:09:17,848 [DEBUG] W-9000-titan_1.0 org.pytorch.serve.wlm.WorkerThread - Backend worker monitoring thread interrupted or backend worker process died., startupTimeout:120sec java.lang.InterruptedException: null at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2056) ~[?:?] at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2133) ~[?:?] at java.util.concurrent.ArrayBlockingQueue.poll(ArrayBlockingQueue.java:432) ~[?:?] at org.pytorch.serve.wlm.WorkerThread.run(WorkerThread.java:234) [model-server.jar:?] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?] at java.lang.Thread.run(Thread.java:829) [?:?] 2025-01-20T18:09:17,848 [WARN ] W-9000-titan_1.0 org.pytorch.serve.wlm.BatchAggregator - Load model failed: titan, error: Worker died. 2025-01-20T18:09:17,848 [WARN ] W-9000-titan_1.0 org.pytorch.serve.wlm.BatchAggregator - Load model failed: titan, error: Worker died. 2025-01-20T18:09:17,849 [DEBUG] W-9000-titan_1.0 org.pytorch.serve.wlm.WorkerThread - W-9000-titan_1.0 State change WORKER_STARTED -> WORKER_STOPPED 2025-01-20T18:09:17,849 [DEBUG] W-9000-titan_1.0 org.pytorch.serve.wlm.WorkerThread - W-9000-titan_1.0 State change WORKER_STARTED -> WORKER_STOPPED 2025-01-20T18:09:17,849 [WARN ] W-9000-titan_1.0 org.pytorch.serve.wlm.WorkerThread - Auto recovery failed again 2025-01-20T18:09:17,849 [WARN ] W-9000-titan_1.0 org.pytorch.serve.wlm.WorkerThread - Auto recovery failed again 2025-01-20T18:09:17,849 [INFO ] W-9000-titan_1.0 org.pytorch.serve.wlm.WorkerThread - Retry worker: 9000 in 21 seconds. 2025-01-20T18:09:17,849 [INFO ] W-9000-titan_1.0 org.pytorch.serve.wlm.WorkerThread - Retry worker: 9000 in 21 seconds. 2025-01-20T18:09:17,847 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - Backend worker process died. 2025-01-20T18:09:17,853 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - Traceback (most recent call last): 2025-01-20T18:09:17,853 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - File "/usr/local/lib/python3.11/dist-packages/ts/model_service_worker.py", line 301, in 2025-01-20T18:09:17,853 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - worker.run_server() 2025-01-20T18:09:17,854 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - File "/usr/local/lib/python3.11/dist-packages/ts/model_service_worker.py", line 268, in run_server 2025-01-20T18:09:17,854 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - self.handle_connection(cl_socket) 2025-01-20T18:09:17,854 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - File "/usr/local/lib/python3.11/dist-packages/ts/model_service_worker.py", line 196, in handle_connection 2025-01-20T18:09:17,855 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - service, result, code = self.load_model(msg) 2025-01-20T18:09:17,855 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - ^^^^^^^^^^^^^^^^^^^^ 2025-01-20T18:09:17,856 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - File "/usr/local/lib/python3.11/dist-packages/ts/model_service_worker.py", line 133, in load_model 2025-01-20T18:09:17,856 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - service = model_loader.load( 2025-01-20T18:09:17,856 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - ^^^^^^^^^^^^^^^^^^ 2025-01-20T18:09:17,856 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - File "/usr/local/lib/python3.11/dist-packages/ts/model_loader.py", line 125, in load 2025-01-20T18:09:17,856 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - entry_point, initialize_fn = self._get_class_entry_point(module) 2025-01-20T18:09:17,856 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-01-20T18:09:17,856 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - File "/usr/local/lib/python3.11/dist-packages/ts/model_loader.py", line 178, in _get_class_entry_point 2025-01-20T18:09:17,856 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - raise ValueError( 2025-01-20T18:09:17,856 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - ValueError: Expected only one class in custom service code or a function entry point [, , , , ] 2025-01-20T18:09:17,857 [INFO ] W-9000-titan_1.0-stdout org.pytorch.serve.wlm.WorkerLifeCycle - Stopped Scanner - W-9000-titan_1.0-stdout 2025-01-20T18:09:17,857 [INFO ] W-9000-titan_1.0-stdout org.pytorch.serve.wlm.WorkerLifeCycle - Stopped Scanner - W-9000-titan_1.0-stdout 2025-01-20T18:09:17,884 [INFO ] W-9000-titan_1.0-stderr org.pytorch.serve.wlm.WorkerLifeCycle - Stopped Scanner - W-9000-titan_1.0-stderr 2025-01-20T18:09:17,884 [INFO ] W-9000-titan_1.0-stderr org.pytorch.serve.wlm.WorkerLifeCycle - Stopped Scanner - W-9000-titan_1.0-stderr 2025-01-20T18:09:30,184 [ERROR] Thread-10 org.pytorch.serve.metrics.MetricCollector - Traceback (most recent call last): File "/usr/local/lib/python3.11/dist-packages/ts/metrics/metric_collector.py", line 11, in from ts.metrics import system_metrics File "/usr/local/lib/python3.11/dist-packages/ts/__init__.py", line 10, in from . import version File "/usr/local/lib/python3.11/dist-packages/ts/version.py", line 5, in from pathlib import Path File "/usr/local/lib/python3.11/dist-packages/pathlib.py", line 10, in from collections import Sequence ImportError: cannot import name 'Sequence' from 'collections' (/usr/lib/python3.11/collections/__init__.py) 2025-01-20T18:09:30,184 [ERROR] Thread-10 org.pytorch.serve.metrics.MetricCollector - Traceback (most recent call last): File "/usr/local/lib/python3.11/dist-packages/ts/metrics/metric_collector.py", line 11, in from ts.metrics import system_metrics File "/usr/local/lib/python3.11/dist-packages/ts/__init__.py", line 10, in from . import version File "/usr/local/lib/python3.11/dist-packages/ts/version.py", line 5, in from pathlib import Path File "/usr/local/lib/python3.11/dist-packages/pathlib.py", line 10, in from collections import Sequence ImportError: cannot import name 'Sequence' from 'collections' (/usr/lib/python3.11/collections/__init__.py) 2025-01-20T18:09:38,850 [DEBUG] W-9000-titan_1.0 org.pytorch.serve.wlm.WorkerLifeCycle - Worker cmdline: [/usr/bin/python3, /usr/local/lib/python3.11/dist-packages/ts/model_service_worker.py, --sock-type, unix, --sock-name, /tmp/.ts.sock.9000, --metrics-config, /usr/local/lib/python3.11/dist-packages/ts/configs/metrics.yaml] 2025-01-20T18:09:38,850 [DEBUG] W-9000-titan_1.0 org.pytorch.serve.wlm.WorkerLifeCycle - Worker cmdline: [/usr/bin/python3, /usr/local/lib/python3.11/dist-packages/ts/model_service_worker.py, --sock-type, unix, --sock-name, /tmp/.ts.sock.9000, --metrics-config, /usr/local/lib/python3.11/dist-packages/ts/configs/metrics.yaml] 2025-01-20T18:09:40,971 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - s_name_part0=/tmp/.ts.sock, s_name_part1=9000, pid=12521 2025-01-20T18:09:40,972 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - Listening on port: /tmp/.ts.sock.9000 2025-01-20T18:09:40,988 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - Successfully loaded /usr/local/lib/python3.11/dist-packages/ts/configs/metrics.yaml. 2025-01-20T18:09:40,988 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - [PID]12521 2025-01-20T18:09:40,989 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - Torch worker started. 2025-01-20T18:09:40,989 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - Python runtime: 3.11.11 2025-01-20T18:09:40,989 [DEBUG] W-9000-titan_1.0 org.pytorch.serve.wlm.WorkerThread - W-9000-titan_1.0 State change WORKER_STOPPED -> WORKER_STARTED 2025-01-20T18:09:40,989 [DEBUG] W-9000-titan_1.0 org.pytorch.serve.wlm.WorkerThread - W-9000-titan_1.0 State change WORKER_STOPPED -> WORKER_STARTED 2025-01-20T18:09:40,989 [INFO ] W-9000-titan_1.0 org.pytorch.serve.wlm.WorkerThread - Connecting to: /tmp/.ts.sock.9000 2025-01-20T18:09:40,989 [INFO ] W-9000-titan_1.0 org.pytorch.serve.wlm.WorkerThread - Connecting to: /tmp/.ts.sock.9000 2025-01-20T18:09:40,990 [DEBUG] W-9000-titan_1.0 org.pytorch.serve.wlm.WorkerThread - Flushing req.cmd LOAD repeats 1 to backend at: 1737396580990 2025-01-20T18:09:40,990 [DEBUG] W-9000-titan_1.0 org.pytorch.serve.wlm.WorkerThread - Flushing req.cmd LOAD repeats 1 to backend at: 1737396580990 2025-01-20T18:09:40,991 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - Connection accepted: /tmp/.ts.sock.9000. 2025-01-20T18:09:40,991 [INFO ] W-9000-titan_1.0 org.pytorch.serve.wlm.WorkerThread - Looping backend response at: 1737396580991 2025-01-20T18:09:40,991 [INFO ] W-9000-titan_1.0 org.pytorch.serve.wlm.WorkerThread - Looping backend response at: 1737396580991 2025-01-20T18:09:40,992 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - model_name: titan, batchSize: 1 2025-01-20T18:09:40,994 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - Backend worker process died. 2025-01-20T18:09:40,995 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - Traceback (most recent call last): 2025-01-20T18:09:40,995 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - File "/usr/local/lib/python3.11/dist-packages/ts/model_service_worker.py", line 301, in 2025-01-20T18:09:40,995 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - worker.run_server() 2025-01-20T18:09:40,995 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - File "/usr/local/lib/python3.11/dist-packages/ts/model_service_worker.py", line 268, in run_server 2025-01-20T18:09:40,995 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - self.handle_connection(cl_socket) 2025-01-20T18:09:40,995 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - File "/usr/local/lib/python3.11/dist-packages/ts/model_service_worker.py", line 196, in handle_connection 2025-01-20T18:09:40,995 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - service, result, code = self.load_model(msg) 2025-01-20T18:09:40,995 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - ^^^^^^^^^^^^^^^^^^^^ 2025-01-20T18:09:40,995 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - File "/usr/local/lib/python3.11/dist-packages/ts/model_service_worker.py", line 133, in load_model 2025-01-20T18:09:40,995 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - service = model_loader.load( 2025-01-20T18:09:40,995 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - ^^^^^^^^^^^^^^^^^^ 2025-01-20T18:09:40,996 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - File "/usr/local/lib/python3.11/dist-packages/ts/model_loader.py", line 125, in load 2025-01-20T18:09:40,996 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - entry_point, initialize_fn = self._get_class_entry_point(module) 2025-01-20T18:09:40,996 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-01-20T18:09:40,996 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - File "/usr/local/lib/python3.11/dist-packages/ts/model_loader.py", line 178, in _get_class_entry_point 2025-01-20T18:09:40,996 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - raise ValueError( 2025-01-20T18:09:40,996 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - ValueError: Expected only one class in custom service code or a function entry point [, , , , ] 2025-01-20T18:09:41,004 [INFO ] epollEventLoopGroup-5-1 org.pytorch.serve.wlm.WorkerThread - 9000 Worker disconnected. WORKER_STARTED 2025-01-20T18:09:41,004 [INFO ] epollEventLoopGroup-5-1 org.pytorch.serve.wlm.WorkerThread - 9000 Worker disconnected. WORKER_STARTED 2025-01-20T18:09:41,004 [DEBUG] W-9000-titan_1.0 org.pytorch.serve.wlm.WorkerThread - System state is : WORKER_STARTED 2025-01-20T18:09:41,004 [DEBUG] W-9000-titan_1.0 org.pytorch.serve.wlm.WorkerThread - System state is : WORKER_STARTED 2025-01-20T18:09:41,004 [DEBUG] W-9000-titan_1.0 org.pytorch.serve.wlm.WorkerThread - Backend worker monitoring thread interrupted or backend worker process died., startupTimeout:120sec java.lang.InterruptedException: null at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2056) ~[?:?] at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2133) ~[?:?] at java.util.concurrent.ArrayBlockingQueue.poll(ArrayBlockingQueue.java:432) ~[?:?] at org.pytorch.serve.wlm.WorkerThread.run(WorkerThread.java:234) [model-server.jar:?] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?] at java.lang.Thread.run(Thread.java:829) [?:?] 2025-01-20T18:09:41,004 [DEBUG] W-9000-titan_1.0 org.pytorch.serve.wlm.WorkerThread - Backend worker monitoring thread interrupted or backend worker process died., startupTimeout:120sec java.lang.InterruptedException: null at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2056) ~[?:?] at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2133) ~[?:?] at java.util.concurrent.ArrayBlockingQueue.poll(ArrayBlockingQueue.java:432) ~[?:?] at org.pytorch.serve.wlm.WorkerThread.run(WorkerThread.java:234) [model-server.jar:?] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?] at java.lang.Thread.run(Thread.java:829) [?:?] 2025-01-20T18:09:41,005 [WARN ] W-9000-titan_1.0 org.pytorch.serve.wlm.BatchAggregator - Load model failed: titan, error: Worker died. 2025-01-20T18:09:41,005 [WARN ] W-9000-titan_1.0 org.pytorch.serve.wlm.BatchAggregator - Load model failed: titan, error: Worker died. 2025-01-20T18:09:41,005 [DEBUG] W-9000-titan_1.0 org.pytorch.serve.wlm.WorkerThread - W-9000-titan_1.0 State change WORKER_STARTED -> WORKER_STOPPED 2025-01-20T18:09:41,005 [DEBUG] W-9000-titan_1.0 org.pytorch.serve.wlm.WorkerThread - W-9000-titan_1.0 State change WORKER_STARTED -> WORKER_STOPPED 2025-01-20T18:09:41,005 [WARN ] W-9000-titan_1.0 org.pytorch.serve.wlm.WorkerThread - Auto recovery failed again 2025-01-20T18:09:41,005 [WARN ] W-9000-titan_1.0 org.pytorch.serve.wlm.WorkerThread - Auto recovery failed again 2025-01-20T18:09:41,005 [INFO ] W-9000-titan_1.0 org.pytorch.serve.wlm.WorkerThread - Retry worker: 9000 in 34 seconds. 2025-01-20T18:09:41,005 [INFO ] W-9000-titan_1.0 org.pytorch.serve.wlm.WorkerThread - Retry worker: 9000 in 34 seconds. 2025-01-20T18:09:41,046 [INFO ] W-9000-titan_1.0-stderr org.pytorch.serve.wlm.WorkerLifeCycle - Stopped Scanner - W-9000-titan_1.0-stderr 2025-01-20T18:09:41,046 [INFO ] W-9000-titan_1.0-stderr org.pytorch.serve.wlm.WorkerLifeCycle - Stopped Scanner - W-9000-titan_1.0-stderr 2025-01-20T18:09:41,048 [INFO ] W-9000-titan_1.0-stdout org.pytorch.serve.wlm.WorkerLifeCycle - Stopped Scanner - W-9000-titan_1.0-stdout 2025-01-20T18:09:41,048 [INFO ] W-9000-titan_1.0-stdout org.pytorch.serve.wlm.WorkerLifeCycle - Stopped Scanner - W-9000-titan_1.0-stdout 2025-01-20T18:10:15,006 [DEBUG] W-9000-titan_1.0 org.pytorch.serve.wlm.WorkerLifeCycle - Worker cmdline: [/usr/bin/python3, /usr/local/lib/python3.11/dist-packages/ts/model_service_worker.py, --sock-type, unix, --sock-name, /tmp/.ts.sock.9000, --metrics-config, /usr/local/lib/python3.11/dist-packages/ts/configs/metrics.yaml] 2025-01-20T18:10:15,006 [DEBUG] W-9000-titan_1.0 org.pytorch.serve.wlm.WorkerLifeCycle - Worker cmdline: [/usr/bin/python3, /usr/local/lib/python3.11/dist-packages/ts/model_service_worker.py, --sock-type, unix, --sock-name, /tmp/.ts.sock.9000, --metrics-config, /usr/local/lib/python3.11/dist-packages/ts/configs/metrics.yaml] 2025-01-20T18:10:16,670 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - s_name_part0=/tmp/.ts.sock, s_name_part1=9000, pid=12680 2025-01-20T18:10:16,686 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - Listening on port: /tmp/.ts.sock.9000 2025-01-20T18:10:16,697 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - Successfully loaded /usr/local/lib/python3.11/dist-packages/ts/configs/metrics.yaml. 2025-01-20T18:10:16,698 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - [PID]12680 2025-01-20T18:10:16,698 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - Torch worker started. 2025-01-20T18:10:16,698 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - Python runtime: 3.11.11 2025-01-20T18:10:16,699 [DEBUG] W-9000-titan_1.0 org.pytorch.serve.wlm.WorkerThread - W-9000-titan_1.0 State change WORKER_STOPPED -> WORKER_STARTED 2025-01-20T18:10:16,699 [DEBUG] W-9000-titan_1.0 org.pytorch.serve.wlm.WorkerThread - W-9000-titan_1.0 State change WORKER_STOPPED -> WORKER_STARTED 2025-01-20T18:10:16,699 [INFO ] W-9000-titan_1.0 org.pytorch.serve.wlm.WorkerThread - Connecting to: /tmp/.ts.sock.9000 2025-01-20T18:10:16,699 [INFO ] W-9000-titan_1.0 org.pytorch.serve.wlm.WorkerThread - Connecting to: /tmp/.ts.sock.9000 2025-01-20T18:10:16,701 [DEBUG] W-9000-titan_1.0 org.pytorch.serve.wlm.WorkerThread - Flushing req.cmd LOAD repeats 1 to backend at: 1737396616700 2025-01-20T18:10:16,701 [DEBUG] W-9000-titan_1.0 org.pytorch.serve.wlm.WorkerThread - Flushing req.cmd LOAD repeats 1 to backend at: 1737396616700 2025-01-20T18:10:16,702 [INFO ] W-9000-titan_1.0 org.pytorch.serve.wlm.WorkerThread - Looping backend response at: 1737396616702 2025-01-20T18:10:16,702 [INFO ] W-9000-titan_1.0 org.pytorch.serve.wlm.WorkerThread - Looping backend response at: 1737396616702 2025-01-20T18:10:16,704 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - Connection accepted: /tmp/.ts.sock.9000. 2025-01-20T18:10:16,705 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - model_name: titan, batchSize: 1 2025-01-20T18:10:16,707 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - Backend worker process died. 2025-01-20T18:10:16,708 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - Traceback (most recent call last): 2025-01-20T18:10:16,709 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - File "/usr/local/lib/python3.11/dist-packages/ts/model_service_worker.py", line 301, in 2025-01-20T18:10:16,709 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - worker.run_server() 2025-01-20T18:10:16,710 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - File "/usr/local/lib/python3.11/dist-packages/ts/model_service_worker.py", line 268, in run_server 2025-01-20T18:10:16,711 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - self.handle_connection(cl_socket) 2025-01-20T18:10:16,711 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - File "/usr/local/lib/python3.11/dist-packages/ts/model_service_worker.py", line 196, in handle_connection 2025-01-20T18:10:16,712 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - service, result, code = self.load_model(msg) 2025-01-20T18:10:16,717 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - ^^^^^^^^^^^^^^^^^^^^ 2025-01-20T18:10:16,718 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - File "/usr/local/lib/python3.11/dist-packages/ts/model_service_worker.py", line 133, in load_model 2025-01-20T18:10:16,718 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - service = model_loader.load( 2025-01-20T18:10:16,719 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - ^^^^^^^^^^^^^^^^^^ 2025-01-20T18:10:16,719 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - File "/usr/local/lib/python3.11/dist-packages/ts/model_loader.py", line 125, in load 2025-01-20T18:10:16,720 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - entry_point, initialize_fn = self._get_class_entry_point(module) 2025-01-20T18:10:16,721 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-01-20T18:10:16,721 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - File "/usr/local/lib/python3.11/dist-packages/ts/model_loader.py", line 178, in _get_class_entry_point 2025-01-20T18:10:16,722 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - raise ValueError( 2025-01-20T18:10:16,722 [INFO ] W-9000-titan_1.0-stdout MODEL_LOG - ValueError: Expected only one class in custom service code or a function entry point [, , , , ] 2025-01-20T18:10:16,735 [INFO ] epollEventLoopGroup-5-2 org.pytorch.serve.wlm.WorkerThread - 9000 Worker disconnected. WORKER_STARTED 2025-01-20T18:10:16,735 [INFO ] epollEventLoopGroup-5-2 org.pytorch.serve.wlm.WorkerThread - 9000 Worker disconnected. WORKER_STARTED 2025-01-20T18:10:16,735 [DEBUG] W-9000-titan_1.0 org.pytorch.serve.wlm.WorkerThread - System state is : WORKER_STARTED 2025-01-20T18:10:16,735 [DEBUG] W-9000-titan_1.0 org.pytorch.serve.wlm.WorkerThread - System state is : WORKER_STARTED 2025-01-20T18:10:16,735 [DEBUG] W-9000-titan_1.0 org.pytorch.serve.wlm.WorkerThread - Backend worker monitoring thread interrupted or backend worker process died., startupTimeout:120sec java.lang.InterruptedException: null at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2056) ~[?:?] at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2133) ~[?:?] at java.util.concurrent.ArrayBlockingQueue.poll(ArrayBlockingQueue.java:432) ~[?:?] at org.pytorch.serve.wlm.WorkerThread.run(WorkerThread.java:234) [model-server.jar:?] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?] at java.lang.Thread.run(Thread.java:829) [?:?] 2025-01-20T18:10:16,735 [DEBUG] W-9000-titan_1.0 org.pytorch.serve.wlm.WorkerThread - Backend worker monitoring thread interrupted or backend worker process died., startupTimeout:120sec java.lang.InterruptedException: null at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2056) ~[?:?] at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2133) ~[?:?] at java.util.concurrent.ArrayBlockingQueue.poll(ArrayBlockingQueue.java:432) ~[?:?] at org.pytorch.serve.wlm.WorkerThread.run(WorkerThread.java:234) [model-server.jar:?] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?] at java.lang.Thread.run(Thread.java:829) [?:?] 2025-01-20T18:10:16,736 [WARN ] W-9000-titan_1.0 org.pytorch.serve.wlm.BatchAggregator - Load model failed: titan, error: Worker died. 2025-01-20T18:10:16,736 [WARN ] W-9000-titan_1.0 org.pytorch.serve.wlm.BatchAggregator - Load model failed: titan, error: Worker died. 2025-01-20T18:10:16,736 [DEBUG] W-9000-titan_1.0 org.pytorch.serve.wlm.WorkerThread - W-9000-titan_1.0 State change WORKER_STARTED -> WORKER_STOPPED 2025-01-20T18:10:16,736 [DEBUG] W-9000-titan_1.0 org.pytorch.serve.wlm.WorkerThread - W-9000-titan_1.0 State change WORKER_STARTED -> WORKER_STOPPED 2025-01-20T18:10:16,736 [WARN ] W-9000-titan_1.0 org.pytorch.serve.wlm.WorkerThread - Auto recovery failed again 2025-01-20T18:10:16,736 [WARN ] W-9000-titan_1.0 org.pytorch.serve.wlm.WorkerThread - Auto recovery failed again 2025-01-20T18:10:16,736 [INFO ] W-9000-titan_1.0 org.pytorch.serve.wlm.WorkerThread - Retry worker: 9000 in 55 seconds. 2025-01-20T18:10:16,736 [INFO ] W-9000-titan_1.0 org.pytorch.serve.wlm.WorkerThread - Retry worker: 9000 in 55 seconds. 2025-01-20T18:10:16,787 [INFO ] W-9000-titan_1.0-stderr org.pytorch.serve.wlm.WorkerLifeCycle - Stopped Scanner - W-9000-titan_1.0-stderr 2025-01-20T18:10:16,787 [INFO ] W-9000-titan_1.0-stderr org.pytorch.serve.wlm.WorkerLifeCycle - Stopped Scanner - W-9000-titan_1.0-stderr 2025-01-20T18:10:16,789 [INFO ] W-9000-titan_1.0-stdout org.pytorch.serve.wlm.WorkerLifeCycle - Stopped Scanner - W-9000-titan_1.0-stdout 2025-01-20T18:10:16,789 [INFO ] W-9000-titan_1.0-stdout org.pytorch.serve.wlm.WorkerLifeCycle - Stopped Scanner - W-9000-titan_1.0-stdout 2025-01-20T18:10:30,203 [ERROR] Thread-13 org.pytorch.serve.metrics.MetricCollector - Traceback (most recent call last): File "/usr/local/lib/python3.11/dist-packages/ts/metrics/metric_collector.py", line 11, in from ts.metrics import system_metrics File "/usr/local/lib/python3.11/dist-packages/ts/__init__.py", line 10, in from . import version File "/usr/local/lib/python3.11/dist-packages/ts/version.py", line 5, in from pathlib import Path File "/usr/local/lib/python3.11/dist-packages/pathlib.py", line 10, in from collections import Sequence ImportError: cannot import name 'Sequence' from 'collections' (/usr/lib/python3.11/collections/__init__.py) 2025-01-20T18:10:30,203 [ERROR] Thread-13 org.pytorch.serve.metrics.MetricCollector - Traceback (most recent call last): File "/usr/local/lib/python3.11/dist-packages/ts/metrics/metric_collector.py", line 11, in from ts.metrics import system_metrics File "/usr/local/lib/python3.11/dist-packages/ts/__init__.py", line 10, in from . import version File "/usr/local/lib/python3.11/dist-packages/ts/version.py", line 5, in from pathlib import Path File "/usr/local/lib/python3.11/dist-packages/pathlib.py", line 10, in from collections import Sequence ImportError: cannot import name 'Sequence' from 'collections' (/usr/lib/python3.11/collections/__init__.py)