lbourdois commited on
Commit
ab06693
·
verified ·
1 Parent(s): 1208532

Improve language tag

Browse files

Hi! As the model is multilingual, this is a PR to add other languages than English to the language tag to improve the referencing. Note that 29 languages are announced in the README, but only 13 are explicitly listed. I was therefore only able to add these 13 languages.

Files changed (1) hide show
  1. README.md +74 -63
README.md CHANGED
@@ -1,64 +1,75 @@
1
- ---
2
- license: cc-by-nc-nd-4.0
3
- base_model:
4
- - Qwen/Qwen2.5-1.5B
5
- language:
6
- - en
7
- - de
8
- tags:
9
- - Function_Call
10
- - Automotive
11
- - SLM
12
- - GGUF
13
- ---
14
-
15
- # Qwen2.5-1.5B-Auto-FunctionCaller
16
-
17
- ## Model Details
18
-
19
- * **Model Name:** Qwen2.5-1.5B-Auto-FunctionCaller
20
- * **Base Model:** [Qwen/Qwen2.5-1.5B](https://huggingface.co/Qwen/Qwen2.5-1.5B)
21
- * **Model Type:** Language Model fine-tuned for Function Calling.
22
- * **Recommended Quantization:** `Qwen2.5-1.5B-Auto-FunctionCaller.Q4_K_M_I.gguf`
23
- * This GGUF file using Q4\_K\_M quantization with Importance Matrix is recommended as offering the best balance between performance and computational efficiency (inference speed, memory usage) based on evaluation.
24
-
25
- ## Intended Use
26
-
27
- * **Primary Use:** Function calling extraction from natural language queries within an automotive context. The model is designed to identify user intent and extract relevant parameters (arguments/slots) for triggering vehicle functions or infotainment actions.
28
- * **Research Context:** This model was specifically developed and fine-tuned as part of a research publication investigating the feasibility and performance of Small Language Models (SLMs) for function-calling tasks in resource-constrained automotive environments.
29
- * **Target Environment:** Embedded systems or edge devices within vehicles where computational resources may be limited.
30
- * **Out-of-Scope Uses:** General conversational AI, creative writing, tasks outside automotive function calling, safety-critical vehicle control.
31
-
32
- ## Performance Metrics
33
-
34
- The following metrics were evaluated on the `Qwen2.5-1.5B-Auto-FunctionCaller.Q4_K_M_I.gguf` model:
35
-
36
- * **Evaluation Setup:**
37
- * Total Evaluation Samples: 2074
38
- * **Performance:**
39
- * **Exact Match Accuracy:** 0.8414
40
- * **Average Component Accuracy:** 0.9352
41
- * **Efficiency & Confidence:**
42
- * **Throughput:** 10.31 tokens/second
43
- * **Latency (Per Token):** 0.097 seconds
44
- * **Latency (Per Instruction):** 0.427 seconds
45
- * **Average Model Confidence:** 0.9005
46
- * **Calibration Error:** 0.0854
47
-
48
- *Note: Latency and throughput figures are hardware-dependent and should be benchmarked on the target deployment environment.*
49
-
50
- ## Limitations
51
-
52
- * **Domain Specificity:** Performance is optimized for automotive function calling. Generalization to other domains or complex, non-structured conversations may be limited.
53
- * **Quantization Impact:** The `Q4_K_M_I` quantization significantly improves efficiency but may result in a slight reduction in accuracy compared to higher-precision versions (e.g., FP16).
54
- * **Complex Queries:** May struggle with highly nested, ambiguous, or unusually phrased requests not well-represented in the fine-tuning data.
55
- * **Safety Criticality:** This model is **not** intended or validated for safety-critical vehicle operations (e.g., braking, steering). Use should be restricted to non-critical systems like infotainment and comfort controls.
56
- * **Bias:** Like any model, performance and fairness depend on the underlying data. Biases present in the fine-tuning or evaluation datasets may be reflected in the model's behavior.
57
-
58
- ## Training Data (Summary)
59
-
60
- The model was fine-tuned on a synthetic dataset specifically curated for automotive function calling tasks. Details will be referenced in the associated publication.
61
-
62
- ## Citation
63
-
 
 
 
 
 
 
 
 
 
 
 
64
  TBD
 
1
+ ---
2
+ license: cc-by-nc-nd-4.0
3
+ base_model:
4
+ - Qwen/Qwen2.5-1.5B
5
+ language:
6
+ - zho
7
+ - eng
8
+ - fra
9
+ - spa
10
+ - por
11
+ - deu
12
+ - ita
13
+ - rus
14
+ - jpn
15
+ - kor
16
+ - vie
17
+ - tha
18
+ - ara
19
+ tags:
20
+ - Function_Call
21
+ - Automotive
22
+ - SLM
23
+ - GGUF
24
+ ---
25
+
26
+ # Qwen2.5-1.5B-Auto-FunctionCaller
27
+
28
+ ## Model Details
29
+
30
+ * **Model Name:** Qwen2.5-1.5B-Auto-FunctionCaller
31
+ * **Base Model:** [Qwen/Qwen2.5-1.5B](https://huggingface.co/Qwen/Qwen2.5-1.5B)
32
+ * **Model Type:** Language Model fine-tuned for Function Calling.
33
+ * **Recommended Quantization:** `Qwen2.5-1.5B-Auto-FunctionCaller.Q4_K_M_I.gguf`
34
+ * This GGUF file using Q4\_K\_M quantization with Importance Matrix is recommended as offering the best balance between performance and computational efficiency (inference speed, memory usage) based on evaluation.
35
+
36
+ ## Intended Use
37
+
38
+ * **Primary Use:** Function calling extraction from natural language queries within an automotive context. The model is designed to identify user intent and extract relevant parameters (arguments/slots) for triggering vehicle functions or infotainment actions.
39
+ * **Research Context:** This model was specifically developed and fine-tuned as part of a research publication investigating the feasibility and performance of Small Language Models (SLMs) for function-calling tasks in resource-constrained automotive environments.
40
+ * **Target Environment:** Embedded systems or edge devices within vehicles where computational resources may be limited.
41
+ * **Out-of-Scope Uses:** General conversational AI, creative writing, tasks outside automotive function calling, safety-critical vehicle control.
42
+
43
+ ## Performance Metrics
44
+
45
+ The following metrics were evaluated on the `Qwen2.5-1.5B-Auto-FunctionCaller.Q4_K_M_I.gguf` model:
46
+
47
+ * **Evaluation Setup:**
48
+ * Total Evaluation Samples: 2074
49
+ * **Performance:**
50
+ * **Exact Match Accuracy:** 0.8414
51
+ * **Average Component Accuracy:** 0.9352
52
+ * **Efficiency & Confidence:**
53
+ * **Throughput:** 10.31 tokens/second
54
+ * **Latency (Per Token):** 0.097 seconds
55
+ * **Latency (Per Instruction):** 0.427 seconds
56
+ * **Average Model Confidence:** 0.9005
57
+ * **Calibration Error:** 0.0854
58
+
59
+ *Note: Latency and throughput figures are hardware-dependent and should be benchmarked on the target deployment environment.*
60
+
61
+ ## Limitations
62
+
63
+ * **Domain Specificity:** Performance is optimized for automotive function calling. Generalization to other domains or complex, non-structured conversations may be limited.
64
+ * **Quantization Impact:** The `Q4_K_M_I` quantization significantly improves efficiency but may result in a slight reduction in accuracy compared to higher-precision versions (e.g., FP16).
65
+ * **Complex Queries:** May struggle with highly nested, ambiguous, or unusually phrased requests not well-represented in the fine-tuning data.
66
+ * **Safety Criticality:** This model is **not** intended or validated for safety-critical vehicle operations (e.g., braking, steering). Use should be restricted to non-critical systems like infotainment and comfort controls.
67
+ * **Bias:** Like any model, performance and fairness depend on the underlying data. Biases present in the fine-tuning or evaluation datasets may be reflected in the model's behavior.
68
+
69
+ ## Training Data (Summary)
70
+
71
+ The model was fine-tuned on a synthetic dataset specifically curated for automotive function calling tasks. Details will be referenced in the associated publication.
72
+
73
+ ## Citation
74
+
75
  TBD