cormacb commited on
Commit
3796dba
·
verified ·
1 Parent(s): f50a33f

Upload folder using huggingface_hub

Browse files
.gitattributes CHANGED
@@ -33,3 +33,5 @@ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
33
  *.zip filter=lfs diff=lfs merge=lfs -text
34
  *.zst filter=lfs diff=lfs merge=lfs -text
35
  *tfevents* filter=lfs diff=lfs merge=lfs -text
 
 
 
33
  *.zip filter=lfs diff=lfs merge=lfs -text
34
  *.zst filter=lfs diff=lfs merge=lfs -text
35
  *tfevents* filter=lfs diff=lfs merge=lfs -text
36
+ gemma3-270m-it-q8.litertlm filter=lfs diff=lfs merge=lfs -text
37
+ gemma3-270m-it-q8.task filter=lfs diff=lfs merge=lfs -text
README.md ADDED
@@ -0,0 +1,525 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ ---
2
+ license: gemma
3
+ base_model:
4
+ - google/gemma-3-270m-it
5
+ library_name: transformers.js
6
+ tags:
7
+ - gemma3
8
+ - gemma
9
+ - google
10
+ pipeline_tag: text-generation
11
+ ---
12
+
13
+ # Gemma 3 model card
14
+
15
+ **Model Page**: [Gemma](https://ai.google.dev/gemma/docs/core)
16
+
17
+ **Resources and Technical Documentation**:
18
+
19
+ * [Gemma 3 Technical Report][g3-tech-report]
20
+ * [Responsible Generative AI Toolkit][rai-toolkit]
21
+ * [Gemma on Kaggle][kaggle-gemma]
22
+ * [Gemma on Vertex Model Garden][vertex-mg-gemma3]
23
+
24
+ **Terms of Use**: [Terms][terms]
25
+
26
+ **Authors**: Google DeepMind
27
+
28
+ ## Model Information
29
+
30
+ Summary description and brief definition of inputs and outputs.
31
+
32
+ ### Description
33
+
34
+ Gemma is a family of lightweight, state-of-the-art open models from Google,
35
+ built from the same research and technology used to create the Gemini models.
36
+ Gemma 3 models are multimodal, handling text and image input and generating text
37
+ output, with open weights for both pre-trained variants and instruction-tuned
38
+ variants. Gemma 3 has a large, 128K context window, multilingual support in over
39
+ 140 languages, and is available in more sizes than previous versions. Gemma 3
40
+ models are well-suited for a variety of text generation and image understanding
41
+ tasks, including question answering, summarization, and reasoning. Their
42
+ relatively small size makes it possible to deploy them in environments with
43
+ limited resources such as laptops, desktops or your own cloud infrastructure,
44
+ democratizing access to state of the art AI models and helping foster innovation
45
+ for everyone.
46
+
47
+ ### Inputs and outputs
48
+
49
+ - **Input:**
50
+ - Text string, such as a question, a prompt, or a document to be summarized
51
+ - Images, normalized to 896 x 896 resolution and encoded to 256 tokens
52
+ each
53
+ - Total input context of 128K tokens for the 4B, 12B, and 27B sizes, and
54
+ 32K tokens for the 1B and 270M sizes.
55
+
56
+ - **Output:**
57
+ - Generated text in response to the input, such as an answer to a
58
+ question, analysis of image content, or a summary of a document
59
+ - Total output context up to 128K tokens for the 4B, 12B, and 27B sizes,
60
+ and 32K tokens for the 1B and 270M sizes per request, subtracting the
61
+ request input tokens
62
+
63
+ ### Citation
64
+
65
+ ```none
66
+ @article{gemma_2025,
67
+ title={Gemma 3},
68
+ url={https://arxiv.org/abs/2503.19786},
69
+ publisher={Google DeepMind},
70
+ author={Gemma Team},
71
+ year={2025}
72
+ }
73
+ ```
74
+
75
+ ## Model Data
76
+
77
+ Data used for model training and how the data was processed.
78
+
79
+ ### Training Dataset
80
+
81
+ These models were trained on a dataset of text data that includes a wide variety
82
+ of sources. The 27B model was trained with 14 trillion tokens, the 12B model was
83
+ trained with 12 trillion tokens, 4B model was trained with 4 trillion tokens,
84
+ the 1B with 2 trillion tokens, and the 270M with 6 trillion tokens. The
85
+ knowledge cutoff date for the training data was August 2024. Here are the key
86
+ components:
87
+
88
+ - Web Documents: A diverse collection of web text ensures the model is
89
+ exposed to a broad range of linguistic styles, topics, and vocabulary. The
90
+ training dataset includes content in over 140 languages.
91
+ - Code: Exposing the model to code helps it to learn the syntax and
92
+ patterns of programming languages, which improves its ability to generate
93
+ code and understand code-related questions.
94
+ - Mathematics: Training on mathematical text helps the model learn logical
95
+ reasoning, symbolic representation, and to address mathematical queries.
96
+ - Images: A wide range of images enables the model to perform image
97
+ analysis and visual data extraction tasks.
98
+
99
+ The combination of these diverse data sources is crucial for training a powerful
100
+ multimodal model that can handle a wide variety of different tasks and data
101
+ formats.
102
+
103
+ ### Data Preprocessing
104
+
105
+ Here are the key data cleaning and filtering methods applied to the training
106
+ data:
107
+
108
+ - CSAM Filtering: Rigorous CSAM (Child Sexual Abuse Material) filtering
109
+ was applied at multiple stages in the data preparation process to ensure
110
+ the exclusion of harmful and illegal content.
111
+ - Sensitive Data Filtering: As part of making Gemma pre-trained models
112
+ safe and reliable, automated techniques were used to filter out certain
113
+ personal information and other sensitive data from training sets.
114
+ - Additional methods: Filtering based on content quality and safety in
115
+ line with [our policies][safety-policies].
116
+
117
+ ## Implementation Information
118
+
119
+ Details about the model internals.
120
+
121
+ ### Hardware
122
+
123
+ Gemma was trained using [Tensor Processing Unit (TPU)][tpu] hardware (TPUv4p,
124
+ TPUv5p and TPUv5e). Training vision-language models (VLMS) requires significant
125
+ computational power. TPUs, designed specifically for matrix operations common in
126
+ machine learning, offer several advantages in this domain:
127
+
128
+ - Performance: TPUs are specifically designed to handle the massive
129
+ computations involved in training VLMs. They can speed up training
130
+ considerably compared to CPUs.
131
+ - Memory: TPUs often come with large amounts of high-bandwidth memory,
132
+ allowing for the handling of large models and batch sizes during training.
133
+ This can lead to better model quality.
134
+ - Scalability: TPU Pods (large clusters of TPUs) provide a scalable
135
+ solution for handling the growing complexity of large foundation models.
136
+ You can distribute training across multiple TPU devices for faster and more
137
+ efficient processing.
138
+ - Cost-effectiveness: In many scenarios, TPUs can provide a more
139
+ cost-effective solution for training large models compared to CPU-based
140
+ infrastructure, especially when considering the time and resources saved
141
+ due to faster training.
142
+ - These advantages are aligned with
143
+ [Google's commitments to operate sustainably][sustainability].
144
+
145
+ ### Software
146
+
147
+ Training was done using [JAX][jax] and [ML Pathways][ml-pathways].
148
+
149
+ JAX allows researchers to take advantage of the latest generation of hardware,
150
+ including TPUs, for faster and more efficient training of large models. ML
151
+ Pathways is Google's latest effort to build artificially intelligent systems
152
+ capable of generalizing across multiple tasks. This is specially suitable for
153
+ foundation models, including large language models like these ones.
154
+
155
+ Together, JAX and ML Pathways are used as described in the
156
+ [paper about the Gemini family of models][gemini-2-paper]; *"the 'single
157
+ controller' programming model of Jax and Pathways allows a single Python
158
+ process to orchestrate the entire training run, dramatically simplifying the
159
+ development workflow."*
160
+
161
+ ## Evaluation
162
+
163
+ Model evaluation metrics and results.
164
+
165
+ ### Benchmark Results
166
+
167
+ These models were evaluated against a large collection of different datasets and
168
+ metrics to cover different aspects of text generation. Evaluation results marked
169
+ with **IT** are for instruction-tuned models. Evaluation results marked with
170
+ **PT** are for pre-trained models.
171
+
172
+ #### Gemma 3 270M
173
+
174
+ | **Benchmark** | **n-shot** | **Gemma 3 PT 270M** |
175
+ | :------------------------ | :-----------: | ------------------: |
176
+ | [HellaSwag][hellaswag] | 10-shot | 40.9 |
177
+ | [BoolQ][boolq] | 0-shot | 61.4 |
178
+ | [PIQA][piqa] | 0-shot | 67.7 |
179
+ | [TriviaQA][triviaqa] | 5-shot | 15.4 |
180
+ | [ARC-c][arc] | 25-shot | 29.0 |
181
+ | [ARC-e][arc] | 0-shot | 57.7 |
182
+ | [WinoGrande][winogrande] | 5-shot | 52.0 |
183
+
184
+ [hellaswag]: https://arxiv.org/abs/1905.07830
185
+ [boolq]: https://arxiv.org/abs/1905.10044
186
+ [piqa]: https://arxiv.org/abs/1911.11641
187
+ [triviaqa]: https://arxiv.org/abs/1705.03551
188
+ [arc]: https://arxiv.org/abs/1911.01547
189
+ [winogrande]: https://arxiv.org/abs/1907.10641
190
+
191
+ | **Benchmark** | **n-shot** | **Gemma 3 IT 270m** |
192
+ | :------------------------ | :-----------: | ------------------: |
193
+ | [HellaSwag][hellaswag] | 0-shot | 37.7 |
194
+ | [PIQA][piqa] | 0-shot | 66.2 |
195
+ | [ARC-c][arc] | 0-shot | 28.2 |
196
+ | [WinoGrande][winogrande] | 0-shot | 52.3 |
197
+ | [BIG-Bench Hard][bbh] | few-shot | 26.7 |
198
+ | [IF Eval][ifeval] | 0-shot | 51.2 |
199
+
200
+ [hellaswag]: https://arxiv.org/abs/1905.07830
201
+ [piqa]: https://arxiv.org/abs/1911.11641
202
+ [arc]: https://arxiv.org/abs/1911.01547
203
+ [winogrande]: https://arxiv.org/abs/1907.10641
204
+ [bbh]: https://paperswithcode.com/dataset/bbh
205
+ [bbh]: https://paperswithcode.com/dataset/bbh
206
+ [ifeval]: https://arxiv.org/abs/2311.07911
207
+
208
+ #### Gemma 3 1B, 4B, 12B & 27B
209
+
210
+ ##### Reasoning and factuality
211
+
212
+ | Benchmark | n-shot | Gemma 3 IT 1B | Gemma 3 IT 4B | Gemma 3 IT 12B | Gemma 3 IT 27B |
213
+ |--------------------------------|--------|:-------------:|:-------------:|:--------------:|:--------------:|
214
+ | [GPQA][gpqa] Diamond | 0-shot | 19.2 | 30.8 | 40.9 | 42.4 |
215
+ | [SimpleQA][simpleqa] | 0-shot | 2.2 | 4.0 | 6.3 | 10.0 |
216
+ | [FACTS Grounding][facts-grdg] | - | 36.4 | 70.1 | 75.8 | 74.9 |
217
+ | [BIG-Bench Hard][bbh] | 0-shot | 39.1 | 72.2 | 85.7 | 87.6 |
218
+ | [BIG-Bench Extra Hard][bbeh] | 0-shot | 7.2 | 11.0 | 16.3 | 19.3 |
219
+ | [IFEval][ifeval] | 0-shot | 80.2 | 90.2 | 88.9 | 90.4 |
220
+
221
+ | Benchmark | n-shot | Gemma 3 PT 1B | Gemma 3 PT 4B | Gemma 3 PT 12B | Gemma 3 PT 27B |
222
+ | ------------------------------ |----------|:--------------:|:-------------:|:--------------:|:--------------:|
223
+ | [HellaSwag][hellaswag] | 10-shot | 62.3 | 77.2 | 84.2 | 85.6 |
224
+ | [BoolQ][boolq] | 0-shot | 63.2 | 72.3 | 78.8 | 82.4 |
225
+ | [PIQA][piqa] | 0-shot | 73.8 | 79.6 | 81.8 | 83.3 |
226
+ | [SocialIQA][socialiqa] | 0-shot | 48.9 | 51.9 | 53.4 | 54.9 |
227
+ | [TriviaQA][triviaqa] | 5-shot | 39.8 | 65.8 | 78.2 | 85.5 |
228
+ | [Natural Questions][naturalq] | 5-shot | 9.48 | 20.0 | 31.4 | 36.1 |
229
+ | [ARC-c][arc] | 25-shot | 38.4 | 56.2 | 68.9 | 70.6 |
230
+ | [ARC-e][arc] | 0-shot | 73.0 | 82.4 | 88.3 | 89.0 |
231
+ | [WinoGrande][winogrande] | 5-shot | 58.2 | 64.7 | 74.3 | 78.8 |
232
+ | [BIG-Bench Hard][bbh] | few-shot | 28.4 | 50.9 | 72.6 | 77.7 |
233
+ | [DROP][drop] | 1-shot | 42.4 | 60.1 | 72.2 | 77.2 |
234
+
235
+ [gpqa]: https://arxiv.org/abs/2311.12022
236
+ [simpleqa]: https://arxiv.org/abs/2411.04368
237
+ [facts-grdg]: https://goo.gle/FACTS_paper
238
+ [bbeh]: https://github.com/google-deepmind/bbeh
239
+ [ifeval]: https://arxiv.org/abs/2311.07911
240
+ [hellaswag]: https://arxiv.org/abs/1905.07830
241
+ [boolq]: https://arxiv.org/abs/1905.10044
242
+ [piqa]: https://arxiv.org/abs/1911.11641
243
+ [socialiqa]: https://arxiv.org/abs/1904.09728
244
+ [triviaqa]: https://arxiv.org/abs/1705.03551
245
+ [naturalq]: https://github.com/google-research-datasets/natural-questions
246
+ [arc]: https://arxiv.org/abs/1911.01547
247
+ [winogrande]: https://arxiv.org/abs/1907.10641
248
+ [bbh]: https://paperswithcode.com/dataset/bbh
249
+ [drop]: https://arxiv.org/abs/1903.00161
250
+
251
+ ##### STEM and code
252
+
253
+ | Benchmark | n-shot | Gemma 3 IT 1B | Gemma 3 IT 4B | Gemma 3 IT 12B | Gemma 3 IT 27B |
254
+ |----------------------------|--------|:-------------:|:-------------:|:--------------:|:--------------:|
255
+ | [MMLU][mmlu] (Pro) | 0-shot | 14.7 | 43.6 | 60.6 | 67.5 |
256
+ | [LiveCodeBench][lcb] | 0-shot | 1.9 | 12.6 | 24.6 | 29.7 |
257
+ | [Bird-SQL][bird-sql] (dev) | - | 6.4 | 36.3 | 47.9 | 54.4 |
258
+ | [Math][math] | 0-shot | 48.0 | 75.6 | 83.8 | 89.0 |
259
+ | HiddenMath | 0-shot | 15.8 | 43.0 | 54.5 | 60.3 |
260
+ | [MBPP][mbpp] | 3-shot | 35.2 | 63.2 | 73.0 | 74.4 |
261
+ | [HumanEval][humaneval] | 0-shot | 41.5 | 71.3 | 85.4 | 87.8 |
262
+ | [Natural2Code][nat2code] | 0-shot | 56.0 | 70.3 | 80.7 | 84.5 |
263
+ | [GSM8K][gsm8k] | 0-shot | 62.8 | 89.2 | 94.4 | 95.9 |
264
+
265
+ | Benchmark | n-shot | Gemma 3 PT 4B | Gemma 3 PT 12B | Gemma 3 PT 27B |
266
+ | ------------------------------ |----------------|:-------------:|:--------------:|:--------------:|
267
+ | [MMLU][mmlu] | 5-shot | 59.6 | 74.5 | 78.6 |
268
+ | [MMLU][mmlu] (Pro COT) | 5-shot | 29.2 | 45.3 | 52.2 |
269
+ | [AGIEval][agieval] | 3-5-shot | 42.1 | 57.4 | 66.2 |
270
+ | [MATH][math] | 4-shot | 24.2 | 43.3 | 50.0 |
271
+ | [GSM8K][gsm8k] | 8-shot | 38.4 | 71.0 | 82.6 |
272
+ | [GPQA][gpqa] | 5-shot | 15.0 | 25.4 | 24.3 |
273
+ | [MBPP][mbpp] | 3-shot | 46.0 | 60.4 | 65.6 |
274
+ | [HumanEval][humaneval] | 0-shot | 36.0 | 45.7 | 48.8 |
275
+
276
+ [mmlu]: https://arxiv.org/abs/2009.03300
277
+ [agieval]: https://arxiv.org/abs/2304.06364
278
+ [math]: https://arxiv.org/abs/2103.03874
279
+ [gsm8k]: https://arxiv.org/abs/2110.14168
280
+ [gpqa]: https://arxiv.org/abs/2311.12022
281
+ [mbpp]: https://arxiv.org/abs/2108.07732
282
+ [humaneval]: https://arxiv.org/abs/2107.03374
283
+ [lcb]: https://arxiv.org/abs/2403.07974
284
+ [bird-sql]: https://arxiv.org/abs/2305.03111
285
+ [nat2code]: https://arxiv.org/abs/2405.04520
286
+
287
+ #### Multilingual
288
+
289
+ | Benchmark | n-shot | Gemma 3 IT 1B | Gemma 3 IT 4B | Gemma 3 IT 12B | Gemma 3 IT 27B |
290
+ |--------------------------------------|--------|:-------------:|:-------------:|:--------------:|:--------------:|
291
+ | [Global-MMLU-Lite][global-mmlu-lite] | 0-shot | 34.2 | 54.5 | 69.5 | 75.1 |
292
+ | [ECLeKTic][eclektic] | 0-shot | 1.4 | 4.6 | 10.3 | 16.7 |
293
+ | [WMT24++][wmt24pp] | 0-shot | 35.9 | 46.8 | 51.6 | 53.4 |
294
+
295
+ | Benchmark | Gemma 3 PT 1B | Gemma 3 PT 4B | Gemma 3 PT 12B | Gemma 3 PT 27B |
296
+ | ------------------------------------ |:-------------:|:-------------:|:--------------:|:--------------:|
297
+ | [MGSM][mgsm] | 2.04 | 34.7 | 64.3 | 74.3 |
298
+ | [Global-MMLU-Lite][global-mmlu-lite] | 24.9 | 57.0 | 69.4 | 75.7 |
299
+ | [WMT24++][wmt24pp] (ChrF) | 36.7 | 48.4 | 53.9 | 55.7 |
300
+ | [FloRes][flores] | 29.5 | 39.2 | 46.0 | 48.8 |
301
+ | [XQuAD][xquad] (all) | 43.9 | 68.0 | 74.5 | 76.8 |
302
+ | [ECLeKTic][eclektic] | 4.69 | 11.0 | 17.2 | 24.4 |
303
+ | [IndicGenBench][indicgenbench] | 41.4 | 57.2 | 61.7 | 63.4 |
304
+
305
+ [mgsm]: https://arxiv.org/abs/2210.03057
306
+ [flores]: https://arxiv.org/abs/2106.03193
307
+ [xquad]: https://arxiv.org/abs/1910.11856v3
308
+ [global-mmlu-lite]: https://huggingface.co/datasets/CohereForAI/Global-MMLU-Lite
309
+ [wmt24pp]: https://arxiv.org/abs/2502.12404v1
310
+ [eclektic]: https://arxiv.org/abs/2502.21228
311
+ [indicgenbench]: https://arxiv.org/abs/2404.16816
312
+
313
+ ##### Multimodal
314
+
315
+ | Benchmark | Gemma 3 IT 4B | Gemma 3 IT 12B | Gemma 3 IT 27B |
316
+ |-----------------------------------|:-------------:|:--------------:|:--------------:|
317
+ | [MMMU][mmmu] (val) | 48.8 | 59.6 | 64.9 |
318
+ | [DocVQA][docvqa] | 75.8 | 87.1 | 86.6 |
319
+ | [InfoVQA][info-vqa] | 50.0 | 64.9 | 70.6 |
320
+ | [TextVQA][textvqa] | 57.8 | 67.7 | 65.1 |
321
+ | [AI2D][ai2d] | 74.8 | 84.2 | 84.5 |
322
+ | [ChartQA][chartqa] | 68.8 | 75.7 | 78.0 |
323
+ | [VQAv2][vqav2] (val) | 62.4 | 71.6 | 71.0 |
324
+ | [MathVista][mathvista] (testmini) | 50.0 | 62.9 | 67.6 |
325
+
326
+ | Benchmark | Gemma 3 PT 4B | Gemma 3 PT 12B | Gemma 3 PT 27B |
327
+ | ------------------------------ |:-------------:|:--------------:|:--------------:|
328
+ | [COCOcap][coco-cap] | 102 | 111 | 116 |
329
+ | [DocVQA][docvqa] (val) | 72.8 | 82.3 | 85.6 |
330
+ | [InfoVQA][info-vqa] (val) | 44.1 | 54.8 | 59.4 |
331
+ | [MMMU][mmmu] (pt) | 39.2 | 50.3 | 56.1 |
332
+ | [TextVQA][textvqa] (val) | 58.9 | 66.5 | 68.6 |
333
+ | [RealWorldQA][realworldqa] | 45.5 | 52.2 | 53.9 |
334
+ | [ReMI][remi] | 27.3 | 38.5 | 44.8 |
335
+ | [AI2D][ai2d] | 63.2 | 75.2 | 79.0 |
336
+ | [ChartQA][chartqa] | 63.6 | 74.7 | 76.3 |
337
+ | [VQAv2][vqav2] | 63.9 | 71.2 | 72.9 |
338
+ | [BLINK][blinkvqa] | 38.0 | 35.9 | 39.6 |
339
+ | [OKVQA][okvqa] | 51.0 | 58.7 | 60.2 |
340
+ | [TallyQA][tallyqa] | 42.5 | 51.8 | 54.3 |
341
+ | [SpatialSense VQA][ss-vqa] | 50.9 | 60.0 | 59.4 |
342
+ | [CountBenchQA][countbenchqa] | 26.1 | 17.8 | 68.0 |
343
+
344
+ [coco-cap]: https://cocodataset.org/#home
345
+ [docvqa]: https://www.docvqa.org/
346
+ [info-vqa]: https://arxiv.org/abs/2104.12756
347
+ [mmmu]: https://arxiv.org/abs/2311.16502
348
+ [textvqa]: https://textvqa.org/
349
+ [realworldqa]: https://paperswithcode.com/dataset/realworldqa
350
+ [remi]: https://arxiv.org/html/2406.09175v1
351
+ [ai2d]: https://allenai.org/data/diagrams
352
+ [chartqa]: https://arxiv.org/abs/2203.10244
353
+ [vqav2]: https://visualqa.org/index.html
354
+ [blinkvqa]: https://arxiv.org/abs/2404.12390
355
+ [okvqa]: https://okvqa.allenai.org/
356
+ [tallyqa]: https://arxiv.org/abs/1810.12440
357
+ [ss-vqa]: https://arxiv.org/abs/1908.02660
358
+ [countbenchqa]: https://github.com/google-research/big_vision/blob/main/big_vision/datasets/countbenchqa/
359
+ [mathvista]: https://arxiv.org/abs/2310.02255
360
+
361
+ ## Ethics and Safety
362
+
363
+ Ethics and safety evaluation approach and results.
364
+
365
+ ### Evaluation Approach
366
+
367
+ Our evaluation methods include structured evaluations and internal red-teaming
368
+ testing of relevant content policies. Red-teaming was conducted by a number of
369
+ different teams, each with different goals and human evaluation metrics. These
370
+ models were evaluated against a number of different categories relevant to
371
+ ethics and safety, including:
372
+
373
+ - **Child Safety**: Evaluation of text-to-text and image to text prompts
374
+ covering child safety policies, including child sexual abuse and
375
+ exploitation.
376
+ - **Content Safety:** Evaluation of text-to-text and image to text prompts
377
+ covering safety policies including, harassment, violence and gore, and hate
378
+ speech.
379
+ - **Representational Harms**: Evaluation of text-to-text and image to text
380
+ prompts covering safety policies including bias, stereotyping, and harmful
381
+ associations or inaccuracies.
382
+
383
+ In addition to development level evaluations, we conduct "assurance
384
+ evaluations" which are our 'arms-length' internal evaluations for responsibility
385
+ governance decision making. They are conducted separately from the model
386
+ development team, to inform decision making about release. High level findings
387
+ are fed back to the model team, but prompt sets are held-out to prevent
388
+ overfitting and preserve the results' ability to inform decision making.
389
+ Assurance evaluation results are reported to our Responsibility & Safety Council
390
+ as part of release review.
391
+
392
+ ### Evaluation Results
393
+
394
+ For all areas of safety testing, we saw major improvements in the categories of
395
+ child safety, content safety, and representational harms relative to previous
396
+ Gemma models. All testing was conducted without safety filters to evaluate the
397
+ model capabilities and behaviors. For both text-to-text and image-to-text, and
398
+ across all model sizes, the model produced minimal policy violations, and showed
399
+ significant improvements over previous Gemma models' performance with respect
400
+ to ungrounded inferences. A limitation of our evaluations was they included only
401
+ English language prompts.
402
+
403
+ ## Usage and Limitations
404
+
405
+ These models have certain limitations that users should be aware of.
406
+
407
+ ### Intended Usage
408
+
409
+ Open vision-language models (VLMs) models have a wide range of applications
410
+ across various industries and domains. The following list of potential uses is
411
+ not comprehensive. The purpose of this list is to provide contextual information
412
+ about the possible use-cases that the model creators considered as part of model
413
+ training and development.
414
+
415
+ - Content Creation and Communication
416
+ - Text Generation: These models can be used to generate creative text
417
+ formats such as poems, scripts, code, marketing copy, and email drafts.
418
+ - Chatbots and Conversational AI: Power conversational interfaces
419
+ for customer service, virtual assistants, or interactive applications.
420
+ - Text Summarization: Generate concise summaries of a text corpus,
421
+ research papers, or reports.
422
+ - Image Data Extraction: These models can be used to extract,
423
+ interpret, and summarize visual data for text communications.
424
+ - Research and Education
425
+ - Natural Language Processing (NLP) and VLM Research: These
426
+ models can serve as a foundation for researchers to experiment with VLM
427
+ and NLP techniques, develop algorithms, and contribute to the
428
+ advancement of the field.
429
+ - Language Learning Tools: Support interactive language learning
430
+ experiences, aiding in grammar correction or providing writing practice.
431
+ - Knowledge Exploration: Assist researchers in exploring large
432
+ bodies of text by generating summaries or answering questions about
433
+ specific topics.
434
+
435
+ ### Limitations
436
+
437
+ - Training Data
438
+ - The quality and diversity of the training data significantly
439
+ influence the model's capabilities. Biases or gaps in the training data
440
+ can lead to limitations in the model's responses.
441
+ - The scope of the training dataset determines the subject areas
442
+ the model can handle effectively.
443
+ - Context and Task Complexity
444
+ - Models are better at tasks that can be framed with clear
445
+ prompts and instructions. Open-ended or highly complex tasks might be
446
+ challenging.
447
+ - A model's performance can be influenced by the amount of context
448
+ provided (longer context generally leads to better outputs, up to a
449
+ certain point).
450
+ - Language Ambiguity and Nuance
451
+ - Natural language is inherently complex. Models might struggle
452
+ to grasp subtle nuances, sarcasm, or figurative language.
453
+ - Factual Accuracy
454
+ - Models generate responses based on information they learned
455
+ from their training datasets, but they are not knowledge bases. They
456
+ may generate incorrect or outdated factual statements.
457
+ - Common Sense
458
+ - Models rely on statistical patterns in language. They might
459
+ lack the ability to apply common sense reasoning in certain situations.
460
+
461
+ ### Ethical Considerations and Risks
462
+
463
+ The development of vision-language models (VLMs) raises several ethical
464
+ concerns. In creating an open model, we have carefully considered the following:
465
+
466
+ - Bias and Fairness
467
+ - VLMs trained on large-scale, real-world text and image data can
468
+ reflect socio-cultural biases embedded in the training material. These
469
+ models underwent careful scrutiny, input data pre-processing described
470
+ and posterior evaluations reported in this card.
471
+ - Misinformation and Misuse
472
+ - VLMs can be misused to generate text that is false, misleading,
473
+ or harmful.
474
+ - Guidelines are provided for responsible use with the model, see the
475
+ [Responsible Generative AI Toolkit][rai-toolkit].
476
+ - Transparency and Accountability:
477
+ - This model card summarizes details on the models' architecture,
478
+ capabilities, limitations, and evaluation processes.
479
+ - A responsibly developed open model offers the opportunity to
480
+ share innovation by making VLM technology accessible to developers and
481
+ researchers across the AI ecosystem.
482
+
483
+ Risks identified and mitigations:
484
+
485
+ - **Perpetuation of biases**: It's encouraged to perform continuous
486
+ monitoring (using evaluation metrics, human review) and the exploration of
487
+ de-biasing techniques during model training, fine-tuning, and other use
488
+ cases.
489
+ - **Generation of harmful content**: Mechanisms and guidelines for content
490
+ safety are essential. Developers are encouraged to exercise caution and
491
+ implement appropriate content safety safeguards based on their specific
492
+ product policies and application use cases.
493
+ - **Misuse for malicious purposes**: Technical limitations and developer
494
+ and end-user education can help mitigate against malicious applications of
495
+ VLMs. Educational resources and reporting mechanisms for users to flag
496
+ misuse are provided. Prohibited uses of Gemma models are outlined in the
497
+ [Gemma Prohibited Use Policy][prohibited-use].
498
+ - **Privacy violations**: Models were trained on data filtered for removal
499
+ of certain personal information and other sensitive data. Developers are
500
+ encouraged to adhere to privacy regulations with privacy-preserving
501
+ techniques.
502
+
503
+ ### Benefits
504
+
505
+ At the time of release, this family of models provides high-performance open
506
+ vision-language model implementations designed from the ground up for
507
+ responsible AI development compared to similarly sized models.
508
+
509
+ Using the benchmark evaluation metrics described in this document, these models
510
+ have shown to provide superior performance to other, comparably-sized open model
511
+ alternatives.
512
+
513
+ [g3-tech-report]: https://arxiv.org/abs/2503.19786
514
+ [rai-toolkit]: https://ai.google.dev/responsible
515
+ [kaggle-gemma]: https://www.kaggle.com/models/google/gemma-3
516
+ [vertex-mg-gemma3]: https://console.cloud.google.com/vertex-ai/publishers/google/model-garden/gemma3
517
+ [terms]: https://ai.google.dev/gemma/terms
518
+ [safety-policies]: https://ai.google/static/documents/ai-responsibility-update-published-february-2025.pdf
519
+ [prohibited-use]: https://ai.google.dev/gemma/prohibited_use_policy
520
+ [tpu]: https://cloud.google.com/tpu/docs/intro-to-tpu
521
+ [sustainability]: https://sustainability.google/operating-sustainably/
522
+ [jax]: https://github.com/jax-ml/jax
523
+ [ml-pathways]: https://blog.google/technology/ai/introducing-pathways-next-generation-ai-architecture/
524
+ [sustainability]: https://sustainability.google/operating-sustainably/
525
+ [gemini-2-paper]: https://arxiv.org/abs/2312.11805
gemma3-270m-it-q8.litertlm ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:757e9119fa5bd667a2774fb470ac4afcd3190a21c677f8e69a5d6bc908abdd63
3
+ size 304005120
gemma3-270m-it-q8.task ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:0f7147f1c22eaf758b819bbf7841793e4c90096c9352cde7fbe5c631f2265ef5
3
+ size 303950933