add model, tokenizer files, and model card

README.md

@@ -0,0 +1,43 @@
+---
+language: th
+license: cc-by-sa-4.0
+tags:
+- word segmentation
+datasets:
+- best2010
+- lst20
+- tlc
+- vistec-tp-th-2021
+- wisesight_sentiment
+pipeline_tag: token-classification
+---
+
+# Multi-criteria BERT base Thai with Lattice for Word Segmentation
+
+This is a variant of the pre-trained model [BERT](https://github.com/google-research/bert) model.
+The model was pre-trained on texts in the Thai language and fine-tuned for word segmentation based on [bert-base-multilingual-cased](https://huggingface.co/bert-base-multilingual-cased).
+This version of the model processes input texts with character-level with word-level incorporated with a lattice structure.
+
+The scripts for the pre-training are available at [tchayintr/latte-ptm-ws](https://github.com/tchayintr/latte-ptm-ws).
+
+## Model architecture
+
+The model architecture is described in this [paper](https://www.jstage.jst.go.jp/article/jnlp/30/2/30_456/_article/-char/ja).
+
+## Training Data
+
+The model is trained on multiple Thai word segmented datasets, including best2010, lst20, tlc (tnhc), vistec-tp-th-2021 (vistec2021) and wisesight_sentiment (ws160).
+The datasets can be accessed as follows: 
+- [best2010](https://thailang.nectec.or.th)
+- [lst20](https://huggingface.co/datasets/lst20)
+- [tlc](https://huggingface.co/datasets/tlc)
+- [vistec-tp-th-2021](https://github.com/mrpeerat/OSKut/tree/main/VISTEC-TP-TH-2021)
+- [wisesight_sentiment](https://huggingface.co/datasets/wisesight_sentiment).
+
+## Licenses
+
+The pre-trained model is distributed under the terms of the [Creative Commons Attribution-ShareAlike 4.0](https://creativecommons.org/licenses/by-sa/4.0/).
+
+## Acknowledgments
+
+This model was trained with GPU servers provided by [Okumura-Funakoshi NLP Group](https://lr-www.pi.titech.ac.jp).

config.json

@@ -0,0 +1,32 @@
+{
+  "_name_or_path": "data/ptm/bert-base-multilingual-cased",
+  "architectures": [
+    "BertModel"
+  ],
+  "attention_probs_dropout_prob": 0.1,
+  "classifier_dropout": null,
+  "directionality": "bidi",
+  "hidden_act": "gelu",
+  "hidden_dropout_prob": 0.1,
+  "hidden_size": 768,
+  "initializer_range": 0.02,
+  "intermediate_size": 3072,
+  "layer_norm_eps": 1e-12,
+  "max_position_embeddings": 512,
+  "model_type": "bert",
+  "num_attention_heads": 12,
+  "num_hidden_layers": 12,
+  "output_hidden_states": true,
+  "pad_token_id": 0,
+  "pooler_fc_size": 768,
+  "pooler_num_attention_heads": 12,
+  "pooler_num_fc_layers": 3,
+  "pooler_size_per_head": 128,
+  "pooler_type": "first_token_transform",
+  "position_embedding_type": "absolute",
+  "torch_dtype": "float32",
+  "transformers_version": "4.15.0",
+  "type_vocab_size": 2,
+  "use_cache": true,
+  "vocab_size": 277964
+}

pytorch_model.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:cf4bf78eceb5bcf685db96e138942f82f7b5ee344ea12a5f09b345e73df2b0e1
+size 1198149361

special_tokens_map.json

@@ -0,0 +1 @@
+{"bos_token": "[BOS]", "eos_token": "[EOS]", "unk_token": "[UNK]", "sep_token": "[SEP]", "pad_token": "[PAD]", "cls_token": "[CLS]", "mask_token": "[MASK]", "additional_special_tokens": ["[UNC]", "[LEN]", "[LPP]", "[BEST2010]", "[LST20]", "[TNHC]", "[VISTEC-TPTH2021]", "[WS160]"]}

tokenizer.pkl

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:d7c818a62877090642e2ae669cab30a90a0ceaa2b0f42a3c113a128c5c17e917
+size 15059005

tokenizer_config.json

@@ -0,0 +1 @@
+{"do_lower_case": false, "do_basic_tokenize": true, "never_split": null, "unk_token": "[UNK]", "sep_token": "[SEP]", "pad_token": "[PAD]", "cls_token": "[CLS]", "mask_token": "[MASK]", "tokenize_chinese_chars": true, "strip_accents": null, "special_tokens_map_file": null, "tokenizer_file": "data/ptm/bert-base-multilingual-cased/tokenizer.json", "name_or_path": "data/ptm/bert-base-multilingual-cased", "tokenizer_class": "BertTokenizer"}