cglez/bert-s140-uncased

Model Card: BERT-Sentiment140

An in-domain BERT-base model, pre-trained from scratch on the Sentiment140 dataset text.

Model Details

Description

This model is based on the BERT base (uncased) architecture and was pre-trained from scratch (in-domain) using the text in Sentiment140 dataset, excluding its test split. Only the masked language modeling (MLM) objective was used during pre-training.

• Developed by: Cesar Gonzalez-Gutierrez
• Funded by: ERC
• Architecture: BERT-base
• Language: English
• License: Apache 2.0
• Base model: BERT base model (uncased)

Checkpoints

Intermediate checkpoints from the pre-training process are available and can be accessed using specific tags, which correspond to training epochs and steps:

Epoch	Step	Tags
1	15000	epoch-1	step-15000
2	30000	epoch-2	step-30000
3	45000	epoch-3	step-45000
5	75000	epoch-5	step-75000
10	150000	epoch-10	step-150000
15	225000	epoch-15	step-225000
20	300000	epoch-20	step-300000
25	375000	epoch-25	step-375000

To load a model from a specific intermediate checkpoint, use the revision parameter with the corresponding tag:

from transformers import AutoModelForMaskedLM

model = AutoModelForMaskedLM.from_pretrained("<model-name>", revision="<checkpoint-tag>")

Sources

• Paper: [Information pending]

Training Details

For more details on the training procedure, please refer to the base model's documentation: Training procedure.

Training Data

All texts from Sentiment140 dataset, excluding the test partition.

Training Hyperparameters

• Precision: fp16
• Batch size: 32
• Gradient accumulation steps: 3

Uses

For typical use cases and limitations, please refer to the base model's guidance: Inteded uses & limitations.

Bias, Risks, and Limitations

This model inherits potential risks and limitations from the base model. Refer to: Limitations and bias.

Environmental Impact

• Hardware Type: NVIDIA Tesla V100 PCIE 32GB
• Runtime: 36.5 h
• Cluster Provider: Artemisa
• Compute Region: EU
• Carbon Emitted: 6.79 kg CO2 eq.

Citation

BibTeX:

[More Information Needed]