ronalhung commited on
Commit
e3f1e15
·
verified ·
1 Parent(s): e84362a

Add SetFit ABSA model

Browse files
README.md CHANGED
@@ -6,19 +6,20 @@ tags:
6
  - text-classification
7
  - generated_from_setfit_trainer
8
  widget:
9
- - text: '- they use fresh mozzarella instead of the:The pizza is delicious - they
10
- use fresh mozzarella instead of the cheap, frozen, shredded cheese common to most
11
- pizzaria''s.'
12
- - text: 'refinement: Food, though somewhat:An oasis of refinement: Food, though
13
- somewhat uneven, often reaches the pinnacles of new American fine cuisine - chef''s
14
- passion (and kitchen''s precise execution) is most evident in the fish dishes
15
- and soups.'
16
- - text: We had the lobster sandwich and it was:We had the lobster sandwich and it
17
- was FANTASTIC.
18
- - text: The fish is fresh but:The fish is fresh but the variety of fish is nothing
19
- out of ordinary.
20
- - text: with classic upscale Italian decor.:Nice restaurant overall, with classic
21
- upscale Italian decor.
 
22
  metrics:
23
  - accuracy
24
  pipeline_tag: text-classification
@@ -26,7 +27,7 @@ library_name: setfit
26
  inference: false
27
  base_model: sentence-transformers/all-MiniLM-L6-v2
28
  model-index:
29
- - name: SetFit Polarity Model with sentence-transformers/all-MiniLM-L6-v2
30
  results:
31
  - task:
32
  type: text-classification
@@ -37,13 +38,13 @@ model-index:
37
  split: test
38
  metrics:
39
  - type: accuracy
40
- value: 0.7244094488188977
41
  name: Accuracy
42
  ---
43
 
44
- # SetFit Polarity Model with sentence-transformers/all-MiniLM-L6-v2
45
 
46
- This is a [SetFit](https://github.com/huggingface/setfit) model that can be used for Aspect Based Sentiment Analysis (ABSA). This SetFit model uses [sentence-transformers/all-MiniLM-L6-v2](https://huggingface.co/sentence-transformers/all-MiniLM-L6-v2) as the Sentence Transformer embedding model. A [LogisticRegression](https://scikit-learn.org/stable/modules/generated/sklearn.linear_model.LogisticRegression.html) instance is used for classification. In particular, this model is in charge of classifying aspect polarities.
47
 
48
  The model has been trained using an efficient few-shot learning technique that involves:
49
 
@@ -53,8 +54,8 @@ The model has been trained using an efficient few-shot learning technique that i
53
  This model was trained within the context of a larger system for ABSA, which looks like so:
54
 
55
  1. Use a spaCy model to select possible aspect span candidates.
56
- 2. Use a SetFit model to filter these possible aspect span candidates.
57
- 3. **Use this SetFit model to classify the filtered aspect span candidates.**
58
 
59
  ## Model Details
60
 
@@ -63,10 +64,10 @@ This model was trained within the context of a larger system for ABSA, which loo
63
  - **Sentence Transformer body:** [sentence-transformers/all-MiniLM-L6-v2](https://huggingface.co/sentence-transformers/all-MiniLM-L6-v2)
64
  - **Classification head:** a [LogisticRegression](https://scikit-learn.org/stable/modules/generated/sklearn.linear_model.LogisticRegression.html) instance
65
  - **spaCy Model:** en_core_web_lg
66
- - **SetFitABSA Aspect Model:** [setfit-absa-aspect](https://huggingface.co/setfit-absa-aspect)
67
- - **SetFitABSA Polarity Model:** [ronalhung/setfit-absa-restaurants-aspect](https://huggingface.co/ronalhung/setfit-absa-restaurants-aspect)
68
  - **Maximum Sequence Length:** 256 tokens
69
- - **Number of Classes:** 4 classes
70
  <!-- - **Training Dataset:** [Unknown](https://huggingface.co/datasets/unknown) -->
71
  <!-- - **Language:** Unknown -->
72
  <!-- - **License:** Unknown -->
@@ -78,19 +79,17 @@ This model was trained within the context of a larger system for ABSA, which loo
78
  - **Blogpost:** [SetFit: Efficient Few-Shot Learning Without Prompts](https://huggingface.co/blog/setfit)
79
 
80
  ### Model Labels
81
- | Label | Examples |
82
- |:---------|:-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
83
- | negative | <ul><li>'But the staff was so horrible:But the staff was so horrible to us.'</li><li>', forgot our toast, left out:They did not have mayonnaise, forgot our toast, left out ingredients (ie cheese in an omelet), below hot temperatures and the bacon was so over cooked it crumbled on the plate when you touched it.'</li><li>'did not have mayonnaise, forgot our:They did not have mayonnaise, forgot our toast, left out ingredients (ie cheese in an omelet), below hot temperatures and the bacon was so over cooked it crumbled on the plate when you touched it.'</li></ul> |
84
- | positive | <ul><li>"factor was the food, which was:To be completely fair, the only redeeming factor was the food, which was above average, but couldn't make up for all the other deficiencies of Teodora."</li><li>"The food is uniformly exceptional:The food is uniformly exceptional, with a very capable kitchen which will proudly whip up whatever you feel like eating, whether it's on the menu or not."</li><li>"a very capable kitchen which will proudly:The food is uniformly exceptional, with a very capable kitchen which will proudly whip up whatever you feel like eating, whether it's on the menu or not."</li></ul> |
85
- | neutral | <ul><li>"'s on the menu or not.:The food is uniformly exceptional, with a very capable kitchen which will proudly whip up whatever you feel like eating, whether it's on the menu or not."</li><li>'to sample both meats).:Our agreed favorite is the orrechiete with sausage and chicken (usually the waiters are kind enough to split the dish in half so you get to sample both meats).'</li><li>'to split the dish in half so:Our agreed favorite is the orrechiete with sausage and chicken (usually the waiters are kind enough to split the dish in half so you get to sample both meats).'</li></ul> |
86
- | conflict | <ul><li>'The food was delicious but:The food was delicious but do not come here on a empty stomach.'</li><li>"The service varys from day:The service varys from day to day- sometimes they're very nice, and sometimes not."</li></ul> |
87
 
88
  ## Evaluation
89
 
90
  ### Metrics
91
  | Label | Accuracy |
92
  |:--------|:---------|
93
- | **all** | 0.7244 |
94
 
95
  ## Uses
96
 
@@ -109,8 +108,8 @@ from setfit import AbsaModel
109
 
110
  # Download from the 🤗 Hub
111
  model = AbsaModel.from_pretrained(
112
- "setfit-absa-aspect",
113
  "ronalhung/setfit-absa-restaurants-aspect",
 
114
  )
115
  # Run inference
116
  preds = model("The food was great, but the venue is just way too busy.")
@@ -145,14 +144,12 @@ preds = model("The food was great, but the venue is just way too busy.")
145
  ### Training Set Metrics
146
  | Training set | Min | Median | Max |
147
  |:-------------|:----|:--------|:----|
148
- | Word count | 6 | 21.3594 | 43 |
149
 
150
- | Label | Training Sample Count |
151
- |:---------|:----------------------|
152
- | conflict | 2 |
153
- | negative | 19 |
154
- | neutral | 25 |
155
- | positive | 82 |
156
 
157
  ### Training Hyperparameters
158
  - batch_size: (128, 128)
@@ -175,23 +172,26 @@ preds = model("The food was great, but the venue is just way too busy.")
175
  ### Training Results
176
  | Epoch | Step | Training Loss | Validation Loss |
177
  |:------:|:----:|:-------------:|:---------------:|
178
- | 0.0147 | 1 | 0.2915 | - |
179
- | 0.7353 | 50 | 0.1973 | 0.2179 |
180
- | 1.4706 | 100 | 0.0533 | 0.2569 |
181
- | 2.2059 | 150 | 0.0209 | 0.2677 |
182
- | 2.9412 | 200 | 0.0095 | 0.2641 |
183
- | 3.6765 | 250 | 0.0042 | 0.2705 |
184
- | 4.4118 | 300 | 0.0031 | 0.2717 |
 
 
 
185
 
186
  ### Framework Versions
187
- - Python: 3.11.11
188
  - SetFit: 1.1.2
189
  - Sentence Transformers: 3.4.1
190
- - spaCy: 3.7.5
191
- - Transformers: 4.48.3
192
- - PyTorch: 2.5.1+cu124
193
  - Datasets: 3.5.0
194
- - Tokenizers: 0.21.0
195
 
196
  ## Citation
197
 
 
6
  - text-classification
7
  - generated_from_setfit_trainer
8
  widget:
9
+ - text: food:What is even better, is that the prices are very affordable as well,
10
+ and the food is really good.
11
+ - text: 'cuisine:An oasis of refinement: Food, though somewhat uneven, often reaches
12
+ the pinnacles of new American fine cuisine - chef''s passion (and kitchen''s precise
13
+ execution) is most evident in the fish dishes and soups.'
14
+ - text: check:Had we been stalling I could understand where they were coming from,
15
+ but we had been there less than an hour and they hadn't even brought us a check
16
+ yet!
17
+ - text: back waiters:I understand the area and folks you need not come here for the
18
+ romantic, alluring ambiance or the five star service featuring a sommlier and
19
+ a complicated maze of captain and back waiters - you come for the authentic foods,
20
+ the tastes, the experiance.
21
+ - text: wine selection:The wine selection (by the glass and bottle) is wonderful and
22
+ I always recommend that friends make a reservation if they're going to be in town.
23
  metrics:
24
  - accuracy
25
  pipeline_tag: text-classification
 
27
  inference: false
28
  base_model: sentence-transformers/all-MiniLM-L6-v2
29
  model-index:
30
+ - name: SetFit Aspect Model with sentence-transformers/all-MiniLM-L6-v2
31
  results:
32
  - task:
33
  type: text-classification
 
38
  split: test
39
  metrics:
40
  - type: accuracy
41
+ value: 0.8539823008849557
42
  name: Accuracy
43
  ---
44
 
45
+ # SetFit Aspect Model with sentence-transformers/all-MiniLM-L6-v2
46
 
47
+ This is a [SetFit](https://github.com/huggingface/setfit) model that can be used for Aspect Based Sentiment Analysis (ABSA). This SetFit model uses [sentence-transformers/all-MiniLM-L6-v2](https://huggingface.co/sentence-transformers/all-MiniLM-L6-v2) as the Sentence Transformer embedding model. A [LogisticRegression](https://scikit-learn.org/stable/modules/generated/sklearn.linear_model.LogisticRegression.html) instance is used for classification. In particular, this model is in charge of filtering aspect span candidates.
48
 
49
  The model has been trained using an efficient few-shot learning technique that involves:
50
 
 
54
  This model was trained within the context of a larger system for ABSA, which looks like so:
55
 
56
  1. Use a spaCy model to select possible aspect span candidates.
57
+ 2. **Use this SetFit model to filter these possible aspect span candidates.**
58
+ 3. Use a SetFit model to classify the filtered aspect span candidates.
59
 
60
  ## Model Details
61
 
 
64
  - **Sentence Transformer body:** [sentence-transformers/all-MiniLM-L6-v2](https://huggingface.co/sentence-transformers/all-MiniLM-L6-v2)
65
  - **Classification head:** a [LogisticRegression](https://scikit-learn.org/stable/modules/generated/sklearn.linear_model.LogisticRegression.html) instance
66
  - **spaCy Model:** en_core_web_lg
67
+ - **SetFitABSA Aspect Model:** [ronalhung/setfit-absa-restaurants-aspect](https://huggingface.co/ronalhung/setfit-absa-restaurants-aspect)
68
+ - **SetFitABSA Polarity Model:** [ronalhung/setfit-absa-restaurants-polarity](https://huggingface.co/ronalhung/setfit-absa-restaurants-polarity)
69
  - **Maximum Sequence Length:** 256 tokens
70
+ - **Number of Classes:** 2 classes
71
  <!-- - **Training Dataset:** [Unknown](https://huggingface.co/datasets/unknown) -->
72
  <!-- - **Language:** Unknown -->
73
  <!-- - **License:** Unknown -->
 
79
  - **Blogpost:** [SetFit: Efficient Few-Shot Learning Without Prompts](https://huggingface.co/blog/setfit)
80
 
81
  ### Model Labels
82
+ | Label | Examples |
83
+ |:----------|:--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
84
+ | aspect | <ul><li>'staff:But the staff was so horrible to us.'</li><li>"food:To be completely fair, the only redeeming factor was the food, which was above average, but couldn't make up for all the other deficiencies of Teodora."</li><li>"food:The food is uniformly exceptional, with a very capable kitchen which will proudly whip up whatever you feel like eating, whether it's on the menu or not."</li></ul> |
85
+ | no aspect | <ul><li>"factor:To be completely fair, the only redeeming factor was the food, which was above average, but couldn't make up for all the other deficiencies of Teodora."</li><li>"deficiencies:To be completely fair, the only redeeming factor was the food, which was above average, but couldn't make up for all the other deficiencies of Teodora."</li><li>"Teodora:To be completely fair, the only redeeming factor was the food, which was above average, but couldn't make up for all the other deficiencies of Teodora."</li></ul> |
 
 
86
 
87
  ## Evaluation
88
 
89
  ### Metrics
90
  | Label | Accuracy |
91
  |:--------|:---------|
92
+ | **all** | 0.8540 |
93
 
94
  ## Uses
95
 
 
108
 
109
  # Download from the 🤗 Hub
110
  model = AbsaModel.from_pretrained(
 
111
  "ronalhung/setfit-absa-restaurants-aspect",
112
+ "ronalhung/setfit-absa-restaurants-polarity",
113
  )
114
  # Run inference
115
  preds = model("The food was great, but the venue is just way too busy.")
 
144
  ### Training Set Metrics
145
  | Training set | Min | Median | Max |
146
  |:-------------|:----|:--------|:----|
147
+ | Word count | 4 | 17.9296 | 37 |
148
 
149
+ | Label | Training Sample Count |
150
+ |:----------|:----------------------|
151
+ | no aspect | 71 |
152
+ | aspect | 128 |
 
 
153
 
154
  ### Training Hyperparameters
155
  - batch_size: (128, 128)
 
172
  ### Training Results
173
  | Epoch | Step | Training Loss | Validation Loss |
174
  |:------:|:----:|:-------------:|:---------------:|
175
+ | 0.0059 | 1 | 0.311 | - |
176
+ | 0.2959 | 50 | 0.2716 | 0.2559 |
177
+ | 0.5917 | 100 | 0.1808 | 0.2302 |
178
+ | 0.8876 | 150 | 0.0201 | 0.2321 |
179
+ | 1.1834 | 200 | 0.0027 | 0.2249 |
180
+ | 1.4793 | 250 | 0.0017 | 0.2325 |
181
+ | 1.7751 | 300 | 0.0008 | 0.2361 |
182
+ | 2.0710 | 350 | 0.0007 | 0.2496 |
183
+ | 2.3669 | 400 | 0.0006 | 0.2306 |
184
+ | 2.6627 | 450 | 0.0005 | 0.2370 |
185
 
186
  ### Framework Versions
187
+ - Python: 3.11.12
188
  - SetFit: 1.1.2
189
  - Sentence Transformers: 3.4.1
190
+ - spaCy: 3.8.5
191
+ - Transformers: 4.51.3
192
+ - PyTorch: 2.6.0+cu124
193
  - Datasets: 3.5.0
194
+ - Tokenizers: 0.21.1
195
 
196
  ## Citation
197
 
config.json CHANGED
@@ -1,5 +1,4 @@
1
  {
2
- "_name_or_path": "sentence-transformers/all-MiniLM-L6-v2",
3
  "architectures": [
4
  "BertModel"
5
  ],
@@ -19,7 +18,7 @@
19
  "pad_token_id": 0,
20
  "position_embedding_type": "absolute",
21
  "torch_dtype": "float32",
22
- "transformers_version": "4.48.3",
23
  "type_vocab_size": 2,
24
  "use_cache": true,
25
  "vocab_size": 30522
 
1
  {
 
2
  "architectures": [
3
  "BertModel"
4
  ],
 
18
  "pad_token_id": 0,
19
  "position_embedding_type": "absolute",
20
  "torch_dtype": "float32",
21
+ "transformers_version": "4.51.3",
22
  "type_vocab_size": 2,
23
  "use_cache": true,
24
  "vocab_size": 30522
config_sentence_transformers.json CHANGED
@@ -1,8 +1,8 @@
1
  {
2
  "__version__": {
3
  "sentence_transformers": "3.4.1",
4
- "transformers": "4.48.3",
5
- "pytorch": "2.5.1+cu124"
6
  },
7
  "prompts": {},
8
  "default_prompt_name": null,
 
1
  {
2
  "__version__": {
3
  "sentence_transformers": "3.4.1",
4
+ "transformers": "4.51.3",
5
+ "pytorch": "2.6.0+cu124"
6
  },
7
  "prompts": {},
8
  "default_prompt_name": null,
config_setfit.json CHANGED
@@ -1,11 +1,9 @@
1
  {
 
2
  "labels": [
3
- "conflict",
4
- "negative",
5
- "neutral",
6
- "positive"
7
  ],
8
  "spacy_model": "en_core_web_lg",
9
- "normalize_embeddings": false,
10
- "span_context": 3
11
  }
 
1
  {
2
+ "span_context": 0,
3
  "labels": [
4
+ "no aspect",
5
+ "aspect"
 
 
6
  ],
7
  "spacy_model": "en_core_web_lg",
8
+ "normalize_embeddings": false
 
9
  }
model.safetensors CHANGED
@@ -1,3 +1,3 @@
1
  version https://git-lfs.github.com/spec/v1
2
- oid sha256:95bec7361c4e0094bc34b1fed6ee352c06feb5de60a9ef6e89ab83e0f532050b
3
  size 90864192
 
1
  version https://git-lfs.github.com/spec/v1
2
+ oid sha256:3b18d16aca34ce420c1788dc1b084dc9dae7d95dc3d0656b272a85c8107817d1
3
  size 90864192
model_head.pkl CHANGED
@@ -1,3 +1,3 @@
1
  version https://git-lfs.github.com/spec/v1
2
- oid sha256:bdd4039a9c00e09f3e4c90dde222d2015695c03dddf2020e01aceaca7a9b711f
3
- size 13287
 
1
  version https://git-lfs.github.com/spec/v1
2
+ oid sha256:5fe8b5e479477a6226d5bce3f8d401bd7bdfa0c71b6e66857b1d6fe4ce24e75b
3
+ size 3919