SA-Yur-or commited on
Commit
bd9e8d7
·
1 Parent(s): 1998f3f

[doc]: update examples of usage the model

Browse files
Files changed (1) hide show
  1. README.md +34 -8
README.md CHANGED
@@ -71,10 +71,13 @@ In training, a primary objective was to maximize prediction accuracy while speci
71
 
72
  **Pre-requirements**: \
73
  Install *generated_text_detector* \
74
- Run following command: ```pip install git+https://github.com/superannotateai/generated_text_detector.git@v1.0.0```
 
 
75
 
76
  ```python
77
  from generated_text_detector.utils.model.roberta_classifier import RobertaClassifier
 
78
  from transformers import AutoTokenizer
79
  import torch.nn.functional as F
80
 
@@ -82,16 +85,20 @@ import torch.nn.functional as F
82
  model = RobertaClassifier.from_pretrained("SuperAnnotate/ai-detector-low-fpr")
83
  tokenizer = AutoTokenizer.from_pretrained("SuperAnnotate/ai-detector-low-fpr")
84
 
 
 
85
  text_example = "It's not uncommon for people to develop allergies or intolerances to certain foods as they get older. It's possible that you have always had a sensitivity to lactose (the sugar found in milk and other dairy products), but it only recently became a problem for you. This can happen because our bodies can change over time and become more or less able to tolerate certain things. It's also possible that you have developed an allergy or intolerance to something else that is causing your symptoms, such as a food additive or preservative. In any case, it's important to talk to a doctor if you are experiencing new allergy or intolerance symptoms, so they can help determine the cause and recommend treatment."
86
 
 
 
87
  tokens = tokenizer.encode_plus(
88
- text_example,
89
- add_special_tokens=True,
90
- max_length=512,
91
- padding='longest',
92
- truncation=True,
93
- return_token_type_ids=True,
94
- return_tensors="pt"
95
  )
96
 
97
  _, logits = model(**tokens)
@@ -101,6 +108,25 @@ proba = F.sigmoid(logits).squeeze(1).item()
101
  print(proba)
102
  ```
103
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
104
  ## Training Detailes
105
 
106
  A custom architecture was chosen for its ability to perform binary classification while providing a single model output, as well as for its customizable settings for smoothing integrated into the loss function.
 
71
 
72
  **Pre-requirements**: \
73
  Install *generated_text_detector* \
74
+ Run following command: ```pip install git+https://github.com/superannotateai/generated_text_detector.git@v1.1.0```
75
+
76
+ ### Native Usage
77
 
78
  ```python
79
  from generated_text_detector.utils.model.roberta_classifier import RobertaClassifier
80
+ from generated_text_detector.utils.preprocessing import preprocessing_text
81
  from transformers import AutoTokenizer
82
  import torch.nn.functional as F
83
 
 
85
  model = RobertaClassifier.from_pretrained("SuperAnnotate/ai-detector-low-fpr")
86
  tokenizer = AutoTokenizer.from_pretrained("SuperAnnotate/ai-detector-low-fpr")
87
 
88
+ model.eval()
89
+
90
  text_example = "It's not uncommon for people to develop allergies or intolerances to certain foods as they get older. It's possible that you have always had a sensitivity to lactose (the sugar found in milk and other dairy products), but it only recently became a problem for you. This can happen because our bodies can change over time and become more or less able to tolerate certain things. It's also possible that you have developed an allergy or intolerance to something else that is causing your symptoms, such as a food additive or preservative. In any case, it's important to talk to a doctor if you are experiencing new allergy or intolerance symptoms, so they can help determine the cause and recommend treatment."
91
 
92
+ text_example = preprocessing_text(text_example)
93
+
94
  tokens = tokenizer.encode_plus(
95
+ text_example,
96
+ add_special_tokens=True,
97
+ max_length=512,
98
+ padding='longest',
99
+ truncation=True,
100
+ return_token_type_ids=True,
101
+ return_tensors="pt"
102
  )
103
 
104
  _, logits = model(**tokens)
 
108
  print(proba)
109
  ```
110
 
111
+ ### Usage in Detector Wrapper
112
+
113
+ ```python
114
+ from generated_text_detector.utils.text_detector import GeneratedTextDetector
115
+
116
+
117
+ detector = GeneratedTextDetector(
118
+ "SuperAnnotate/ai-detector-low-fpr",
119
+ device="cuda",
120
+ preprocessing=True
121
+ )
122
+
123
+ text_example = "It's not uncommon for people to develop allergies or intolerances to certain foods as they get older. It's possible that you have always had a sensitivity to lactose (the sugar found in milk and other dairy products), but it only recently became a problem for you. This can happen because our bodies can change over time and become more or less able to tolerate certain things. It's also possible that you have developed an allergy or intolerance to something else that is causing your symptoms, such as a food additive or preservative. In any case, it's important to talk to a doctor if you are experiencing new allergy or intolerance symptoms, so they can help determine the cause and recommend treatment."
124
+
125
+ res = detector.detect_report(text_example)
126
+
127
+ print(res)
128
+ ```
129
+
130
  ## Training Detailes
131
 
132
  A custom architecture was chosen for its ability to perform binary classification while providing a single model output, as well as for its customizable settings for smoothing integrated into the loss function.