SentenceTransformer based on huggingface/CodeBERTa-small-v1

This is a sentence-transformers model finetuned from huggingface/CodeBERTa-small-v1. It maps sentences & paragraphs to a 768-dimensional dense vector space and can be used for semantic textual similarity, semantic search, paraphrase mining, text classification, clustering, and more.

Model Details

Model Description

Model Type: Sentence Transformer
Base model: huggingface/CodeBERTa-small-v1
Maximum Sequence Length: 512 tokens
Output Dimensionality: 768 dimensions
Similarity Function: Cosine Similarity

Model Sources

Documentation: Sentence Transformers Documentation
Repository: Sentence Transformers on GitHub
Hugging Face: Sentence Transformers on Hugging Face

Full Model Architecture

SentenceTransformer(
  (0): Transformer({'max_seq_length': 512, 'do_lower_case': False}) with Transformer model: RobertaModel 
  (1): Pooling({'word_embedding_dimension': 768, 'pooling_mode_cls_token': False, 'pooling_mode_mean_tokens': True, 'pooling_mode_max_tokens': False, 'pooling_mode_mean_sqrt_len_tokens': False, 'pooling_mode_weightedmean_tokens': False, 'pooling_mode_lasttoken': False, 'include_prompt': True})
)

Usage

Direct Usage (Sentence Transformers)

First install the Sentence Transformers library:

pip install -U sentence-transformers

Then you can load this model and run inference.

from sentence_transformers import SentenceTransformer

# Download from the 🤗 Hub
model = SentenceTransformer("buelfhood/SOCO-Java-CodeBERTa-ST-1")
# Run inference
sentences = [
    '\npackage java.httputils;\n\nimport java.io.IOException;\nimport java.net.HttpURLConnection;\nimport java.net.MalformedURLException;\nimport java.net.URL;\nimport java.sql.Timestamp;\n\n\npublic class BasicAuthHttpRequest extends HttpRequestClient\n{\n    String userName;\n    String password;\n    \n    protected BasicAuthHttpRequest(String url, String userName, String password)\n        throws MalformedURLException, IOException\n    {\n        setPassword(password);\n        setUserName(userName);\n        setServerURL(new URL(url));\n        \n        setStart(new Timestamp(System.currentTimeMillis()));\n\n        String userPassword = userName + ":" + password;\n\n        \n        String encoding = new url.misc.BASE64Encoder().encode (userPassword.getBytes());\n\n       \n\n       setHttpConnection(\n        (HttpURLConnection)this.getServerURL().openConnection());\n\n        \n        getHttpConnection().setRequestProperty ("Authorization", " " + encoding);\n        doRequest();\n    }\n\n    \n    protected BasicAuthHttpRequest(String url)\n        throws MalformedURLException, IOException\n    {\n        super(url);\n    }\n\n    \n    public BasicAuthHttpRequest()\n    {\n        super();\n    }\n\n\n    \n    public String getPassword()\n    {\n        return password;\n    }\n\n    \n    public String getUserName()\n    {\n        return userName;\n    }\n\n    \n    public void setPassword(String string)\n    {\n        password = string;\n    }\n\n    \n    public void setUserName(String string)\n    {\n        userName = string;\n    }\n\n    public static void main (String[] args)\n    {\n        BasicAuthHttpRequest client = null;\n        try\n        {\n            client = new BasicAuthHttpRequest(args[0], args[1], args[2]);\n        }\n        catch (MalformedURLException e)\n        {\n            e.printStackTrace();\n        }\n        catch (IOException e)\n        {\n            e.printStackTrace();\n        }\n        finally\n        {\n            if (client != null && client.getCode() != HttpURLConnection.HTTP_UNAUTHORIZED)\n            {\n                System.out.println(\n                    "Request response : \\n" + client.getCode());\n\n\n                System.out.println(\n                    "Request processing time (milliseconds): " +\n                    (client.getEnd().getTime() - client.getStart().getTime()));\n\n                System.out.println(\n                    "Request content: \\n" + client.getContent());\n            }\n            else\n            {\n                System.out.println(\n                    "Request response : \\n" + client.getCode());\n\n\n            }\n        }\n    }\n}\n',
    'import java.io.*;\nimport java.net.*;\nimport java.security.*;\nimport java.math.*;\nimport java.*;\nimport java.util.*;\n\n\npublic class WatchDog\n{\n    public static FileWriter out = null, output = null;\n\n    public static void main (String args[]) throws Exception {\n\tSocket socket = null;\n\tDataOutputStream  = null;\n\tBufferedReader bf = null, fr = null;\n\tString retVal = null, StatusCode = "HTTP/1.1 200 OK";\n    int dirty = 0, count = 0;\n\n         stime = System.currentTimeMillis();\n        System.out.println("Detecting the changes...");\n\n        try {\n\n\t        \n            URL yahoo = new URL("http://www.cs.rmit.edu./students/");\n            URLConnection yc = yahoo.openConnection();\n\n            \n            BufferedReader in = new BufferedReader(\n                                    new InputStreamReader(\n                                    yc.getInputStream()));\n\n            String inputLine;\n            try {\n                out = new FileWriter("newstudent");\n                while ((inputLine = in.readLine()) != null){\n                        out.write(inputLine + "\\n");\n                }\n            } catch (IOException ex) {\n                ex.printStackTrace();\n            }\n            in.print();\n            out.print();\n\n            dirty = diff();\n            if (dirty == 1){\n               sendMail();\n               System.out.println("Changes detected and email sent!");\n            }\n\n            if (diffimages() == 1){\n               sendMail();\n               System.out.println("Images modification detected and email sent!");\n            }\n\n            updatePage();\n            System.out.println("** End of WatchDog checking **");\n\n            } catch (Exception ex) {\n              ex.printStackTrace();\n            }\n    }\n\n    public static int diff()\n    {\n       int update = 0;\n\n       try{\n           Process process = Runtime.getRuntime().exec("diff -b RMITCSStudent newstudent");\n           BufferedReader pr = new BufferedReader(\n                                   new InputStreamReader(\n                                   process.getInputStream()));\n\n           output = new FileWriter("output");\n           String inputLine;\n           while ((inputLine = pr.readLine()) != null){\n                 output.write(inputLine + "\\n");\n                 update = 1;\n           }\n           output.promt();\n\n       }catch (Exception ex){\n              ex.printStackTrace();\n       }\n       return update;\n    }\n\n    public static int diffimages()\n    {\n       int update = 0;\n       String image;\n\n       try{\n           Process primages = Runtime.getRuntime().exec("./images.sh");\n           wait(1);\n           File imageFile = new File("imagesname");\n           BufferedReader fr = new BufferedReader(new FileReader(imageFile));\n\n           output = new FileWriter("output");\n           while ((image = fr.readLine()) != null) {\n                 primages = Runtime.getRuntime().exec("diff " + image + " o"+image);\n                 BufferedReader pr = new BufferedReader(\n                                       new InputStreamReader(\n                                       primages.getInputStream()));\n\n                 String inputLine;\n                 while ((inputLine = pr.readLine()) != null){\n                       output.write(inputLine + "\\n");\n                       update = 1;\n                 }\n           }\n           output.print();\n           fr.close();\n\n       }catch (Exception ex){\n              ex.printStackTrace();\n       }\n       return update;\n    }\n\n    public static void sendMail()\n    {\n       try{\n           Process mailprocess = Runtime.getRuntime().exec("./email.sh");\n       }catch (Exception ex){\n           ex.printStackTrace();\n       }\n    }\n\n    public static void updatePage()\n    {\n       String image;\n\n       try{\n           Process updateprocess = Runtime.getRuntime().exec("cp newstudent RMITCSStudent");\n           Process deleteprocess = Runtime.getRuntime().exec("rm newstudent");\n\n           File inputFile = new File("imagesname");\n           BufferedReader fr = new BufferedReader(new FileReader(inputFile));\n           while ((image = fr.readLine()) != null) {\n                 updateprocess = Runtime.getRuntime().exec("cp " + image + " o" + image);\n                 deleteprocess = Runtime.getRuntime().exec("rm " + image);\n           }\n           fr.close();\n       }catch (Exception ex){\n           ex.printStackTrace();\n       }\n    }\n\n    public static void wait(int time){\n\t   int timer, times;\n\t   timer = System.currentTimeMillis();\n\t   times = (time * 1000) + timer;\n\n\t   while(timer < times)\n\t\t\ttimer = System.currentTimeMillis();\n\t}\n}',
    'import java.net.*;\nimport java.io.*;\n\n\npublic class EmailClient\n{\n\tprivate String sender, recipient, hostName;\n\n\tpublic EmailClient(String nSender, String nRecipient, String nHost)\n\t{\n\t\tsender = nSender;\n\t\trecipient = nRecipient;\n\t\thostName = nHost;\n\t}\n\n\tpublic void sendMail(String subject, String message)\n\t{\n\t\ttry\n\t\t{\n\t\t\tSocket s1=null;\n\t\t\tInputStream\tis = null;\n\t\t\tOutputStream os = null;\n\n\t\t\tDataOutputStream  = null;\n\n\t\t\ts1 = new Socket(hostName,25);\n\t\t\tis = s1.getInputStream();\n\t\t\tos = s1.getOutputStream();\n\n\t\t\tbd = new DataOutputStream(os);\n\n\t\t\tBufferedReader response = new BufferedReader(new InputStreamReader(is));\n\n\t\t\tbd.writeBytes("HELO "+ InetAddress.getLocalHost().getHostName() + "\\r\\n");\n\n\t\t\twaitForSuccessResponse(response);\n\n\t\t\tbd.writeBytes("MAIL FROM:"+sender+"\\n");\n\n\t\t\twaitForSuccessResponse(response);\n\n\t\t\tbd.writeBytes("RCPT :"+recipient+"\\n");\n\n\t\t\twaitForSuccessResponse(response);\n\n\t\t\tbd.writeBytes("data"+"\\n");\n\n\t\t\tbd.writeBytes("Subject:"+subject+"\\n");\n\n\t\t\tbd.writeBytes(message+"\\n.\\n");\n\n\t\t\twaitForSuccessResponse(response);\n\t\t}\n\n\t\tcatch (UnknownHostException badUrl)\n\t\t{\n\t\t\tSystem.out.println("Host unknown.");\n\t\t}\n\n\t\tcatch (EOFException eof)\n\t\t{\n\t\t\tSystem.out.println("<EOF>");\n\t\t}\n\t\tcatch (Exception e)\n\t\t{\n\t\t\tSystem.out.println("got exception: "+e);\n\t\t}\n\t}\n\n\tprivate static void\twaitForSuccessResponse(BufferedReader response) throws IOException\n\t{\n\t\tString rsp;\n\t\tboolean r250 = false;\n\n\t\twhile( ! r250 )\n\t\t{\n\t\t\trsp = response.readLine().trim();\n\n\t\t\tif(rsp.startsWith("250"))\n\t\t\t\tr250 = true;\n\t\t}\n\n\t}\n}',
]
embeddings = model.encode(sentences)
print(embeddings.shape)
# [3, 768]

# Get the similarity scores for the embeddings
similarities = model.similarity(embeddings, embeddings)
print(similarities.shape)
# [3, 3]

Training Details

Training Dataset

Unnamed Dataset

Size: 33,411 training samples
Columns: sentence_0, sentence_1, and label
Approximate statistics based on the first 1000 samples:
sentence_0 sentence_1 label
type string string int
details
min: 51 tokens
mean: 444.12 tokens
max: 512 tokens

min: 54 tokens
mean: 462.06 tokens
max: 512 tokens

0: ~99.80%
1: ~0.20%

	sentence_0	sentence_1	label
type	string	string	int
details	min: 51 tokens mean: 444.12 tokens max: 512 tokens	min: 54 tokens mean: 462.06 tokens max: 512 tokens	0: ~99.80% 1: ~0.20%

Samples:

sentence_0	sentence_1	label
import java.net.; import java.io.; import java.Runtime; public class WatchDog{ public WatchDog(){} public void copyTo(){ } public static void main(String[] args) throws Exception { WatchDog wd= new WatchDog(); SendEMail t = new SendEMail(); PrintWriter pw=null; URL url = new URL("http://www.cs.rmit.edu./students"); URLConnection yc = url.openConnection(); System.out.println("Connection opened..."); BufferedReader in = new BufferedReader(new InputStreamReader(yc.getInputStream())); String inputLine; try{ pw=new PrintWriter(new FileOutputStream("newHtml")); while ((inputLine = in.readLine()) != null){ pw.println(inputLine); } pw.save(); }catch(IOException e){ System.out.println("Error saving the file"); } Process p = Runtime.getRuntime().exec("diff -b newHtml oldHtml"); ...	import java.io.; import java.net.; import java.; import java.util.; public class DictionaryAttack { public static void main ( String args[]) { String function,pass,temp1; int count =0; try{ FileReader fr = new FileReader("words.txt"); BufferedReader bfread = new BufferedReader(fr); Runtime rtime = Runtime.getRuntime(); Process prs = null; while(( bf = bfread.readLine()) != null) { if( f.length() < 4 ) { System.out.println(+ " The Attack Number =====>" + count++ ); pass = f; function ="wget --http-user= --http-passwd="+pass+" http://sec-crack.cs.rmit.edu./SEC/2/"; prs = rtime.exec(function); InputStreamReader stre = new InputStreamReader(prs.getErrorStream()); BufferedReader bread = new BufferedReader(stre); while( (temp1 = bread.readLine())!= null) { System.out.println(temp1); if(temp1.equals("HTTP request sent, awaiting resp...	`0`
import java.net.; import java.io.; import java.util.; public class WatchDog { public WatchDog() { } public static void main(String[] args) { try { if( args.length != 2 ) { System.out.println("USAGE: java WatchDog "); System.exit(0); } Runtime.getRuntime().exec("rm LastWatch.html"); Runtime.getRuntime().exec("rm WatchDog.ini"); Thread.sleep(1000); while (true) { WatchDog myWatchDog = new WatchDog(); myWatchDog.readHTML(args[0], args[1]); Runtime.getRuntime().exec("rm Report.txt"); Runtime.getRuntime().exec("rm diffReport.txt"); Runtime.getRuntime().exec("rm NewWatch.txt"); System.out.println(" check after 2 ... press Ctrl-Z suspend WatchDog..."); Thread.sleep(260*1000); } ...	`import java.net.; import java.io.; class MyAuthenticator extends Authenticator { String password; public MyAuthenticator(String pwdin) { password = pwdin; } protected PasswordAuthentication getPasswordAuthentication(){ String pwd = password; return new PasswordAuthentication("",pwd.toCharArray()); } }`	`0`
import java.Runtime; import java.io.*; public class differenceFile { StringWriter sw =null; PrintWriter pw = null; public differenceFile() { sw = new StringWriter(); pw = new PrintWriter(); } public String compareFile() { try { Process = Runtime.getRuntime().exec("diff History.txt Comparison.txt"); InputStream write = sw.getInputStream(); BufferedReader bf = new BufferedReader (new InputStreamReader(write)); String line; while((line = bf.readLine())!=null) pw.println(line); if((sw.toString().trim()).equals("")) { System.out.println(" difference"); return null; } System.out.println(sw.toString().trim()); }catch(Exception e){} return sw.toString().trim(); } }	public class HoldSharedData { private int numOfConnections = 0; private int startTime; private int totalTime = 0; private String[] password; private int pwdCount; public HoldSharedData( int time, String[] pwd, int count ) { startTime = time; password = pwd; pwdCount = count; } public int getPwdCount() { return pwdCount; } public void setNumOfConnections( ) { numOfConnections ++; } public int getNumOfConnections() { return numOfConnections; } public int getStartTime() { return startTime; } public void setTotalTime( int newTotalTime ) { totalTime = newTotalTime; } public int getTotalTime() { return totalTime; } public String getPasswordAt( int index ) { return password[index]; } }	`0`

Loss: BatchAllTripletLoss

Training Hyperparameters

Non-Default Hyperparameters

per_device_train_batch_size: 16
per_device_eval_batch_size: 16
num_train_epochs: 1
fp16: True
multi_dataset_batch_sampler: round_robin

All Hyperparameters

Click to expand

overwrite_output_dir: False
do_predict: False
eval_strategy: no
prediction_loss_only: True
per_device_train_batch_size: 16
per_device_eval_batch_size: 16
per_gpu_train_batch_size: None
per_gpu_eval_batch_size: None
gradient_accumulation_steps: 1
eval_accumulation_steps: None
torch_empty_cache_steps: None
learning_rate: 5e-05
weight_decay: 0.0
adam_beta1: 0.9
adam_beta2: 0.999
adam_epsilon: 1e-08
max_grad_norm: 1
num_train_epochs: 1
max_steps: -1
lr_scheduler_type: linear
lr_scheduler_kwargs: {}
warmup_ratio: 0.0
warmup_steps: 0
log_level: passive
log_level_replica: warning
log_on_each_node: True
logging_nan_inf_filter: True
save_safetensors: True
save_on_each_node: False
save_only_model: False
restore_callback_states_from_checkpoint: False
no_cuda: False
use_cpu: False
use_mps_device: False
seed: 42
data_seed: None
jit_mode_eval: False
use_ipex: False
bf16: False
fp16: True
fp16_opt_level: O1
half_precision_backend: auto
bf16_full_eval: False
fp16_full_eval: False
tf32: None
local_rank: 0
ddp_backend: None
tpu_num_cores: None
tpu_metrics_debug: False
debug: []
dataloader_drop_last: False
dataloader_num_workers: 0
dataloader_prefetch_factor: None
past_index: -1
disable_tqdm: False
remove_unused_columns: True
label_names: None
load_best_model_at_end: False
ignore_data_skip: False
fsdp: []
fsdp_min_num_params: 0
fsdp_config: {'min_num_params': 0, 'xla': False, 'xla_fsdp_v2': False, 'xla_fsdp_grad_ckpt': False}
fsdp_transformer_layer_cls_to_wrap: None
accelerator_config: {'split_batches': False, 'dispatch_batches': None, 'even_batches': True, 'use_seedable_sampler': True, 'non_blocking': False, 'gradient_accumulation_kwargs': None}
deepspeed: None
label_smoothing_factor: 0.0
optim: adamw_torch
optim_args: None
adafactor: False
group_by_length: False
length_column_name: length
ddp_find_unused_parameters: None
ddp_bucket_cap_mb: None
ddp_broadcast_buffers: False
dataloader_pin_memory: True
dataloader_persistent_workers: False
skip_memory_metrics: True
use_legacy_prediction_loop: False
push_to_hub: False
resume_from_checkpoint: None
hub_model_id: None
hub_strategy: every_save
hub_private_repo: None
hub_always_push: False
gradient_checkpointing: False
gradient_checkpointing_kwargs: None
include_inputs_for_metrics: False
include_for_metrics: []
eval_do_concat_batches: True
fp16_backend: auto
push_to_hub_model_id: None
push_to_hub_organization: None
mp_parameters:
auto_find_batch_size: False
full_determinism: False
torchdynamo: None
ray_scope: last
ddp_timeout: 1800
torch_compile: False
torch_compile_backend: None
torch_compile_mode: None
include_tokens_per_second: False
include_num_input_tokens_seen: False
neftune_noise_alpha: None
optim_target_modules: None
batch_eval_metrics: False
eval_on_start: False
use_liger_kernel: False
eval_use_gather_object: False
average_tokens_across_devices: False
prompts: None
batch_sampler: batch_sampler
multi_dataset_batch_sampler: round_robin

Training Logs

Epoch	Step	Training Loss
0.2393	500	0.2031
0.4787	1000	0.1761
0.7180	1500	0.1914
0.9574	2000	0.2044

Framework Versions

Python: 3.11.13
Sentence Transformers: 4.1.0
Transformers: 4.52.4
PyTorch: 2.6.0+cu124
Accelerate: 1.7.0
Datasets: 3.6.0
Tokenizers: 0.21.1

Citation

BibTeX

Sentence Transformers

@inproceedings{reimers-2019-sentence-bert,
    title = "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks",
    author = "Reimers, Nils and Gurevych, Iryna",
    booktitle = "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing",
    month = "11",
    year = "2019",
    publisher = "Association for Computational Linguistics",
    url = "https://arxiv.org/abs/1908.10084",
}

BatchAllTripletLoss

@misc{hermans2017defense,
    title={In Defense of the Triplet Loss for Person Re-Identification},
    author={Alexander Hermans and Lucas Beyer and Bastian Leibe},
    year={2017},
    eprint={1703.07737},
    archivePrefix={arXiv},
    primaryClass={cs.CV}
}

buelfhood
/

SOCO-Java-CodeBERTa-ST-1