Spaces:
Running
Running
File size: 5,041 Bytes
e137e27 feb6fe5 e137e27 feb6fe5 e137e27 |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 |
from fasthtml.common import *
from fasthtml.components import *
intro_div = Div(
H2("Perplexity Evaluation on Duplicate Data"),
H3("Model based Quality Estimation"),
P("We took one of the model-based data quality evaluation strategies adopted by [DataComp-LM](https://arxiv.org/abs/2406.11794), which used perplexity filtering as a candidate for quality filtering. DataComp-LM followed [CCNet’s](https://arxiv.org/abs/1911.00359) practice to use a 5-gram Kneser-Ney model as implemented in the [KenLM](https://github.com/kpu/kenlm) library for efficient perplexity calculation. Following this practice, we estimated data quality by taking a KenLM model (from [edugp/kenlm](https://huggingface.co/edugp/kenlm)) trained on English Wikipedia data to compute perplexity on data with different duplication patterns. Lower perplexity is regarded as a signal of higher quality."),
H3("Sampling Strategy"),
P("We started from a processed Common Crawl (CC) ablation dataset divided by the number of duplicates of each document. For each CC dump, we have different buckets each holding chunks of document with different duplicate count ranges (1-1, 2-5, 6-10, 11-100, 101-1000, 1001-30000000). We sampled the first 10k documents from each chunk with their meta data."),
)
perp1_div = Div(
Section(
H3("Perplexity vs Buckets"),
P("For each bucket, we aggregated all the chunks that belong to a single year and calculated the average perplexity for each (bucket, year) data point."),
Img(src="images/prep-diff-buckets-global.png", height = "300", width = "600" ),
),
Section(
H3("Perplexity vs Years"),
P("Taking the same data, we can convert it into a graph indicating the yearly trend. For most buckets, the average perplexity of dumps from more recent years seem to be lower than that of former years."),
Img(src="images/prep-across-diff-year-global-dup-buckets.png", height = "300", width = "600" ),
),
Section(
H3("Perplexity vs Document Duplication"),
P("We can also break each bucket into distinct document counts. The graph becomes a bit noisy at the end because of insufficient samples with larger duplication counts."),
Img(src="images/prep-across-diff-docs-dup-count-global.png", height = "300", width = "600" ),
),
Section(
H3("Perplexity vs Dump Duplication"),
P("We are also interested in how the number of dumps a document is in affect data quality. From the graph below we can see that documents that are duplicated across around 40 - 60 dumps usually have lower perplexity."),
Img(src="images/prep-across-diff-dump-dup-counts-global.png", height = "300", width = "600" ),
),
Section(
H3("Perplexity vs Local Buckets"),
P("Previously we have seen that documents in recent dumps tend to have lower perplexity. This might be related to the way how global deduplication was implemented. During global deduplication, we only keep copy in the latest dump. Hence documents that are duplicated across multiple dumps only appear in the latest one. To avoid bias brought by this strategy, we tried to recover the states before the global deduplication by reading the metadata attached with each document."),
Img(src="images/prep-across-diff-buckets-local.png", height = "300", width = "600" ),
),
Section(
H3("Perplexity vs Local Dump Duplication"),
P("Following the same practice, we can plot the local version of the graph of average perplexity with respect to dump duplication."),
Img(src="images/prep-diff-dump-dump-counts-local.png", height = "300", width = "600" ),
),
)
llama_div = Div(
Section(
H2("Llama 3.1 8B"),
P("For comparison purpose, we run the same perplexity evaluation with llama 3.1 8B model.")
),
Section(
H3("Perplexity vs Buckets"),
Img(src="images/perp-across-diff-buckets-global.png", height = "300", width = "600" ),
),
Section(
H3("Perplexity vs Years"),
Img(src="images/prep-across-diff-years-global.png", height = "300", width = "600" ),
),
Section(
H3("Perplexity vs Dump Duplication"),
Img(src="images/prep-vs-dump-dup-global.png", height = "300", width = "600" ),
),
Section(
H3("Perplexity vs Local Buckets"),
Img(src="images/prep-diff-buckets-local.png", height = "300", width = "600" ),
),
Section(
H3("Perplexity vs Local Dump Duplication"),
Img(src="images/prep-vs-dump-dup-global.png", height = "300", width = "600" ),
),
)
def results():
return Div(
Section(
intro_div,
perp1_div,
llama_div,
id="inner-text"
)
)
|