Fine-French: La Crème de la Crème du Web Français

How we created the first industrial-scale French corpus using 100% AI curation - keeping only the cream of the crop

Introduction

The French AI ecosystem has long struggled with a fundamental problem: the lack of high-quality, large-scale French training data. While English datasets like The Pile and C4 have enabled remarkable progress in English language models, French researchers and practitioners have been forced to work with either translated datasets, multilingual corpora with limited French content, or smaller, manually curated collections.

Today, we're excited to introduce Fine-French, the first industrial-scale French web corpus that addresses this challenge head-on. With 66 million high-quality French documents filtered from an initial 125 million using GPT-4 synthetic annotation, Fine-French represents a new paradigm in dataset creation: keeping only la crème de la crème through AI judging AI at scale.

The Problem: Web Data Quality Crisis

Current State of French Training Data

Most existing French datasets suffer from critical quality issues:

• Commercial pollution: Promotional content, discount codes, and marketing spam
• Structural artifacts: Poorly filtered HTML, navigation menus, and website boilerplate
• Multilingual dilution: French content mixed with other languages, reducing density
• Translation artifacts: Poor-quality machine translations from English sources

Our analysis of raw French web data revealed that 85% of content contains significant quality issues that negatively impact language model training.

The Cost of Poor Training Data

Training language models on noisy data leads to:

• Reduced model performance on downstream tasks
• Commercial bias in generated content
• Inconsistent language quality and style
• Wasted computational resources on meaningless patterns

Our Solution: GPT-4 as Quality Arbiter

The Fine-French Pipeline

Fine-French introduces a revolutionary three-phase approach to dataset curation:

Phase 1: Synthetic Annotation at Scale

We developed a comprehensive prompt system for GPT-4-o to evaluate French web content across multiple quality dimensions:

Quality Assessment Framework:

1. Linguistic Purity (25 points)

   • Grammar and spelling accuracy
   • Vocabulary richness and appropriateness
   • Syntactic complexity and correctness

2. Educational Value (25 points)

   • Information density and depth
   • Learning potential for language models
   • Contribution to French language understanding

3. Structural Integrity (25 points)

   • Logical organization and coherence
   • Complete thoughts and ideas
   • Absence of HTML artifacts

4. Commercial Filtering (25 points)

   • Detection of promotional content
   • Identification of marketing language
   • Filtering of transactional text

Sample GPT-4 Evaluation:

{
  "text": "L'architecture gothique française se caractérise par une recherche constante de verticalité...",
  "quality_score": 92,
  "rationale": {
    "linguistic_purity": "Excellent French vocabulary and grammar",
    "educational_value": "High information density about French cultural heritage",
    "structural_integrity": "Well-organized explanatory text",
    "commercial_content": "Zero promotional elements detected"
  },
  "bad_prompt_detected": 0
}

Phase 2: Knowledge Distillation

The 2 million GPT-4 annotations served as training data for a specialized CamemBERT-large classifier:

Training Configuration:

• Base Model: CamemBERT-large (110M parameters)
• Training Data: 2M synthetic annotations (80/10/10 split)
• Optimization: AdamW with cosine scheduling
• Performance: 94.2% accuracy on held-out test set

Classification Performance:

Overall Performance:

• Accuracy: 94.2%
• Processing Speed: 1,247 documents/second

Phase 3: Industrial-Scale Filtering

The trained classifier processed the entire FineWeb-2 French corpus, generating quality predictions for each document:

Filtering Results:

• Input: 125,020,619 documents
• High Quality: 66,234,891 documents (53%)
• Filtered Out: 58,785,728 documents (47%)

Quality Analysis and Validation

Content Distribution Comparison

Before Filtering (FineWeb-2):

After Filtering (Fine-French):

Language Model Performance Impact

We evaluated the impact of Fine-French on downstream model performance:

Training Efficiency Comparison:

Geographic and Domain Coverage

Fine-French maintains diversity across French-speaking regions and various content domains. The filtering process preserves content from multiple francophone sources while eliminating commercial pollution that was present in the original dataset.

The dataset includes content spanning educational materials, technical documentation, news articles, cultural content, and government resources, with the key difference being the systematic removal of promotional and commercial content that dominated the original corpus.

Technical Implementation

Dataset Schema

{
    'text': str,                    # Main textual content
    'id': str,                      # Unique document identifier
    'url': str,                     # Source URL
    'date': str,                    # Crawl timestamp
    'language_score': float,        # French confidence score
    'bad_prompt_detected': int,     # Quality flag (0=keep, 1=filter)
    'minhash_cluster_size': int,    # Deduplication cluster size
    'dump': str,                    # CommonCrawl dump identifier
    'file_path': str               # Original file location
}

Usage Examples

Loading High-Quality Content Only:

from datasets import load_dataset

# Load only the curated, high-quality content
dataset = load_dataset("legmlai/finefrench").filter(
    lambda x: x['bad_prompt_detected'] == 0
)

print(f"High-quality documents: {len(dataset['train']):,}")
# Output: High-quality documents: 66,234,891

Training Data Preparation:

def prepare_training_text(examples):
    """Prepare text for language model training"""
    return {
        'text': [
            text.strip() for text in examples['text'] 
            if len(text.strip()) > 100  # Minimum length filter
        ]
    }

training_data = dataset.map(
    prepare_training_text,
    batched=True,
    remove_columns=['id', 'url', 'date', 'file_path']
)

Quality Analysis:

import pandas as pd

df = dataset['train'].to_pandas()

# Analyze retention by domain
df['domain'] = df['url'].str.extract(r'https?://(?:www\.)?([^/]+)')
retention_by_domain = df.groupby('domain')['bad_prompt_detected'].agg(['count', 'mean'])

print("Top domains by quality retention:")
print(retention_by_domain.sort_values('mean').head(10))

Comparison with Existing Datasets

Scale and Quality Comparison

Innovation Advantages

Fine-French's Unique Value Propositions:

1. AI-Native Curation: First dataset created entirely through AI evaluation
2. Commercial-Free: Systematic elimination of promotional content
3. Quality Consistency: 94.2% accuracy in quality detection
4. French-Optimized: Designed specifically for French language model training
5. Industrial Scale: Largest curated French corpus available
6. Reproducible Process: Fully automated pipeline for future updates

Impact on French AI Ecosystem

Democratizing High-Quality Training Data

Fine-French addresses several critical challenges in the French AI ecosystem:

Research Impact:

• Enables researchers to train competitive French language models
• Reduces computational waste from poor-quality training data
• Provides benchmark for future French dataset development

Industry Applications:

• Powers development of French customer service AI
• Enables creation of French content generation tools
• Supports French legal and financial AI applications

Educational Benefits:

• Provides students with clean, pedagogical French text
• Enables development of French language learning tools
• Supports computational linguistics research

Performance Benchmarks

The methodology used to create Fine-French - filtering out commercial content and low-quality text - is designed to improve training efficiency and model performance. The 22.4% perplexity improvement mentioned earlier demonstrates the value of training on curated, high-quality content versus raw web data.

Future Considerations

Dataset Maintenance

Fine-French represents our current approach to French data curation. The methodology we've developed could potentially be applied to:

• Quality Assessment: Ongoing evaluation of web content quality
• Content Updates: Assessment of newly crawled French content
• Methodology Refinement: Improvements to the filtering pipeline

Lessons Learned

The development of Fine-French has validated several key principles:

• AI-Native Curation: Language models can effectively evaluate training data quality
• Commercial Content Detection: Automated systems can reliably identify promotional content
• Quality vs. Quantity: Smaller, curated datasets often outperform larger, noisy ones
• French-Specific Needs: Language-specific curation provides better results than generic approaches

Conclusion

Fine-French represents a paradigm shift in dataset creation: from manual curation and simple heuristics to AI-native quality assessment at scale. By leveraging GPT-4's language understanding capabilities, we've created the first industrial-scale French corpus that prioritizes la crème de la crème over raw quantity.

Key Achievements:

• ✅ 66 million high-quality French documents
• ✅ 53% retention rate from intelligent filtering
• ✅ 22.4% improvement in model perplexity
• ✅ Zero human annotation required
• ✅ 94.2% classification accuracy
• ✅ Complete commercial content elimination

The release of Fine-French democratizes access to high-quality French training data, enabling researchers, startups, and enterprises to build better French language models. More importantly, it demonstrates that AI systems can effectively curate la crème de la crème of training data for other AI systems, opening new possibilities for automated dataset creation.

This methodology shows that we no longer need to accept noisy, commercial-polluted datasets. Instead, we can systematically identify and preserve only the highest quality content for language model training.

Try Fine-French today: https://huggingface.co/datasets/legmlai/finefrench

---

Fine-French is developed by legml.ai and expertly curated by Mohamad Alhajar. The dataset is released under ODC-By 1.0 license.

Citation

@dataset{finefrench2024,
  title={Fine-French: La Crème de la Crème du Web Français},
  author={Alhajar, Mohamad and {legml.ai}},
  year={2024},
  publisher={Hugging Face},
  url={https://huggingface.co/datasets/legmlai/finefrench},
  note={AI-curated French web corpus filtered from 125M to 66M high-quality documents using GPT-4 synthetic annotation},
  license={ODC-By 1.0}
}