Update Splade Index

.gitattributes

@@ -33,3 +33,5 @@ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
 *.zip filter=lfs diff=lfs merge=lfs -text
 *.zst filter=lfs diff=lfs merge=lfs -text
 *tfevents* filter=lfs diff=lfs merge=lfs -text
+corpus.jsonl filter=lfs diff=lfs merge=lfs -text
+corpus.mmindex.json filter=lfs diff=lfs merge=lfs -text

README.md

@@ -0,0 +1,60 @@
+---
+language: en
+library_name: splade-index
+tags:
+- splade
+- splade-index
+- retrieval
+- search
+- sparse
+---
+
+# Splade-Index
+
+This is an index created with the [splade-index](https://github.com/rasyosef/splade-index) library (version `0.2.0`)
+
+## Installation
+
+You can install the `splade-index` library with `pip`:
+
+```bash
+pip install "splade-index==0.2.0"
+
+# Include extra dependencies like stemmer
+pip install "splade-index[full]==0.2.0"
+
+# For huggingface hub usage
+pip install huggingface_hub
+```
+
+## Load this Index
+
+You can use the following code to load this SPLADE index from Hugging Face hub:
+
+```python
+import os
+from sentence_transformers import SparseEncoder
+from splade_index import SPLADE
+
+# Download the SPLADE model that was used to create the index from the HuggingFace Hub
+model_id = "the-splade-model-id" # Enter the splade model id
+model = SparseEncoder(model_id)
+
+# Set your huggingface token if repo is private
+token = os.environ["HF_TOKEN"]
+repo_id = "rasyosef/natural_questions_3m_splade_index"
+
+# Load a SPLADE index from the Hugging Face model hub
+retriever = SPLADE.load_from_hub(repo_id, model=model, token=token)
+```
+
+## Stats
+
+This dataset was created using the following data:
+
+| Statistic | Value |
+| --- | --- |
+| Number of documents | 2681468 |
+| Number of tokens | 464573223 |
+| Average tokens per document | 173.25 |
+

corpus.jsonl

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:a8538ecb90cb39efe5c714fffc7f33fccc9c0a2cfa26fefe78719984b817b0f7
+size 1352178313

corpus.mmindex.json

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:fc815836a535e07102b65018672d4a39cd5bfe4ce2f2f582b1eea86ea1fc64cb
+size 27329599

csc.index.npz

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:4f38dd48ccdd3178d991ba25d6f8c57ec2d6fe7b912825b2db7066469d1ea160
+size 2517313766

params.index.json

@@ -0,0 +1,7 @@
+{
+    "dtype": "float32",
+    "int_dtype": "int32",
+    "num_docs": 2681468,
+    "version": "0.2.0",
+    "backend": "numpy"
+}