feat: Add entropy scoring functionality

[davidberenstein1957]

Inspired by approach described here: https://arxiv.org/abs/2312.15685

• Added a filter_by_score method that takes records and filters them against by computing a score based on the average distance to k nearest neighbours in an index. This should probably be extended with a threshold to avoid bias for high-density regions pre-deduplication. After, it filters them based on a budget, which can be a percentage or an exact number to keep post-filtering.
• Added a self_filter_by_score method that does the same but within the existing index.

from datasets import Dataset, load_dataset

from benchmarks import data
from semhash import SemHash

# Load the dataset
dataset = load_dataset("imdb")

# Initialize SemHash with the columns to deduplicate
semhash = SemHash.from_records(records=dataset["train"].to_list(), columns=["text"])

# Deduplicate the records
deduplicated_records = semhash.self_deduplicate(threshold=0.8).deduplicated

# Filter the records
filtered_records = semhash.filter_by_score(records=deduplicated_records, budget=0.8).selected

# Save the filtered records
train_dataset = Dataset.from_list(filtered_records)

Benchmarking. I could not find any benchmarks for this but I thought we would be able to do something with benchmarks on common tasks like text-classification and QnA pre-and post filtering with certain percentages.

from datasets import Dataset, load_dataset

from benchmarks import data
from semhash import SemHash

# Load the dataset
dataset = load_dataset("imdb")


for threshold in [0.2, 0.4, 0.6, 0.8]:
    for budget in [0.2, 0.4, 0.6, 0.8]:
        # Initialize SemHash with the columns to deduplicate
        semhash = SemHash.from_records(records=dataset["train"].to_list(), columns=["text"])

        # Deduplicate the records
        deduplicated_records = semhash.self_deduplicate(threshold=0.8).deduplicated

        # Filter the records
        filtered_records = semhash.filter_by_score(records=deduplicated_records, budget=0.8).selected

        # Save the filtered records
        train_dataset = Dataset.from_list(filtered_records)

       # train and evaluate squad

[davidberenstein1957]

@Pringled should be ready to go :)