Pipeline for language-distribution based sampling of tokenized datasets

[ajude2s]

This PR introduces a pipeline for sampling and filtering tokenized datasets based on a specified language distribution, with validation to ensure correctness.

Key features:

• Language-distribution-based sampling: Samples a target number of documents per language from annotated JSONL files, ensuring the dataset reflects the desired language proportions.
• Document ID tracking: Maintains consistent mapping between annotated JSONL files and tokenized .pbin files using the document_id = <file_hash>_<row_index> convention.
• Tokenized dataset filtering: Filters .pbin files to include only the sampled documents while preserving ordering.
• Validation of filtered data: Ensures that the filtered tokenized .pbin data matches the sampled raw data by tokenizing the raw documents and comparing tokens for correctness.
• Folder structure preservation: Maintains the original folder structure for input and output datasets (raw_data, annotated, tokenized) and writes filtered .pbin files to a separate output folder.
• Hash mapping integration: Uses a CSV-based hash mapping to locate JSONL files based on document hashes.

This pipeline guarantees reproducible, language-balanced sampling, produces correctly filtered tokenized datasets, and validates that the tokenized output corresponds exactly to the sampled documents.