LLM-Sanitizer-Pro

Clean, de-identify and curate LLM training data at scale: PII & secrets redaction (regex + GLiNER NER), exact/fuzzy/semantic dedup, language ID, FineWeb-Edu/DCLM quality scoring, Gopher/C4/FineWeb rules, benchmark decontamination, chat validation. Streaming, parallel, resumable CLI + Python API with run manifests & dataset cards

GitHub

Official source

Start locally with the CLI guide