A fast AWS S3 browser, with inspiration from s5cmd

Unified Efficient Fine-Tuning of 100+ LLMs & VLMs (ACL 2024)

Rapid fuzzy string matching in Python using various string metrics

Access a database of word frequencies, in various natural languages.

🚀 Efficient implementations for emerging model architectures

utilities for batched llm calls with retries

High performance and CommonMark compliant HTML to Markdown converter. Maintained by the Kreuzberg team. Kreuzberg is a fast, polyglot document intelligence engine with a Rust core. It extracts stru…

Augmentation pipeline for rendering synthetic paper printing, faxing, scanning and copy machine processes

High-performance In-browser LLM Inference Engine

Nano vLLM

An open-source framework for detecting, redacting, masking, and anonymizing sensitive data (PII) across text, images, and structured data. Supports NLP, pattern matching, and customizable pipelines.

Next-generation Punkt sentence boundary detection with zero dependencies

OCR & Document Extraction using vision models

OCR Benchmark

OLMost every training recipe you need to perform data interventions with the OLMo family of models.

qpdf: A content-preserving PDF document transformer

A pipeline for performing OCR on historical newspapers

LaTeXML: a TeX and LaTeX to XML/HTML/ePub/MathML translator.

A computer algebra system written in pure Python

Tile primitives for speedy kernels

Toolkit for linearizing PDFs for LLM datasets/training

📦 Repomix is a powerful tool that packs your entire repository into a single, AI-friendly file. Perfect for when you need to feed your codebase to Large Language Models (LLMs) or other AI tools lik…

Synthetic data curation for post-training and structured data extraction

Code at the speed of thought – Zed is a high-performance, multiplayer code editor from the creators of Atom and Tree-sitter.

Parallel S3 and local filesystem execution tool.

SGLang is a high-performance serving framework for large language models and multimodal models.

A pure-python PDF library capable of splitting, merging, cropping, and transforming the pages of PDF files

Streaming replication for SQLite.

LightLLM is a Python-based LLM (Large Language Model) inference and serving framework, notable for its lightweight design, easy scalability, and high-speed performance.