Adding a new loader with improved content detection (MinerU)

[Sonxia3]

Hi! I've been experimenting with a custom loader based on another open-source project (MinerU) that significantly improves the parsing of PDFs. It captures content more accurately than current loaders, including equations, tables, figures, and their captions, along with structural metadata. The results are noticeably richer, especially for technical or academic documents.

That said, there are some trade-offs. The loader is noticeably slower than the existing ones, and it depends on components that must be run locally, as the underlying tool doesn’t provide a public API. It also introduces heavier dependencies compared to the current lightweight approach of Kotaemon.

Given these benefits and limitations, I wanted to ask if the team would be open to including a loader like this in the project.

MinerU repository link: https://github.com/opendatalab/MinerU