Academic
LLM Web Parser
Modular web data pipeline for LLM ingestion. Features domain-specific parsers, gold-standard evaluation, and FastAPI microservices.
restpythoncrawl4aidockerfastapinlp

Modular web data pipeline for LLM ingestion, built in the Computer Engineering Laboratory at Sapienza University of Rome under Prof. Roberto Navigli, together with Michele Pio Forlani and Salvatore Iavarone. It retrieves, cleans and structures web content so it can feed LLM training and retrieval workflows.
Highlights
- AI-assisted extraction with Crawl4AI for LLM-friendly crawling and content retrieval.
- Domain-specific parsers that strip boilerplate, scripts and styles, and normalise text.
- Gold-standard evaluation to measure parser output quality.
- FastAPI microservices with full Docker orchestration for reproducible deployment.
- Structured output (plain text or JSON) ready for annotation tools and datasets.