An ML-powered system built with a team at the NCAA to digitize decades of historical basketball stat reports — turning scanned, inconsistently formatted PDFs into accurate, searchable, ingestion-ready player-season records.
The NCAA holds decades of historical basketball stat reports as scanned PDFs — inconsistent formats, faded text, and handwritten corrections across generations of paper archives. OCR could turn them into text, but the real challenge was structure and governance: preserving raw evidence, automating safe transformations, and surfacing ambiguous cases instead of hiding them.
Decades of report formats change table boundaries, spans, and row alignment. The system first has to isolate the player-stat table from surrounding noise, then resolve header ambiguity — labels like FG, PCT, OPP, FT, and Pos Ht need deterministic mapping to one standard header. On top of that, frequent OCR errors from messy handwriting, scan artifacts, and document blemishes had to be caught and corrected rather than silently passed through.
Azure Document Intelligence's prebuilt-layout model extracts tables, cells, and geometry from each scanned report. A geometry-aware step then isolates the actual player-stat table from surrounding noise, comparing tables by Y-range, alignment, and vertical overlap. Header mapping uses fuzzy matching against a reviewed dictionary to collapse hundreds of inconsistent OCR column labels — 113 variants of "Avg" alone — into one standardized field, and a rules-based correction layer fixes recurring numeric and name errors, with every edit logged for auditability.
End-to-end: source PDFs → OCR (Azure Document Intelligence) → target table extraction → header mapping → error correction → final JSON output mapped to official NCAA stat IDs, ready for database ingestion.
Once documents were processed with Azure Document Intelligence, the resulting JSON was moved into Azure Blob Storage and queried directly with Azure Synapse serverless SQL. I wrote SQL against these files to validate and clean extracted records — flagging missing values, inconsistencies, and low-confidence OCR outputs before they moved further down the pipeline. That querying layer supported the broader ETL process, turning raw OCR output into structured datasets for analysis and helping integrate the digitized historical records into a centralized, queryable database.
We built two internal React apps to keep humans in the loop at scale: LayoutSort, a sandbox for inspecting table layouts and testing extraction logic against real OCR output, and a Dictionary Mapping app for reviewing and approving ambiguous header matches before they're promoted into the production dictionary.
About 18,445 distinct header labels still need review, 70 source files require manual edge-case handling, and the framework is being generalized beyond basketball — treating each new sport as a controlled pilot before its rules move into the shared pipeline.