Intelligent Document Extraction Pipeline

This little project was born out of a suggestion by a friend who was looking for an internal tool for his group at work. Academic paper writing and reviewing were something this group did often, and they wanted a tool to handle the tedious process of confirming the existence of the papers cited in the references section (and identifying potentially hallucinated references). The general idea behind the tool was a simple web application for submitting PDF papers and prompts, with a pipeline running via AWS Lambda on the backend.

The user prompt would ask the LLM for analysis of the academic paper’s contents in terms of overall substance, validity, scientific novelty, etc. However, the project was really more of an exercise in designing a pipeline to process unstructured data by utilizing an LLM to do some of the heavy lifting for the tedious reference-checking task. This was the most technically interesting part from my perspective; leveraging the LLM to handle one of the jobs it does best, dealing with somewhat ambiguous text and making sense of it.

There are multiple ways to skin this particular proverbial cat, and the non-agentic LLM-based pipeline approach that I ultimately used in the project was probably a little bit over-engineered. A more general, elegant solution is certainly possible by leaning a little bit more heavily on the LLM component and using a powerful model. However, part of my goal was to develop a pipeline engineered to be usable with even a very lightweight, less powerful model by giving it smaller, more constrained tasks. This required a bit more work on the pipeline to reduce the amount of interpretation required in each model call, which meant narrowing the range of possible LLM inputs and specifying the expected structured output. Also, the specification was that it all should be executable via an AWS Lambda function, which has its own set of opinions about how this all should be implemented.

Interesting Lessons

1. Designing a system is all about tradeoffs, and a pipeline is no different

   I needed to design a document-processing pipeline that would be systematic but also general enough not to impose too many assumptions about the structure and formatting of the various academic papers being submitted to the app. There is a good deal of variance in what papers look like, how a reference section might be named and formatted, not to mention subtle inconsistencies between document representations produced by the PDF-to-markdown conversion step. The pipeline had to be robust to these differences and be able to detect the start of the references section as well as the end boundary, all without relying on the LLM up to that point.

   The main lesson I took from this entire exercise was one related to system design and tradeoffs. I needed the pipeline to convert the PDF to Markdown, pass the converted paper to the LLM for user-directed analysis, and separately isolate and parse the references section for bibliographic checking. How much work was to be left to the LLM? How much should the user be involved? How general could the text processing be made to handle edge cases without over-engineering or requiring too much human input?

   There is an obvious tension between these three aspects and their relative costs. Too much generality and the code sprawls to accommodate edge cases. Too much human-in-the-loop interaction and the pipeline becomes cumbersome for the user. Too much reliance on the LLM for parsing means a greater risk that weaker models fall short and hallucinated errors appear. Ultimately, I think the balance I found was reasonable.

   First, I could save substantial hassle trying to account for every possible name for a references section by simply asking the user to specify it when submitting their prompt. This was only a very marginal additional step for the user, and very unlikely to be burdensome.

   For section boundary detection, the task was made reasonably robust by using some regex magic. Finding the end boundary could be a little tricky, however, and required a few structural heuristics to account for various PDF formats and artifacts (like page numbers and running headers). If these heuristics failed to identify the end boundary of the references section, a conservative fallback was employed. In that case, the references extracted and passed to the LLM would include all remaining text from the start of the references section until the end of the document. This was a sort of maximal boundary fallback. The system prompt included instructions for the LLM to ignore this extraneous text when producing structured outputs from the extracted references. This is a generally conservative solution to the risk of boundary detection failing, as it is better to include too much text than too little, while introducing only a little additional “entropy” that even a weaker LLM should be able to deal with.

2. LLMs can be very effective at fuzzy matching using context

   This lesson was more concrete and involved the effectiveness of using an LLM as a fuzzy match tool versus relying on exact string comparisons or painstakingly hand-crafted similarity rules to determine whether differently formatted bibliographic records refer to the same paper. If a particular extracted reference’s title returned a potential match from Semantic Scholar, extracted metadata (authors, dates, and titles) was compared using the LLM to do the comparison. The model was asked if two records refer to the same real-world paper, with instructions to allow for some drift between the metadata, including differences in author count or order, small title variations, and publication year discrepancies of up to one year that can arise between publication versions. This left the LLM to judge if differences were due to allowed drift, or whether they were more likely to be different papers. Any papers lacking results from Semantic Scholar, or detected as presenting too strong a degree of drift, are then returned back to the user for a much reduced subset to conduct manual checks on.

Before starting the project, I expected more of my interesting learning takeaways to be focused on prompting and structured output aspects of the process, or on efforts to make the pipeline work with even a weaker, lightweight model. But the most interesting lesson ended up being focused on the higher-level aspects of the project via an important lesson in the art of system design: balancing tradeoffs. In this case, it was a balance between user convenience and involvement, generality and robustness, and code maintainability and clarity. Ultimately, it was about finding a reasonable balance between these competing considerations to produce an effective pipeline.