Prototyping with AI to test information-classification options

Managing an interim challenge

Our client, a state-level early education and care agency, was preparing to move its teacher-certification applications review team from an existing system to a new portal Mighty Acorn Digital engineers had built. During the transition, agency staff needed to review about 100 nonstandard email threads each day, determine whether an educator’s certification application was approved or incomplete, and manually update its status in the new portal. The process required up to 4 team members and had created a backlog of approximately 3,000 entries.

Mighty Acorn engineers Josh Fink, Xaq Rothman, Francisco Morales, and Gerell Troche addressed the challenge by exploring ways to reduce repetitive manual work while retaining human reviews where they mattered most. The initial concept was an AI-assisted workflow, which would retrieve email threads through the Microsoft Graph API, preprocess each thread, and send relevant text to a locally hosted large language model (LLM). Next, the model would classify each application as approved, incomplete, or inconclusive, producing a CSV file containing the application ID, classification, and flags for further review. A separate script could use that file to make controlled updates to applications in the new portal.

Francisco described the challenge in determining how to parse the emails:

We knew this script needed to run on a daily basis, but it couldn’t fetch all the emails because there were too many. Because of that, we had to assume that we didn’t have all the context per message. Luckily for us, each message’s body field included all the context of the thread up to that point. Additionally, since we processed the latest emails first, we could also assume that the first message we see for a specific application was the latest decision. We iterated through many solutions.

Testing the AI approach

The team chose a publicly available, open-source model running locally through Ollama, keeping sensitive constituent information inside the project environment rather than sending it to an external service. Taking this approach supports careful stewardship of constituent data while addressing an immediate operational need.

After reviewing the varied email styles used by 4 state-employed application reviewers, our team identified edge cases that could affect an application’s status. Francisco and Xaq worked through those variations, learning that the start of an email thread often contained the most relevant determination, while the end usually included templated material.

Xaq also highlighted the constraints posed by accessing a live inbox that contains personally identifiable information:

Because we’re using email messages, which include personal information—and it’s a live working inbox for folks—you can imagine we’ve had a bit of trouble figuring out exactly what the right sort of access protocol is for this. We created a swappable input plug-in system that let us test email content from different sources, whether exported from a client or generated from plausible data.

While refining inputs and prompts improved the model’s performance, the prototype revealed a core limitation. Even with approximately 98 percent accuracy, the LLM could return different classifications for the same input on different runs. A false positive could incorrectly mark an application as approved, while a false negative could leave the application to languish in the “review queue.”

Choosing a dependable solution

After reviewing the results, the team decided against pursuing an LLM-based production solution. Deterministic regular-expression matching ended up providing more reliable and repeatable results.

A rules-based approach offered a classification process they could inspect, adjust, and validate as email formats changed. This approach also reduced uncertainty because the same input produced the same result each time. Human review remained available for the cases that didn’t match the established rules.

As Josh summarized: “This wasn’t AI for its own sake. We didn’t end up using the solution, but it felt nice to find a golden slot for an LLM. Ultimately, however, a deterministic solution won the day. Running against the same sample set should produce identical results for this use case.”

Not every experiment needs to lead with AI power. The investigative nature of prototyping can lead to surprising results—the original approach may need a complete revision. In this case, testing an LLM helped the team understand the full spectrum of the issue. The simpler production path ended up being the best fit, matching the client’s need for consistency, transparency, and safe operational use.