Large Language Model(LLM) Integration: The integration of LLM into DonorsChoose’s project creation process enhanced the approval workflow by providing real-time, AI-powered suggestions and feedback. This streamlined the process, making it easier for teachers to refine and submit their projects, ultimately improving project approval rates and reducing time spent in the submission phase.
The challenge:
As part of DonorsChoose's policy, projects submitted by teachers are reviewed by our volunteer screening team, who identify any issues and either approve the project or return it to draft for revisions.
Over time, we noticed that a significant number of projects were being sent back to draft because teachers had not mentioned their requested items in their essay. This is a required step to ensure donors can clearly understand where their contributions are going.
The solution:
I collaborated with our operations team, who manage the volunteer screening process, and the engineering team to integrate LLM into the project creation workflow.
The LLM analyzes the teacher’s essay for relevant keywords and context related to the requested items. If it detects missing references, it prompts teachers to update their essay to clearly mention the items they’ve requested.
More details about this project are outlined below. It’s live, and we’re currently collecting data!
Click on the images below to view them at a larger size
The details:
Ensuring integrity and equity
Our screening team reviews projects to maintain integrity and ensure that teachers are genuinely requesting supplies for their students.
Additionally, our data revealed that minority teachers were being drafted at higher rates than their white counterparts. While our current metrics make it difficult to determine the cause, our goal is to analyze the data collected through this feature and use those insights to guide future iterations—helping us address this discrepancy and improve the process for all teachers.
Context setting
An LLM is only as effective as the data it’s trained on. To ensure it recognized contextual requests, I collaborated with the engineering and data teams to curate a training set of real projects. For example, a teacher might request supplies for a classroom role-playing activity, such as a fictional coffee shop, including snacks and magazines. An early version of the model didn’t recognize that magazines made sense in this context, leading to unnecessary rejections. Training the LLM with diverse project examples helped refine its accuracy and improve the approval process.
My role in this project:
I was the lead designer and product manager, focusing on user experience, usability, front-end styling, and defining success metrics.
