188: Exploring ChatGPT’s Feasibility for Aligning Research Proposals with Publication Outcomes: A Pilot Study

The Clinical and Translational Science Award (CTSA) in the U.S. encompasses more than 60 medical research institutions. CTSA evaluators face challenges in determining alignment between CTSA funded and supported work and subsequent publication outcomes. This pilot study investigates the feasibility of employing ChatGPT-4o (paid version) and ChatGPT-3.5 (free version) to align between CTSA-supported research proposals and subsequent publications. We designed a retrospective observational study using historical data from our CTSA pilot award survey. A sample of 50 proposals with potentially related publications went through zero-shot and few-shot testing. Performance metrics were applied to evaluate, train, and reevaluate ChatGPT output regarding accuracy, precision, and recall. Results will be compared and reported by performance metrics within and across two ChatGPT versions. We will discuss the implication of our findings and offer insights into using GenAIs to assist CTSA research evaluators.