UNIVERSITY PARK, Pa. — The captions that explain images and figures in scientific papers could be far more useful, with more pertinent information conveyed in a more comprehensible way, according to Ting-Hao "Kenneth" Huang, associate professor in the Penn State College of Information Sciences and Technology (IST). The Alfred P. Sloan Foundation recently awarded Huang a $246,000 grant for his proposal, “Generating and Validating Captions for Scientific Figures Based on Scientific Claims,” which continues work improving how such captions are developed. 

Huang submitted the proposal in response to a public request for information from the Chan Zuckerberg Initiative and the Sloan Foundation about existing applications of artificial intelligence (AI) to transform scientific work. 

Since 2019, Huang — along with C. Lee Giles, professor emeritus in the College of IST, and their students — has been working on developing a web-based system that makes it easier for scholars and researchers to compose high-quality captions for figures in academic papers.  

“Scientists use figures like bar charts, pie charts and line charts to communicate the key findings of their work,” Huang said. “However, our studies have shown that the text accompanying critical figures — the figure captions — are often poorly written, with more than half of the captions deemed ‘unhelpful’ by the doctoral students who reviewed them.” 

The project was initially funded by a seed grant from the College of IST and further supported by gift funds from Adobe Research. The grant from the Sloan Foundation will enable the researchers to get more users to access and try the figure-captioning system.  

“Within this effort, we hope to help paper authors — especially those for whom English is not their first language — be more intentional about what specific scientific claims they aim to support with figures and compose effective captions that align with their communication goals," Huang said. “The grant is a strong encouragement for us, and we’re very happy that we can bring this work to more people.”