[3G4-GS-6] Language media processing:

[3G4-GS-6-05] Uncertainty Estimation of Mathematical Problem Answers by Semantic Entropy Probes Learned with Recipe Sentences

〇Yulia Tamura¹, Takuya Matsuzaki¹ (1. Tokyo University of Science)

Keywords:Hallucination, semantic entropy

This study examines the feasibility of applying semantic entropy-based evaluation models across domains, from recipe to mathematical texts. Initial experiments showed that estimating semantic entropy for mathematical solutions is challenging. To address this, we leveraged Semantic Entropy Probes (SEPs) and hypothesized that recipe texts, which share a multi-step structure akin to mathematical texts, could provide an indirect estimation method. We thus trained a logistic regression model on recipe texts and tested its applicability to mathematical texts. Results suggest this model at least partially estimates uncertainty in mathematical solutions.