[4L3-OS-38] OS-38

[4L3-OS-38-05] Human-LLM Hybrid Answer Aggregation for Crowd Annotations

〇Jiyi Li^1,2 (1. Hokkaido University, 2. University of Yamanashi)

Keywords:Crowdsourcing, LLM, Human-in-the-Loop AI

Whether Large Language Models (LLMs) can surpass crowdsourcing in data annotation tasks has gained interest recently. Some works verified this issue with the average performance of individual crowd workers and LLMs on some specific tasks. However, the aggregated answers are the eventually collected annotations, rather than the crowd answers themselves. The scenarios involving crowd answer aggregation need further study. Our studies concentrated on two types of annotations including categorical labels and text answers. On the one hand, we studied the scenario of answer aggregation on the crowd categorical labels in the classification tasks and LLMs are used as creators of the labels. We propose a Crowd-LLM hybrid aggregation method, finding that We propose a Crowd-LLM hybrid aggregation method, finding that adding LLM labels from good LLMs to existing crowdsourcing datasets can enhance the quality of the aggregated labels of the datasets. On the other hand, we also explored text answer aggregation and assess LLMs as aggregators in close-ended crowd text answer scenarios. We proposed a hybrid aggregation approach within a Creator-Aggregator Multi-Stage (CAMS) framework. Experiments demonstrate that our approach can further improve the answer quality based on the combinations of three resources of workers and answers. These findings have been published at ICASSP2024 and EMNLP2024.