[4G1-GS-6] Language media processing:

[4G1-GS-6-04] Verification of Detection Performance of LLM-Generated Text in Japanese

Rio Iwata¹, 〇Takashi Tsunakawa¹, Masafumi Nishida¹ (1. Shizuoka University)

Keywords:LLMs, GPT, AI-generated text detection

The recent spread of large language models (LLMs), such as ChatGPT, is expected to increase the amount of LLM-generated text on the Internet, including spam and misinformation. In this study, we applied Binoculars, a state-of-the-art zero-shot detection method, to Japanese text discrimination and evaluated its detection performance for Japanese text generated by LLM. We also propose a method to employ focal loss for calculating the loss function. We evaluated the detection performance on a dataset consisting of human-written text extracted from the OSCAR corpus and LLM-generated text using GPT-3.5 Turbo, and found that both the accuracy rate and F1-score remained above 0.94 for texts longer than 200 characters, and the detection performance decreased as the text became shorter. Furthermore, a proposed method based on focal loss improved the accuracy rate and F1-score for any number of characters.