[3E1-GS-10] AI application: Entertainment

[3E1-GS-10-05] Production of MusicXML from Locally Inclined Sheetmusic Photo Image by Using Measure-based Multi-modal Deep-learning-driven Assembly Method

img2Mxml App for playing music from smartphone sheetmusic photo images

〇Tomoyuki Shishido^1,5, Fehmiju Fati², Daisuke Tokushige³, Yasuhiro Ono⁴, Itsuo Kumazawa¹ (1. Tokyo Institute of Technology, 2. Musician, 3. SKIP Law Firm, 4. Enspirea LLC, 5. Shishido&Associates)

Keywords:sheetmusic, score, smartphone, deep learning, measure, photo

Deep learning has been applied to optical music sheet recognition (OMR). However, OMR processing from various sheet-music images still lacks precision to be widely applicable. We propose a measure-based multimodal deep-learning-driven assembly (MMdA) method enabling end-to-end OMR processing from various images including inclined photo images. Using this method, measures are extracted using a deep-learning model, aligned, and resized to be used for inference of given musical-symbol components by using multiple deep-learning models in sequence or in parallel. The use of each standardized measure enables efficient training of the deep-learning models and accurate adjustment of five staff lines in each measure, which enables locally inclined sheet-music images to be precisely positioned. Thus, a score can be reproduced from the inclined image with the proposed MMdA method while current OMR applications cannot. Multiple musical-symbol-component deep-learning feature-category models with a small number of feature types can represent a diverse set of notes and other musical symbols including chords. The proposed MMdA method provides a solution to end-to-end OMR processing and enhances the utility of OMR of mobile phone- based sheet-music photo images.