Score-to-audio music performance synthesis is the focus of Deep Performer.

Abstract

Synthesizing a musical score into an authentic performance is the goal of music performance synthesis. This article introduces the Deep Performer, a new method for converting musical scores into audio performance synthesis, by drawing on recent developments in text-to-speech synthesis. Unlike speech, music typically incorporates polyphony and lengthy notes. Therefore, we provide two novel approaches to handle polyphonic inputs and provide granular conditioning in a transformer encoder-decoder paradigm. We provide a novel violin dataset that includes paired recordings, scores, and predicted alignments to train our proposed system. We demonstrate that the model we have suggested is capable of synthesising polyphonic and harmonically structured music. In a hearing test, we outperform the baseline model—a conditional generative audio model—in terms of noise level, pitch accuracy, and timbre. On top of that, when tested on an existing piano dataset, our suggested model achieves far better overall quality than the baseline.
Index Terms— Audio synthesis, computer music, music information retrieval, machine learning, neural network