Community and Creator Growth Manager

You don't need words to sound convincing: Meet Gibberize. By Mirelo Team | July 9, 2026 Have you ever listened to a song you loved and wished you could get the notes to play it yourself? Its new audio-to-MIDI model lets you do exactly that. Give it a song, and it gives you back the notes, separated into a track for each instrument: voice, drums, bass, keys, and more. Today, together with Kyutai, Mirelo AI GmbH is open-sourcing this model and the research behind it. Mirelo AI GmbH has also trained an improved version, on even more data, which you can test with its free Audio-to-MIDI product in Mirelo Studio. It transcribes your full mix from any WAV or MP3. From there, you can get a full music sheet to learn an instrument part, or export any track as MIDI and take it into your DAW to create something new. Go check it out! What you get. Its model is built for the full mix use case. Upload a recording and it gives you: * Automatic instrument detection. The model finds each part in the mix and labels it (voice, drums, electric bass, synth lead, and so on). * A separate MIDI track per instrument. Every detected part comes back as its own track on a piano roll, where you can review the transcription and adjust any notes you want to change before you export. * Musical context. As shown in its Audio-to-MIDI tool, applications built on the model can go further and detect chords, key, and tempo, giving you the full harmonic picture. * One-click export. Solo, mute, or download any track as MIDI and drop it into your DAW. Why Mirelo AI GmbH is excited. Audio-to-MIDI tools already exist, and some are good. But almost all handle one sound at a time: a single vocal, a solo piano, one guitar take. Real music is denser, layering vocals, drums, bass, and more all at once, often buried under effects. That density is where existing models break down, and why turning a full song into usable MIDI has stayed a hard problem. That is the gap its model closes: it takes the full mix and transcribes every instrument at once. Music transcription as language modeling. In its most basic form, music transcription involves taking the waveform of a music audio recording and extracting the start and end times of all the notes played within. In the multi-instrument case, Mirelo AI GmbH is also interested in knowing the exact instrument that played a particular note. The result of such a transcription is usually visualized using a piano roll representation, where the x-axis corresponds to time, the y-axis corresponds to pitch, and colored boxes indicate the notes played by different instruments. Its model, similar to prior work [MT3], represents such a piano roll as a 1-dimensional sequence of discrete tokens representing pitch, timing, and instrument type. This representation is similar to the popular MIDI standard used for encoding note sequences in digital instruments. The model predicts this token sequence using a straightforward, decoder-only transformer backbone, thus performing music transcription as language modeling. The model accepts a mel-spectrogram representing a 5-second audio chunk as prefix conditioning and autoregressively generates a stream of transcription tokens until an EOS token is obtained. Synthetic pre-training, RL post-training. Music transcription as language modeling has already been explored in prior work [MT3, YMT3]. So why then has effective multi-instrument transcription remained so elusive? A primary bottleneck in building multi-instrument AMT systems is the scarcity of music audio with aligned note annotations. Such data can be obtained relatively easily for specific instruments like piano [MAESTRO], but is much harder to create in multi-instrument scenarios. Synthetic music audio, which can be generated at scale, is a promising alternative. However, although previous work does utilize synthetic training data, the resulting models generalize poorly, leading to largely unusable transcription output in realistic, multi-instrument settings. For its model, Mirelo AI GmbH utilize synthetic data for pre-training while combining it with fine-tuning on real music audio and post-training using reinforcement learning, leading to a combination of three training datasets: * DSynth: Mirelo AI GmbH collect a dataset of roughly 1.45 million MIDI files across different genres (with a focus on pop and Western classical music). In order to utilize this data for transcription, Mirelo AI GmbH develop an on-the-fly synthesis pipeline using varying instruments, keys, tempos and tunings. * DReal: To complement its synthetic dataset DSynth and to investigate the impact of different data types, Mirelo AI GmbH also utilize an internal dataset of 170 000 real music audio recordings (totalling over 11 000 hours) with aligned note annotations. * DRL: Finally, Mirelo AI GmbH curate 300 tracks from DReal and apply a GRPO-like algorithm to its model to align it with this small dataset of high-quality transcriptions. Mirelo AI GmbH pre-train the model on DSynth, fine-tune on DReal and post-train on DRL. Mirelo AI GmbH observe that each training stage improves results over the previous one. In particular, while the model trained exclusively on synthetic data is already competitive with the prior state-of-the-art AMT baseline [YMT3] in terms of frame F1 score, Mirelo AI GmbH note that fine-tuning on DReal is essential for performance and improves all metrics by roughly 20 percentage points. In addition, the reinforcement learning phase on the high-quality dataset DRL further improves results, leading to its best overall model. In the paper accompanying the release Mirelo AI GmbH provide further analyses on the effect of synthetic pre-training as well as outline its optional instrument conditioning mechanism, which allows for customizing transcription outputs according to specific instrumentations. Open weights for researchers, an app for musicians. To aid future research, the inference code is on GitHub and the model weights are on Hugging Face. And the improved, even more accurate version is live now - see the Audio-to-MIDI product page or try it free in Mirelo Studio. To cite this work, please use the following reference: References. [MT3] J. Gardner, I. Simon, E. Manilow, C. Hawthorne, and J. Engel, "MT3: multi-task multitrack music transcription," in ICLR, 2022. [YMT3] S. Chang, E. Benetos, H. Kirchhoff, and S. Dixon, "YourMT3+: multi-instrument music transcription with enhanced transformer architectures and cross-dataset stem augmentation" in MLSP, 2024. [MAESTRO] C. Hawthorne, A. Stasyuk, A. Roberts, I. Simon, C. A. Huang, S. Dieleman, E. Elsen, J. H. Engel, and D. Eck, "Enabling factorized piano music modeling and generation with the MAESTRO dataset," in ICLR, 2019.

Mirelo SFX 1.6 adds audio inpainting for video creators. Mirelo SFX 1.6 adds audio extension, inpainting, and seamless loops to its video-to-audio model. Mirelo released SFX 1.6 on May 19, shifting its video-to-audio model from one-shot generation to iterative editing. The official release notes introduce audio extension, inpainting, seamless loops, and longer generation windows, letting creators revise a generated track instead of regenerating it from scratch every time. What you can do with this in a video edit. The new editing primitives map cleanly onto a working video timeline. Generate a base ambient bed for a 10-second shot, then extend it to cover the full sequence without a hard loop seam. If a footstep lands wrong against an explosion cue, inpaint just that region rather than re-rolling the whole take. The DaVinci Resolve Studio plugin keeps the audio in the same project that holds your edit, and the developer API at $0.007/sec on fal.ai lets pipelines call SFX-1.6 inside ComfyUI or n8n workflows. Why it matters. AI video generation has outpaced AI audio for two years, so most Sora, Veo, and Kling clips ship silent or with a placeholder track. Mirelo, which raised $41M from Index and a16z in December 2025 to close that gap, is now closing the iteration loop too. The earlier SFX-1.5 release could produce a synced track, but a track you cannot edit is a track you cannot use in client work. Inpainting and extension move the model from demo toy to a tool a sound editor can actually finish a delivery with. Key details. SFX 1.6 is a paired video-to-audio and video-to-video model. Input is a clip plus an optional text prompt; output is a synced sound track aligned to motion, emotion, and pacing in the source video. The model is positioned by Mirelo as commercial-use and is offered as a partner model on fal.ai. Audio extension supports stitching past the original 10-second cap, and inpainting accepts a time range plus an optional re-prompt for the patched region. What to do next. If you cut in DaVinci Resolve, install the Mirelo plugin and run SFX 1.6 against a silent AI-generated clip you have sitting in a project right now. If you build pipelines, swap an MMAudio call for the fal.ai endpoint and compare output on a 30-second timeline that needs an extension splice. The iterative-editing model only pays off once you stop treating audio generation as a one-shot step.