by Yonghyun Kim
Automatic Piano Transcription has been a longstanding focus within the MIR community. While today’s audio-based systems perform remarkably well when it comes to estimating pitch, onset, and velocity, the sustain pedal creates a physical bottleneck: because sound persists long after a key is released, extracting complete and accurate MIDI note durations from audio alone is practically challenging.
Visual Piano Transcription (VPT) offers a direct workaround by observing the physical state of the keys. A pressed key is visually distinct from a released one, regardless of whether the pedal is engaged. Furthermore, video remains robust in situations where audio is noisy, degraded, or contains multiple instruments.
However, VPT remains underexplored, and existing systems have a few critical gaps:
- Limited Context: They process very short temporal windows — at most 0.2 seconds of video at a time.
- Lopsided Accuracy: Offset (key release) accuracy lags behind onset (key press) detection.
- Missing Dynamics: No prior VPT system actually reports note-level velocity.
To address these limitations, we are introducing V2N (Video to Notes), the first complete VPT system capable of full MIDI extraction. Instead of just supervising the center frame of a small video window, V2N processes a full 1-second temporal context and applies supervision to every single frame.
The core components of our architecture include:
- Multi-Task Heads: We use four dedicated parallel heads for onset, offset, key hold, and velocity. Training these jointly not only enables offset and velocity prediction, but it also improves the onset accuracy.
- Offset-Guided Decoding: Instead of relying on a simple “key hold” threshold — which is the visual equivalent of audio frame-activity — V2N explicitly terminates notes at the peaks predicted by the offset head. This improves the model’s ability to track physical key releases.
- Longer Temporal Context: By using a Conformer-style backbone alongside multi-frame loss, the model utilizes a broader temporal context. Our ablations show this longer context yields performance gains.
TV2N sets a new state-of-the-art on both the PianoVAM and R3 datasets. We are the first video-only system to report note-level velocity F1 scores, and we substantially improve physical key-release predictions.
While there are still open challenges to solve — namely, cross-dataset generalization where models struggle with varying camera geometries and angles — this is a major step forward for visual piano transcription. The work will be presented at the 27th International Society for Music Information Retrieval Conference (ISMIR) in Abu Dhabi, UAE.