August, 2026 | Music Informatics Group

Multi-Task Multi-Frame Visual Piano Transcription

by Yonghyun Kim

Automatic Piano Transcription has been a longstanding focus within the MIR community. While today’s audio-based systems perform remarkably well when it comes to estimating pitch, onset, and velocity, the sustain pedal creates a physical bottleneck: because sound persists long after a key is released, extracting complete and accurate MIDI note durations from audio alone is practically challenging.

Visual Piano Transcription (VPT) offers a direct workaround by observing the physical state of the keys. A pressed key is visually distinct from a released one, regardless of whether the pedal is engaged. Furthermore, video remains robust in situations where audio is noisy, degraded, or contains multiple instruments.

However, VPT remains underexplored, and existing systems have a few critical gaps:

Limited Context: They process very short temporal windows — at most 0.2 seconds of video at a time.
Lopsided Accuracy: Offset (key release) accuracy lags behind onset (key press) detection.
Missing Dynamics: No prior VPT system actually reports note-level velocity.

To address these limitations, we are introducing V2N (Video to Notes), the first complete VPT system capable of full MIDI extraction. Instead of just supervising the center frame of a small video window, V2N processes a full 1-second temporal context and applies supervision to every single frame.

The core components of our architecture include:

Multi-Task Heads: We use four dedicated parallel heads for onset, offset, key hold, and velocity. Training these jointly not only enables offset and velocity prediction, but it also improves the onset accuracy.
Offset-Guided Decoding: Instead of relying on a simple “key hold” threshold — which is the visual equivalent of audio frame-activity — V2N explicitly terminates notes at the peaks predicted by the offset head. This improves the model’s ability to track physical key releases.
Longer Temporal Context: By using a Conformer-style backbone alongside multi-frame loss, the model utilizes a broader temporal context. Our ablations show this longer context yields performance gains.

TV2N sets a new state-of-the-art on both the PianoVAM and R3 datasets. We are the first video-only system to report note-level velocity F1 scores, and we substantially improve physical key-release predictions.

While there are still open challenges to solve — namely, cross-dataset generalization where models struggle with varying camera geometries and angles — this is a major step forward for visual piano transcription. The work will be presented at the 27th International Society for Music Information Retrieval Conference (ISMIR) in Abu Dhabi, UAE.

H2H Music Improv: A Communication Model and Audio-Visual Dataset for Music Improvisation

by Aleksandra Teng Ma

Improvisation has always been a powerful way to create new kinds of music: musicians feed off each other in real time, and new forms emerge from that back-and-forth. Machine learning could push this even further. Since ML models represent musical structure differently from how humans understand it, they have potential to open up creative territory that’s genuinely new, not just music generated “in the style of” existing artists. But realizing that potential in improvisation requires humans and machine learning systems to communicate meaningfully in real time. This is where free improvisation is most demanding: it removes any pre-agreed style, tempo, key, or role; a significant part of the performance has to emerge from interaction and communication itself. Most current real-time AI improvisation systems, however, treat communication as an afterthought, layering interaction strategies onto a generative algorithm through static, predefined modes like trading or call-and-response, rather than building communication into the algorithm design itself. To close this gap, we looked at how expert human musicians actually communicate with each other during free improvisation, in hopes of eventually building that understanding into generative systems at the design level.

Through a six-month co-design process with expert improvisers, we formalized a communication model built from two point-in-time actions, initiation and acknowledgement, which compose into three temporal states: negotiation (initiations without uptake), proposal (an initiation awaiting acknowledgment), and stability (the shared space that follows). The figure below shows the model itself, alongside a real clip from our dataset, annotated independently by both musicians who played it.

To ground the model in real playing, we recorded the H2H (Human-to-Human) Music Improvisation Dataset: six hours of duo free improvisation across five expert musicians, captured with clean per-player audio stems, and synchronized multi-camera video. Each musician annotated their own intentions and their perception of their partner’s intentions to capture the intent-perception gap.

Our project page provides an interactive preview and downloadable dataset, along with the paper: https://h2himprov.github.io/. This work will also be presented at the 2026 International Society for Music Information Retrieval Conference (ISMIR).

Beyond Static Benchmarks: A Post-Leaderboard Evaluation Paradigm for Generative Music and Creative AI

by Alexander Lerch

Generative AI has rapidly transformed the way we create music and other creative artifacts. Yet, behind these impressive advancements lies a fundamental hurdle that the research community has yet to solve: How do we actually evaluate these systems?

The Evaluation Crisis in Creative AI

Right now, the AI field relies heavily on static benchmarks and single-score leaderboards. We often rush to adopt uninterpretable, single-score proxies, such as the Fréchet Audio Distance (FAD), to evaluate highly complex, multidimensional cultural artifacts (Lerch, 2025).

But creative AI and music pose unique challenges. Trying to compress musical quality into a single number leads to what I call an evaluation crisis characterized by four systemic failures:

Validity Failure: Automated metrics often fail to measure the actual musical or user-facing qualities they are intended to target.
Leaderboard Failure: Optimizing for a single score encourages overfitting and obscures critical trade-offs.
Comparability Failure: The use of different datasets, features, and metrics makes cross-paper comparison unreliable.
Governance Failure: No shared process exists to update evaluation benchmarks as models and dataset standards evolve.

A New Paradigm: Shifting to a Dynamic Ecosystem

To solve this, I propose a paradigm shift from rigid benchmarking to an evolving, multidimensional, and community-governed evaluation ecosystem. Rather than boxing researchers into a single static metric, the proposed framework cleanly decouples three core components: reference data, feature representations, and distance metrics. This modularity ensures the pipeline can easily extend and grow alongside generative models.

The core components of the proposed framework are:

Reference Benchmark Profiles (RBPs): To maintain cross-study standardization without sacrificing domain flexibility, the framework introduces RBPs. These are curated, versioned configurations that define the appropriate datasets, metrics, and features for specific musical genres or use cases (e.g., music for advertisements).
A Dual-Track Feature Strategy: We shouldn’t have to choose between interpretability and performance. The framework implements an explanatory track utilizing interpretable features (like timbre, pitch chroma, and tempo descriptors) alongside a dense track utilizing learned embeddings (like CLAP or MERT).
Multidimensional Assessment and Visualizations: No more single-score rankings. The framework aggregates metrics into a multi-axis vector, allowing researchers to view model performance across distinct musical and technical dimensions using intuitive radar charts.
Community Governance: An open-source framework is only as good as the community behind it. The ecosystem is designed to be steered by a consortium of institutional and community stakeholders to approve, version, and deprecate profiles transparently.

By establishing evaluation as a transparent, evolving process rather than a static metric, this framework outlines a practical roadmap toward the meaningful assessment of creative AI. Ensuring that our tools are flexible, multidimensional, and community-steered allows the field to move past uninterpretable metrics and build an authentic foundation for future innovation.

I will present this concept at the ACM AI Leadership Summit. The preprint is available here.