Category: project

Multi-Task Multi-Frame Visual Piano Transcription

by Yonghyun Kim

Automatic Piano Transcription has been a longstanding focus within the MIR community. While today’s audio-based systems perform remarkably well when it comes to estimating pitch, onset, and velocity, the sustain pedal creates a physical bottleneck: because sound persists long after a key is released, extracting complete and accurate MIDI note durations from audio alone is practically challenging.

Visual Piano Transcription (VPT) offers a direct workaround by observing the physical state of the keys. A pressed key is visually distinct from a released one, regardless of whether the pedal is engaged. Furthermore, video remains robust in situations where audio is noisy, degraded, or contains multiple instruments.

However, VPT remains underexplored, and existing systems have a few critical gaps:

  • Limited Context: They process very short temporal windows — at most 0.2 seconds of video at a time.
  • Lopsided Accuracy: Offset (key release) accuracy lags behind onset (key press) detection.
  • Missing Dynamics: No prior VPT system actually reports note-level velocity.

To address these limitations, we are introducing V2N (Video to Notes), the first complete VPT system capable of full MIDI extraction. Instead of just supervising the center frame of a small video window, V2N processes a full 1-second temporal context and applies supervision to every single frame.

The core components of our architecture include:

  • Multi-Task Heads: We use four dedicated parallel heads for onset, offset, key hold, and velocity. Training these jointly not only enables offset and velocity prediction, but it also improves the onset accuracy.
  • Offset-Guided Decoding: Instead of relying on a simple “key hold” threshold — which is the visual equivalent of audio frame-activity — V2N explicitly terminates notes at the peaks predicted by the offset head. This improves the model’s ability to track physical key releases.
  • Longer Temporal Context: By using a Conformer-style backbone alongside multi-frame loss, the model utilizes a broader temporal context. Our ablations show this longer context yields performance gains.

TV2N sets a new state-of-the-art on both the PianoVAM and R3 datasets. We are the first video-only system to report note-level velocity F1 scores, and we substantially improve physical key-release predictions.result comparison

While there are still open challenges to solve — namely, cross-dataset generalization where models struggle with varying camera geometries and angles — this is a major step forward for visual piano transcription. The work will be presented at the 27th International Society for Music Information Retrieval Conference (ISMIR) in Abu Dhabi, UAE.

H2H Music Improv: A Communication Model and Audio-Visual Dataset for Music Improvisation

by Aleksandra Teng Ma

Improvisation has always been a powerful way to create new kinds of music: musicians feed off each other in real time, and new forms emerge from that back-and-forth. Machine learning could push this even further. Since ML models represent musical structure differently from how humans understand it, they have potential to open up creative territory that’s genuinely new, not just music generated “in the style of” existing artists. But realizing that potential in improvisation requires humans and machine learning systems to communicate meaningfully in real time. This is where free improvisation is most demanding: it removes any pre-agreed style, tempo, key, or role; a significant part of the performance has to emerge from interaction and communication itself. Most current real-time AI improvisation systems, however, treat communication as an afterthought, layering interaction strategies onto a generative algorithm through static, predefined modes like trading or call-and-response, rather than building communication into the algorithm design itself. To close this gap, we looked at how expert human musicians actually communicate with each other during free improvisation, in hopes of eventually building that understanding into generative systems at the design level.

Through a six-month co-design process with expert improvisers, we formalized a communication model built from two point-in-time actions, initiation and acknowledgement, which compose into three temporal states: negotiation (initiations without uptake), proposal (an initiation awaiting acknowledgment), and stability (the shared space that follows). The figure below shows the model itself, alongside a real clip from our dataset, annotated independently by both musicians who played it.

Annotation example

To ground the model in real playing, we recorded the H2H (Human-to-Human) Music Improvisation Dataset: six hours of duo free improvisation across five expert musicians, captured with clean per-player audio stems, and synchronized multi-camera video. Each musician annotated their own intentions and their perception of their partner’s intentions to capture the intent-perception gap.

photo of the recording studio setupOur project page provides an interactive preview and downloadable dataset, along with the paper: https://h2himprov.github.io/. This work will also be presented at the 2026 International Society for Music Information Retrieval Conference (ISMIR).

Beyond Static Benchmarks: A Post-Leaderboard Evaluation Paradigm for Generative Music and Creative AI

by Alexander Lerch

Generative AI has rapidly transformed the way we create music and other creative artifacts. Yet, behind these impressive advancements lies a fundamental hurdle that the research community has yet to solve: How do we actually evaluate these systems?

The Evaluation Crisis in Creative AI

Right now, the AI field relies heavily on static benchmarks and single-score leaderboards. We often rush to adopt uninterpretable, single-score proxies, such as the Fréchet Audio Distance (FAD), to evaluate highly complex, multidimensional cultural artifacts (Lerch, 2025).

But creative AI and music pose unique challenges. Trying to compress musical quality into a single number leads to what I call an evaluation crisis characterized by four systemic failures:

  • Validity Failure: Automated metrics often fail to measure the actual musical or user-facing qualities they are intended to target.
  • Leaderboard Failure: Optimizing for a single score encourages overfitting and obscures critical trade-offs.
  • Comparability Failure: The use of different datasets, features, and metrics makes cross-paper comparison unreliable.
  • Governance Failure: No shared process exists to update evaluation benchmarks as models and dataset standards evolve.

A New Paradigm: Shifting to a Dynamic Ecosystem

To solve this, I propose a paradigm shift from rigid benchmarking to an evolving, multidimensional, and community-governed evaluation ecosystem. Rather than boxing researchers into a single static metric, the proposed framework cleanly decouples three core components: reference data, feature representations, and distance metrics. This modularity ensures the pipeline can easily extend and grow alongside generative models.

The core components of the proposed framework are:

  • Reference Benchmark Profiles (RBPs): To maintain cross-study standardization without sacrificing domain flexibility, the framework introduces RBPs. These are curated, versioned configurations that define the appropriate datasets, metrics, and features for specific musical genres or use cases (e.g., music for advertisements).
  • A Dual-Track Feature Strategy: We shouldn’t have to choose between interpretability and performance. The framework implements an explanatory track utilizing interpretable features (like timbre, pitch chroma, and tempo descriptors) alongside a dense track utilizing learned embeddings (like CLAP or MERT).
  • Multidimensional Assessment and Visualizations: No more single-score rankings. The framework aggregates metrics into a multi-axis vector, allowing researchers to view model performance across distinct musical and technical dimensions using intuitive radar charts.
  • Community Governance: An open-source framework is only as good as the community behind it. The ecosystem is designed to be steered by a consortium of institutional and community stakeholders to approve, version, and deprecate profiles transparently.

By establishing evaluation as a transparent, evolving process rather than a static metric, this framework outlines a practical roadmap toward the meaningful assessment of creative AI. Ensuring that our tools are flexible, multidimensional, and community-steered allows the field to move past uninterpretable metrics and build an authentic foundation for future innovation.

I will present this concept at the ACM AI Leadership Summit. The preprint is available here.

FoleySet: A Multi-Level Human-Annotated Foley Sound Dataset

by Sunshiyu Wang

FoleySet is a new publicly available dataset of 10,000 human-annotated Foley audio clips designed to address a key gap in Foley research: the lack of structured, fine-grained, human-annotated Foley sound datasets. As evolving machine learning approaches create new opportunities for Foley sound classification, retrieval, synchronization, and generation, high-quality datasets are essential for developing and evaluating reliable models. FoleySet provides this foundation through a two-level taxonomy containing 9 major categories and 73 sub-categories, covering sounds such as footsteps, clothing movement, human-produced sounds, liquid sounds, metal interactions, clicks, break/drop events, material interactions, and open/close mechanisms.

Because Foley does not have a universally unambiguous definition, we define it as sounds arising from human-related actions, including human-produced sounds and human interactions with materials, objects, and surfaces. Based on this definition, we developed a Foley-specific taxonomy by reviewing prior research, Foley practice, and commercial Foley sound libraries. Common Foley-related keywords were extracted, normalized, and manually refined into the final two-level taxonomy. Using this taxonomy, FoleySet was built through a multi-step pipeline: candidate clips were collected from Creative Commons Zero-licensed Freesound recordings, manually screened, standardized through audio processing, and annotated with major-category, sub-category, and one-shot/multi-shot labels. The final dataset includes training, validation, and test splits, along with metadata such as Freesound tags, descriptions, source IDs, uploader information, and URLs.

FoleySet: Content categories and statistics
FoleySet: Content categories and statistics

We also provide baseline classification results using a PaSST-based audio embedding model. The model performs well on the 9-way major-category classification task, while the more fine-grained 73-way sub-category task remains challenging, especially for acoustically similar or underrepresented Foley sounds. Overall, FoleySet provides a practical dataset and Foley-specific taxonomy for developing models that retrieve, classify, and generate detailed sounds of human action and material interaction, with the goal of supporting future Foley research and contributing to broader audiovisual production workflows.

Dataset link: FoleySet

Paper link: Paper

Quantifying Spatial Audio Quality Impairment

by Karn Watcharasupat

Spatial audio quality is a highly multifaceted concept (see this for a very long list of things to consider). “Geometrical” components of spatial audio quality are perhaps the least subjective aspect of spatial audio quality to quantify, yet there have been very little attempt at dealing withit since BSS Eval came out almost 20(!) years ago.

Even the geometrical component of spatial audio quality is not trivial to quantify. We resorted to only considering the interchannel time differences (ITD) and interchannel level differences (ILD) of the test signal relative to a reference signal. With this, it is actually possible to construct a signal model to isolate _some_ of the spatial distortion. By using a combination of Weiner-style least-square optimization and good ol’ correlation maximization, we propose a signal decomposition method to isolate the spatial error, in terms of interchannel gain leakages and changes in relative delays, from a processed signal. These intermediates parameters can then be used as a diagnostic tool to identify the nature of the spatial distortion and to quantify the spatial quality impairment.

Methods

Evaluation of Generative Models in Music

by Alexander Lerch

Generative Artificial intelligence is increasingly capable of composing music, from short melodies to full songs. Despite the increasing number of new, “superior” models, there has been no consensus on how to measure this progress. How do we know if one model is indeed better than the other?

Evaluating AI-generated music is challenging because music perception is inherently subjective. There is no single “correct” or “best” version of a song, and people’s tastes vary widely and objectively evaluating elusive properties such as aesthetics, musicality, creativity or emotional impact is ultimately pointless. The language of music is complex and abstract, and its perception subjective.

Evaluation Targets

The paper breaks evaluation into two main categories:
• System Output: focusing on the generated output of a system and its properties
• User Experience: focusing on how people interact with a generative system.
Researchers use both subjective and objective methods. Subjective methods include listening tests, surveys, and Turing-style tests where listeners try to guess whether a piece was composed by a human or a machine. Objective methods use mathematical metrics to compare the AI’s output to human-composed music, measuring things like pitch distribution, rhythm patterns, and audio fidelity.

Challenges and Conclusion

There are several major challenges in evaluating generative music systems. First, the validity of existing methodologies is limited. Second, existing metrics have limited and/or unknown musical and perceptual meaning. Third, there is no standard set of metrics, which makes it hard to compare different systems. In addition, there are concerns around the topic of responsible AI.

There is a need for more consistent, interdisciplinary approaches to evaluating generative music. It highlights the need for better metrics, more transparent research practices, and deeper collaboration between computer scientists, musicians, and psychologists.

Resources

Please find the open access survey paper for more details.

Latte: Cross-framework Python Package for Evaluation of Latent-Based Generative Models

by Karn N. Watcharasupat and Junyoung Lee

Controllable deep generative models have promising applications in various fields such as computer vision, natural language processing, or music. However, implementations of evaluation metrics for these generative models remain non-standardized. Evaluating disentanglement learning, in particular, might require implementing your own metrics, possibly entangling you more than when you started.

Latte (for latent tensor evaluation) is a package designed to help both you and your latent-based model to stay disentangled at the end of the day (YMMW, of course). The Python package is, by design, created to work with both PyTorch and TensorFlow. All metrics are first implemented in NumPy with minimal dependencies (like scikit-learn) and then a wrapper is created to turn these NumPy functions into TorchMetrics or Keras Metric modules. This way, each metric is always calculated in the exact same way regardless of the deep learning framework being used. In addition, the functional NumPy API is also exposed, so that post-hoc evaluation or models from other frameworks can also enjoy our metric implementations.

Currently, our package supports the classic disentanglement metrics: Mutual Information Gap (MIG), Separate Attribute Predictability (SAP), Modularity. In addition, several dependency-aware variants of MIG proposed here and here are also included. These metrics are useful for situations where your semantic attributes are inherent dependent with respect to one another, a situation where traditional metrics might penalize a latent space that has correctly learned the nature of the semantic attributes. Latte also implements interpolatability metrics which evaluate how smoothly or monotonically your decoder translates the latent vectors into generated samples.

To make life simpler, Latte also comes equipped with metric bundles which are optimized implementations of multiple metrics commonly used together. The bundles optimize away duplicate computation of identical or similar steps in the metric computation, reducing both the lines of code needed and the runtime. We are working to add more metrics and bundles into our package. The most updated list can always be found at our GitHub repository.

Latte can be easily installed via pip using pip install latte-metrics. We have also created a few Google Colab notebooks demonstrating how you can use Latte to evaluate an attribute-regularized VAE for controlling MNIST digits, using vanilla PyTorch, PyTorch Lightning, and TensorFlow. The full documentation of our package can be found here.

Resources

From labeled to unlabeled data – on the data challenge in automatic drum transcription

by Chih-Wei Wu

Automatic Drum Transcription (ADT) is an on-going research topic that concerns the extraction of drum events from music signals. After roughly three decades of research on this topic, many methods and several datasets have been proposed to address this problem. However, similar to many other Music Information Retrieval (MIR) research topics, the availability of realistic and carefully curated datasets is one of the bottlenecks for advancing the performance of ADT systems.

In our previous blog post, we briefly discussed this challenge in the context of ADT. With a standard annotated dataset (i.e., ENST drums) and a small collection of unlabeled data, we demonstrated the possibility of harnessing unlabeled music data for improvements in the context of ADT.
In this paper, we explore this idea further in the following directions:

  1. Identify the major types of ADT systems and investigate generic methods for integrating unlabeled data into these systems accordingly
  2. Train the systems using a large scale unlabeled music dataset
  3. Evaluate all systems using multiple labeled datasets currently available

The intention is to validate the idea of using unlabeled data for ADT in a large scale. To achieve this goal, we present two approaches.

Method

To show that the benefit of using unlabeled data can be generalized to most ADT systems, the first thing is to identify the most popular ADT approaches. To this end, we reviewed existing ADT systems (for more information, please refer to this recent publication), and we found that the two most popular approaches can be categorized as Segment-and-classify ( classification-based) and Separate-and-detect (activation-based).

Based on these two types of approaches, two learning paradigms for incorporating unlabeled data are evaluated. These are

    1. Feature Learning and
    2. Student Teacher Learning.

As shown in the figure below, both paradigms may extract information from unlabeled data and transfer them to ADT systems through different mechanisms; the feature learning paradigm learns a feature extractor that computes distinctive features from audio signals, whereas the student teacher learning paradigm focuses on generating “pseudo ground truth” (i.e., soft targets) using teacher models and passing them onto student models. Different variants of both paradigms are evaluated.

Results 1: we need more labeled data

In the first part of the experiment results, the averaged performance across all evaluated systems for each labeled dataset (e.g, ENST drums, MIREX 2005, MDB drums, RBMA) is presented. As shown in the following figure, for each individual drum instrument such as Kick Drum (KD), Snare Drum (SD), and HiHat (HH), the averaged performances differ from dataset to dataset. This result not only shows the relative difficulties of these datasets, but also implies the danger of relying on solely one dataset (which is exactly the case in many prior ADT studies). This result highlights the need for more diverse labeled datasets!

Results 2: unlabeled data is useful

In the second part of the experiment results, different systems for each learning paradigm are compared under the controlled conditions (e.g., training methods, the number of unlabeled examples used, etc.). For feature learning paradigm, as shown in the following table, both evaluated systems outperformed baseline systems on SD for the averaged F-Measure. This improvement was confirmed through a statistical check. This result suggests Segment-and-classify ADT systems can successfully benefit from unlabeled data through feature learning.

Role System HH BD SD
Baseline MFCC 0.61 0.62 0.40
Baseline CONV-RANDOM 0.61 0.54 0.39
Evaluated CONV-AE 0.61 0.62 0.42
Evaluated CONV-DAE 0.61 0.61 0.42

For student-teacher learning paradigm, encouraging results can also be found (this time on HH). In the following table, it is shown that all student models performed better on HH compared to both teacher models. This result indicates the possibility of obtaining students that are better than teachers with the help of unlabeled data. Additionally, this result confirms the finding in our previous paper in a larger scale.

Role System HH BD SD
Teacher PFNMF (SMT) 0.47 0.61 0.45
Teacher PFNMF (200D) 0.47 0.67 0.40
Student FC-200 0.56 0.57 0.44
Student FC-ALL 0.53 0.59 0.42
Student FC-ALL (ALT 0.55 0.58 0.44

Conclusion

According to the results, both learning paradigms can potential improve the ADT performances with the addition of unlabeled data. However, each paradigm seems to benefit different instruments. In other words, it is not easy to conclude which paradigm is “the way to go” when it comes to harnessing unlabeled resources. To simply put it, unlabeled data certainly has potential in improving ADT systems, and further investigation is worthwhile.

If you are interested in learning more details about this work, please refer to our paper. The code is available on github.

On the evaluation of generative models in music

by Li-Chia Richard Yang

Generative modeling among creative systems has research interest in a wide variety of tasks. Just as deep learning has reshaped the whole field of artificial intelligence, it has reinvented generative modeling in recent years, e.g., in music or painting. Regardless, however, of the research interest in generative systems, the assessment and evaluation of such systems has proven challenging.

In recent research on music generation, various data-driven models have shown promising results. As a quick example, here are two generated samples from two distinct systems:

Magenta (Attention RNN)
Magenta (Lookback RNN)

 

Now, how can we analyze and compare the behavior of these models?
As the ultimate judge of creative output is the human (listener or viewer), subjective evaluation is generally preferable in generative modeling. However, the general drawbacks of subjective evaluation can be summarized as various issues related to the required amount of resources and to the experiment design. Furthermore, objective evaluation has the advantage of providing a systematic, repeatable measurement across a significant amount of generated samples.

The proposed evaluation strategy

The proposed method does not aim at assessing musical pieces in the context of human-level creativity nor does it attempt to model the aesthetic perception of music.

It rather applies the concept of multi-criteria evaluation in order to provide metrics that assess basic technical properties of the generated music and help researchers identify issues and specific characteristics of both model and dataset. In a first step, we define two collections of samples as our datasets (in case of objective evaluation, one dataset contains the generated samples, the other contains samples from the training dataset). Then, we extract a set of features based on musical domain-knowledge for two main targets of the proposed evaluation strategy:

Absolute MeasurEments

Absolute measurements give insights into properties and characteristics of a generated or collected set of data.
During the model design phase of a generative system, it can be of interest to investigate absolute metrics from the output of different system iterations or of datasets as opposed to a relative evaluation. A typical example is the comparison of the generated results from two generative systems: although the model properties cannot be determined precisely for a data-driven approach, the observation of the generated samples can justify or invalidate a system design.
For instance, an absolute measurement can be a statistic analysis of note length transition histogram of each sample of a given dataset. In the following figure, we can easily observe the difference of such this feature among datasets from two different genres.

Relative Measurements

In order to enable the comparison of different sets of data, the relative measure generalizes the result among features with various dimensions.
We first perform pairwise exhaustive cross-validation for each feature and smooth the histogram results into probability density functions (PDFs) for a more generalizable representation. If the cross-validation is computed within one set of data, we will refer to it as intra-set distances. If each sample of one set is compared with all samples of the other set, we call it the inter-set distances.
Finally, we measure the similarity between these distributions for the application of evaluating music generative systems, and compute two metrics between the target dataset’s intra-set PDF and the inter-set PDF: the Kullback-Leibler Divergence (KLD) and overlapped area (OA) of two PDFs.
Take the following visualized figure as an example: assume set1 is the training data, while set2 and set3 correspond to generated results from two systems. The analysis can provide a quick observation It can be easily observed that the system that generates generated set2 produces results more in line has a closer behavior in such feature with the training data (in the context of this feature).

Find out more

Check out our paper for detailed use-case demonstration and the released toolbox for further application.

Instrument Activity Detection in Polyphonic Music

by Siddharth Gururani

Most forms of music are rendered as a mixture of acoustic and electronic instruments. The human ear, for the most part, is able to discern the instruments being played in a song fairly easily. However, the same is not true for computers or machines. The task of recognizing instrumentation in music is still an unsolved and active area of research in Music Information Retrieval (MIR).

The applications of such a technology are manifold:

  • Metadata which includes instrumentation enables instrument-specific music discovery and recommendations.
  • Identifying regions of activity of specific instruments in a song allows easy browsing for users. For example, a user interested in a guitar solo or vocals in a song can easily browse to the relevant part.
  • Instrument activity detection may serve as a helpful pre-processing step for other MIR tasks such as automatic transcription and source separation.

In our work, we propose a neural network-based system to detect activity for 18 different instruments in polyphonic music.

Challenges in Instrument Activity Detection

A big challenge in building algorithms for instrument activity detection is the lack of appropriate datasets. Until very recently, the IRMAS dataset was used as the benchmark dataset for instrument recognition in polyphonic music. However, this dataset is not suitable for an instrument activity detection because of the following reasons:

  • The test set contains 3 to 10 second snippets of audio that are only labeled with instruments present instead of a fine-grained instrument activity annotation.
  • The training clips are labeled with a single ‘predominant’ instrument even if more than one instrument is active in the clip.

We overcome this challenge by leveraging multi-track datasets such as the MedleyDB and Mixing Secrets dataset. These multi-track datasets contain the mixes as well as the stems accompanying them. Therefore, annotations for fine-grained stem activity may be automatically obtained by applying envelope tracking on the instrument stems.

In addition, we identify metrics that allow easier comparison of models for instrument activity detection. Traditional metrics such as precision, recall and f1-score are both threshold dependent and not ideal for multi-label classification scenarios. We use label-ranking average precision (LRAP) and area under the ROC curve (AUC) for comparison between different model architectures. Both these metrics are threshold agnostic and are suitable for multi-label classification.

Method and Models

We propose a rather simple pipeline for our instrument activity detection system. The block diagram below shows the high-level processing steps in our approach. First, we split our all the multi-tracks into artist conditional splits. We obtain 361 training tracks and 100 testing tracks. During training, the various models are fed with log-scaled mel-spectrograms for 1 second clips for the training tracks. We train these models to predict all the instruments present in a 1 second clip. We compare Fully Connected, Convolutional (CNN) and Convolutional-Recurrent (CRNN) Neural Networks in this work.

During testing, a track is split into 1 second clips and fed into the model. Once all 1 second level predictions are obtained from the model, we evaluate the predictions at different time-scales: 1 s, 5 s, 10 s and track-level. We aggregate over time by max-pooling the predictions and annotations for longer time-scale evaluation.

Results

As expected, the CNN and CRNN models outperform the Fully Connected architectures. The CNN or the CRNN perform very similarly and we attribute that to the choice of input time context. For only a 1 second input, there are only a few time-steps for the recurrent network to learn temporal features from, hence the insignificant change in performance over the CNN. An encouraging finding was that the models perform well for rare instruments also.

We also propose a method for visualizing confusions in a multi-label context, shown in the figure above. We visualize the distribution of false negatives for all instruments conditioned on a false positive of a particular instrument. For example, the first row in the matrix represents the distribution of false negatives of all instruments conditioned on the acoustic guitar false positives. We observe several cases of confusions that make sense musically, for example: different guitars, tabla and drums, synth and distorted guitars being confused.

For more details on the various processing steps, detailed results and discussion, please check out the paper here! Additionally, a 3 and a half minute lightning talk given at the ISMIR conference is accessible here.