This artist statement grew out of three characteristics I’ve noticed in AI artworks I find compelling:

1. Unequivocally AI. AI is a new generative medium. Build new things that feel AI.

2. Artistic intent. Make deliberate choices, not low-effort content.

3. Culturally rooted. Connect with your audience.

Subscribe now

Unequivocally AI

Generative AI enables experiences and aesthetics that were previously not possible.

The role of the artist is to rigorously push the boundaries of what “unequivocally AI” means.

Artistic intent

AI media is often criticized as low-effort and for threatening artists’ livelihoods. The narrative around AI art must evolve:

• Low-effort content → AI as a medium for storytelling

• Replacing artists → AI in service of artistic agency

Culturally rooted

For AI art to have cultural impact, we must build compelling narratives and make work people want to experience, share, and talk about.

Otherwise, our feeds will be flooded with low-effort AI content while artists struggle to build sustainable careers around (AI) art.

My practice

I’m personally interested in exploring better interactive (active listening) music experiences as the music industry converges on AI remixing.

AI remixing is a poor interactive experience that is fully functional (covers in other genres) instead of artistic (unexpected, novel interactions with music).

I explore interactive games that turn active listening into a familiar experience, relying on beats inspired by popular culture to meet audiences where they already are.

Subscribe now

Disclaimer. The views expressed are my own and do not reflect the opinions or positions of my employer.

All em dashes were typed manually.

In high school, I noticed that the transcription in the music textbook was four semitones off. That’s when I first thought I might have absolute pitch.

This also became apparent when I started playing the clarinet. Unlike the piano, the clarinet is a transposing instrument: the note written on the page is not the pitch that sounds. My ear immediately recognized the sounding pitch, even though it did not match the note I was reading.

For most instruments and situations, I can quickly name notes without a reference tone.

Yet, I struggle with:

• Black-key notes (accidentals)

• Non-Western / non-classical instruments

• Chords and full mixes

• Long motifs

So why does this happen to me and to many others?

In this post, I review the science behind my “absolute pitch impostor syndrome”:

• Absolute pitch might be a spectrum

• The role of classical piano training

• Semitone errors are common

• Perceptual biases shape pitch perception

What is absolute pitch?

Absolute pitch (also known as perfect pitch) is the rare ability to identify or produce musical notes without a reference.

For example, if I hear a 440 Hz tone, I can immediately recognize it as an A.

The absolute pitch spectrum

In psychological science, human abilities are often simplified into binary categories: you either have an ability or you don’t.

Early work by Bachem (1937) challenged this view by proposing four levels of absolute pitch:

• Genuine absolute pitch: Fast, certain, and accurate, even with chords.

• “Pseudo” absolute pitch: Slower, uncertain, and approximate.

• “Quasi” absolute pitch: Approximate pitch estimation with a reference tone.

• No absolute pitch.

More recent research by Van Hedger et al. (2020) suggests that genuine and “pseudo” absolute pitch are not separate abilities, but different points along the same continuum.

Piano training shaped my perception

The piano is an instrument through which many learn music, and its Western classical repertoire for beginners is mostly in C major, G major, D major, or F major.

As a result, many musicians develop stronger pitch associations for white-key notes.

Several studies have found that people with absolute pitch identify white-key notes not only more accurately, but also more quickly than black-key notes.

The same learning bias appears with timbre.

Many people with absolute pitch develop it through piano training. As a result, pitch labels can become partly associated with the piano’s timbre, a phenomenon sometimes called “absolute piano”. Previous research showed that some people who could reliably identify piano notes performed much worse with pure sine tones (which lack the harmonic structure of piano tones).

This explains why I am much more comfortable identifying notes played on a piano than on percussive/electronic sounds, or non-Western instruments.

Semitone errors

Absolute pitch errors are often off by a semitone.

Semitone errors occur most often on black-key notes, which is consistent with the idea that early piano training emphasizes white-key notes.

Further, semitone errors also become more common with age.

One of the earliest reports comes from Vernon, who documented his own absolute pitch over more than 50 years. Around the age of 52, he began hearing every note about a semitone higher. By age 71, the shift had reached almost a whole tone.

Larger studies have since confirmed the trend: pitch-labeling errors become more common with age, and consistent semitone shifts are not unusual among older adults.

Perceptual biases and pitch perception

I often struggle to name every note in a fast solo. But once it ends, I can usually identify the final note. This is consistent with the recency effect, a well-known phenomenon in cognitive psychology.

A similar effect appears in chords and recordings with many instruments.

I also find that the highest note is often easier to identify than the notes below it. Research shows that listeners naturally pay more attention to the highest voice in polyphonic music. This is known as the high-voice superiority effect.

So... do I have absolute pitch?

I’ve never been fully confident about my absolute pitch abilities.

While I can quickly identify most notes without a reference, I struggle with black-key notes, unfamiliar timbres, dense mixes, and long musical phrases.

I assumed those limitations meant I didn’t really have absolute pitch. But the scientific literature suggests a different explanation: I might simply be somewhere on the absolute pitch spectrum.

Further, years of classical piano training shaped my ear around white-key notes and familiar timbres. This also explains why piano notes feel effortless, while non-Western / non-classical instruments or black-key notes feel harder.

Previous research also helps explain why I am sometimes a semitone off, and why those errors have become more common as I’ve gotten older.

For years, I thought I was an impostor. But now I understand that my challenges are well-documented in the scientific literature and shared by many others.

Subscribe now

References

• Various Types of Absolute Pitch by Bachem in JASA (1937)

• Tone height and tone chroma as two different pitch qualities by Bachem in Acta Psychologica (1950)

• Absolute pitch: A case study by Vernon in British Journal of Psychology (1977)

• Practically perfect pitch by Lockhead & Byrd in Journal of the Acoustical Society of America (1981)

• Recency and suffix effects in serial recall of musical stimuli by Greene & Samuel in Journal of Experimental Psychology (1986)

• Musical pitch identification by absolute pitch possessors by Miyazaki in Perception & Psychophysics (1988)

• Effects of Timbre and Pitch Region by Miyazaki in Music Perception (1989)

• Explaining the high voice superiority effect in polyphonic music: Evidence from cortical evoked potentials and peripheral auditory models by Trainor et al. in Hearing Research (2013)

• Music Proficiency and Quantification of Absolute Pitch: A Large-Scale Study among Brazilian Musicians by Leite et al. in Frontiers in Neuroscience (2016)

• Revisiting discrete versus continuous models of human behavior: The case of absolute pitch by Van Hedger et al. in PLoS One (2020)

Disclaimer. The views expressed are my own and do not reflect the opinions or positions of my employer.

After spending the last few months experimenting with agents for writing the Stable Audio 3 paper, I think scientific writing has never been bottlenecked by typing speed.

It's bottlenecked by our ability to communicate ideas clearly.

If you don't have anything to say, agents won't fix that. If you do, they can help you convey your ideas more effectively.

But if your goal is to use agents for writing papers faster, you’ll probably end up with a generic article that fails to communicate the nuances of your contribution.

Clarity in scientific writing means designing figures that tell a story, building tables that highlight the important numbers, finding related work that frames your contributions, fixing awkward sentences, or removing typos.

TL;DR. Let agents handle the parts you struggle with, so you can write better. Keep doing what you enjoy and are good at, or agents won't make you faster.

Agentic Loop

1. Start with a draft.

2. Have one agent review it as a top machine learning conference reviewer.

3. Have a second agent rewrite another draft based on the review, acting as a scientific writer.

4. Repeat the review–rewrite cycle until the reviewer is satisfied.

The reviewer evaluates each draft across dimensions that I define: clarity, logical flow, strength of the arguments, scientific rigor, citation quality, or length. This agentic loop typically requires less than 20 iterations, and I also save intermediate versions so I can inspect how the manuscript evolved over time.

This has been an interesting workflow to experiment with, but it also comes with important drawbacks that I discuss below.

Why better not faster?

Because you’ll still have to review everything. Every paragraph, every change, every citation, every equation, every figure, every table.

Guided agents reliably get 80% of the work done, but the final edits belong to me. The last 20% is what some call “unslopifying”.

I tried hard to automate that last 20%, and I’m not sure it’s worth the effort.

Despite all the time I spent setting up better agentic loops, they never quite captured the final layer of coherence, emphasis, and style that I wanted. Working with agents can be slow and expensive, because building agentic workflows can take more time and tokens than actually using them.

LaTeX Figures

During my PhD, I spent far too much time working on LaTeX figures.

I loved the aesthetics, and LaTeX ended my fear of printing a pixelated conference poster. Thanks to vector graphics, I knew LaTeX posters would come out perfectly. That’s why I once made an entire poster using LaTeX. It took me several days, and I loved the result. But never did it again.

I simply couldn’t justify spending that much time on something that would be displayed for just a few hours. So I switched to PowerPoint or other vector editors like everyone else. They weren’t as cool, but I could finish a poster in a morning instead of a few days.

Now LaTeX figures are viable for me again, because agents handle the tedious part. Yet, they don’t get it right in one shot. You have to iterate, give feedback, and refine the output. And after a few rounds and a bit of taste, figures speak for themselves.

I can use as many figures as I want to support the paper's storytelling. The only constraint left is the conference page limit…

LaTeX Tables

I’m a LaTeX table storyteller.

During my PhD I could easily spend an entire day polishing a table. Because tables matter, because good tables tell a story before anyone reads the paragraph.

It is painful to turn a spreadsheet into a clean, readable LaTeX table. Multi-row headers. Bold values. Missing entries. Spacing. Endless copy-paste. Today, agents can easily handle spreadsheets to generate LaTeX tables. After a few iterations, agents can generate a polished table without the tedious copy-paste or formatting work.

Now I spend my time thinking about whether tables communicate the right story instead of counting ampersands.

References

One of my biggest fears is missing someone else’s prior work.

Agents are surprisingly good at surfacing papers I might have overlooked, especially from adjacent fields that I don’t naturally follow. But they are not a shortcut to avoid reading those papers. If anything, they create more work because they're often filled with unfamiliar terminology that takes time to digest.

Agents also handle the tedious last-minute work: cleaning references, unifying citation styles, and catching typos in author names. But references are also a hallucination minefield. So every citation or edit an agent suggests needs to be verified, and every referenced paper still needs to be read.

Typos

Whenever I revisit one of my papers, I always find a typo. Always. In fact, yesterday we found one in the abstract of Stable Audio 3 (published only a few weeks ago).

Remember that agents are good and fast at catching typos. Don’t make the same mistake we did, and run one final proofreading pass with an agent before you submit.

What Agents Don’t Do

Agents won't figure out your scientific contribution. They won't build your narrative or give you your voice. You are basically limited by your own creativity.

If you have nothing to say, no agent will find it for you.

Papers written entirely by agents read fine on the surface but fail to emphasize the nuances that actually matter, leaving you with a generic, forgettable paper.

Let agents handle the parts you struggle with so you can write better. Keep doing what you enjoy and are good at, or you won’t be faster.

In my case, I enjoy storytelling and writing, but I struggle with building LaTeX tables and figures, cleaning references, and hunting for typos. That’s where agents help the most. The moment I try to automate storytelling and writing, I end up spending more time reviewing than writing.

That’s why agents are for writing better, not for writing faster.

Subscribe now

Disclaimer. The views expressed are my own and do not reflect the opinions or positions of my employer.

It comes from people abusing tools in creative ways.

Here are fourteen ways the community is misusing Stable Audio 3.

1. Runs on MacBook Pro

2. Loop variations

3. Use SFX models to make music

4. Drawing sounds

5. Mine the glitches

6. Become a prompt jockey

7. Generate covers

8. Turn recordings into experimental samples

9. Your voice as an interface

10. Combine ComfyUI tools

11. As a guitar effect

12. Mood vectors as controls

13. Runs on Arduino

14. Train a LoRA

Subscribe now

1. Runs on MacBrook Pro

Install Stable Audio 3 on your MacBook Pro with this one-liner.

2. Loop variations

3. Generate SFXs to make music

The most interesting drum and percussive sounds I’ve got from Stable Audio didn’t come from music models. They came from models trained on sound effects.

4. Drawing sounds

5. Mine the glitches

6. Become a prompt jockey

7. Generate covers with the “silence trick”

• Pad your reference audio with two seconds of silence at the end.

• Use inpaitnting to continue this padded reference audio.

• The model is forced to start clean (from silence) but still sees the reference.

• Combine with a prompt and you get a cover driven by the reference and the prompt simultaneously.

8. Turn recordings into experimental samples

9. Your voice as an interface

10. Combine ComfyUI tools

11. As a guitar effect

Use the audio-to-audio feature to transform the sound of your guitar.

Sounds around 1m30s are done using this technique.

12. Mood vectors as controls

13. Runs on Arduino

14. Train a LoRA

The stable-audio-tools repo includes LoRA training. Take your own catalog (which you own and have the legal right to use) and overfit.

The result: a model that sounds unmistakably like you. Not a sample of you. A model of you. Sample distributions, not audio!

Subscribe now

Disclaimer. The views expressed are my own and do not reflect the opinions or positions of my employer.

Subscribe now

How it works?

The model takes three inputs: a text prompt (“a jazz piano solo”), the duration in seconds, and optional editing controls. With that, it generates stereo audio at 44.1 kHz.

Three model sizes ship: small (459M parameters, up to 2 minutes), medium (1.4B, up to 6m 20s), and large (2.7B, also 6m 20s). The first two are open weights.

Efficiency unlock: variable-length

Here’s the way most diffusion audio models work: pick a maximum length (say 120s), and every single generation runs at that length. Even if the user only wants 5 seconds. The remaining 115 seconds get padded with silence. You pay full compute cost for a short clip.

Stable Audio 3 addresses this challenge. Ask for 9 seconds, the model generates 9 seconds of compute. Ask for 100, you get 100. Compute scales with what the user actually needs.

It’s not just a generator, it’s an editor

Most creators don’t need one-shot generators. They want to iterate. For this reason, Stable Audio 3 supports audio editing via inpainting.

You hand it an audio clip plus a mask saying “regenerate this part”, and the model fills in the masked region while keeping everything else intact. They support three modes:

• Single segment: re-roll one part of a track (a transition, a transient, one weird cymbal hit).

• Multi-segment: edit multiple sections at once.

• Continuation: feed in 5 seconds, let the model extend it.

One-shot generation is cool, but editing is what lets artists finish a piece of work.

Three-stage training

1. Flow matching pre-training (the base model). It learns to turn noise into audio over many small steps. Problem: it’s slow at inference, as it requires 50 to 100 steps per generation.

2. Distillation warmup. Train the model to do one step generation, instead. Faster, but outputs are not (yet) good.

3. Adversarial post-training. Bring in a second model (a discriminator) that rates how realistic the audio sounds, and fine-tune the generator to fool it. This recovers the sharpness that distillation smoothed away, without going back to multi-step inference.

The result: te resulting adversarially post-trained model can generate audio in a single pass. Yet, one step generations remain challenging.

Stable Audio 3 uses 8 inference steps while producing better outputs than the 50-step base model.

How 8 inference steps?

Even after distillation and adversarial post-training, going from pure noise to clean audio in one step is challenging.

So, at inference, ping-pong sampling is used: denoise, add a little noise back, denoise again, add less noise, denoise, done. Eight iterations total.

Runs on your laptop

• small runs on a MacBook Pro M4. Takes 5.92 seconds to generate 120 seconds of audio. With CoreML acceleration, that drops to 3 seconds.

• medium needs about 6.5 GB of VRAM at 120s. It fits on an RTX 4060, RTX 3060, or RTX 4070. It takes a few seconds to generate 6 minutes of audio.

If you’re building creative tools, you can now experiment with on-device AI audio features. With no API costs, no network latency, and built-in privacy.

If you’re a musician or sound designer, try inpainting. Iterate, edit, and regenerate parts to stay in control and execute your vision.

In the technical report and model repository, we provide further details:

• Stable Audio 3: arXiv (technical report) and GitHub (inference code and weights).

Subscribe now

Disclaimer. The views expressed are my own and do not reflect the opinions or positions of my employer.

Through Counterpoint, the studio he runs with Samuel Diggins, he has collaborated with brands like Roland and Moog, and with AI music pioneers like YACHT, Holly Herndon, and Mat Dryhurst. This work has appeared at Ars Electronica, CTM Festival, and SFMOMA. Yet, his name remains quietly known.

Subscribe now

His online installations let you experience generative music concepts by recreating classic tools (tapes, loops, or delays) using JavaScript. Beyond the technical execution, a refined sense of design runs through everything he creates. His visuals are beautiful and inspiring.

From recreating iconic works by Reich, Riley, and Eno to reviving Laurie Spiegel’s Music Mouse online, his projects are a crash course in the history of generative music. These web-based sketches, as he calls them, are educational and fun. They allow you to explore complex concepts in a hands-on, interactive way.

After following his work for many years, I finally have the pleasure of interviewing Tero Parviainen.

Q: Most of your work lives online as interactive websites rather than as static tracks on streaming platforms. Could you share how your background led you to this intersection of software and sound, and how you define your artistic practice today?

I have a software engineering background. I spent the early stages of my professional life as a generalist software consultant, before about a decade ago I decided I needed a switch. I felt a strong pull towards music and the arts more generally, so that’s where I went. It wasn’t really a career change as much as just a decisive concentration of all my focus and energy on this one thing. To apply the software skills I’d picked up to an area of personal interest.

Because of this path, the center of gravity for my creative identity is as a software maker, more so than a musician or an artist. Software is my primary craft. Projects tend to start as ideas of interactions, systems, procedures, and protocols, and the music then springs from that initial conception. The natural end result of this kind of process is rarely a static track, and usually something adaptive, reactive, or just open and unbounded by time. A lot of the time it’s a web app, as I find the web to be the optimal medium for casual interactive music experiences. It’s open, standardised, cross-platform, free, and with very capable primitives for both visuals and audio.

Q: Could you walk us through one specific project? From the initial idea to the technical and musical decisions that shaped the final result?

The Digital Electronium was our first collaboration with Yuri Suzuki, back in 2019. He had an idea about building a new version of The Electronium, a mythical generative music machine from the 1960s, conceived and made by Raymond Scott. This was a machine Scott worked on for decades, including for some time at Motown Records as their head of electronic music.

Yuri had secured permission from the Raymond Scott estate, and received a pile of Scott’s sketches and documents from his friend Mark Mothersbaugh, who also owns the only (semi-)functional original Electronium machine today. We proceeded to study all that material, to learn how the original machine might have worked.

Yuri didn’t want the digital version to be an exact copy, but to make use of modern day affordances where it made sense and honoured the spirit of the original machine. What he did not want to do, however, is strip back and simplify. A big part of the Electronium’s aesthetic as a piece of kit is that it looks like an airplane cockpit. It has hundreds of knobs, switches, and buttons arranged in multiple panels. We tried to reproduce this rather faithfully as a UI, spread over three large interconnected touchscreen displays. This was the main design constraint on the audio backend: all these hundreds of controls had to do something meaningful, as this was designed to be poked at and enjoyed by casual users - visitors to an exhibition in The Barbican where it was to be presented.

The audio system powering that UI runs in Ableton Live. Within it we have a bunch of Max patches, as Max for Live devices. They communicate with the touchscreens over OSC, doing control flow and generative patterning. Audio DSP is synths and effects in Ableton - some stock, some plugins.

So most of the generative stuff in there is what you might call good old-fashioned generative: algorithms in Max. But there’s also a bit of generative AI in there too. Scott’s machine had a “counterpoint” section, whose purpose was to generate automatic contrapuntal accompaniment to whatever patterns you had punched in. Our version of this was powered by Coconet, an open model by our friends at Google Magenta, which is a convnet trained to inpaint four-part harmony. So whenever you engage the counterpoint section, we run a Coconet inference to generate a unique harmonisation.

Q: Your work also reflects a strong interest in wellbeing music and the history of generative music.

Yes, a good chunk of the work we’ve done at Counterpoint has been around making music systems in the service of wellbeing: psychedelic therapy, sleep, spa treatments, exercise. This is a natural fit for adaptive music systems, because the music is there to facilitate potentially unpredictable situations and may need to readjust in the moment. Software can do that. Working in these contexts also often feels prosocial, because you’re getting to contribute to embodied practices that help people regulate their nervous systems in an increasingly mad world. It’s music as medicine.

The interest in generative music history has been about self-education, first and foremost. When I made the decision to concentrate in music ten years ago, my strategy was to try and learn from the masters. I wanted to try and figure out what they had done, and how, and why. Studying the work of people like Brian Eno, Laurie Spiegel, and Terry Riley helped me understand how to think about music as systems, which gave me an angle that made sense to me. We’ve continued this in our projects with Yuri Suzuki, which has given us opportunities to explore Raymond Scott’s Electronium, as well as classic music tech by Roland and Moog.

Q: Finally, like many other OG musicians in our community, you also spent time working in the NFT space. How do you reflect on that period now?

Yeah, that was an interesting time. It’s kind of receded into the distance with the whole pandemic fever dream. But for a while there, there was a lot of generative art happening on blockchains. A lot of artists were able to make a living out of it outside of institutional support. A few got very rich, but many more were just able to support themselves. That was a bit of an anomaly. We’ve done a few pieces too, with people like Boreta, Aaron Penne, Bright Moments, and Amon Tobin.

The scene is still going strong, but now it’s become more of a true believer’s affair. There’s a lot less money going around, and the upside of that is a lot of the scamminess and speculation has washed away too. I remain cautiously bullish on crypto and blockchains in general. There’s a lot of deep thinking on decentralisation and distributed trust happening in that world, and that seems like something we should be thinking about in a world where it’s increasingly clear there are no adults in charge.

Q: NFTs are an interesting concept when tied to generative digital art, as they represent a unique, fixed snapshot of a system that is otherwise dynamic and changing.

It seems like an appropriate medium for code-based art, for sure. In communities like Art Blocks the model is that as an artist, you make an algorithm that’s capable of producing many iterations of an artwork. Sometimes an infinite amount of them. Then for everyone who buys one, the algorithm is run with a different random seed, so everyone gets to own their unique piece that no one has seen before. Tyler Hobbs coined the phrase “long-form generative art“ for this kind of thing.

The Rituals collection we worked on for Aaron Penne and Boreta is an example of this. It’s a slow, meditative, audiovisual piece with 1000 versions owned by different people. When it was launched, there was a special event in LA where you could get one and it was revealed to you as you’re sitting alone in a room with a big screen and sound system. But beyond that, it lives forever as code on the Ethereum blockchain and runs in a web browser.

Subscribe now

Disclaimer. The views expressed in this content are those of Tero Parviainen and do not reflect my own opinions or those of my employer. Only the introduction was written by me.

But recently proposed models require supervised TRAINING or FINETUNING with data that is challenging to collect.

Given the high costs of training or finetuning generative models, INFERENCE-TIME control methods have been introduced.

We explore inference-time controls through a “guidance-based” approach that relies on “selective TFG” and “Latent-Control Heads (LatCHs)”.

• Guidance? Use the gradient of a target distance function with respect to the diffusion process to guide sampling. For example, use RMS to guide sampling.

• Selective TFG? Only apply TFG to a few early steps. TFG is a type of guidance that unifies most guidance-based frameworks, see below.

• Latent-Control Heads (LatCHs)? Audio-based guidance is costly because it requires backpropagation through audio decoders during sampling. We instead train latent-control heads to guide diffusion through latent features directly.

TL;DR: LatCHs guide the diffusion process directly in latent space, avoiding the expensive decoder step, and requiring minimal training resources (7M parameters and ≈ 4 hours of training). Experiments with Stable Audio Open demonstrate effective control over intensity, pitch, and beats (and a combination of those) while maintaining generation quality. We are the first to use latent-control heads for guidance in few, selected diffusion steps.

Subscribe now

Latent-Control Heads (LatCHs)

Guidance-based methods rely on a pretrained latent diffusion model and introduce an external control signal (e.g., RMS) that the model was not explicitly trained to follow.

To enforce this control, such methods compute the gradient of a target signal (e.g., RMS) with respect to the diffusion process and use it to guide sampling.

End-to-end guidance can be slow and VRAM intensive as it requires backpropagating through the VAE decoder.

LatCHs, however, are compute-efficient because they directly predict control features from the latent space. There is no need to decode latents into audio.

TFG

Our work uses TFG as the guidance method.

TFG unifies most guidance-based frameworks (DPS, MPGD, LGD, UGD) within a shared hyperparameter space:

• Guidance strength (ρ and µ).

• Guidance accuracy strength (γ, controlled by adding noise).

• Niter and Nrecur, which define how many times guidance is applied.

If ρ or µ are too high, the sampling process may drift off-manifold. Keeping ρ and µ small preserves output quality while still following the control signal.

We found that increasing γ was crucial in some cases. When guidance follows the control too aggressively, audio quality may degrade (off-manifold drift). While reducing ρ or µ could weaken control adherence, increasing γ provides an alternative way to relax guidance precision while preserving both audio quality and control.

TFG is compatible with CFG scaling, which we set to 7.

Note: TFG stands for Training-Free Guidance. While TFG itself is training-free, we additionally train Latent-Control Heads.

Selective TFG

One key factor omitted from TFG’s hyperparameter space is which diffusion steps to apply guidance.

Since a control signal may perceptually arise at specific points during sampling, limiting guidance to those steps reduces the risk of drifting off-manifold.

By applying TFG guidance only on a few selected steps, we reduce computational overhead while balancing control accuracy and audio quality.

• Applying guidance to more steps improves control alignment but increases the risk of drifting off-manifold.

• Applying guidance to fewer steps reduces drift risk but may weaken control precision.

We call this selective TFG, where guidance is applied only to a subset of steps. In our experiments, we use guidance on just 20% of the early sampling steps.

Results

• We explored intensity (RMS), beat-tracking, pitch (CREPE), tags (PANNs) and harmony (chroma) as control features.

• LatCHs achieve competitive performance in audio quality, prompt adherence, control alignment, and efficiency.

• End-to-end also performs well, but at a high compute cost.

• MOS results are generally good for beats and intensity, and those quality metrics are comparable to SAO.

• Multiple guidance controls also works (e.g., beats+intensity).

• The studied methods are more reliable with gradual or low-frequency controls, such as intensity or beats.

• Controls with greater variability, such as pitch, which involves rapid note changes, pose challenges, and show worse performance across quality metrics.

Note that intensity and beats consist of 1D outputs and pitch outputs 160 pitch classes. We also explored PANNs (527 tags) and Chroma (12 pitches) feature extractors with mixed results. Based on those observations, we hypothesize that 1D outputs, rather than sparse high-dimensional ones, could be more suited for guidance.

Subscribe now

1. DIFFUSION MODEL: supports multiple tasks such as generation, remixing, and editing.

2. LANGUAGE MODEL: supports reprompting and semantic tokens generation.

3. DATA PREPARATION: detailed annotation of 27M songs.

4. OPEN WEIGHTS: a range of models is available, with support for LoRA fine-tunes.

   Subscribe now

How does it work?

First, ACE-Step 1.5 relies on the LANGUAGE MODEL to generate structured metadata (e.g., genre, instruments, mood, structure) from unstructured user prompts.

In a second pass, the same LANGUAGE MODEL generates semantic tokens conditioned on the above described metadata.

The authors refer to these two stages as “chain-of-thought” or “planning” or “structuring the music” before the audio is generated.

Important: The LANGUAGE MODEL generates both text tokens (first pass) and semantic tokens (second pass).

Finally, the (LATENT) DIFFUSION MODEL generates 48 kHz stereo audio.

Depending on its configuration, the DIFFUSION MODEL can perform text-to-music generation, cover generation, continuations, or instrument addition/removal.

Diffusion Model

The generative model is a LATENT diffusion model with these components:

• CONVOLUTIONAL VAE: compresses 48kHz stereo audio into a compact 64-dimensional latent at 25Hz, trained with an adversarial objective.

• DIFFUSION TRANSFORMER: with odd layers using sliding window attention to capture local features, while even layers use global group query attention to model long-term structure.

It is conditioned on:

• Caption embeddings, produced by a frozen Qwen3-0.6B model.

• Timbre and lyric encoders, trained jointly with the system.

ACE-Step 1.5 can handle multiple tasks simultaneously, including: text-to-music, cover generation, repainting, vocal-to-background separation, track extraction, or layer addition. To that end, the model processes multiple inputs:

• SOURCE: latent of the audio to generate/edit.

• NOISED TARGET: required input for diffusion.

• MASK: defines structure, required for repainting.

Training proceeds as follows:

1. Pre-training with 20M text-to-music pairs.

2. Multi-task fine-tuning with best 10M songs and 6M stem-separated tracks.

3. High quality fine-tuning with a subset of 2M songs.

4. Few-step distillation, for fast inference (from 50 diffusion steps to 8).

5. “Intrinsic” reinforcement learning (DiffusionNFT) that does not rely on human preferences.

Few-step distillation is based on distribution matching distillation (DMD2) plus adversarial training.

The “intrinsic” reinforcement learning algorithm is interesting as it does not require training an additional reward model nor human preferences.

Instead, it uses the consensus across multiple cross-attention heads (between lyrics tokens and latent-audio frames) as a reward signal for lyric-audio synchronization:

• High consensus → sharp, coordinated attention peaks across heads → high alignment scores.

• Low consensus → blurry, scattered attention patterns → low alignment scores.

Traditional RL for diffusion models requires traversing the full denoising path to compute a reward. In contrast, DiffusionNFT leverages intermediate steps, eliminating the need to traverse the entire path.

Language Model

Remember that prompt interpretation is handled by the language model, while the diffusion model focuses on audio generation.

From unstructured user prompts, the language model generates structured YAML metadata (e.g., BPM, key, duration, and song structure).

By offloading prompt understanding to the language model, the system ensures the DiT receives clear, unambiguous conditioning, maximizing its generative potential.

In addition, the language model generates “semantic tokens”. The authors refer to these as “audio codes”, but given their very low temporal resolution (5 Hz), I refer to them as “semantic tokens” to remain consistent with previous works.

Semantic tokens are obtained by downsampling latent representations using an attention pooler, followed by quantization with finite scalar quantization (FSQ).

Training proceeds as follows:

• Train the audio tokenizer to get semantic tokens (“audio codes”).

• Pre-training from Qwen3 models.

• Supervised fine-tuning.

• “Intrinsic” reinforcement learning (GRPO) that does not rely on human preferences.

Note that the original language model operates only on text tokens. Through pre-training, supervised fine-tuning, and reinforcement learning, the language model learns to handle semantic tokens (5 Hz “audio codes”). For example:

• Text-to-tokens: given text captions, it generates semantic tokens.

• Tokens-to-text: given semantic tokens, it predicts the corresponding captions.

During RL, this bidirectional capability (text-to-tokens and tokens-to-text) allows designing a self-consistency reward without requiring an external reward model.

The reward is based on pointwise mutual information (PMI), which penalizes generic descriptions and rewards captions that are specific to the given semantic tokens.

Generative Reward Policy Optimization (GRPO) generates multiple outputs for the same prompt, ranks them by reward (PMI scores), and updates the policy using relative comparisons within the group.

Samples better than the group average get reinforced and worse ones get suppressed, eliminating the need for a separate critic network.

Data Preparation

1. Annotate 5M audios with Gemini 2.5.

2. Fine-tune Qwen2.5-Omni to create ACE-Captioner.

3. Fine-tune Qwen2.5-Omni to create ACE-Transcriber. This is used to transcribe lyrics.

4. Reward model used to further train ACE-Captioner and ACE-Transcriber with reinforcement learning.

5. Use ACE-Captioner and ACE-Transcribe to caption and transcribe the lyrics of 27M songs.

6. Reward model to discart low-alignement pairings.

Open Weights

Several different models have been made available.

The distilled turbo models are fast, generating a full song in just 2 seconds on an A100 and under 10 seconds on an RTX 3090.

They can also run locally on machines with less than 4 GB of VRAM.

ACE-Step 1.5 also supports LoRA fine-tuning, allowing users to easily personalize the model for their own audio or style preferences.

Subscribe now

Disclaimer. The paper provides limited implementation details. To better understand the underlying methodology, I studied both the paper and the released codebase. Yet, the reinforcement learning components, parts of the training pipeline, and the training data are not publicly available. I welcome feedback or corrections if others understand parts of ACE-Step 1.5 differently. The views expressed are do not reflect the opinions or positions of my employer.

It consists of 4 components:
1. HeartCodec: low-rate audio tokens.
2. HeartMuLa: autoregressive generative model.
3. HeartTranscriptor: lyrics tokens.
4. HeartCLAP: text and reference-audio tokens.

Subscribe now

1. HeartCodec

HeartCodec compresses audio (48 kHz) into tokens at a low rate (12.5 Hz).

It operates at a low rate, which is crucial for generating long music sequences. A lower rate means fewer tokens need to be generated, making the process more efficient.

Let’s go through the HeartCodec diagram, following the colored blocks.

Yellow. It leverages semantic representations across multiple abstraction levels:
- MuEncoder → musical (layer 11) and acoustic (layer 2) embeddings.
- WavLM and Whisper → two different phonetic embeddings.

Purple. The downsampling schema is based on insert/extract query. After every two consecutive frames, a learnable query token is inserted. Then, the two consecutive frames are extracted (removed). This reduces the frame rate by two and produces a downsampled representation which is then discretized using residual vector quantization (RVQ).

Blue. Each “Layer” (from 0 to K-1) represents one of the K RVQ codebooks.

Green. A two-step decoder is used to turn RVQ tokens into waveforms:
- Flow matching generating SQ-Codec continuous latents.
- SQ-Codec decoder generating waveforms.

HeartCodec is trained in 3 stages:
- Pre-training and finetuning with 600k songs.
- Reflow Distillation with 50k high-quality songs.
- Finetuning the SQ-Codec decoder.

2. HeartMuLa

HeartMuLa is an autoregressive generative model, and it’s hierarchical.

It operates on discrete audio tokens produced by HeartCodec and is conditioned on:
- Tags.
- Lyrics.
- Reference audio.

RVQ consists of K codebooks, and the K=0 one captures the coarse “semantic” information. The rest of residual layers, provide acoustic details. The hierarchical structure of the model exploits this fact, to predict first all K=0 tokens (first global step) and then predict the rest of the tokens (second local step).

Training involves 4 stages:
- Warmup with 30 sec segments.
- Pretraining with full songs.
- Supervised Finetuning with high-quality subset.
- Direct Preference Optimization (DPO).

DPO reformulates the RL objective as supervised learning, optimizing the policy directly from preference pairs without needing a reward model.

3. HeartTranscriptor

HeartTranscriptor is a lyric recognition model optimized for musical audio. It is a fine-tuned version of Whisper trained on the high-quality dataset of music recordings.

4. HeartCLAP

HeartCLAP consists of two main components: a text encoder and a music encoder. Both are initialized with pre-trained weights from MuQ-MuLan and are used to extract text and reference-audio tokens.

Dataset

It consists of three parts: music with lyrics, instrumental music, and TTS datasets, totaling around 600k songs and 50k segments (about 100k hours) of high-quality audio.

HeartTranscriptor was used to align lyrics with the music.

SongFormer was used to annotate the music’s structure and sections, with prompts defined at the section level.

Tags were generated using Qwen2.5-Omni.

Results

The abstract claims to “reproduce a Suno-level, commercial-grade system using academic-scale data and GPU resources”.

And the results show it is pretty close, congrats to the team!

Subscribe now

Disclaimer. The views expressed are my own and do not reflect the opinions or positions of my employer.

I’ve been conducting research in deep learning and music for many years now, and recently I’ve begun to question our work’s impact on music.

Have we contributed with ‘slop’ machines? I hope not. Yet, I cannot ignore the disconnect between my enthusiasm and the public’s perception:

• I’m excited about AI’s artistic potential.

• The world perceives AI output as ‘slop’.

For those who aren’t familiar, ‘slop’ is a popular term used to describe AI-generated art as disposable or low-value content.

Feeling both worried and uncertain about this, I stepped back last year to explore:

• The current state of AI music and emerging artistic trends.

• The broader history of algorithmic and generative music, long before AI.

To realize that the most interesting AI music outlets fall into the realm of Interactive AI Music:

• Generative albums, such as Kunst Kaputt (SENAIDA), which listeners can actively interact with.

• Live performances, like Prompt Jokeys (Dadabots) or MusicFX DJ (Magenta), where the audience or performers generate music live with AI.

• Interactive radios, exemplified by Suno’s or Sonauto’s AI radios.

• New streaming services, as seen in Udio’s recent move toward an interactive, AI-based streaming model.

• Interactive installations, such as Space DJ (Magenta) or TECHNE: The Vivid Unknown (John Fitzgerald), where music (and visuals) respond to audience input.

• Or even new bands! Like BRNRT Collective, our Interactive AI Music band :)

What do these projects have in common? They use AI to create new, interactive musical experiences that were previously not possible.

In this post I define Interactive AI Music, a term that brings together my favorite artistic projects in AI and music.

Definition

Interactive AI Music involves three important characteristics:

• Interactivity: it has the ability to change in response to user input, allowing audiences or performers to influence music in real-time.

• Artistic agency: musicians compose both the music and the interaction, ensuring that the resulting AI experience remains artistically meaningful.

• Generative AI: instead of playing pre-recorded tracks, AI is used to create new melodies, rhythms, harmonies, or sounds on-the-fly.

Artistic agency

AI music is often criticized for being soulless and threatening to musicians’ livelihoods. But AI music does not need to be like that.

I strongly believe that it can be used to create art that was previously not possible, and that it will be equally appealing as previous art forms.

However, the narrative and tools have to evolve:

• soulless → AI to tell stories

• replace musicians → AI in service of artistic agency

For this reason, storytelling and artistic agency lie at the heart of Interactive AI Music. As such, artists do not only compose music—but also compose the interaction.

Interactive AI Music is composed with interactivity in mind.

The result? An immersive, interactive experience fully directed and composed by the artist to tell a story with the help of AI.

Generative

Generative music is also known as indeterminate music, chance music, stochastic music, or aleatoric music. In essence, random processes shape generative music.

Accordingly, no two interactive experiences are the same—as generative models allow each interaction to unfold uniquely.

In Interactive AI Music, generativity and interactivity are closely intertwined. Modern generative AI systems unlock forms of interaction that were previously not possible.

Yet, generative music is not new and interactive systems can exist without AI.

Generative modular setups is one example.

Distribution formats

Legacy distribution formats feel obsolete as new playback ideas emerge.

• MP3s made digital audio easy to store and share in the early days of the internet.

• Dolby Atmos facilitates spatial audio that works across different speaker layouts.

AI can be seen as a new format where instead of distributing a fixed audio file, artists distribute generative AI models or applications that make music on-the-fly.

With this new paradigm, a musical piece now becomes a generative system rather than a single recording.

Such new formats would allow music to adapt culturally and contextually, giving music a longer lifespan by making each piece more flexible and reusable:

• Share personalized remixes on social media.

• Start your day with acoustic versions.

• Energize workouts with EDM remixes.

Accordingly, the listener shifts from being a passive consumer to an active co-creator.

This opens the door to new super-fan experiences, aligning with the industry’s growing focus on deeper, more intimate relationships between artists and audiences.

With new media distribution formats, art consumers are becoming co-creators.

New artistic medium

What’s an artistic medium? It’s the thing artists use to make art. Like canvas for painters, clay for sculptors, video for directors, or recordings for musicians.

A new artistic medium → AI
One that’s alive → generative

Artists are releasing AI models or interactive applications or generative albums. For inviting fans to co-create with them.

Music designed for interaction—opening up new interactive, artistic possibilities.

Interacting with SENAIDA’s Kunst Kaputt generative album (video above) feels like playing a game. Hence, Interactive AI Music is new artistic medium where you don’t play music, you play with music.

Analogy

Think of movies and games. Movies are a passive experience, games are interactive. Music is typically like a movie, but Interactive AI Music is more like a game.

A director creates a movie designed to be consumed in a very specific way. Every shot, sound, and edit is crafted so you experience the story exactly as intended.

A game studio, on the other hand, creates a space for you to explore. The story is there, but it’s flexible. You make decisions and experience it in your own way.

Interactive AI Music works the same way.

It might not be the default way to experience music, just as games aren’t the default way to consume stories. But when you engage with it, it offers something movies and traditional albums can’t: exploration and a sense of co-creation.

“Music that you play with, not music that you play” — Mikey Shulman (CEO at Suno).

Not a new concept

Interactive, adaptive, generative, or algorithmic music is not new.

For example, musical improvisation has existed since the 9th century, with performers generatively shaping music in real-time interacting with other performers.

Adaptive music is also common in video games. Music can change in volume, tempo, or arrangement, often shifting the tone and intensity during events like combats.

One of the earliest examples is Space Invaders (1978), where a simple repeating four-note motif speeds up as the aliens descend.

In recent years, interactive music has evolved from early tape and computer experiments into sophisticated digital systems. Today, we are exploring AI for combining art and technology to reimagine how we experience music.

Limitations

Interactive AI Music faces two main challenges:

• New listening habits are required: from passive to active listeners.

• Both interactive formats and distribution remain undefined.

Music consumption has historically been passive. Most listeners want to relax when they listen to music, just as they do when watching a movie or reading a book.

Interactive AI Music, which demands active participation, conflicts with how most people prefer to experience music.

At the same time, today’s music streaming platforms are highly consolidated and built around passive listening, and changing global listening habits will be difficult.

Interactive AI music may not become the default way people experience music, but it can coexist with traditional formats, like videogames do with movies and books.

Also, standardized formats and distribution have yet to be defined.

It’s unclear whether current streaming platforms will support interactive experiences, as Netflix has begun doing with video games. Or whether new platforms will emerge to disrupt the industry, similar to how Steam did in gaming.

For now, musicians rely on their own websites to share interactive online installations. This works for frontier artists, but limits access to wider audiences due to format and distribution gaps.

Possibilities

A musical medium is emerging through Interactive AI Music, enabling new ideas like generative albums and radios, installations, or interactive performances.

Its value lies in unlocking interactions that were previously unattainable, from new gamified experiences or installations to innovative live formats that are interactive.

Music can thus become a participatory and collective process rather than a fixed outcome.

Much like digitalization democratized music production (bedroom producers) or enabled unconventional live show formats (DJ sets), Interactive AI Music can offer a fresh perspective on how music is created, performed, and experienced.

This moment offers a unique opportunity to shape a nascent artistic scene. Musicians, startups, researchers, and corporations are now exploring how AI-driven interactive music can help expand audiences and inspire unexpected artistic directions.

A provocation

Finally, I want to leave you, the reader, with a provocation.

The future of AI music won’t shape itself. It’s up to us to imagine what it can become. If its current state feels like ‘slop,’ consider creating something different yourself.

Explore, experiment, and propose original ideas. While this essay has focused on Interactive AI Music, I’m sure there’s still much more to discover.

Subscribe now

Acknowlegment. Thanks to Zack Zukowsky (Dadabots) for the analogy comparing videogames and cinema.

Disclaimer. The views expressed are my own and do not reflect the opinions or positions of my employer.

We were both there for the AI Song Contest, and he sat down with us after a shy hello. It would have been easy to finish breakfast, exchange polite goodbyes, and go our separate ways. After all, it was early in the morning and our brains were still booting up.

Subscribe now

But after complaining about the cold weather and the mediocre buffet coffee, we began a conversation about our shared passion: AI music.

We discussed Pure Data, generative music, and online communities. He described sharing agency and controlling generative AI processes with a constellation of parameters. His unique vocabulary revealed that his background was different from mine. He is neither from academia nor industry, but from Berlin and the Internet.

His name is Martin Heinze (marts~), and I’m glad our paths crossed that morning.

A simple breakfast became an unexpected reminder that ordinary moments often bring the most interesting encounters.

Today, I’m sharing a (written) interview with Martin, starting with the breakfast we had when we first met.

Q: Do you remember what you ate the day we met? I remember having a delicious apple pie.

I had Croissant with jam, Pain au chocolate and then moved on to Musli but got so excited talking to you guys I forgot to finish up.

Q: For those who don’t know you, can you tell us about your artistic background?

I graduated at the University of Media Arts and Design (HfG) Karlsruhe which is closely entangled with ZKM, one of the key institutions for media art and technology in Germany. During that time I naturally came into contact with a lot of concepts, works, artistic practices and of course people. All of this had a lasting impression, even though at the time I found it quite difficult to transform it into something practical.

At the same time I started producing electronic music in the periphery of the Drum & Bass continuum, had releases on various music labels, played DJ gigs internationally and launched my own record label Weevil Neighbourhood.

Over the years I found it more and more difficult to enjoy writing music in my established routines. This was when earlier influences and maybe implicit knowledge from my time in Karlsruhe came into play. I began diving into concepts of generative music and algorithmic composition, changed my technical setup, and basically transitioned from production to development. This happened pretty much at the same time that early practical examples of generative AI in music (not only in the audio domain) started to surface, e.g., Dadabots’ Relentless Doppelganger YouTube livestream.

At that point I had written and produced various albums, EPs, singles and remixes in a traditional way that makes for a decent audio data set to train neural nets with - and now there were deep learning algorithms that would allow building novel kinds of instruments based on audio data for use in real-time settings. So it became one of my main practical research topics to create these instruments based on my own music and learn how to use them to spawn new audio items.

Q: In your work, you put a strong emphasis on ethics. Can you tell us more about your stands and how those translate to your artistic practice? I’m particularly interested in your actions to compensate the environmental footprint, as you are the first AI musician I’m hearing to talk about that.

I wouldn’t say that my artistic practice is primarily guided by ethics, but I generally feel inclined to reflect on privilege and responsibility. Privilege comes in various forms, sometimes it’s structurally determined, sometimes arbitrary; it can also be temporary. Acting responsibly considering your own privileges just seems reasonable for the long run, be it in regards to the creative work of other individuals when curating your training datasets or the environment we all live in.

The yearly environmental footprint of one person living in Germany is currently more than 10 tons of CO2 equivalents while Paris 1.5°C targets required less than 2 tons per person globally by 2030. I find that enough reason to aim at reducing that footprint no matter whether you have trained a neural net in your life or not. In the EU, the most effective regulatory instrument for additionally compensating for your environmental footprint at the moment is buying (and retiring) fractions of EU Allowances for CO2 emissions, which I chose to do.

Q: AI should be used to create art in ways that are both ethical and compelling. Ethics are important, but being compelling is just as important. What’s your compelling reason to use AI in your artistic practice?

Expanding creativity. Working with neural nets has radically changed my practice and perspective on music creation from writing pieces to “finding” them in or “spawning” them from the reproductive amalgamations of generative AI. This is primarily an artist centric view. But I think it is also noticeable in the results and might change listening expectations over time. For me it also leads away from a tradition of creating recorded artifacts more towards a spontaneous, participatory and ephemeral music creation and consumption practice. My optimistic intuition is that this might actually result in more humans prosuming music rather than machines flooding services with arbitrarily generated music for algorithms to consume.

Q: Could you share an example where AI expanded your creativity? Nao Tokui wrote that once he got goosebumps after his AI DJ picked two tracks that he couldn’t imagine working together. Did you find yourself in similar situations?

Absolutely. This happens to me each time I load a newly trained neural audio model for the first time after a successful training run. First, I usually just feed it some random noise and let it run for a while, listening, observing, learning about its character and the sound information it can reproduce.

That stimulation of creativity for me comes by finding a sweet spot in sharing agency when making music with neural audio models but not fully giving it up. I learned that when I embrace a certain lack of control I’m used to from deterministic production routines, I’m being rewarded with surprising results I wouldn’t have aimed for in the first place. And you could say that sharing agency already starts in the training process, where you as a human agent determine the training data and the general set up in which the training itself is being carried out. However, what exactly happens in the training process is out of your hands.

Q: You also release your tools open source. Can you tell us more about your technical setup?

I train models on cloud services with a set of Jupyter notebooks i’ve created.

For composition, I’m using Pure Data almost exclusively these days. I’ve developed a set of custom abstractions I use to build frameworks for semi-generative or algorithmic use cases. These are basic interfaces to the abstract and obscure latent space of the model types I’m employing (RAVE, vschaos2, AFTER, MSPrior). I inject information that mimicks latent embeddings and temporarily conserve parameter constellations that lead to musically interesting results - it’s a bit like tuning and eventually playing an instrument.

I maintain a YouTube channel where I showcase patches and frameworks that are built with these abstractions and how I engage into improvisation with the models. I call this “Latent Jamming” because it works a bit like a jam session with multiple agents inside latent space.

Q: Could you share some videos with us and walk us through them?

Certainly. On a baseline technical level, it all revolves around finding the right parameter constellations on signal level while operating in the latent space of the models.

In all three examples you can see similar approaches at play: create noise (or even more generally random arrays of values between -1 and 1), scale values and add an offset per latent dimension. This is mainly the tuning part: playing around with different value arrays, noise seeds, and offsets helps in understanding the model’s general sonic behaviour. Then, macro scale comes into focus by establishing repetition or rhythmicality; technically, this is done by retriggering noise seeds or value arrays or by changing offsets gradually or abruptly on certain triggers - think sequencing here.

The first example shows earlier experiments with this; these resulted in a compilation of recordings called “Spoor”.

I was particularly interested in different lengths of loops or rhythmical figures and how they would still be recognizable when applied in latent space.

In the second example I was already generalizing earlier experiments into reusable abstractions; also, I started working with multiple models in one setup, trying to turn the general sound aesthetics into something less abstract. Output documentation of this is “Saatgut Proxy” or “Latent Jams”.

The third video shows an example of my most recent setup. I’m using between 3 and 5 models in one framework now and added another control layer that works either in analogy to a randomized automation in a given range of parameter values or more sophisticated interaction with latent mimikry. This latest framework is the one I created my contribution to the AI Song Contest 2025 with.

Q: What are the main challenges that you face as an AI musician?

Finding simple words for what the AI part in my music is. Considering reservations against AI, especially in the creative domain, I tend to use other terminology like deep learning, neural audio synthesis etc. - that usually makes it worse because I need to go more into the technical details making it even less tangible.

Q: Interesting you mention language. I see, for example, that you use the word “spawning” introduced by Holly Herndon or introduce the concepts of “parameter constellations” or “latent-jamming”. Do you think those new concepts make our music less accessible to wider audiences? Maybe the term generative music, its classical definition, beyond AI, already defines our music.

I think for the time being these terms can help to differentiate seemingly novel aspects of music creation against conventional ones primarily to encourage dialogue and discourse. We might find out on the way that they’re not necessary because existing terminology and concepts are sufficient. There are indicators that this is the case when thinking of concepts of generative music, as you said. I use “spawning” for example because I like the implicit meaning of arbitrariness in how entities come into being as opposed to “creating”, “writing”, “producing”, “developing” or other terminology that implies creative intent. For me it also implies a closed system: spawning new audio entities structurally cannot go beyond what the models have seen during training, it requires outside-system interaction or system-bending in order to create something new - pretty much the same as with conventional instruments. In “Latent Jamming”, the improvisation itself is certainly not a new approach, but as of now I think that acting with intent inside an unstable complex system while embracing and enforcing its instability might qualify as new. Does all this terminology make it less accessible to wider audiences? Maybe. Can it help fostering discourse and dialogue with the outcome of a more differentiated view on “AI Music”? Same.

Subscribe now

Disclaimer. The views expressed in this content are those of Martin Heinze and do not reflect my own opinions or those of my employer. Only the introduction was written by me.

Subscribe now

It always felt limiting that general audio source separation models are source-specific.

For example, speech separation models are speaker-agnostic because you don’t want a different model for each speaker in the world. It won’t scale.

Why, then, does general audio source separation need to be source-specific? Why do we need a different model for drums, for guitars, for dogs, or for speech? Wouldn’t it be cool to address general audio source separation with a single model? That’s exactly what SAM Audio aims to do.

SAM Audio separates a target source (one source at a time) based on text, visual, or temporal prompts.

• Text: specify what to separate using a description.

• Visual: use a visual mask from video frames to indicate what to separate.

• Temporal: select an audio region to indicate when to separate.

These modalities can be used independently or together.

Hence, SAM Audio uses a single general model conditioned on text, visual, and temporal prompts, instead of relying on separate source-specific models.

SAM Audio is also generative.

Non-generative (mask prediction) approaches have historically been used for source separation—because they do not “hallucinate”, ensuring that the output strictly reflects the input. They are also typically faster, enabling real-time applications, though they can produce filtering artifacts.

Generative approaches, however, have not historically been used for source separation due to the lack of large training datasets and pre-trained base (generative) models. They are typically slower, which limits their applications, and they can “hallucinate” content.

SAM Audio also uses latent diffusion transformers (DiTs).

Three model variants are considered: small (500M parameters), base (1B), and large (3B). More specifically, SAM Audio trains a flow matching model on large-scale audio dataset covering speech, music, and general sounds.

The model outputs the target source and the residual audio, which are modeled jointly.

It is conditioned as follows:

• Mixture, visual and temporal prompts are first encoded into frame-aligned feature sequences, which are concatenated with the noisy latents before being passed into the DiT.

• Text prompt is encoded into a global textual embedding with cross-attention layers within the DiT.

Used encoder types:

• Visual: Perception Encoder.

• Text: T5-base.

• Autoencoder is DAC-like but replaces the quantizer (RVQ) with a variational autoencoder (VAE). At 25 Hz with latent size 128.

• Temporal span sequences encode whether the target event is silent or active at frame t.

Not all training examples include all three prompt modalities. To handle this, they apply dropout to the conditioning types during training, so each modality can be used independently at inference time.

SAM Audio uses a combination of losses:

• A generative loss: the standard flow matching loss.

• An alignment loss: that projects intermediate representations into the embedding space of an external audio event detection model.

But most of the work on SAM Audio went into the training data.

The structure of the collected data is (x_mix, x_target, x_residual, conditioning), where the conditioning includes (text, video, temporal span).

Training data comes from two sources:

1. A large-scale, medium-quality audio–video corpus (≈1M hours).

2. A collection of small- to medium-scale high-quality audio datasets (≈20k hours).

Where only a small subset includes ground-truth stems for music and speech mixtures.

The rest of mixtures are synthetic: noisy music (music+non-music), speech (speaker+speaker or speaker+noise), and general sounds (audio+audio).

After training with this data, they generate more training data by pseudo-labelling the large-scale audio-video corpus as follows:

1. Caption dataset to find text-prompts.

2. Separate mixtures containing multiple sound events with an undertrained SAM Audio version.

3. Remove low-quality candidates.

4. Prepare (text, visual, temporal) prompts.

Criteria to filter low-quality candidates:

• Low text-audio alignment using CLAP.

• Bad audio cleanliness as measured with the “audiobox-aesthetic” model.

• Overly silent outputs.

• Inisufficient audio-visual correspondence.

Prompts are prepared as follows:

• Text: obtained through automatic captioning.

• Visual: masks are obtained using SAM3, relying on the text prompt.

• Temporal: select audio regions based on RMS.

This results in ≈1.1k hours of additional training data (out of the original ≈1M hours).

The model is pre-trained on synthetic audio mixtures, while the remaining high-quality data and all pseudo-labeled data are used exclusively for fine-tuning.

Inference uses a 16-step ODE solver without classifier-free guidance. They run 8 different separations and select the best.

Subscribe now

Temporal Span Clarification

The temporal span prompt is a bit unusual.

It appears almost like a “cheat code” that tells the model exactly when to separate—rather than learning both what and when to separate. Providing the temporal span allows the model to focus solely on what to separate.

It is encoded as a sequence, where each frame is marked as either active (in yellow below) or not.

Subscribe now

Disclaimer. The views expressed are my own and do not reflect the opinions or positions of my employer.

And is the result of fusing Generative Adversarial Networks (“GAN”) with “Phantasmo”.

Subscribe now

They are Daniel Joaquin Trujillo and Josh Valenzuela.

And their artistic work blurs the lines between AI music and XR technology.

GANTASMO also nods to “Phantasmo”, the 1940s comic hero who could separate his consciousness from his body.

Like Phantasmo, they exist in tension:

• Between control and surrender with AI.

• Between physicality and abstraction with XR.

GANTASMO also explores the uncanny in sound.

Tracks are polished, precise, but unmistakably strange—beautifully uncanny.

Live, they translate AI’s complexity into something tangible in XR.

Because AI music is challenging to defend live, XR becomes a bridge.

I’m particularly interested in their work because they:

• Uniquely address the AI live performance challenge with XR.

• Explore AI music through the uncanny sound.

Here’s the full (written) interview we did following our meeting at the AI Song Contest and at the Barcelona AI Music Meetup.

What inspired your artistic proposal around AI?

Here at GANTASMO Industries, our mission is to make accessible, high-energy bangers with technically composed, yet accessible, experimental work. Our music is made to be… well, actually listenable to begin with, but still experience a broad range of topics, feelings, even thought experiments. We make songs for when you wanna laugh, cry, mosh, or mueve tu culo.

Our hope is to make interesting yet listenable music to explore historical, emotional, psychoacoustic, and narrative territories that would be difficult or impossible to access with traditional techniques alone.

Told through a near future, quasi-cyberpunk lore, nearly all of GANTASMO’s catalog lyrically and conceptually addresses topics, the such as love, loss, environmental collapse, the futility of war, the impact of creative obsession, internal conflict, war, and the psychopathy of nuclear weapons.

What sets your artistic work apart in the XR music scene?

Wait wait wait, hold on a second… There’s an XR Music scene?

XR is our audience facing medium for performance. Using modified Meta Quest 3 headsets, which enable the use of gestural recognition and hand tracking to manipulate and modulate virtual instruments, SFX, and VFX as if they were real instruments in 3D space. A single backpack containing a headset, a laptop, and a rat’s nest of cables/adapters replace what is often truckloads of DJ/audio equipment. This evaporation of hardware allows our shows to be extremely portable, and rapidly deployable in venues of all sizes. Plus, these crazy face computers compliment the futuristic aesthetic at the backbone of GANTASMO as performers, dev team, and composers.

A lengthy history in various tech and music fields is supported by our duo of seasoned Hackathon veterans – with dozens of XR hack victories including a win at MIT Reality Hack 2023, and making it to the 2015 DARPA Robotics Challenge Finals. Combine that with years working in robotics, web dev, video game dev, music, film, 3D animation, and on and on provides GANTASMO with the tools for harnessing this uncanny AI sound.

What is the most misunderstood aspect of AI/XR music that you’d like to clarify for the general public?

There seems to be a common belief that using AI in music (or any other artistic domain) is either a shortcut for the untalented & lazy, or an outright replacement for artists, like “push button, get song, profit.”

Quite often AI use is framed as this inescapable dichotomy where you’re an adversary or ally, “Real Artist” or “AI Slopmiser”. Personally and through GANTASMO we use AI in countless and novel ways throughout our lives, quite possibly way more than we assume most readers could even fathom, and maybe more than we’d like to admit. However, we continue to hand paint, animate in Blender, write songs, program, answer interview questions and all that as ourselves. Between the two of us we have a museum commissioned artist, touring frontman in multiple bands, award winning photographer, pro skateboarder, songwriter, multiple award winning films including a 2025 AI Film Award at Cannes, and have founded a couple successful (and maybe a few less successful) non-AI businesses. If all that is negated just because we use AI then I guess we are in fact: lazy and talentless.

People see AI as some executioner of all creative endeavors, and the only way to stop AImageddon is to demean, belittle, or shame anyone even considering its use. Since 2018, GANTASMO has been using AI to restore audio from 19th-century wax cylinders, repair decaying images and video for museums and estates, reconstruct dinosaur fossils, simulate proteins for drug discovery.

Expression through music is one of the most personal, powerful, raw human things we have, and people should be allowed to express themselves in whatever way they want so long as it doesn’t harm anyone or anything living. Whether it’s made with washboards and spoons, a grand piano, or some computer based mumbo jumbo.Nobody should be denied access to the joy of creation because of your fear caused by a silly 1980s robot movie, though we do love those movies. Whether or not someone knows what the devil’s chord is, has the money to buy a saxophone, or makes music you do or don’t enjoy, lazily or not – shouldn’t be a reason to prevent them from making music, good or not.

Art belongs to everyone, and it belongs to no one. Creative expression is essentially a human right in our eyes.

As an AI/XR musician, what challenges do you face?

One of the biggest challenges is that we’ve been bombarded with so many performance and speaking opportunities, while trying to maintain “normal” jobs, that it’s essentially impossible to keep up with our own lives. Keeping our development ahead of the curve, while juggling making music, art, social media, and everything that GANTASMO requires to smoothly operate turns essentially every show into a ‘testing in production’ nightmare – all while trying not to lose sleep, sanity, friendships, or romantic relationships along the way.

In short: our triumvirate of challenges as AI/XR artists/innovators are time, tedious organization, and finances for travel and R&D.

Bonus answer: there’s a perception gap that we address in “Just Give Up” where-in we reference a few infrequent occurrences of venues being afraid to book us, even backing out of shows at the last minute because they’re afraid of potential anti-AI backlash.

Though there are valid concerns, it seems like most of the loud anti-AI nay-sayers are forming this “AI = Die” type worldview that’s crystallizing into a performative, holier-than-thou amalgamation of criticism, confusion, and fear. Our art is our way of expressing grief, ambition, love, loss, burnout, and hope just like you. We just happen to be some extremely good looking nerds dorking around in XR headsets expressing themselves through some new-fangled face computers.

Your tracks often have an uncanny sound. Can you elaborate on your creative process and sound design?

Ahhhhh yes, Unified Neural Composition through Amplitude Nonlinearity and Neuroacoustic Yield, aka UNCANNY.

The process involves pipeline iteration: running audio through sequential models and feeding outputs back as inputs, forcing the models to compound distortions and invent new sound possibilities. We create quasi-LORAs (custom fine-tunes) intentionally biased toward generating anomalous sounds in regions of latent space.

We’ve also trained custom AI tools to handle tedious aspects of sound design. One example: mapping the 650+ parameters of complex VST synthesizers like Serum. We analyze a sound, create a recipe, and instantly configure the VST settings, accelerating production dramatically.

Thematically, our content ranges from satirical lore in “Ultrasonic Cavitation” (presenting our project as a Generative Adversarial Network Tactical Acoustic Super-Intelligence and Molecular Orchestrator), environmental activism in “Nature’s Tomb” made from samples of extinct species and destructive technologies, the futility of war in “And Ares Smiles”, loss and phantom limb syndrome as in “Something Lingers”, personal struggle such as “Rubble and Debris” motivational defiance set forth in “The Future”, chronicles of nuclear horror “Miracle Mile”, and heartbreak “Will I Dream”.

There’s an art/activism organization called Cetadelic for which we wrote “Hell is a Very Small Place” recursive soundscapes fragment and compress, mirroring psychological, and personal experiences to bring awareness to the permanent psychological and physiological effects of solitary confinement in our prison systems and that of animal captivity, particularly that of Kshamenk “The Loneliest Orca”.

“Nam-shub of Enki” uses an overly complex process of transliterating Sumerian based on modern agglutinative languages, precisely fermenting heavy subharmonic Khoomei/digeridoo style bass, generations of molding stable latent attractors states, and much much more. This resulted in rhythms containing phantom notes existing more in perception than waveform, instruments that recite verses in Sumerian and a curse on all of mankind using this 4000yr old reconstructed dead language.

Oh, yeah, we used AI to put curse on the entire planet, we probably should’ve mentioned that sooner.

What do you think will be the impact of AI in the history of music?

We foresee too many possible futures for one article to contain, but assuming this uncanny pursuit of the sonically unfamiliar continues, music in the year 2069 might be such a huge cocktail of different cultures, that it could result in much more universally acceptable and respectful forms. Or realtime generation powered by forecasting from an individual’s current biometric readings and historical data could result in an unimaginable level of fragmentation and hyper-personalization.

Either way, AI enables preservation and evolution of musical lineages that predate modern recording infrastructure.

GANTASMO has an upcoming version of “House of the Rising Sun.” You know, that Animals song from the ‘60s? Well, that was inspired by a Dave Van Ronk rendition, and Woody Guthrie and Lead Belly released versions in the ‘40s, then Georgia Turner in the ‘30s, which is remarkably similar to this old British folk song “The Unfortunate Rake”, then a handful of similar ballads from the 16th century, predating New Orleans and blah blah blah. Artist, stripper, musician, scientist, fish monger or whatever and use AI while honoring history and passing the torch to the next generation in whatever modern form to which it has transmogrified.

We apply this same reverent evolution to massively influential yet underappreciated figures: Frank Zappa’s genre-defying hyper-technical experimentation, Scott Joplin’s ragtime innovation, Tom Lehrer’s inimitable razor sharp political satire/educational crossover, and Sandy Nelson’s generation defining drum chops.

Additionally, our tenure in the XR world, and just being generally swell fellas has led to us collaborating with an OG of XR Bobby Carlton wherein we reimagine and refactor selections from his decades long catalog set to release on an upcoming single and EP under his moniker The Darkk Llord.

Following your finalist position in the AI Song Contest, what’s next for you? What are GANTASMO’s future plans?

The AI Song Contest isn’t over until they kick us out. Even then, that’s debatable.

Let me start by saying that there’s a possibility that we may be possessed by some AI phantom from the future, or we’re hopelessly addicted to learning, programming, and making music. We don’t know.

So, pending GANTASMO’s institutionalization and/or exorcism, you’ll likely find us in our underground lair, writing and releasing new material, tightening up our live sets, compiling notes for upcoming speaking engagements, doing house shows, mentoring hackathons, collaborations, bar mitzvahs, and whatever formats make sense, and likely some that don’t.

In parallel, we’ll keep building and refining the tools we use on stage and in the studio, because a lot of what we need still doesn’t exist off the shelf. We plan to open-source these tools for others to use, break, and improve upon. We just love meeting developers, artists, researchers, and nerds interested in this space and would like to continue doing so.

However, we know that we must remain flexible enough to adapt to all updates and new tech that may emerge.

GANTASMO has been around since 2018, but as a publicly performing act, we turned 1 year old October 5, and the response has already been way bigger than we expected. As overwhelming as it may be sometimes, being invited to speak, teach, and experiment with different communities has been one of the best parts of this whole thing.

There is no master plan for the future of GANTASMO, but we know the general direction: never ending R&D, push the limits of AI/XR convergence, hard drives overflowing with the latest models, and obviously – more and more dank music.

Speaking of dank music… our single “Memento Mori pt I” is live on all major streaming platforms today November 26, which also happens to be JoshTASMO’s birthday. This track is from our upcoming album “UNCANNY” scheduled to release everywhere on Monday December 8th.

GANTASMO is the vehicle, but the goal is simple: never stop learning, disseminate knowledge and tools, build a career out of this, communicate and build a more equitable future for creative expression alongside any like-minded individuals willing to participate in potentially shaping history.

We will conclude this transmission by saying…

The future doesn’t exist. However, the future starts today, and the future is in your hands.

Subscribe now

Disclaimer. The views expressed in this content are those of GANTASMO and do not reflect my own opinions or those of my employer. Only the introduction was written by me.

I was a master’s student at the Music Technology Group (MTG), spending my days discovering the world of sound and music computing.

I didn’t know it then, but Freesound was about to play a massive role in my life.

For those who don’t know Freesound, it is an online community and library of user-uploaded sounds.

Freesound is the largest open audio database on the internet.

And is hosted by the Music Technology Group (MTG), where I was studying.

At the time, I was just discovering deep learning. And studying how to apply it for music and audio problems was fun!

I spent three months in Hannover as part of my master’s thesis, and I created LaTeX slides to compare NMF against these new deep learning methods, and I ended up doing most of my work from my bed.

Then I started a PhD at the MTG, and DRAMA unfolded.

Subscribe now

No GPUs, No Data

By 2015, the deep learning revolution was in full swing.

But to do meaningful work as a researcher, I needed two things:

1. Computing Power (GPUs)

2. Data

We had neither.

Expertise was fleeing to the industry and we were left wondering how to compete.

In 2017, I found a potential solution: join Pandora for an internship.

Suddenly, I had access to:

• 1.2 million songs annotated for audio-tagging research

• 16 GPUs

But it was a walled garden.

When I returned to Barcelona and the MTG, I couldn’t bring the data or GPUs with me.

I was back to square one. No GPUs. No data.

Freesound

But the MTG hosts Freesound.

With hundreds of thousands of recordings under Creative Commons licenses.

In 2018, Freesound had about 400,000 uploads.

Could we leverage it to build an open dataset for reproducible audio-tagging research?

And we rolled up our sleeves and started annotating.

The result was FSD50K:

• Over 50,000 clips.

• Open licenses.

• Annotated by humans.

It became a standard.

Freesound to the rescue.

Generative AI

But technology moves fast.

Fast forward to the 2020s, the field shifted from audio-tagging to generative AI.

We weren’t just classifying audio anymore, we were generating it.

And the requirements for this new era were even steeper.

More GPUs. And data-intensive.

But by 2024, Freesound had grown to over 700,000 audio files.

And a subset of 500,000 recordings is under permissive licenses:

• CC0

• CC-BY

• CC-Sampling+

Once again, Freesound to the rescue.

This data would allow training a competitive generative AI model that we could actually release.

Not just a paper. Not just an API. But the weights, such that artists and researchers could play with it.

And we trained Stable Audio Open with Freesound data.

The result? It became one of the most liked and downloaded text-to-audio models on Hugging Face.

Happy birthday!

My journey—from a master’s student struggling for data to releasing state-of-the-art generative models—was only possible because of the Freesound community.

And this year, Freesound turned 20. Congratulations!

Every time you uploaded a field recording, a sound effect, or a musical loop, you weren’t just sharing a file. You were fueling the future of open audio research. Thanks!

Freesound to the rescue, indeed.

Subscribe now

Disclaimer. The views expressed are my own and do not reflect the opinions or positions of my employer.

AI experts can’t stop talking about copyright and AI ethics, arguing the importance of licensing training data.

But ethical alternatives only matter if the AI solution actually works. Right?

So far, the best (and most used) models have been trained on “publicly available data” under the umbrella of “fair use”. For example:

• Suno is accused of using copyrighted music to train its models.

• Sora is trained on publicly available content, licensed content, and content generated by human reviewers.

Competing is challenging for those training on licensed data, as it is often scarce and typically sourced from stock libraries (rather than commercial music or films).

This can limit both the diversity and the quality of the output generations.

How far is too far for ethical options that fall short of consumers’ expectations?

Further, AI is increasingly influencing our everyday lives through products in the marketplace (like chatbots or AI software for creatives).

Accordingly… small startups, large corporations, and (even) artists are subject to market forces, impacting stakeholders in varying ways.

Several initiatives (such as public regulation or private certifications) are being introduced to prevent abuse.

In this post, I explore the narratives and power dynamics behind why many artists still hesitate to embrace AI.

Why most artists don’t even consider AI for their work?

Finally, in the past few days, we’ve seen major moves in the music industry:

• Spotify partners with major labels to develop AI music products.

• Udio teams up with Universal Music Group to create an AI streaming platform.

• Stability AI and Universal Music Group partner to develop AI music creation tools.

Hence, major labels are now actively developing AI solutions—an involvement that could reshape the industry and redefine the narratives around AI art.

And, most importantly, this marks a new era. Opening the possibility of training high-quality models on licensed commercial music.

AI models are no longer constrained to the style of stock libraries.

Subscribe now

Public Regulation

• The private development of AI through the free market inquires the risk that leading companies define, de facto, the very definition and nature of AI. The economic incentives are huge.

• Public investment in AI could lead to centralized systems that governments may use for reelection or social control.

In both scenarios, it seems reasonable to enable mechanisms of democratic control over the economy/power.

Will this kill innovation? Perhaps. But maybe that’s exactly the point. Everyone calls it a bug—but for regulators, it’s a feature.

Be aware of your bubble.

Private Certifications

• Social trust is essential for AI adoption, and demands a strong ethical approach.

• AI is currently being rolled out through the free market, that is challenging to intervene.

If it’s difficult to agree on new mechanisms of democratic control, why not promoting certificates around ethics and AI?

Note the parallelism when buying cage-free or free-range eggs.

Rather than adding new restrictions, bring in new incentives.

The goal of AI certificates is to add value to those products that are developed following ethical principles.

Ethically sourced, grassfed, organic AI music eh? — Kuraido (@MailliwNos).

The problem is that consumers may place limited value on certifications, prioritizing instead AI solutions that are effective and reliable.

Narratives

Marketing efforts are also contributing to the social definition of AI.

In a context where AI technology is still underdeveloped and society is unprepared for its critical adoption, leading companies are pushing their (optimistic) vision of AI to the world.

This (legitimate) push to increase the company’s share price is creating hype.

And fear.

Because a (pessimistic) narrative of concern has also emerged, often fueled by media and publishers seeking clicks.

This surprisingly well-coordinated combo is actually driving people apart.

Power Asymmetries

Current AI market is full of power asymmetries, which influence how artists perceive AI-generated works.

• Example 1: Can your artist friend build an AI-based installation?

No, because she does not even understand which are the building blocks of this technology and she is concerned it could threaten their work.

Artists often face pushback for using AI, as it is still not widely accepted in many creative circles.

• Example 2: Can you train an AI model based on your artworks?

Yes, if you know how to code, read scientific papers, and can buy expensive hardware.

Only a few people can imagine and develop new AI systems.

And those have the capacity (ab)use their dominance to define what AI (art) means.

A New Era

Major music labels are now actively involved in shaping the development of AI.

Chronology of recent news…

• 2 October 2025: Financial Times reports upcoming AI licensing deals involving major labels.

• 16 October 2025: Spotify partners with major labels.

• 29 October 2025: Udio partners with Universal Music Group.

• 30 October 2025: Stability AI partners with Universal Music Group.

Interestingly, Spotify is a streaming platform.

And Udio is becoming a streaming service, as Udio discontinued the download feature.

Stability AI, on the other hand, is focusing on professional music creation tools.

Based on these announcements, major labels appear focused on:

• Using AI to engage (super)fans through (interactive AI) streaming services.

• Enable new forms of music creation (with AI) for artists.

Yet, what comes next remains uncertain. Especially as the Financial Times reported that major labels are also in talks with other companies.

Subscribe now

Disclaimer. The views expressed are my own and do not reflect the opinions or positions of my employer.

Today’s quotes are from: ‘A Year with Swollen Appendices’ by Brian Eno (1996).

Subscribe now

Brian Eno thought deeply about generative music during the 90s—and he also wanted to urinate on Duchamp’s Fountain 🙂

For those reasons alone, I (really, REALLY) had to read ‘A Year with Swollen Appendices’.

In this book..

• Defines generative and ambient music.

• Reflects on being an artist, and questions what culture even means.

• Shares his thoughts on Duchamp’s Fountain.

• Includes letters exchanged with fellow musicians.

Half the book is essays, letters, and stories. The other half is a diary from 1995.
Yes, from 1995! Offering a rare glimpse into his thoughts and creative process.

In his diary, he also shares his frustrations:

• Critics mocking his work.

• The limitations of the medium for his generative artistic experiments.

“This was regarded by many English music critics as a kind of arty joke, and they had a lot of fun with it. (…) Those early seeds have contributed to a rich forest of music.”

“However, (…) this would have involved persuading my listeners to buy four or five CD-players instead of just one, and then buy the set of four or five CDs to play on them.”

This book has been a huge inspiration. Both as an aspiring author, and as a musician experimenting with AI.

I felt comforted by the familiarity of his struggles:

• Traditional mediums also felt constraining as we experimented with Interactive AI Music.

• Copyright issues dominate the conversation today, leaving little room for AI artistic exploration.

Generative Music

Definition…

“From now on there are three alternatives: live music, recorded music and generative music. Generative music enjoys some of the benefits of both its ancestors. Like live music, it is always different. Like recorded music, it is free of time-and-place limitations – you can hear it when you want and where you want.”

Innovation..

“In my audio-visual installations I found another way of making ever-changing music. I distributed the pre-recorded musical elements over several (usually four to eight) audio cassettes of different lengths. These were all played back simultaneously.”

Frustration..

“My records, however, were always recordings of the output of one or another of these combinatorial systems: though it could produce original music forever, what went on the record was a 30-minutes section of its output, which would then of course be identical each time you played it. However, what I always wanted to do was to sell the system itself, so that a listener would know that the music was always unique. Since this would have involved persuading my listeners to buy four or five CD-players instead of just one, and then buy the set of four or five CDs to play on them, I didn’t spend too much time on the project.”

Ambient Music

Definition..

“Ambient Music must be able to accommodate many levels of listening attention without enforcing one in particular; it must be as ignorable as it is interesting.”

Frustration..

“This was regarded by many English music critics as a kind of arty joke, and they had a lot of fun with it. (…) Those early seeds have contributed to a rich forest of music.”

What’s Culture?

Definition..

“Culture is everything we don’t have to do.”

Examples..

“We have to eat, but we don’t have to have ‘cuisines’, Big Macs or Tournedos Rossini. We have to cover ourselves against the weatehr, but we don’t have to be so concerned as we are about weather we put on Levi’s or Yves Saint-Laurent.”

Duchamp’s Fountain

A fact..

“The attempts to keep art special become increasingly bizarre.”

A question..

“Duchamp had explicitly been saying, ‘I can call any old urinal—or anything else for that matter—a piece of art’, and yet curators acted as though only this particular urinal was A Work Of Art. If that wasn’t the case, then why not exhibit any urinal?”

An intention..

“I always wanted to urinate on that piece of art, to leave my small mark on art history.”

Axis Thinking

Definition..

“An axis is a name for a continuum of possibilities between two extreme positions: so the axis between black and white is a scale of grays.”

Examples..

“I can illustrate this idea by applying it to the description of haircuts: natural-contrived, rebel-conformist, wild-civilised, futuristic-nostalgic, businesslike-bohemian. Each of these exists as a “dimension” in the haircut space, which now becomes multidimensional.”

Subscribe now

Previous Bright Bites:

• ‘Tasty Algorithm: Artificial Intelligence in Gastronomy’ by Eneko Axpe (2025).

• ‘Surfing human creativity with AI’ by Nao Tokui (2023).

• ‘Exploring Artistic Frontiers in the Era of Artificial Intelligence’ by V. Işık (2024).

Disclaimer. The excerpts shared here reflect Brian Eno’s perspectives, as published in ‘A Year with Swollen Appendices’. They are not my own and do not reflect the opinions or positions of my employer.

But while strolling my child around or watching him sleep, I can sneak in brief (and admittedly irresponsible) phone editing sessions.

Subscribe now

How it went?

The good:
- Mobile editing tools are capable AF.
- Generative AI video apps are now accessible and improving fast.

The bad:
- Making precise edits was uncomfortable.
- Pushed my phone’s hardware to its limits.

The ugly:
- Total cost: 81,98€.
- My bandmates roasted me for editing on a phone.

Concept

The ¿CULO O CODO? meme from the 90s plays on the confusion between: the butt ("CULO") and the elbow ("CODO").

The meme shows a close-up shot that looks ambiguous.

And you can’t really tell if it’s a codo (elbow) or a culo (butt).

Our idea is to use AI to zoom out and reveal the truth.

Brainrot AI Aesthetics

Fast forward to 2025, internet culture revolves around Brainrot AI.

Brainrot AI is a genre of silly, repetitive, low-effort, uncanny content that fuels dumbscrolling.

The ¿CULO O CODO? meme, with its mix of absurd confusion and familiar discomfort, can be seen as a precursor for Brainrot AI content.

Our song critiques internet habits, highlighting how our engagement with digital culture often reduces to unproductive consumption.

For this reason, we decided to explore Brainrot AI aestetics.

Meet the star of our show… Culozino Niccolocodo.

Choosing Hailuo 02

We need an image-to-video model because our concept involves turning elbow images into zooming-out videos.

In late July, we tested several tools, including Kling AI, Runway, and Pixverse.

But ultimately chose Hailuo 02 for its generous free credits and superior Brainrot AI aesthetics.

One prompt we used frequently was: “Zoom out into a creepy character”.

I was also impressed by how well it maintained character consistency.

Starting from a single character image, prompts like “A creepy character [action]” produce results that stay true to the original character while performing the action.

Our characters perform actions such as running, climbing, or eating spaghetti.

I also enjoyed how the camera controls allowed for dynamic and more dramatic shots.

Hailuo 02’s visual style, controls, and image-to-video features are perfect for the concept and aesthetics we are pursuing.

Using CapCut

Turns out, editing on a phone actually works surprisingly well.

Video editing on phones is becoming incredibly powerful, with advanced features like layers, effects, transitions, and color correction.

That said, precision edits were a nightmare. Tiny trims, syncing beats…

Also, my old phone almost gave up. Exporting the video was a struggle.

In the end, ~90% of the videoclip was done on my phone.

The last 10% (mostly precision editing and exporting) happened on my computer.

This hybrid setup saved my sanity.

Costs

Two months of Pro subscription to HailuoAI: 40.99€ x 2 = 81.98€.

But if you know what you’re doing, you could probably pull off the entire videoclip in under a month.

You might even manage with the Standard subscription instead of Pro.

I was able to save some money with CapCut, though.

For the last tweaks, I had to switch to my computer. And for a hybrid phone/computer workflow, a subscription is required.

Luckily, I managed to finish those final edits within the trial period.

So, CapCut cost me exactly 0€.

Subscribe now

Disclaimer. The views expressed are my own and do not reflect the opinions or positions of my employer.

1. Interactive.

2. AI sounding.

3. Relatable.

Subscribe now

We experiment with three main ideas:

1. Interactive with generative AI. We use AI not merely as a tool, but as a dynamic artistic medium. Our goal is to make the listening experience interactive by relying on modern generative AI. Beyond mastering and sound design, AI is used during playback. Transforming the audience from passive listeners into creative participants.

2. Brainrot AI sound. We embrace AI’s ability to make sound that’s both uncanny and brainrot.

3. Relatable Rhythm. Even with strange textures or generative sounds, we make music with relatable danceable beats.

Experience our interactive/generative AI music piece online:

• https://brnrt-player.duckdns.org

I made this project with:

• Pablo Alonso is a researcher at the Music Technology Group of Universitat Pompeu Fabra.

• Raimon Aranyó is an engineer at Dolby Laboratories.

• Guillermo Cambara is a researcher working at Amazon.

Having first met at Universitat Pompeu Fabra and later at Dolby, we share a common interest in AI, music, and engineering.

Making music is always better with friends :)

CULOCODO

This is the name of our song.

The ¿CULO O CODO? meme from the 90s plays on the confusion between: the butt ("CULO") and the elbow ("CODO").

The meme typically presents a visual that ambiguously resembles either one, provoking a mix of amusement and disorientation.

This deliberate uncertainty taps into a feeling that is both familiar and strange, invoking what scholars call the uncanny: an experience where something appears recognizable yet unsettling because it challenges our usual perceptions.

Fast forward to 2025, internet culture revolves around uncanny brainrot AI.

Brainrot AI is a genre of silly, repetitive, low-effort, uncanny content that fuels dumbscrolling, where people mindlessly scroll through endless meaningless posts.

The ¿CULO O CODO? meme, with its mix of absurd confusion and familiar discomfort, can be seen as a precursor for brainrot, uncanny content.

Our song critiques internet habits, highlighting how our engagement with digital culture often reduces to unproductive consumption.

Videoclip

Our videoclip looks at the ¿CULO O CODO? meme through the lens of AI.

We used AI to zoom out and reveal: butt ("CULO") or elbow ("CODO")?

Using Hailuo AI’s image-to-video tool, we turned photos our elbows into uncanny characters to answer… ¿CULO O CODO?

Human-AI co-creation process

Our main goal is to make the listening experience interactive by relying on the capabilities of modern generative AI.

Our interactive/generative AI track is available online: https://brnrt-player.duckdns.org

To align with the guidelines of the AI Song Contest, we submitted an audio file that represents a rendition that we obtained through our interface.

Yet, to fully experience the project as intended, we recommend visiting our interactive/generative AI track online.

Generative music is also called indeterminate music. Or chance music. Or stochastic. And has a long history rooted in algorithmic and experimental approaches to composition. See previous post.

We view AI as an artistic medium that is generative, stochastic, and ephemeral, drawing inspiration from the pioneering ideas of generative music artists.

We used AI exclusively for sound design, via generative audio-to-audio transformations with Stable Audio and Neutone Morpho.

We also generated samples with Stable Audio Open.

The BRAINROT AI transformations in our website are audio-to-audio transformations of the stems in our original composition CULOCODO.

All aspects of the composition, including melody, harmony, rhythm, and arrangement, were entirely created by us without the use of AI.

We did not use AI for mixing, and used LNDR for mastering.

Live performance opportunities

Interactive AI music opens up exciting possibilities for live performances.

With generative AI, no two tracks are ever the same.

And interactive AI music could transform concerts into unique, ephemeral experiences, where the audience influences the music in real time.

Play with our generative and interactive AI music release :) It’s an instrument!

Subscribe now

Disclaimer. The views expressed are my own and do not reflect the opinions or positions of my employer.

• Full paper (arXiv)

• Database of collected AI Music (GitHub)

• Video (below!)

Subscribe now

We study how musicians use artificial intelligence (AI) across formats like:

• Singles

• Albums

• Performances

• Installations

• AI voices

• Operas

• Soundtracks

We collect 337 music artworks and categorize them based on AI usage:

• AI composition

• Co-composition

• Sound design

• Lyrics generation

• Translation.

Here are our key insights:

• Artistic agency: Most of the collected musical artworks use AI as a tool for co-composition or sound design, allowing artists to maintain creative control. Others, however, have explored the absence of artistic agency as part of their creative process.

• Aesthetics: AI music often embraces AI's flaws and aesthetics, but at times it can be nearly indistinguishable from non-AI music. Some explored AI's imperfections and artifacts to evoke the uncanny. Others used AI's multi-genre generation capabilities to produce music in multiple (and potentially new) genres.

• Sound design: Artists curate datasets to train their own models, similar to how producers program synthesizers to create their unique sounds. This example reveals the common technical challenges AI musicians face, a shared difficulty among those in the field.

• Innovative uses include multilanguage song releases and exploring its potential in live performances and installations, bringing AI into formats like opera, soundtracks, ballets, and online installations.

• Artistic medium: Another key innovation is using AI as an artistic medium, allowing generative AI albums to become a dynamic medium that evolves with each listen. AI voice releases are also an artistic medium that enables new interactions between artists and fans, raising interesting questions around authorship and digital identity.

We also discuss the importance of casual music creators and their role in the commodification of music creation—in a context where millions of tracks are being generated weekly, with our study covering 337 artworks.

We also find that parallels with past narratives showcase that current trends and cultural pushback are not new.

We hope our work contributes to understanding how early adopters have used AI so far and serves as inspiration for future AI musicians, encouraging the creation of music that leaves a lasting, positive impact on music history.

Subscribe now

Disclaimer. The views expressed in this blog do not reflect the opinions or positions of my employer.

In music, a beef is a battle—not fought with punches, but with sharp, pointed lyrics.

Subscribe now

They diss.
They respond.
Back and forth.
Track after track.

Tragic Beef

First up: a look at a key historical beef—with no AI involved.

West Coast vs. East Coast.
Diss tracks turned into headlines.
Headlines turned into funerals.

In the 1990s, the East-West Coast hip-hop rivalry, centered on Tupac and Biggie, escalated from music to real violence.

Both artists were killed in drive-by shootings within a year of each other, turning a lyrical feud into one of music’s most tragic chapters.

That tragic beef set the stage for a new kind of beef powered by AI.

Decades after Tupac's death, his voice resurfaced in an AI-generated diss track by Drake.

Deepfake Beef

On April 19, 2024, Drake released a diss track called 'Taylor Made Freestyle' aimed at Kendrick Lamar.

The song featured AI-generated vocals that mimicked the voices of Tupac and Snoop Dogg.

The estate of Tupac urged Drake to take down the song. A few days after, Drake pulled the track from all platforms.

Well.. not from all platforms. You can listen to the track in the following video:

On April 30, 2024, Kendrick Lamar responded with a diss track titled 'Euphoria'.

He directly addresses Drake’s use of AI: 'Am I battlin’ ghost or AI?'

On May 5, 2024, Metro Boomin dropped a beat titled 'BBL Drizzy'.

He did so shortly after Drake called him out in 'Push Ups': "Metro, shut your hoe ass up and make some drums".

The song’s title references a rumor that Drake (aka Drizzy) had plastic surgery, with "BBL" being slang for Brazilian Butt Lift.

He samples an AI-generated track from Udio, an AI music platform currently facing legal beef from major record labels.

And Metro Boomin turns the track into a contest—$10K for the best diss against Drake. And the internet runs wild.

Drake claps back by rapping over the same beat weeks later with 'U My Everything'.

King Willonius (who created the original track with Udio) is credited as a songwriter of 'U My Everything' as it contains a sample from 'BBL Drizzy'.

Note that King Willonius will be compensated—not Metro Boomin, who popularized the track.

Legal Beef

Major labels vs. Udio & Suno → link, link
Authors Guild vs. OpenAI & Microsoft → link
New York Times vs. OpenAI & Microsoft → link

AI is at the center of a growing legal debate. Right holders argue that their copyrighted works are being used without permission to train AI models.

We've covered this topic enough → link, link
Let's move on to a lesser-known beef in AI music.

Academic Beef

David Cope developed an AI system in the 80s which generated music in the style of Bach, Beethoven, or Chopin.

For those unfamiliar with Cope, he’s one of the early pioneers of AI music. And he sadly passed away this May.

Yet, in 2008, AI music researcher Geraint Wiggins called Cope’s work pseudoscience.

In the paper above, Wiggins argues that Cope’s claims in his book are ‘vague, exaggerated, or untestable’.

I highly recommend reading the paper—it’s my favorite beef in AI music.

Subscribe now

Disclaimer. The views expressed are my own and do not reflect the opinions or positions of my employer.