I was a music AI sceptic – until I actually used it

With artificial intelligence programs that can now generate entire songs^[1] on demand, you’d be forgiven for thinking AI might eventually lead to the decline of human-made music.

But AI can still be used ethically to help human musicians challenge themselves and grow their music-making abilities^[2]. I should know. As a composer and music educator, I was an AI sceptic until I started working with the technology^[3].

Two sides of the argument

If you can write a text prompt, you can use AI to create a track in any genre, for almost any musical application.

Besides generating full tracks, music AI can be used in sound analysis, noise removal, mixing and mastering, and to create entire sound palettes (such as for use in video games and podcasts). Suno, Beatoven, AIVA, Soundraw and Udio are some of the companies currently leading in the AI music space.

In many cases, the outputs don’t have to be excellent, they just have to be good enough, and they can undercut the services of real musicians and sound designers.

The music industry is understandably concerned. In April 2024, the US-based Artist Rights Alliance^[4] published an open letter^[5], signed by more than 200 artists, calling for developers to stop training their AIs with copyrighted work (as this would allow companies to emulate artists’ music and image, and therefore deplete the royalties paid to artists).

At the same time, music AI companies claim to lower the barrier to making music, such as by removing the need for physical equipment and traditional music education.

In an interview from January^[6], Suno’s chief executive Mikey Shulman said:

it’s not really enjoyable to make music now. It takes a lot of time. It takes a lot of practice […] the majority of people don’t enjoy the majority of the time they spend making music.

This is far from the message I want to send my students. However, it does unfortunately reflect the increasing pressure musicians feel to master their craft as soon as possible, in an increasingly fast-paced world that’s geared towards an intangible end goal, rather than enjoying the process of making mistakes and learning.

From a sceptic to a reluctant advocate

In 2023, I was commissioned by the Sydney Opera House create a new work with Sydney-based design company Kopi Su^[7], and to develop a new generative music AI tool in the process. This tool, called Koup Music^[8], is now in beta testing.

I accepted the opportunity – but with quite a few hesitations, as I wasn’t really interested in working with AI. Would this be a huge waste of time, or end with my data added to some mysterious AI data pool? Or would it open up new creative directions for me?

The tool was based on a text-to-image diffusion model called Riffusion^[9]. It takes a text prompt and generates a spectrogram, which is a visual representation of the various frequencies in an audio signal as they change through time. This is then converted to audio.

First, I would upload my own recorded sample to the AI, and then choose a text prompt to transform it into a new five-second sample.

For example, I could upload a short vocal melody and ask the AI to turn it into an insect, or re-contextualise it for a “hip hop” style. Sometimes the generated samples sounded very similar to my own voice (due to the vocals I uploaded).

The following insect voice output became the subject of the musical piece below it.

Somewhere between a voice and an insect.

At the time of the project, the outputs could only be 5 or 10 seconds long – not long enough to make a full track. I considered this a positive, as it meant I had to incorporate the samples into my own larger work.

Some samples were catchy. Some were funny. Others were boring. Some came out with scratchy, harsh timbres. The imperfection of it all gave me permission to have fun.

I focused on generating separate musical elements with my text prompts, rather than fully arranged samples. A generated drum beat or melody line could be enough to inspire a completely new musical track in a style I would never have attempted otherwise.

This output was used in the track How Things Grow.

Sometimes, one generated sample was enough. Other times, I challenged myself to use only AI-generated sounds to create a full track. In these cases, I used techniques such as filtering and looping small snippets to tease out the sounds I wanted.

For instance, I used the following audio samples to create the track below:

These snippets were used in the track Boom Boom Boom.

The process felt like a collaboration – like I was making music with a kooky colleague. This took away the pressure to make “perfect” music, and instead allowed me to focus on new creative possibilities.

My takeaways

I’ve concluded it’s not a bad idea to know what large music AIs are capable of. We can use them to further our own musical understanding, such as by studying how they use stylistic trends and mixing techniques, or how they translate musical ideas to suggest different genres.

For me, the key to quashing my AI scepticism was using an AI that didn’t take over the entire working process. I remained flexible to its suggestions, while using my own knowledge to retain creative control.

My experience isn’t isolated. Multiple^[10] studies^[11] have^[12] found that users of music AIs reported feeling satisfied with programs that allowed them to retain a sense of ownership over the composing process.

The connecting factor across these projects was that the AI did not generate entire musical works in one go. Instead, a limited amount of musical information was generated (such as rhythms, melodies or chords), allowing the user to dictate the final result.

The beauty in human imperfection

Despite Shulman’s claims, the key to a meaningful relationship with music AI is to work alongside it – not to let it do all the work.

Do I think every music student should start incorporating AI into their daily practice? No. But under the right circumstances, it can provide the tools to produce something truly creative.

Making “imperfect” art that takes time – and hard work – is the price of being human. And I’m grateful for that.

References

1. ^{^} generate entire songs (theconversation.com)
2. ^{^} grow their music-making abilities (theconversation.com)
3. ^{^} working with the technology (link.springer.com)
4. ^{^} Artist Rights Alliance (artistrightsalliance.org)
5. ^{^} open letter (musicrow.com)
6. ^{^} interview from January (www.youtube.com)
7. ^{^} Kopi Su (www.kopisustudio.com)
8. ^{^} Koup Music (www.koupmusic.com)
9. ^{^} Riffusion (en.wikipedia.org)
10. ^{^} Multiple (dl.acm.org)
11. ^{^} studies (arxiv.org)
12. ^{^} have (zenodo.org)