Text-to-audio generation is here. One of the next big AI disruptions could be in the music industry

The past few years have seen an explosion in applications of artificial intelligence to creative fields. A new generation of image and text generators is delivering impressive^[1] results^[2]. Now AI has also found applications in music, too.

Last week, a group of researchers at Google released MusicLM^[3] – an AI-based music generator that can convert text prompts into audio segments. It’s another example of the rapid pace of innovation in an incredible few years for creative AI.

With the music industry still adjusting to disruptions caused by the internet and streaming services, there’s a lot of interest in how AI might change the way we create and experience music.

Read more: Neil Young’s ultimatum to Spotify shows streaming platforms are now a battleground where artists can leverage power^[4]

Automating music creation

A number of AI tools now allow users to automatically generate musical sequences or audio segments. Many are free and open source, such as Google’s Magenta^[5] toolkit.

Two of the most familiar approaches in AI music generation are:

1. continuation, where the AI continues a sequence of notes or waveform data, and

2. harmonisation or accompaniment, where the AI generates something to complement the input, such as chords to go with a melody.

Similar to text- and image-generating AI, music AI systems can be trained on a number of different data sets. You could, for example, extend a melody by Chopin using a system trained in the style of Bon Jovi – as beautifully demonstrated in OpenAI’s MuseNet^[6].

Such tools can be great inspiration for artists with “blank page syndrome”, even if the artist themselves provide the final push. Creative stimulation is one of the immediate applications of creative AI tools today.

But where these tools may one day be even more useful is in extending musical expertise. Many people can write a tune, but fewer know how to adeptly manipulate chords to evoke emotions, or how to write music in a range of styles.

Although music AI tools have a way to go to reliably do the work of talented musicians, a handful of companies are developing AI platforms for music generation.

Boomy^[7] takes the minimalist path: users with no musical experience can create a song with a few clicks and then rearrange it. Aiva^[8] has a similar approach, but allows finer control; artists can edit the generated music note-by-note in a custom editor.

There is a catch, however. Machine learning techniques are famously hard to control, and generating music using AI is a bit of a lucky dip for now; you might occasionally strike gold while using these tools, but you may not know why.

An ongoing challenge for people creating these AI tools is to allow more precise and deliberate control over what the generative algorithms produce.

New ways to manipulate style and sound

Music AI tools also allow users to transform a musical sequence or audio segment. Google Magenta’s Differentiable Digital Signal Processing^[9] library technology, for example, performs timbre transfer.

Timbre^[10] is the technical term for the texture of the sound – the difference between a car engine and a whistle. Using timbre transfer, the timbre of a segment of audio can be changed.

Such tools are a great example of how AI can help musicians compose rich orchestrations and achieve completely new sounds. In the first AI Song Contest^[11], held in 2020, Sydney-based music studio Uncanny Valley^[12] (with whom I collaborate), used timbre transfer to bring singing koalas into the mix.

Timbre transfer has joined a long history of synthesis techniques that have become instruments in themselves.

Taking music apart

Music generation and transformation are just one part of the equation. A longstanding problem in audio work is that of “source separation”. This means being able to break an audio recording of a track into its separate instruments.

Although it’s not perfect, AI-powered source separation has come a long way. Its use is likely to be a big deal for artists; some of whom won’t like that others can “pick the lock” on their compositions.

Meanwhile, DJs and mashup artists will gain unprecedented control over how they mix and remix tracks. Source separation start-up Audioshake^[13] claims this will provide new revenue streams for artists who allow their music to be adapted more easily, such as for TV and film.

Artists may have to accept this Pandora’s box has been opened, as was the case when synthesizers and drum machines first arrived and, in some circumstances, replaced the need for musicians in certain contexts.

But watch this space, because copyright laws do offer artists protection from the unauthorised manipulation of their work. This is likely to become another grey area in the music industry, and regulation may struggle to keep up^[14].

New musical experiences

Playlist popularity has revealed how much we like to listen to music that has some “functional” utility^[15], such as to focus, relax, fall asleep, or work out to.

The start-up Endel^[16] has made AI-powered functional music its business model, creating infinite streams to help maximise certain cognitive states.

Endel’s music can be hooked up to physiological data such as a listener’s heart rate. Its manifesto^[17] draws heavily on practices of mindfulness and makes the bold proposal we can use “new technology to help our bodies and brains adapt to the new world”, with its hectic and anxiety-inducing pace.

Other start-ups are also exploring functional music. Aimi^[18] is examining how individual electronic music producers can turn their music into infinite and interactive streams.

Aimi’s listener app invites fans to manipulate the system’s generative parameters such as “intensity” or “texture”, or deciding when a drop happens. The listener engages with the music rather than listening passively.

It’s hard to say how much heavy lifting AI is doing in these applications – potentially little. Even so, such advances are guiding companies’ visions of how musical experience might evolve in the future.

The future of music

The initiatives mentioned above are in conflict with several long-established conventions, laws and cultural values regarding how we create and share music.

Will copyright laws be tightened to ensure companies training AI systems on artists’ works compensate those artists? And what would that compensation be for? Will new rules apply to source separation? Will musicians using AI spend less time making music, or make more music than ever before?

If there’s one thing that’s certain, it’s change. As a new generation of musicians grows up immersed in AI’s creative possibilities, they’ll find new ways of working with these tools.

Such turbulence is nothing new in the history of music technology, and neither powerful technologies nor standing conventions should dictate our creative future.

Read more: No, the Lensa AI app technically isn’t stealing artists' work – but it will majorly shake up the art world^[19]

References

1. ^{^} impressive (theconversation.com)
2. ^{^} results (www.theguardian.com)
3. ^{^} released MusicLM (google-research.github.io)
4. ^{^} Neil Young’s ultimatum to Spotify shows streaming platforms are now a battleground where artists can leverage power (theconversation.com)
5. ^{^} Magenta (magenta.tensorflow.org)
6. ^{^} MuseNet (soundcloud.com)
7. ^{^} Boomy (boomy.com)
8. ^{^} Aiva (www.aiva.ai)
9. ^{^} Differentiable Digital Signal Processing (magenta.tensorflow.org)
10. ^{^} Timbre (www.youtube.com)
11. ^{^} AI Song Contest (www.aisongcontest.com)
12. ^{^} Uncanny Valley (uncannyvalley.com.au)
13. ^{^} Audioshake (www.digitalmusicnews.com)
14. ^{^} struggle to keep up (www.tandfonline.com)
15. ^{^} has some “functional” utility (pitchfork.com)
16. ^{^} Endel (endel.io)
17. ^{^} manifesto (manifesto.endel.io)
18. ^{^} Aimi (www.aimi.fm)
19. ^{^} No, the Lensa AI app technically isn’t stealing artists' work – but it will majorly shake up the art world (theconversation.com)