I. Linear adaptive music.
Video games have what many consider a non-linear structure, meaning that the timing and outcome of events are unknown and are not predetermined by being set on a single timeline. This sets a major challenge for composers who wish to write for this medium as music’s ability to create emotions and tell stories relies primarily on the linear development of harmonic, rhythmic and melodic concepts. That being said, I personally think that the structure of a game needs to be considered ‘linear adaptive’ rather than 'non-linear' as players still experience a linear series of events. The game adapts to their actions, choices, dexterity, ability to solve puzzles etc. thus creating a single path that leads to one of many outcomes. Thinking about it that way means that there would be a way to make music retain an overall linear structure as it adapts to a series of non-predetermined events.
The idea of a non-predetermined structure in media is one that dates prior to video games and was first explored in storytelling. Some popular books published in the 1930s allowed the reader to choose between multiple endings; these evolved into gamebooks or “choose your own adventure” books that gained popularity during the 1980s and still have fans in the role-playing community. These books contain branching stories with multiple alternative plots. Their narrative was often organised in story fragments. At the end of one fragment, the reader - depending on her choice - is directed to continue reading a different one. One segment can branch out into a multitude of options, eventually leading to one of several possible endings.
Diamantis Sellis 2019
This narrative technique - if well executed - gives the reader a sense of immersion and makes them feel part of the story. It is still used today in many story-driven video games.
In music, the idea of composing and reorganising fragments had been explored during the 18th century. In 1792 Nikolaus Simrauk (Mozart’s publisher) released a book entitled Musikalisches Würfelspiel (German for "musical dice game"). In this book, segments of music are organised in a table - the player has to roll a dice to determine a succession of single measures to be played. He would eventually end up with a randomly generated Minuet or Trio. Here is a website on which you can play the game: http://www.playonlinedicegames.com/mozart
This youtube video from the channel ComposingMusic.net illustrates how the game works: Figures - Many - Difference - Mozart's Dice Game
For this game, the musical outcome is randomly determined, however, it is easy to realise how an audio programmer could decide to use a similar method to create a system that adapts to multiple game events or states. A composer could use Mozart’s method to carefully produce fragments of varying lengths that can be combined and use the branching method to make the final piece adapt to the multiple paths a video game offers. Famous game composer Winifred Phillips called this technique ‘Horizontal Re-Sequencing’ in her book ‘A Composer’s Guide to Game Music'. Here is an illustration of how it could be applied to an enemy encounter in an action/adventure game or RPG.
This method is - in my opinion - the most effective way to deliver adaptive music. It makes it retain its primarily horizontal structure, allowing the composer to effectively use harmonic and rhythmic development. However, it can be difficult to implement as it comes with a few challenges and drawbacks:
Composing multiple fragments that can be combined, shuffled and can branch out to several other fragments requires a lot of compositional skill and organisation.
Not all fragments will be able to seamlessly combine, forcing the composer to compose additional segments that act as transitions.
Even a short piece may require the creation of a large number of music files, therefore, significantly increasing the composer’s workload. Good workflow, effective file management and large disk space will also be required.
Some adaptive systems can become very complex and lead to musical limitations for the composer such as forcing them to remain fixed to a single key or harmonic progression.
The composer needs to be able to design and keep track of an adaptive music system.
The composer will often have to learn to use technology that allows them to implement their adaptive system into the medium they are working with.
These challenges show how much the modern composer needs to be skilful in both music composition and audio technology.
Youtuber Scruffy made this incredibly detailed video deconstructing the boss battle pieces of the game Pikmin 3. The music composed for the ‘Armoured Mawdad’ (first boss in the game), is a perfect illustration of how the technique discussed above can be used. It also shows how a composer really needs to hone her craft in order to write complex musical ideas adaptively.
Scruffy, 2020
II. Layered adaptive music.
One other commonly used method - often in conjunction with the linear adaptive approach - is to rely on the addition and/or subtraction of vertical layers. In its simplest form, this technique could consist of exporting a piece as multiple Stems (Stereo Mixes) of the different instruments or groups of instruments that, together, form the full arrangement. An orchestral piece could be separated into Strings, Woodwinds, Brass and Percussion. The multiple layers could then be arranged in different combinations that match the intensity of gameplay. Although easy to implement, this method presents a couple of limitations:
Adding a new instrument every few bars to give the illusion of development and intensity is the oldest trick in the composer’s handbook. It has worked for centuries and still works today, however, it is only effective for a short duration and cannot be relied on as a long term method.
No matter how many layers of music are created, if the underlying rhythm and harmony remain the same, the piece might start to feel repetitive.
The music might lose its meaning if some stems are taken out of the mix or mixed inappropriately.
A composer, therefore, needs to write with the underlying knowledge that the stems are going to be played in multiple configurations. Some planning will be required and decisions need to be made regarding the role of each stem, their instrumentation, harmonic, melodic and rhythmic content. It might also be a good idea to make sure that each stem is playable as a standalone piece and combinable with any of the other stems, giving complete freedom to whoever is going to manipulate them.
Here is Winnifred Phillips’ vertical system for the game ‘The Maw’
Winifred Phillips, 2014
Notice how each layer contains at least one rhythmic, one harmonic and one melodic instrument. These combinations would allow the stems to be used individually as well as combined.
Moving forward, a composer or developer may come up with creative ways to work with vertical layers. One idea could be to use layers of the same piece of music but arranged with different instrumentation. Here the layers are not supposed to play together but rather fade in and out of each other. Many Nintendo games have featured this trick over the years. Here’s a relatively recent example from Mario Kart 8:
Notice how the instrumentation changes as the player drives through the thunder cloud. This method is easy to implement as the two tracks simply need to play simultaneously on the same timeline and crossfade.
A more advanced technique would consist of mapping and aligning the harmonic rhythm of each stem and then figuring out where some gaps could be filled. This method can allow a composer to create vertical layers that add complementary harmony, giving the sense that the music is moving forward. Here’s a vertical system that composer Chance Thomas used when working on the score for Dota 2:
Chance Tomas, 2016
Here, each new layer doubles the tempo and the harmonic ‘wholes’ of the previous one are filled with new chords. The core harmony remains aligned, making sure that the layers can be crossfaded from any point.
When working on ‘My time at Sandrock’ I used a similar method. The idea we had for this life sim/RPG was to create a contrast between the vivid atmosphere of the main town and the openness of the wilderness that surrounds it whilst keeping transitions relatively seamless for the player. The wilderness layer would be playing at all times and the town layer would fade in on top of it as the player approaches it. This meant that the musical material playing in the wilderness layer would need to be able to match or complement whatever would be playing in the town layer at any point. This method led to unique polychords and created interesting harmonic colours.
Claude Ruelle, 2022
We also used the vertical approach to make certain parts of each theme more modular. For example, melodic lines were given 3 or 4 variations that can be swapped based on the developer's need. This was also done in an attempt to avoid repetition and ear fatigue.
Conclusion
These are the most common ways composers and developers have managed to make music adaptive, however, many more techniques have not yet been discovered. As gaming technology gets more advanced and future interactive media is being created, it becomes more and more important for the modern musician to understand how to create songs that adapt to a wide array of possibilities and outcomes whilst retaining their integrity. It is not impossible that, in the near future, music will be able to adapt in real-time to people’s social interactions, environment, activities, mood etc.
Resources:
Exploring the story structure of branching books - Diamantis Sellis, 2019: https://scholar.google.com/citations?view_op=view_citation&hl=en&user=3JyzlaoAAAAJ&citation_for_view=3JyzlaoAAAAJ:bz8QjSJIRt4C
Musical Dice Game:https://imslp.org/wiki/Musikalisches_W%C3%BCrfelspiel,_K.516f_(Mozart,_Wolfgang_Amadeus)
Adaptive music techniques - Brown, Andrew & Kerr, Thorin (2009): https://eprints.qut.edu.au/29978/
Youtube series on adaptive music by Kajero: https://www.youtube.com/watch?v=G4ileEPzfpA&list=PLNpTJ4IJdJqfio1aAwphtAIczWYs59op3