Soundgarden

Soniferous Gardens

"A garden is a place where nature is cultivated. It is a humanized treatment of landscape. Trees, fruit, flowers, grass are sculpted organically from the wilderness by art and science. Sometimes the garden is drastically clipped and manicured, as in the harshly classical gardens of Versailles and Vienna; elsewhere man has restricted his touch to assisting certain characteristic features of the landscape to flourish. A true garden is a feast for all the senses." - R. Murray Schafer, The Soundscape, p. 246.

Creating a soniferous garden, an "acoustically designed park" (Schafer, p. 247), requires awareness and understanding of the given immediate sonic environment. Its acoustic qualities stem from a manifold range of features, including geometry, materials, time of day, weather, and so on. In principle, we can model and control these features using modern game engines, which is exactly what I will do in this project.

This project is an attempt to adapt the concept of the soniferous garden into a shortform videogame, while also testing out some hypotheses from music-emotion research regarding the effect of certain musical structures on emotion.

Adapting the concept

A quick Google search on "soniferous garden video games" lead me to this video of a VR installation:

Soniferous Garden: Procedural Music Demo

According to the artist's website, the project features field recordings from the Yellowstone and Badlands National Parks paired with generative music. I find it interesting that the artist incorporated generative music, although the reasoning behind it is not mentioned. However, a case for generative music in a virtual adaptation of a soniferous garden can certainly be made if we look at Schafer's examples, where he explicitly mentions water organs and wind harps. I will interpret these as forms of generative music machines, powered by the elements, where water and wind become the streams of data that generate an ever-changing flow of music. Therefore, the idea of incorporating generative music into a virtual soniferous garden is reasonable. (On this note, it would be interesting to conduct a deeper investigation into soniferous gardens in video games/soniferous gardens as video games in the future.)

Running into a (game) jam

Originally I had planned to put together a simple proof of concept, developed over a weekend in game jam style. I wanted to:

1. Upgrade my Procedural Music System
2. Build a game design around the music system

This did not work out as planned. I simply put too much on my plate here, trying to do two major things at the same time. Also, I realized it had been a while since I created a game from scratch. For the last two years I had been mostly involved in developing ongoing projects. I've become rusty. I have to make sure that all required tools are ready to use during the jam. As one of the two main tasks is also one of the fundamental tools, I now understand I need to focus on the tool first. My goal is to work on it until it can generate 'pleasant-sounding' music.

Nevertheless, here is a screenshot of the first attempt: Musical Pac-Man? 🤔

Research

I watched this video on the soundtrack of Mini Metro, showing how the game state and game events can be translated into musical pieces that are both pleasant and meaningful.

Serialism & Sonification in Mini Metro (GDC 2018)

Meaningful because the resulting music is a representation of what is happening the game: Trains moving between stations, new stations popping up; the tempo of the music is derived from the length of an in-game hour. These are only a few examples, all providing inspiration for this project regarding adaptive and generative music techniques.

FRACT OSC was a big inspiration for me and put me on the trajectory of making games and making music with/in games. Therefore, it made sense to take a look at it for this project. One object seemed especially interesting to me in the context of the soniferous garden:

Fract Osc 2024 01 31 12 16 26

This beam of light feels alive. It has a pulse that triggers movement and sound, and it reacts to you with a filter sweep, as you move closer to and away from it. I would like to have something similar to that in my garden.

Building the music system

I am using ProcMu, a procedural music system I built for my master's thesis, as a foundation for the music system in this project. This provides me with a basic structure that I plan to improve on by adding more sophisticated generative algorithms. Instead of CSound, which ProcMu originally used for music sequencing and sound generation, I am now using a Unity plugin called AudioHelm. This removes the step of translating parameters and sending them to CSound. Further, AudioHelm comes with a versatile synthesizer that can produce some decent quality sounds.

I will add documentation for the music system in this project as I work on it. Until then, here is the documentation of my master's project which contains information about the different components of ProcMu, many of which also will be found in the upgraded version.

More about the development of the music generators to come in a future entry.

Upcoming

Next steps:

• Finish implementing generative algorithms
• Design distinctive sound settings for each area/music zone

Further design considerations:

• Synchronize footstep sounds with the sequencer. Does the player then also become a part of the soniferous garden? Would be interesting from both game design and theoretical perspectives.
• Place dynamic sound objects in the environments, possibly also synchronized with the music, similar to the example in FRACT OSC above.
• Create a music-emotion interface that maps valence/arousal to music configurations, each representing an emotion on Russell's scale.

Mapping things out

The picture above shows the first draft of the game environment, an island with three distinct areas:

• Swamp (middle, green)
• Labyrinth (bottom right, blue)
• Hill (top left, red)

The player is confined to these three areas. It is not possible for them to wander the empty spaces near the beaches. The idea was to use height differences to maximize the distance between areas while keeping the footprint small and paths short. Further, I hope to create a sense of scale and make the map appear larger than it actually is. Another option would be to go fully procedural, generating a map, placing points of interest randomly. This is still a possibility for the future.

Since this soundscape-based game takes place on an island, the player should have some way of interacting with the sound of the waves. Therefore, in the second version of the map, I removed the labyrinth area and walls, making the beaches accessible.

Configuring music

Below you can see the music configuration object I created in Unity, holding various generative parameters.

The generation of rhythm and melody is currently separated from each other. Rhythm, or, when a note is played is generated [using a Euclidean algorithm](banff.pdf (mcgill.ca)) . For the melodic part, i.e., pitch, I implemented several approaches that I aim to test in terms of output quality and fit, since it is possible that some techniques might work better for certain musical tracks, for example bass vs. melody. Melodic parts are always derived from a global musical scale, ensuring harmonic unity. Here is a brief overview of the generative techniques I used.

• Random: A simple algorithm that just randomly selects notes from a given scale.
• L-system: Generating melodies from arbitrary grammars.
• Arpeggio: Follows the scale in a specified direction (up, down, and combinations).
• Perlin: Using Perlin noise to generate continuous melodic lines.

Despite all the work up to this point, I felt like I had been missing a sense of direction for this project. I knew roughly where I wanted to be but could not find an exciting and interesting path to get there, getting stuck with too much technical implementation and too little conceptual incentive. However, all I needed was...

A hard reset

Wednesday, February 14, 20:55. As my Cheetos-stained fingers type these words, I finally got a sense of direction. My original plan for today was to test a build of the project, to see if the audio system I was using would actually work. I never really doubted that. In the worst case, a Windows build would always work.

Welp, that was not the case, meaning most of the implementation work of the past weeks was for nothing...?

Not quite. Enter again, Unreal. I had originally planned to use it, mainly because its MetaSounds system is the only real-time audio solution that is integrated within a game engine, and quite similar to PureData or MaxMSP that I have extensively worked with before. However, I didn't feel confident enough to use Unreal on a new, time-constrained project. What mainly stopped me was that I had to create a generative configuration interface, as seen above, and I just couldn't wrap my head around connecting the Unreal GUI builder and MetaSounds in this short time.

However, I found there are simpler but no less effective ways to control the music. Everything else was actually quite easy to do, after I had already "practiced" using Unreal for a few months. Working on the Unity project had helped my idea to become clearer. Within two days, I was able to whip up a basic set up that would resemble what I already had in Unity, music and all, and then some, all without implementing a GUI.

Reducing complexity

I solved the music configuration issue by moving away from a monolithic structure. Instead of controlling all parameters in a central system concurrently, I'm going to implement different modules that affect individual music parameters. For example, one module could affect rhythmic patterns, based on how the players moves around, another could choose a scale based on the player's location. In addition, "sound landmarks" that are placed in the environment supply various areas with a sound signature, meant to facilitate player orientation. It is much easier to set up a principal musical structure that is supported by several environmental sound objects, instead of creating and assigning whole sets of configurations per area.

Natural/diegetic sounds?

I have paired these two words to complement each other, only to realize a moment later that diegetic and natural seem almost diametrically opposed to each other. Diegetic is often used in the context of something artificial (i.e., not natural), a human-made medium. For instance, music in a movie can be diegetic and non-diegetic, coming from on- or off-screen. The same can be the case for music in games. A sound may be defined as natural in its diegetic context, such as the howling wind that blows over a meadow, rustling the grass.

To overcome this opposition, let's frame things. Games are an interactive medium that simulate dynamic worlds, i.e., virtual environments. Since the given virtual environment represents the totality of what exists in that context, would that not have a resemblance of nature? What if we assume this virtual totality as nature, and therefore, all the entities it holds as the natural foundation that a soniferous garden is build upon?

Natural sounds

My concept for environmental sound objects is to combine two sound layers: The first layer would aim to emulate a sound one would generally associate with a natural origin. For example, using noise to generate something that represents waves, or wind. The second layer would present a melodic sound. Melodic sound layers could be harmonically linked to an overall musical scale. Any melodic and rhythmic aspects, both of "natural" and melodic sounds, could be synchronized with a sequencer. This sequencer would control both diegetic and non-diegetic music/sounds. Some of these objects can be the aforementioned sound landmarks, others may hold less meaning but could enhance the environment with dynamic and playful behaviors.

Reflecting on reflection

Tuesday, February 20, 23:20. These things are hard to write. I feel like there is so much (many different concurrent processes) and so little (overall process is incremental) happening at the same time that it becomes difficult to tie the different directions of thought together. This writing is not only meant as a reflection on the development process of my unnamed semester project - work title: Soundgarden - it is also a reflection on how to polish my writing to the point that reflection becomes clearer. Originally, I had wanted to put all the above text into one coherent story before I decided to roughly separate it according to the featured themes and chronology. That seemed to have made the text a lot more coherent already. I would expect that further writings will help in improving coherence and quality further. But is coherence and quality all I want in a text? I feel like I am constantly forgetting relevant things, since I'm dealing with so many topics. So there should also be some form of completeness, no step left out in documentation. I think that writing a section at the end of a work day could help improve my process.

Music heard while working on the project

I often listen to music while I work. I am putting together a playlist of fitting and inspiring music that I come across as I work on the project. The tunes in the list at the time of writing this feature repetitive structures that may lend themselves well to be adapted algorithmically.

Here it is. It will grow as I continue to work on this project.

Some of the featured tracks were at least partially recorded using modular synthesizers. There is a striking similarity between modular synthesis, MetaSounds, and pretty much any visual programming system but that's a topic for another time...

Upcoming

Next steps:

• Make the music pleasant to listen to
• Implement gameplay: Hiding and finding objects, the overall music is meant to affectively influence players, environmental sounds may hold clues about the hidden object's positions.

Further considerations:

• How does the project connect to Schafer's Soniferous Garden?
• What principles of acoustic design apply here?
• What does it mean in the bigger picture of game soundscapes and how does it relate, or link back to real environments?

Recap

What I did since last time:

• Populated the world with assets
• Refined the gameplay
• Designed more musical elements
• Connected gameplay to generative music parameters
• Further developed the area-specific sounds
• Conducted research into soundscapes and acousmatics

Iterating on gameplay

The original gameplay loop is as follows: An object is hidden in the environment. Whenever the player finds and touches it, the object moves to another location, waiting to be found again. As a pseudocode-loop it would look something like this:

while(true){
	Game.SpawnObjectAtRandomLocation();
	while(!Player.HasFoundObject()){
		if(Player.HasFoundObject())
			break;
	}
}

This mechanic aims to give the player an incentive to move through the game environment. There is no time limit to find the object, allowing players to engage with the garden at their own pace. I also consider adding an option for players to disable the hidden object game.

Moving around and looking for an object is all the player does. This seems quite boring to me. Even though I hope that generative music will make gameplay less boring, I'm afraid this might not be enough to be engaging.

I ended up replacing a glowing collectable orb with stones that blend into the environment. Because of their smaller size and ability to be hidden in plain sight, the player will likely need to pay more attention to their surroundings. This also provides an opportunity to employ the affective qualities of the generative music. Because the rocks are not easily visible from afar, musical guidance becomes more important. When the player gets closer to the object to find, the music could turn to a style designed to slow them down and look more closely. Also, I'm planning to add a deliberate 'pick-up' action, so the player does not accidentally collect an object by walking over it, without having actually consciously found it.

Barriers

I realized that giving the player access to all parts of the island from the beginning might be detrimental when it comes to finding small objects that are scattered around.

Instead, I implemented barriers that disappear one after another, depending on how many objects the player has found. This way, players are introduced to the environment gradually, which might help facilitate stepwise familiarization with the different parts of the map. Also, this will give a purpose to collecting the collectibles.

Expanding the accessible area will also expand the area where the collectible can spawn. I'm wondering whether to remain with spawning one collectible at a time, or several (maybe even lots??...). It could make sense to have several objects concurrently, also considering they are quite small. Finding the needle in a haystack doesn't seem like a fun activity to me. However, if there is more than one collectible, then where should the music system guide the player? To the closest collectible? If we have lots of rocks, the general area where most collectibles are?

Once the player has access to all parts of the map, the stones could be used for something else. For example, maybe the player can thrown them in the ocean, once they gain access to the beach. Maybe there could even be specific actions for each area?

Mapping

When I switched the project over to Unreal, I had to come up with a new map. It still follows the same concept, however, Unreal's spline-based mapping tools help achieving a generally smoother outcome. Hoping that tinting the world in a certain style would inspire some musical ideas, I focused on the visual design.

The unity map had a swamp in it. In this new version, it became a sea cove since it was easier to set up with Unreal's water body system and I didn't have any suitable assets anyway.

However, once I had figured out Unreal some more, and found a suitable free asset pack, the swamp returned.

All the rock geometry in the map comes from a single asset, in various positions, scales, and rotations. At first, my dressing up of the terrain looked like blatantly obvious copypasting, which I aim to either fix or cover up with some vegetation.

As my technique was improving throughout the process, however, things ended up looking much smoother. Notice the difference between the picture above, or below on the left side, where I had started, and the rocks below on the right, where I finished covering the island.

If I were to use a texture mapping technique that depends on the world position of the object, and possibly uses some form of randomization, I can make the texture repetitions disappear, and the asset repetitions less obvious. At the end of the day, it's just shapes.

When mapping, I like to consider the the logic of the environment. How did the water get into the center of the island? How does it stay there?

I'm imagining that occasional rainfall, along with a rock structure that isolates the body of water in the center of the island from the sea lead to the emergence of the swamp. Of course, the shape of the island and the distribution of the objects and paths may suggest an influence beyond the natural.

So let this island exist at the threshold between the real and the magical (which is where games operate anyway, right?). Maybe that's something I can play with in future iterations. Thinking of colorful fog and psychedelic weather phenomena.... Oh and maybe a day/night cycle...

Simplifying the sound setup

Since the beginning of the project, I was struggling with controlling the overall musical outcome. My approach was to have music configurations, i.e., sets of parameters, that would define how music would be generated in each area of the game world. This proved to be difficult, since there are many parameters. Also, I'm not familiar with Unreal's GUI building tools enough to implement a complex UI. The solution? Influence individual generative music parameters, rather than all of them at once. Components of the system should be able to influenced and influence each other without relying on some monolithic master structure.

Beyond that, I decided to use some pre-recorded or -composed elements, to aid with generating a more coherent and possibly pleasant-sounding outcome, without having to worry too much about the complex task of coming up with a fully generative solution. This also opens up the possibility of high quality samples that could act as a 'filler' in the music, bridging the gaps between sounds while increasing overall sound quality. I am still implementing generative approaches wherever it makes sense without proving to be a technical burden and might exchange non-generative elements with generative ones as I work more on the music.

I have also looked into mapping player actions to variables which can then be used as parameters to control the music system. The game should check whether the player is walking to switch between "walking" and "resting" themes. Such themes could come in form of specific drum patterns, activation of some instruments, or by affecting multiple generative parameters in order to generate music that sounds more intense.

Avenues of music-emotion research

What does it mean to have 'intense' music? Thinking about how to use music to affect player behavior, I turned to music-emotion research in order to find initial approaches.

I'm using Russell's Circumplex Model of Affect, where emotions are defined by valence, i.e., positivity (happy vs. sad), and arousal, i.e., strength. By musical intensity, I refer to a combination of valence and arousal. This simplification is informed by the fact several studies in music-emotion research showed increases/decreases in both valence and arousal dimensions triggered by a single feature. For example, research showed that a staccato articulation (many short notes) would yield both a perceived higher valence and higher arousal response vs. legato articulation (long, connected notes). I will use research such as the one linked as guidelines for designing the music to purposefully affect the player's emotions and behavior, aiming to help them find the collectibles quicker.

On working with MetaSound

One of my approaches in developing sounds is to design and ear-test melodic and rhythmic patterns, as well as sound designs in external software and then recreate them in the engine. While Unreal's MetaSound offers some capacity to design sounds in real-time, changing the connections between modules, adding/removing modules, or introducing new variables will prompt a reset of the music system and interrupt audio playback. Designing sounds in real-time without interruptions and reloads within MetaSound requires the setup to be mostly complete, using float variables as inputs, as Unreal allows for these to be changed via knobs in real-time.

MetaSounds is quite similar to a Eurorack system, or at least, this is how I see it. This view allows me to approach a musical task in a technical setting while remaining within a musician's mind set.

When it comes to (electronic) music there is a tradeoff between working with hardware and software. With hardware it's much easier to design a sound. Turning a real knob is more immediate than using the mouse to click and turn a virtual one. The tradeoff, however, goes both ways. With MetaSounds, whenever I find myself in need of a certain sound, I can build the module to generate it, something I often do whenever the need for a particular function or sound arises. With Eurorack hardware, I can't just solder together a new module and buying modules can be prohibitively expensive!

Upcoming

Next steps

• Create more connections between player behavior/game state and music parameters
• Add more variation to the music
• Theory: situate my approach of combining natural/musical sounds within the soniferous garden and acousmatics
• Develop a plan to test and evaluate the project

Tutorial time 📖

I made some adjustments to the tutorial and overall gameplay structure. At the start of the game, the collectible stones spawn in the player's view, leading them across the island. As the player collects these initial stones, barriers that unlock access to more parts of the island disappear visibly in front of them. My intention here is to 1. teach the player that collecting stones leads to some outcome, 2. gradually familiarize the player with the layout of the island.

I have also placed an illustration in the tutorial area, showing a cryptic depiction of the stone collecting task that might even add some sense of mystery to this lonely island.

The drawing was generated with ChatGPT, using the following prompt: Create a cave drawing, the drawing only, without a background (which could be plain white for example). The drawing explains the viewer to look for stones and take them.

I hope the tutorial will make gameplay self-explanatory without resorting to text. There's still no explicit pick-up action, and I wonder if it's needed after all. However, without text, I'm not sure how to communicate an eventual pickup mechanic to the player. It could be through an adjustment of the collection illustration, maybe depicting a key that needs to be pressed. Since there is gamepad support, this would mean at least two additional illustrations for the keyboard key and gamepad button.

There's also the issue of adjusting graphics settings. While the game certainly looks nice, thanks to Unreal, it's relatively demanding in terms of hardware. As with the text, I would like to have an in-game menu for that. It would be nice to set graphics settings via a splash screen on game start, so it's removed from the actual game. On that note, maybe this is where I could add an explicit gameplay instruction like "pick up stones with left click/right trigger".

Soundwalking 🚶

I conducted a soundwalk on the island. Not listen to what's there, but rather to what isn't.

I was looking at the leaves that weren't rustling,

I was listening to the wind that wasn't howling,

Saw a swamp as still and lifeless as ever.

I was a ghost that walked weightlessly upon the sand, the rocks, the mud, the dirt.

There's an endless ocean all around - but no sound.

This little exercise in sound mindfulness lead not only to the spontaneous poetic outpour above, it has helped me to come up with a list of sounds to fill the sonic void.

What was already there

• Music
• Ocean: musical waves
• Hill: musical wind

Added after the soundwalk

• Hill: parrots
• Swamp: frogs
• Trees: rustling leaves
• World tone: a subtle pink noise that is always heard. Helps retain an "island feel", even when the ocean is more distant

Still to add

• Player footsteps
• Stone pickup sound

Also to consider

• How sound travels and reverberates through the environment
• How the soundscape changes over time

Frogs 🐸

Whenever it makes sense, I want to combine a natural and musical sound, or connect the natural sound to the music system in a meaningful way. For the frogs, I wanted to pitch the calls so that they're in harmony with the music. For that, I cut apart a sample featuring multiple frog calls and changed the pitch of every frog call sample to correspond with a C note. This allowed for pitching the frog sounds dynamically along the given musical scale. However, testing the sound in-game showed that the pitch of the sound is not clear enough for the harmony to be audible, so I will need to either add another layer with a clearer discernible pitch, or use different frog sounds.

The frogs follow a specific rhythmic pattern: one source is triggered in intervals at the length of one bar in the overall music. Three more sources each use increasingly (minimally) longer intervals. The small differences in intervals accumulate, leading to an ever-repeating moving rhythmic cycle where the sounds continuously run in and out of sync.

Parrots 🦜

Here, I adapted a field recording into a procedural version of itself. I listened to the original recording, analyzing the frequency and duration of birdcalls that would usually appear in clusters. After extracting the individual bird calls from the sample and grouping them into loud and quiet calls, I created a MetaSound patch which would play back these calls in a similar manner as heard in the original sample. With everything being procedural now, played bird samples are randomized. I have also added an intensity parameter that modulates the occurence and duration of clustered calls (intensity 0 = no calls at all, intensity 1 = call clusters consisting of many calls played successively). I want to connect this to an eventual day/night cycle.

Trees 🌴

As above, I cut apart and procedurally recreated the original sample. I would like to connect the actual swaying of the trees, which is currently random and controlled by a shader. To change this, I would need to implement a wind system and adjust the tree shader, which is out of the scope for now. Also, the illusion of an actual connection between the swaying movement and sound is good enough for now.

Soundwalking also made me think about the sounds the player would make, which lead to general thoughts on player embodiment.

Player embodiment 🧘

I was thinking about player embodiment: How to evoke a sense of player presence? How to make the player feel as if they are on the island? There is an inherent act of player embodiment that stems from the interactive nature of a game: moving around the game world, influencing its objects can be seen as acts of embodiment. Isn't it through enhancing and expanding these interactions that we as game designers facilitate embodiment in the worlds we create?

One common way would be the addition of player character sounds. I should add footsteps, and possibly other sounds that would stem from player interaction (e.g., picking up a stone). I'm also thinking about adding microphone support: The player can make sounds which are then shaped by the game environment. This would require a convincing implementation of spatial audio that would benefit any sound regardless. Microsoft Acoustics would be a solution for that. Going to look into that soon.

More thoughts on MDM 📄

I certainly enjoy the newfound creativity that MDM has brought me. I feel like I know what it means to design and make games again, and I love trying out ideas in this context. Frequent journaling and extensive commit messages not only help me get better at formulating my thoughts and tracing design steps, they enable me to actively think about my work. They help me to think ahead, unlike before, where thinking was usually the result of some roadblock, and design merely a reaction.

I also noticed that my thinking is now focused on the bigger picture, and the conceptual background, rather than on specific features and how to implement them. I seem to think in bigger steps: A step is not a task anymore, it is an idea, consisting of several smaller steps. Curiously, this had an impact on my commit pattern. Going from multiple commits in a day, with Soundgarden, there have been commit intervals of as big as one week. Now, I commit once an idea is complete, rather than at every step of incidentally developing it. While I think having the big idea in mind is good, fewer commits are not - overnight data loss is real! And frequent commit messages just facilitate more thorough documentation. I should do commits more often again, even if that means an idea has not been completed yet.

Another thing I noticed that may be due to MDM. I implemented some code for spawning the collectibles which I think I would have implemented in a more efficient manner had I used my agile Trello board. I do think the board helps thinking in terms of more traditional software design terms, which can be handy to know sometimes. Working on Soundgarden, I had a moment where I realized that using a certain data structure would simplify a solution I had implemented. Here, technical knowledge helped to implement a gameplay idea in a technically more efficient way, after I had arguably spent some time on a less efficient and slightly broken solution. However, I find that MDM actually facilitates thought that would lead me to this conclusion. Even if MDM was the reason for spending a little more time than usual on one thing, it also helped me to arrive at this realization in the first place. It is certainly possible to think about good technical implementation whenever it becomes necessary. This is essentially an inversion of my Trello workflow where I would stop and think about design whenever something technical would not be working, whereas now I think about the technical whenever it is in the way of design/gameplay. This makes for a much more creative workflow and more interesting outcomes. MDM rules!

Further thoughts 💭

I was thinking about a game mechanic where the player can speak basic syllables to interact with certain objects in the game environment. I was also considering having combinations of syllables as passwords/magic spells that would, for example, open a pathway or activate some musical object. There should be some way to input syllables without speaking - I could map AEIOU to keys on a keyboard and buttons on a gamepad.

The character controller has jumping enabled by default. I placed some invisible walls so the player won't go for a swim in the ocean or swamp. With some clever jumping in the right location, it's possible to circumvent the tutorial barriers without collecting all stones, and reach other locations as well. I could disable jumping and add even more barriers to prevent this. However, I would much rather reward the player for their curiosity and exploration. I could place some sort of easter egg in such locations, or even a special type of stone collectible that could unlock some more "out there" consequences, like turning the sky green or letting it snow.

Next ⏭️

• Check Microsoft Acoustics and microphone support
• Implement footstep sounds
• Connect environmental sounds with music system
• Add day/night cycle
• More musical frogs/swamp
• Add more things that happen when player collects stones

In this entry, I will focus on the theory behind Soundgarden and outline my approach in adapting Schafer's acoustic design principles, explaining some of the reasoning behind the sounds and systems that I've been silently adding over the previous weeks.

Principles of acoustic design 🔊

Schafer mentions four principles of acoustic design (The Soundscape, p. 238). I have summarized them into single terms and will explain them in the context of my own work with Soundgarden.

Composition: Sounds in a game environment should be chosen and arranged such that they contribute to an immersive and harmonious soundscape, while respecting the player's auditory threshold, i.e., avoiding harmfully loud or imperceptibly quiet sounds.

Meaning: Sounds should be more than mere functional cues. A sound may already hold symbolic depth or may be imbued with it when integrated into a game environment. This may allow for additional emotional layers, and enhancement of existing narratives. Sounds may resonate with players when using auditory symbols, evoking memories, emotions, and personal connections.

Timing: Sounds may occur at different points in time, constantly overlapping or alternating with each other. A sound may have its own rhythm and tempo, and/or may be controlled by the rhythms of other actors and systems in a game.

Self-regulation: Sounds should be capable of moderating themselves to prevent auditory overload. This is especially important when a sound is driven procedurally, where a range of outcomes, including extreme ones, is possible.

What I find interesting about these principles is that sound may be substituted by or combined with other embedded media. These principles appear to fit a systemic view of a game: Schafer speaks of acoustic ecologies, whereas Soundgarden constitutes the ecology of a game environment. I'm not using the term ludic ecology here, since it suggests too much emphasis on the process of play, whereas I see a game environment as playful, in terms of an interplay between all these different elements that comprise it.

Just like bird calls may occur in certain patterns during specific times, marking a location, regulating themselves not to dominate the soundscape, other game events and outputs of various systems may do so as well. Looking at game making through the lens of acoustic design therefore allows me as a game designer to have a systemic - i.e., ecologic - approach to creating a world and its interactions.

From reality to simulated reality by applying observed patterns 🔎

Real ecologies and their phenomena can provide us with pointers for systemic design.

The figure below (Schafer, The Soundscape, p. 231) depicts the audible activity of three different animal groups over a 12-hour period. The curves which are derived from observed animal activity resemble curves that game engines use to set continuous values.

We could now naïvely copypaste these curves into the game engine and have them control the activity of actors directly. However, to adapt a real-world phenomenon, one must understand the relationships that led to its emergence. Specifically looking at the bird activity, there is activity starting at approximately 3 AM and ending at around 10 PM, which appears to correlate with sunrise and sunset. This leads to the conclusion that it is not the time of day that affects activity here (and one does not need to be an ornithologist to know that birds don't wear watches) but rather brightness, affected by the increasing and decreasing intensity of sunlight throughout the day. Instead directly controlling bird activity by a 'birds' curve, the birds are programmed to observe the intensity of the sun. In informatics, an observer pattern is a common solution for having multiple entities observe a certain behavior and run some code when the observed entity dispatches an event. In a way, the optimal technical solution here mimics a real-world behavior. The introduction of sun intensity as a systemic factor may in turn inspire more thinking about other entities in the game world that could be affected by its variations.

On Normalization 🎚️

Whenever a logical component - for example a module in my music system, or the day night cycle indicating sun intensity - uses float parameters for input or output, I map them to a range between 0 and 1, also referred to as a unit interval in mathematics. Not only allows this for easier connection of different components because values don't need to be remapped, it allows for "logical compression", i.e., deriving meaning from a value and its name. 0 will always be the least of something, or off, whereas 1 will always be the maximum, the most intense, just like the parrot calls I described in the previous post. If all parts of the music system, as well as game logic, use normalized values as in/outputs, connecting the individual components becomes trivial. This may facilitate some experimentation - one might try connecting systems for interesting results. What if the sun intensity does not just affect activity of animals? For example, player movement speed could be in proportion to sun intensity, affecting how the player moves and possibly how they perceive their environment, with slower movement perhaps leading to more attention towards the immediate environment... Some connections may lead to profound impacts on gameplay.

Rethinking the music 🎼

Constantly playing background music seems to closely resemble the act of moving through the world with headphones in your ears, possibly shutting off the world around you. Soundgarden is about listening to the interplay of sound and silence, and gathering meaning from it. As of now, music is playing constantly. While I don't think that having constant background music is necessarily against the principles of acoustic design, I do think that its current implementation does not leave enough space to listen to the environment. To address this, and because I think it will lead to more musical variation overall, I have decided to move over the 'extradiegetic' sounds into the world. I'm also thinking of introducing sounds that seem to come from inside the game world but that can't be properly localized, inspired by my own experiences with low-frequency hums (off-topic, but here is an excellent video on these mysterious low-frequency hums that occur in our world). Anyway, bringing more music into the world will mean that I'm going to have to come up with new objects that emanate sound. For those, I will likely move into a slightly more surreal direction. Looking back at some inspirations I described in my first post.

Ch-ch-ch-ch-changes (and additions) ➕

Below is the original layout of the island that I created when I switched to Unreal.

In this version, the island is encircled by a beach, which the player is able to walk on, giving them the possibility to walk all around the island. Collectibles can spawn here as well, which turned out to be a bit of an issue: While the elevated ring path around the swamp and the hillside outlook (southern yellow circle) allow the player to overlook the eastern, northern, and northwestern parts of the beach, this is not the case for the southern and southwestern parts. Furthermore, if the player wanted to get to an elevated position to look for the next collectible, there is only one access point to the inner part of the island, which is at the intersection between the beach, elevated path, and swamp. Testing the game, I realized this led a lot of walking, especially when circling the island via the beach, only to find the next collectible elsewhere.

Addressing these issues, I massively reduced the size of the beach, with only a relatively small patch left in the east of the island. I then added two new areas in the south and west that connect the beach to the elevated pathway. The new area in the south is a cave that can be accessed from the beach and leads to the new area in the west, a rocky path along the ocean.

I also chose to add a cave as a type of environment because it contrasts the outdoor areas. A cave provides opportunity to implement a noticeable reverb, affecting sounds and music. I have also scattered colorful lights around it that I plan to synchronize with the music. I'm also thinking about an unlockable event where paintings appear on the cave walls, depicting the progression and pitch of an audible melody.

The cave leads to a rocky path along the ocean, which leads back to the elevated pathway around the swamp. These changes in the map layout allow the player to reach other parts of the island quicker and reduce overall walking. When it comes to finding the collectibles, the cave and the southern part of the new ocean path cannot be seen from the hillside outlook. However, since these are the only areas that can't be seen from the outlook, the player should be able to deduce that the next collectible is likely in one of these areas.

This new and improved map layout facilitates movement much better than before: The hillside outlook is intended as a point of rest and reflection, facilitated by the fact that it's a dead end, even though the cries of the parrots during daytime might get annoying after a while. All other paths eventually circle back to themselves. The player can essentially walk in several circles, never stopping on their way to find the collectibles.

Apart from changes to the map, I have also added a light orb which appears above the player when it's dark outside, illuminating the way. The orb follows the player with a slight delay. This means that whenever the player is moving, the area in front of them is not as bright as when the player is static and the orb is above them. The reasoning behind this delay is that the diminished lighting during movement may help find the collectibles in the distance better, since they are always slightly lit up, which could be missed if the player's immediate surroundings are too bright. On the other hand, standing still with full brightness above allows players to see their immediate environment clearly if they want to focus on it.

Furthermore, I have added some fireflies to the swamp area, which appear when it's dark. In addition, I have begun to add some 'filler' sounds in various areas of the map, to make the soundscape sound more lush overall. These sounds are generally less involved in terms of implementation, and rely more strongly on playing back samples without some procedural logic behind it. Some sounds are audible to different degrees, based on other factors, such as sun brightness (similar to the animal sounds).

Playing with acoustics 👂

I added the Microsoft Acoustics plugin to the project. This plugin is essentially the equivalent of a light baking solution, only for audio. It simulates the real-world behavior of soundwaves traveling through space, including occlusion, attenuation, sounds reverberation and bouncing off of surfaces. I hoped this would simplify remove the need to set all these parameters by hand, while providing a realistic and much higher fidelity spatial audio. On the downside, baking a scene for acoustics takes much longer than baking lighting. I performed one bake which took three days on my desktop PC. Unfortunately, the results were underwhelming, with no discernible reverberation and errors in sound occlusion. I could've fixed these issues by preparing the bake better, however, I didn't want to risk losing any more time on this.

Using MS Acoustics could've especially been cool with added microphone support: Imagine the player walking through the environment, speaking into the microphone, their voice reverberating through the environment in physically accurate ways. It would sure have added to the aspect of player embodiment!

Once the academic deadline for this project has passed, I might return to this. Until then, I will fake sound propagation using the usual engine tools, creating custom attenuation curves, setting up occlusion, and using impulse response-based reverberation. Impulse responses from real-world environments can be used to simulate the reverberation characteristics of real-world environments. By taking impulse responses from real environments that resemble the individual areas of the island, I can at least create some sense of how sounds would travel were this a real place.

Footsteps...and more? 👣

On the topic of player embodiment, I have implemented a system for footsteps, including support for different surface materials. While the system itself works, I'm still having trouble dealing with the intricacies of UE5, as I'm still learning. Therefore, the footsteps don't work properly yet, however, I'm confident it will all be fine by the time I submit this project in two weeks.

Anyway, unlike what I have often seen in other game projects, I'm using two audio sources for the footsteps instead of one - one for each foot. This way, each step sound would come either slightly from the bottom left or right of the player. In situations where the player might be walking on the edge between multiple surfaces - for example, one foot might step onto sand, and another foot would step onto water - this would likely better reflect the player's immediate environment.

I was also thinking about expanding this approach to other body parts, thinking about shoulders, arms, or hips brushing against close surfaces, for example, when crawling through foliage, or squeezing through a narrow opening. I don't plan to implement this immediately but I'd certainly like to experiment with this sometime in the future!

A few more steps ⚒️

• Add more things that happen when the player collects stones
• Add a musical stinger whenever the player collects a stone. It could change, according to how many stones the player has already collected
• Move more musical elements into the diegetic space
• Add a visualization in the environment that depicts how many stones the player has collected

Post? Mortem? 😵

A bit over a month has passed since I finished the first iteration of Soundgarden. Here is a quick list of the major systems and aspects implemented. Most have potential for improvement.

• Adaptive procedural music system (music output needs improvement still)
• Custom attenuation settings for different sound sources
• Local reverb, with different settings per area (implemented but not quite working yet)
• Gameplay: Collecting stones (should provide more feedback upon collection)
• Rewards: Events happening when the player collects certain amount of stones (still need to add more)
• Day/night system (sun movement is still janky)
• Footstep system (some bugs still occur)
• Audible actors with dynamic parameters
• Music speakers (with attenuation/virtualization issues)

The music speakers were a spontaneous addition. I was looking for ways to play back the different musical layers through the environment. I found some assets on Quixel Bridge. A horn for Sound_Kait and a metal bell, which I use upside down, for Sound_Gamelan.

![](https://github.com/Videon/soundgarden/blob/master/docs/attachments/Pasted%20image%2020240609120053.png

So far throughout this project, it happened many times that the assets that I had available would inform the setting of Soundgarden. Here, I already had sounds and selected the assets that fit them best. A horn for an almost whistling-like sound, or something that may be closer to a wind instrument, and a metal bell to accommodate the metallic sounds of gamelan instruments.

Since finishing up the first iteration, I have worked on various fixes, additions, and optimizations.

• Added a main menu: Currently, the menu is only shown at the start of the game. I have added it to eventually provide the option to change the settings of the game, to make the game playable on weaker hardware.
• Enhanced the music system: I have implemented a Euclidean rhythm module, providing more control and variation in the generated rhythms. I'm now looking to improve the melodic parts.
• Fixing music objects: There are still issues with Unreal's management of sound sources and routing the outputs of the music system. Depending on the configuration, some music objects don't activate if they are outside the attenuation range at the start of the game, or don't reactivate when the player leaves their attenuation range and come back. It appears that some objects may or may not work randomly at times. It seemed that I had fixed the issue but it appears that after updating the project to Unreal 5.4, it returned.
• Optimizing performance: As of now, the only major measure to improve performance was to update the project to Unreal 5.4, which brings rendering optimizations. However, more needs to be done for the project to run smoothly on weaker hardware.

Going forward, I'm thinking about two things:

1. How to improve gameplay and make it fun?
2. What are the artistic/philosophical directions/questions I can explore with this project?

To address the first question, I'm thinking about new interactions to add, both in the form of the environment providing more reactions to the player behavior, and giving the player more ways to affect the environment. Very recently, the developer of FRACT OSC released a commented playthrough of the game. So far, I have only watched half of this four hour long video but the developer is providing some very valuable insights here, which brings some inspiration to help me address question one. Beyond that, this developer commentary might also provide general insights for developing games where sound and music take the lead.

Fract Osc 10 Year Retrospective Playthrough and Director's Commentary

Thinking about the second question, I am currently focused on the duality of sound and its representation. I'm thinking about introducing layers of abstraction. Many 3d software and game editors allow to switch rendering modes, such as showing a wireframe of the scene geometry, or exposing individual rendering steps, such as unlit, or lighting only, reflections only, and so on. These different views, combined with the soundscape that is unaffected by them might put emphasis on the role of sound but also expose the technical underpinnings of the game. I am interested to see how this might affect listeners' perception of the virtual ecology and its soundscape. Related to that, I'm considering a mechanic that would replace individual high-fidelity visual elements with more abstract representations: What if the parrots turn into flying cubes? What if the leaves of the trees, and the trees themselves become camera-oriented 2d sprites? And what if the musical structures themselves expose, deconstruct, and rearrange their generative process, falling apart, and coming together again? Maybe the parrots start singing in saw waves, maybe the ocean waves become slowed down samples of parrot cries. To implement any of these ideas would likely lead to improvements in the music system. For example, I might need to introduce a feature that allows to dynamically rearrange sounds by routing the triggers to any sound in the system.

I moved away from the idea that I want to combine every natural sound with a musical one, because almost all of the music output is now integrated into the game environment, with the exception of percussion, until I find a reasonable way to integrate it into the environment as well. This means that natural and musical sounds area always combined in the soundscape anyway, with the player's position, movement, and gaze constantly affecting the audible composition. Soundgarden is a song that the player can move through and experience from various perspectives. This is in line with Schafer's view of our world as a universal composition.

More thoughts on the process 📝

Especially during the first two months of working on Soundgarden, I was fighting technical issues far more often than anticipated. It began with various issues back when the project still ran in Unity (see 2024-02-14). Then, after switching to Unreal, I would often need to stop and learn a new skill that would allow me to do the things I wanted/needed to do. While I can safely say that I am now much more confident in using Unreal than I was three months ago, I'm also under the impression that these intermittent bouts of forced learning were in the way of developing more creative and experimental approaches for Soundgarden. On the other hand, I wonder if the various workarounds may have also lead to innovative approaches that I would not have though of otherwise.

For this project, I have not only been utilizing commit messages for the first time, I used them quite extensively, making them more like mini journal entries. I noticed how this facilitated a constant reaffirmation and rethinking of my design goals: Sometimes, I would realize how to implement a feature while formulating my intentions, other times I would begin to refine existing ideas as I was describing them. With every commit, I now try to write a corresponding message, even if the commit title is obvious, since it may trigger further thought and refinement.

Things that I still want to do ⚒️

This is a list of things I put together at the end of the development of the first iteration.

• Reverb: Due to some last minute tech issues I couldn't finish the reverb implementation. Everything is in ready to go though, since I have already placed all volumes, made component configs and found impulse response for the different areas. Just need to figure out how sub-mixes work in Unreal.
• More ecology: I would like to explore more generative relationships by adding more elements which react to global changes and influence each other, beyond just checking sun intensity. For example, adding more ways audible entities influence music in some way (like the parrots), or make something else appear/disappear.
• Connecting the ocean and wind system: High wind intensity should lead to higher waves, low wind intensity to lower waves.
• More visual effects: For audible entities that are not visible, such as the frogs in the swamp, I would like to give them more presence, not necessarily by adding animated models to the scene but instead adding particle and sound effects like splashing water, which could be triggered by the sound generator.
• Audio performance optimization: Audio currently doesn't cause any performance issues, however many sources are set to play even if they're out of audible reach because they would not reactivate again otherwise. This could become a problem once I exceed the voice count of 128, since I'm not sure if raising the limit even further might eventually lead to underruns. I think the reactivation issue has to do with routing audio from one source to environmental buses. It seems to break the connection when restarting a source.
• More weather: Some rain would be nice! So many ways to explore that musically...

Return to the island(s) 🏝️

After a few weeks of summer break, I am continuing the development of Soundgarden. I have expanded the concept to include two more islands: One island is meant for the examination and presentation of gameplay interactions that are facilitated through and led by sound and music. The other island is meant to present various ways of interacting with and manipulating sound and music. Over the next months, I will be working on the former. Meanwhile, I will also be conducting research into the various interactions led by sound in existing games. This investigation should yield an overview of what has been done in this context and should allow to extrapolate designs of the sound-led gameplay interactions on the new island.

Moving to the beat 🕺

An initial investigation into games led by sound has yielded that the vast majority of such games utilize rhythm as a guiding gameplay element: Usually this involves pressing a key and/or performing a certain action with the right timing that is synchronized with the music. This mechanism occurs in rhythm games like Guitar Hero or Dance Dance Revolution but has also been featured in genre hybrids like Crypt of the Necrodancer or Metronomicon, that combine RPG mechanics with these timed actions.

Derived from this insight, I have begun to implement rhythm-based gameplay into Soundgarden. My first approach involves a grid-based path that is drawn in sync with a rhythm generator which dictates all sound-led interactions on the island.

Soundgarden Rhythm Grid

Active cells can be walked on, while the player will fall through transparent and inactive cells. On every downbeat, the next usable cell in the grid becomes active, while the previous cell becomes deactivated. On subsequent beats, future active cells are shown to indicate the path for the player. I find the current implementation to be very straightforward, and possibly too close to existing adaptations of rhythm-based gameplay but it's a start! How could variations of this gameplay look like? Perhaps I could combine multiple rhythm grids into a sequence of rhythmic obstacles for the player to overcome. This would allow for adding polyrhythms by having the rhythm grids run at different measures. I could also flip the effects of the grid: Instead of becoming walkable, cells could allow the player to pass through them. Here, I could place multiple grids above each other, with the goal for the player to time their fall through all of the grids by observing the changes propagated by the rhythm and waiting until holes in the grids align vertically.

As I’m thinking about polyrhythms, I wonder if such gameplay could be helpful in music education, teaching about different rhythms, and generally increasing awareness of rhythmic variety. For example, most people will probably find it easy to clap a 4/4- or 3/4 measure. However, what about more difficult time intervals, such as 5/4, 23/4, ...? The rhythm grid must be traversed in sync with the given rhythm. As the player learns to traverse the rhythm grid, they might end up internalizing the rhythm to some extent, especially when it is juxtaposed with a familiar measure such as 4/4. On that note, I should check if there are any educational music games out there.

Moving beyond the beat ✨

Okay, so most music games are rhythm games, but what about other aspects of music? For instance, there is pitch and timbre. What information do these aspects of music hold that we can use gameplay-wise? When it comes to pitch, we can vary it. Individual pitches, spaced out in time create a melody. Melodies can be interpreted by the listener. There could be ascending or descending melodies, which could signify some movement. Maybe the player is climbing a mountain, maybe they're descending into a valley - or maybe, instead of accompanying the player's movement, the melody could be an indicator of what the player has to do. In the game Antichamber, one of the first riddles the player encounters is a staircase that seems to go on forever (think the endless stairs in Super Mario 64). Only if the player changes their direction of movement on the staircase into the opposite direction, they will be able to leave the staircase towards a new area. Now, what if we have ascending or descending melodies that would signify to the player which way to go? Maybe the player could be directed through a magical staircase like then one in Antichamber with music: go up, up, down, up, down, down, up, and then you're through! Timbre, on the other hand, is what can, for example, make a difference between a soft sound (e.g., flute) and a harsh sound (e.g., overdriven electric guitar). We know intuitively, and research confirms this, that different timbres are perceived differently emotion-wise. Very harsh timbres sound more energetic and perhaps even aggressive, whereas soft timbres are more relaxing, soothing, and possibly more sad. To find out how this could be used gameplay-wise, let's turn to this well-known example of Kiki (left) and Bouba (right).

So how can this be made into a game design? Most people intuitively associate the sharper timbre of pronouncing Kiki with the edgy shape on the left, and the softer pronounced Bouba with the round shape on the right. This means that it's likely there is an intuitive mapping of certain sounds to visuals. We could take advantage of this by, for example, having a puzzle where the player has to match shapes and sounds. Similar to the previous idea using melody, there could be a labyrinth with passageways marked by symbols, with the player having to use the passage where a symbol matches the timbre of a given sound. We could also expand the rhythm grid design by employing timbre. Sometimes, a cell that is to be activated could actually be a trap that the player should not step on. This could be indicated by adjusting the timbre of the rhythmic sound, changing to something harsh if a cell will become a trap.

Beyond rhythm, melody, and timbre, there are of course more aspects of sound and music to be explored in designing gameplay, for example, spatialization, which I'm sure I will return to in a future post.

Until then, I will implement some variations on the rhythm grid that employ melody and timbre.

Rhythming ⏹️

At the end of my last post, I mentioned how I would integrate melody and timbre into the rhythm grid gameplay. However, after going back to the drawing board, I ended simplifying the rhythm grid instead, to focus on, well, rhythm.

Now, there is a more robust technical structure to support the rhythm grid gameplay, as well as keeping everything else on the sound-led island in sync. As of now, Rhythmer, the music system for the sound-led island, generates five different rhythms using a custom implementation of the Euclidean algorithm for rhythm generation. I use this algorithm frequently since it's easy to implement and makes for complex and interesting rhythmic structures. Multiple generators can be combined to create polyrhythms, which players can not only hear, but also see when visualized in some way. The rhythm grid is essentially a visualization of a given rhythm, so if we have multiple grids that use different rhythms, we can see their movement offset and overlap at different times.

To recap, the general gameplay of the rhythm grid is as follows: Adjacent cells of a grid are activated, so the player can step on them, forming a path over time. Typically, the player must first observe the sequence in which the cells are activated, to get an idea of where to expect the next cell to step upon. The next 3 cells after the current one are highlighted to make it easier for the player to know where to jump next.

Adding polyrhythms makes the gameplay significantly more difficult, as it requires players to track two grids that move at different times. After several unsuccessful tries of traversing the polyrhythmic grid, I ended up reducing the tempo of the island's music, so that everything would move slower and give players more time to react. On that note, I'm thinking about adjusting the tempo of the music system dynamically, according to the gameplay the player engages in. Or possibly, similar to how players can change the speed of daytime progression in the ecological island, I could add an interactive element that allows adjusting the main tempo of the island's music system. This would allow players to choose whichever level of tempo-based difficulty their comfortable with, without forcing difficult gameplay on them. Since one of the main driving factors behind Soundgarden is to make everything about it accessible - this writing, code, assets, etc. - why not make gameplay accessible as well?

Progressing with progress ➡️

With the arrival of the rhythm grid gameplay, we now have something very ludic in here. There will be more areas with other music-led gameplay which makes me think about an overall progression that the player can achieve on this island. Similar to how collecting rocks eventually allows players to play with daytime tempo, or opens pathways on the ecological island, completing a music-led gameplay area could lead to unlocking new areas or gameplay variations. Maybe there could be something like control panels that players can unlock. These panels could allow players to change some parameters. For example, they could adjust the parameters of the Euclidean rhythm generators, leading to changes in all elements that depend upon them, propagating throughout all the different areas on the island.

(Still trying to move) beyond the beat 🤔 Despite my previous ruminations on Kiki and Bouba, thinking about timbre and melody, I haven't actually come up with a gameplay mechanic that utilizes these ideas on perceptual features. However, I've also been meaning to look further into utilizing spatialization of sounds to inform gameplay. Blind Drive is a rare example of a sound-led game that is not rhythm-based, and that extensively uses panning sounds to provide locational information to the player. While the panning is not necessarily realistic - things that are supposed to be in front of you will be hard panned in either direction to signify in which direction to evade them - it suggests a clear course of action. One of the issues with having a player listening closely to sounds is that oftentimes the visual impact of the game environment, as well as the affordances of navigating it are too distracting for such a task. Blind Drive solves this by removing visuals from the equation altogether. So the question remains: How can we facilitate a deeper awareness and understanding of sound within a highly visual environment?

Possibly, we can use visuals to support the sound, pointing out certain aspects of it. Maybe we can help players to pay attention to certain characteristics by training them on the connections between specific visual features and sound features. For example, showing a certain color or geometric shape could mean for the player to pay attention to the occurrence of a specific timbre, musical note, or rhythmic pattern. It would make sense here to look into perception research, as well as to listening practices and exercises, possibly even meditation styles that foster awareness of aspects of one's inner world and surroundings. On that note, what if we can teach awareness of sound, as well as listening techniques through gameplay?

Side note: On performance ⚙️

I received some feedback about periodic stuttering in Soundgarden. After taking a closer look, it appears that Soundgarden has an inconsistent frame rate that slows down almost periodically, 2-3 times per second. This is of course inacceptable for any real-time application, let alone one that features music and sound-based gameplay. Especially any gameplay that uses rhythm will be heavily affected by this. I was thinking about this when I was playing Hi-Fi Rush recently. In this game, everything is synchronized with the music: The player character's walking animation, environmental animations, and, when fighting enemies, the points at which the player has to press buttons for maximum attack efficiency. For me, the biggest strength of Hi-Fi rush is the almost instant feeling of flow it creates once I got into the groove and forgot about the fact that I have to constantly push buttons to fight enemies. Pressing buttons becomes as intuitive as dancing to a DJ set, almost as if the controller disappears and instead creates a direct connection between my musical sense and the action on the screen. The game runs consistently at 60 fps. If there were fluctuations in the frame rate, even if the music playback was unaffected, the inconsistency of the game's speed would make it more difficult to get into the flow so quickly, if at all. When it comes to immersion and flow, it seems that a stable and consistent performance is even more important in sound and music-based games than it already is in general. Conversely, this means that I should make it one of my top priorities with Soundgarden to ensure a consistently good performance across different hardware, at least until that is the case.

So what's next? A deep dive into performance optimization in UE5, as well as implementing a new gameplay mechanic that is not rhythm-based.

The listening wind 💨

The gameplay I'm about to describe is inspired by several other gameplay experiences: First, the spatialization-based gameplay in Blind Drive (see previous post), second, a section in the puzzle adventure game Riven, where the player must navigate a dark tunnel, and third, the Super Mario 64 remake for Nintendo DS that features a section with spatialized audio to guide the player around a haunted house in order to find a star.

In my new gameplay mechanic, the player must navigate a pitch black cave where the path is not obvious. The player must listen and follow the sound of the wind passing through the cave in order to find the exit. To test this, I have created a gray box prototype of a cave environment (see screenshot, ceiling removed for demonstration). A spline marks the path through the cave. A wind audio emitter object moves along the spline, always positioned on the closest point to the player on the spline.

The wind audio emitter object has four audio sources, arranged in a rectangle, with two "front" sources designated to emit sounds from the direction the wind is heard to be coming from, and two "back" sources to emit sounds indicating the direction the wind is blowing towards. The back sources sound more quiet and muffled compared to the front sources. This is to indicate some directionality. I'm assuming that if the wind sound came only from a single source placed on the spline, the player would have no sense of direction when following the sound, and would be more likely to follow the sound back out where they came from. With the current setup, I'm envisioning the player to stop over and over again as they traverse the cave, listening to and noticing the differences between the two types of wind sounds, and learning to follow the one that sounds sharper and more pronounced. After adjusting the distances between the sources several times, I managed to navigate the cave only by sound. However, I find that this navigation by sound needs to become clearer and easier. Another issue is that with this setup there is potential for the audio sources to intersect with the cave walls, especially in curves. To solve that, I will most likely need to change the positioning of the wind sound sources along the spline. I'm thinking that instead of four sources moving on a fixed rig along the spline, the two pairs of sources should each be placed that they are always a certain distance in front and behind the closest point to the player on the spline. This way, I could place the front and back sources further apart from each other, possibly increasing the spatial audio impression that would help guide the player.

Home improvement ⚒️

I had originally planned to finish the two new islands first before adding improvements to the ecology island. However, now I'm getting regular feedback from testers for both the new in-development content and the existing content. Also, as I'm working on the new islands and figuring out Unreal, I'm constantly getting ideas for improving existing things. I decided it's probably best to act on these ideas now, constantly implementing small improvements while I have a clear vision, rather than dealing with many old ideas at once later. Therefore, I have implemented several improvements in the ecology island.

The most noticeable improvement is in fixing the performance I had described in the previous post. It affected the existing island as well as the separate testing maps and made it significantly more difficult to engage in rhythm-based gameplay due to an inconsistent frame rate. It turned out that global illumination quality settings above medium would trigger a Lumen update 2-3 times per second that would lead to strong drops in frame rate. Lowering the setting to medium removed the issue but also had an impact on the lighting, making the scene darker overall. This required me to adjust the lights in the scene to compensate for that. I also noticed that emissive materials have now stopped affecting the scene lighting. Since I have not been relying on emissive surfaces so much, it's not an issue, also considering that Unreal offers some good alternatives such as real-time area lights. Improving performance is not only important to guarantee a flawless experience of existing content and to facilitate gameplay that requires consistent timing. Since eventually all three islands will be featured on the same map, I have to make sure everything remains smooth once that becomes the case.

Another major improvement is the sound. All music speakers are finally working as intended! Before, some music speaker objects in the scene would either not work at all or not reactivate after being occluded. The issue lied in how I was routing the audio from the music MetaSound to the speakers. Before, every speaker would have a custom MetaSound that would read from an audio bus. This seemed to cause issues with how Unreal handled virtualization for audio sources. Now, I'm using source buses that play directly from the selected audio bus. As a result, there is less delay between the source and the speaker, lower CPU usage, and an overall more lush soundscape - win/win/win!

Last but not least, a gameplay improvement. after receiving feedback that for some players it was not clear they have to collect stones, I have begun to improve that. Now, whenever the player collects a stone, the painting in the swamp area will briefly light up, highlighting the stones. I hope that players will notice this, especially when collecting the stones near the painting at the beginning of the game.

More improvement 🛠️

To recap the general idea: As players collect rocks, parts of the island change, new interaction possibilities appear, and secrets unlock. There need to be more incentives and feedback mechanisms to motivate players to explore further. Some time ago, I was debating whether to add interaction input for the player, for example, letting the player interact with certain objects when clicking on them. My main concern was about communicating this mechanic without words. I think I can find a way by communicating that through the painting, maybe adding even more hints in this cave painting style around the island. If players had to click on the stones to collect them, this would add more potential for adding juice to that interaction. This clicking mechanism could be context sensitive. For example, in the rhythm grid puzzle, aiming at the next platform could highlight it, and clicking could make the player character jump on it immediately, without the player having to press the movement keys to steer the direction of the jump. This would make jumping across platforms more instantaneous and precise. Since the player then would only have to focus on the timing and direction, this might allow for faster and more complex rhythm grids to navigate. As I was also debating whether to let the player jump around freely some time ago, context-sensitive interactions would allow me to remove jumping without sacrificing any potential avenues for jumping-based gameplay.

Next up, there will be more work on the listening wind gameplay and possibly more improvements on the ecology island.

Wind gameplay improvements 🛠️

As I already indicated in my previous post, I reworked the sound emitting rig in the wind path gameplay. To recall, I was using a fixed rig before, essentially a rectangle moving along a spline with sound emitters at each corner. Now, the sound emitter rig is split into a front and a rear part that position themselves dynamically on either side of the closest point on the spline to the player. This allows the sources to move around corners in narrower tunnels, while avoiding clipping the sources through the walls of the cave. At the same time, front and rear sources are further away than they were one the rig, making for more easily distinguishable wind directions. Following the sound of either sound source should lead the player to the respective end of the tunnel, which proved to be much easier using this improved setup. An interesting side effect may be contributing to this improvement: When either pair of sound sources is on either side of the player, i.e., 90 degrees to the left and right of the player character, each stereo pair ends up sounding "mono". However, once the player turns to face one sound, for example, looking towards the front sources are ahead, the sound sources begin to sound "stereo" again. This helps when orienting oneself in the dark cage. However, I should note that while my own testing deems this setup to be effective in leading me through the tunnel, only feedback from other players will tell if the wind path gameplay is "production ready". As of the week of writing this post, there will be some playtests and feedback on this. Will report back once I know more. 🫡

Listening modes 👂

I looked at the different ways we listen to sounds. Chion (Audio-Vision: Sound on Screen) differentiates between causal, codal, and semantic listening. In causal listening, we focus on who or what is making the sound, trying to understand its origin. In codal listening, we derive meaning from a sound based on a set of learned rules and conventions. In reduced listening, we focus on the features of the sound itself, without thinking about its source and meaning. I am thinking about these modes of listening in terms of gameplay, to understand what has been done before and to possibly find a gap for coming up with a new gameplay idea. At a first glance, it appears that we can find many gameplay examples for each of these listening modes. We listen in a causal way when we hear footsteps in a game, trying to make out whether this might be an enemy trying to sneak up on us in some competitive online shooter. Codal listening is what we do when we are listening to a familiar spoken language. Applied to games, this is when we develop certain expectations or make assumptions based on what some sounds mean to us. We listen in a reduced way, for example, when we are playing Guitar Hero, trying to anticipate the structure of a song so we're ready to press the right button on our plastic guitar. To come up with new gameplay that goes beyond these examples, I'm going to brainstorm some ideas for each of the listening modes. I intend to eventually pick out at least one of these ideas and test them out in Soundgarden.

Causal

• Hide and listen: Locating and pointing out the source of an invisible sound in the game environment just by listening.
• Odd one out: Pointing out one sound source that sounds a little different than all other sources. This idea also involves reduced listening, since we first need to familiarize ourselves with the sounds, to make out a difference in the first place.

Codal

• Lost in translation: Listening to different sounds and matching them with abstract symbols according to how much these symbols are perceived to represent the sound. Think the previous example of Kiki and Bouba.
• Hidden information: Certain melodies could be taught to the player and then be inserted dynamically into real-time generated music. These melodies could hold some sort of information, for example, a combination to unlock a door, or indicate certain locations for the player go to.

Reduced

• Sound match: Match the characteristics of one sound, for example a synthesized tone, by adjusting the parameters of another sound.

Next up 🔜

Beyond these brief ideas, I was also thinking about implementing new visual elements that react to sound as additional unlockables on the first island. Also, I was thinking about Schafer's practice of ear cleaning and possibly deriving some game mechanics from that. For example, mapping one's sound environment by consciously acknowledging various surrounding sound sources.

Next, I will evaluate the playtesting feedback and possibly rework the wind path gameplay. Also, I'm planning to design and implement a new sound interaction.

New worlds 🏝️

I have begun work on island 2, which hosts the sound-led game mechanics. To create the new island, I followed a similar approach as with island 1. First, I sculpted the landscape.

Using the landscape tool, I created a rough sectioning of the island, to provide dedicated areas for the various sound-led game mechanics I aim to implement. The steep inclines act as separators, with the intention of focusing the player's view on their current area and minimizing distractions by other areas. At the same time, I'm leaving just enough gaps for players to catch a glimpse of other areas and possibly incite curiosity. Nevertheless, on this island it's especially important for the player to be able to maintain focus, since some of the sound-led interactions require a deep engagement with sound.

The yellow circles in the picture above show where I currently aim to implement gameplay areas. The green circle in the middle marks a hub location, from which the player will be able to reach all other areas. The green arrow in the bottom indicates where the player enters the island. I also made a cut that separates the island into two parts. The reason behind this is that I intend to have the player cross this small ravine by interacting with a sound-led game mechanic. Most likely, I will implement the rhythm grid here, to have the player traverse the grid and get to the other side.

I decided to first implement the wind path game mechanic on this island, for which I designated the circled area in the bottom right. Since I wanted this mechanic to play out in a dark location, I created a cave tunnel underneath the hill there. The dark spots within that circle mark both ends of the tunnel. Before sending the player into a pitch black cave, however, I wanted to give them some time to learn the mechanic of having to listen to the wind for navigation. For this purpose, I built a small outdoor labyrinth that leads to the cave. More on that later.

To add more definition to the shape of the island, I began dressing it up with rock geometry. Just like on the first island, I used a single rock 3D model, copied, rotated, and scaled many times to avoid repetition, while keeping the project size small, using a minimal amount of objects. I did, however, use a different rock model than on island 1, which should help maintain a consistent, yet distinct look for each area in Soundgarden.

On the first island, whenever it gets dark, a light orb appears that follows the player and lights the way. Since this might interfere with the game mechanics of island 2, for example, by lighting the otherwise pitch black wind cave, the light orb will be disabled here. Instead, I have place some lights around the island that will ensure the player can navigate regardless of daylight.

I then proceeded to connect the two islands. At first, I implemented a walkway that would appear after the player completes the tutorial on island 1. However, I noticed that walking between the islands might be a little boring, since there's nothing else to do here. Therefore, I removed the walkway and added teleporters that transport the player between islands. With this in mind, I am already rethinking my concept for island 3, which will be the subject of the next phase of this project.

Since I would still prefer players to walk between the islands, I am considering to make island 3 a lagoon instead. The lagoon would be placed in between the two islands, allowing the player to move between them and would hold a variety of mechanics where the player can affect and change sounds. Come to think of it, I actually don't remember having seen many lagoons as game environments, so this would be an interesting concept to work with overall. Perhaps, the novelty factor of a lagoon would support the experimental nature of the upcoming interactions with sound.

Final ruminations on wind 🌬️

Since the playtest was pushed back by another week, I used the extra time to integrate the wind path mechanic into the new island, as described above. This allowed testers to test it out in a natural environment, as opposed to a grey box level.

As mentioned above, the outdoor labyrinth area is meant to teach the wind path mechanic. The area features some dead ends. However, if the player follows the sound of the wind, they will eventually arrive at the cave entrance. While testers noticed the howling of the wind and enjoyed how it sounded, some did not recognize that the sound was there to guide them, and those who did, did not know how to interpret the sonic information. This means I need to add more guidance here. Just like I did with the illustration on island 1 that indicates to players to collect rocks, I need to think of a fitting pictographic depiction of "you must listen to the sound and follow its direction", preferably in the same archaic style as before to remain stylistically consistent. Since the labyrinth is not too complex, players would still be able to find the cave entrance. However, the issue became even more apparent once players entered the dark cave. Without knowledge of how to interpret the sound, none of the testers actually managed to get to the other end.

I clarified the function and meaning of the sound, explaining how each of the two types of wind sounds indicate the direction of the respective cave ends. However, even with this knowledge, testers reported having difficulties in differentiating between the two sounds, which likely interfered with locating the direction of the wind sound leading to the desired exit. I tried to address the issue by having only the wind sound leading to the opposite exit be audible when entering through the wind path, gradually fading in the second wind sound leading back to where the player got in. I hoped that this would help players "lock on" to one of the sounds. However, this was still not clear enough. I also realized that it may actually be more difficult to follow a sound in a straight line ahead than turning around corners, since sounds coming more from the left or right are just more noticeable.

There are several ways I could address these issues. First, I could focus more on teaching the mechanic. Since I was able to navigate the cave without issues, I know it is absolutely possible to traverse it by ear. Therefore, taking more time to thoroughly teach this mechanic to the player might help. Second, I could add some tonality to the wind sounds, in the same way I did with the beach sound of island 1. Each of the two wind sounds could additionally incorporate a musical pitch that would likely help in differentiating between them. Perhaps this will also mitigate the issue occurring in straight sections. Third, I'm thinking back about my concepts for facilitating a feeling of embodiment. In a previous post, I briefly mentioned the idea of having the player character's body brushing over surrounding surfaces. This could actually come handy in here, since players would have a way of locating walls by brushing against them. Using this additional information in conjunction with the wind sounds might make it easier to navigate the darkness.

Aside from this feedback, testers also noted the good graphics and the soothing atmosphere and sounds of Soundgarden. They also commended how the composition of the landscape guides the players gaze and creates focus. Since I never aimed for good graphics, I think it's funny that my game about sound was noticed for that. However, in terms of the environmental design, I'm glad to know I'm on the right track here.

What's next? 💭

I still need to re-implement the rhythm grid from its grey box version to something that integrates nicely with island 2's environment. Instead of using transparent platforms, I might end up going for something more kinetic. Perhaps rock platforms that slide in/out in the rhythm. I'm also thinking about the next sound-led interaction to prototype. For that, I'm going to have another look at the listening mode-based mechanics I outlined in my previous post.

Sound gates 🚪

The latest gameplay to feature in Soundgarden is called Sound gates and is an implementation of the "odd one out" mechanic I brainstormed previously. Here, the player is presented with a selection of gates - 3 at a time in the current prototype - and must identify the gate that emanates a sound different from the others. Walking through the correct gate will lead the player to the next set of gates, or eventually the exit. Walking through an incorrect gate will block the player's path, requiring them to walk back to the start and start anew. The correct gate is randomized each time the player approaches the area, to make sure the gates are not traversable through trial and error. In the current prototype, only two similar sounds are featured: sine wave and triangle wave. If two gates are playing a sine wave, the correct gate will be marked by a triangle wave and vice versa.

Soundgarden Sound Gates Test

However, I plan to add more differences in sound features in the future. In addition to timbre, the sounds could also differ in pitch or feature slightly differing repeating patterns. I'm also thinking about a progression in difficulty. When using pitch, the odd pitch could at first be represented by a clearly audible interval, which could then be reduced as the player progresses, until the differences in pitch are so miniscule they might require very close and concentrated listening. Same goes for timbre. For example, the sound gates could feature more subtle timbral differences. Instead of pointing out a triangle wave vs. a sine wave, the gates could feature sound waves that are morphed between the two types, leading to more gradual differences. The gates could also feature melodies, where the odd one might feature one different note in the sequence, requiring the player to spend more time listening and memorizing in order to be able to tell a difference. As we can see here, while the mechanic seems simple, there are many variations to play with!

Another recap 🔁

Another year is coming to an end and with it this project... at least for the remainder of this year, as I will be enjoying the holiday season. I am yet to finish phase 2 which focuses on the sound-led island. While I'm happy with how the second island is coming together, there are still more sound-led game mechanics to add. Nevertheless, I would like to recap what happened in this project since I took it up again back in September.

Inspired by the insights I gained researching sound-led games, I developed three gameplay mechanics.

First, the rhythm grid that requires the player to traverse platforms moving to a beat. This seems to be the most straightforward mechanic and is close to what we have seen in many rhythm-based games. While testers reported the traversal to be a bit tricky at times (for example, because the rhythm was too fast), the mechanic was easily understood. I still need to integrate this mechanic into island 2, since it currently only exists as a grey box prototype. Integrating the rhythm grid into the game environment will require me to rethink the floating transparent platforms in favor of something that better fits the natural rocky environment.

Second, the wind path mechanic where players must follow the sound of the wind to reach the exit of a dark tunnel. While I'm personally very happy with how it turned out, being able to easily navigate the tunnel by ear, testers found it far more difficult. On the one hand, there were difficulties in understanding the mechanic - I will need to work on communicating it better. On the other hand, even when testers understood the function of the wind, the provided sonic information did not seem to be enough for most people to get a clear sense of direction. There is clearly more work to be done here. Of the three mechanics listed here, this is the only one that is already integrated on island 2.

Third, the sound gates, that I described above. I am yet to see what testers will think about it. However, I have a feeling that it will be easier to understand - and perhaps easier to solve - than the wind path mechanic.

I still plan to add one or two more sound-led mechanics to island 2. Once they are all integrated and playable for most people, phase 2 will be done and I will move on to work on interactions where players can affect sound. Instead of creating a third island, as I had planned originally, I will build a lagoon that will connect islands 1 and 2 and feature these interactions.

Beyond the new island and its game mechanics, I have also implemented some general improvements for Soundgarden.

First, I fixed some issues with the sound speakers on island 1, making for a fuller soundscape. I have also added a main menu that allows for changing the graphics settings. This was part of my effort to improve Soundgarden's performance. Stemming from this effort is a slight graphics downgrade which led to a massive performance improvement. However, the previous graphics setting can now be selected in the settings menu. Further, we now have save games! This means that players do not need to replay the tutorial every time anymore, which especially makes sense as the world grows, featuring more areas and things to do. Implementing save games took some considerable effort in debugging, since I had to account for the dynamic day-night cycle, player location, and progression (e.g., the amount of collected stones). This required me to refactor the logic for the tutorial and move away from some hard-coded approaches. With the save game system in place, I will need to make sure any new gameplay additions are reflected by the saving system when required.

As the project grows, I will also have to take up old habits. Specifically, I will need to use to-do lists and my light agile approach, in order to track new features and bug fixes. I also want to make the game accessible to more players by supporting OSX and Linux. Recently, I have tested a Mac build that, surprisingly, ran quite decently on my M1 MacBook Air. Still, there are some shader issues I need to fix before adding this version. Once this is done, I have a feeling it will be rather trivial to also make a Linux build as well.

All in all, I'm happy with how the previous months went. I have gained many insights into sound-led game design. I now also have a much better grasp of Unreal Engine and have come to enjoy its paradigms. I know my way around now, to the point that problem solving has become an interesting challenge, rather than some buggy nightmare. Whenever I have a new idea now, I can focus on its implementation, unbothered by the ins and outs of UE5. I learned new systems, such as the procedural generation tool, and I learned how to customize the engine for my needs by creating custom data types and writing my own blueprint nodes in C++. I also know how to do things the right way, which has helped me in fixing old issues, such as the sound speakers on island 1.

I am excited to see where this project is going, and, sometime in summer 2025, how it will all look and sound once it's done!

Old habits? 🧓

I've been meaning to publish this journal entry weeks ago but I got lost in the flow of getting things done - powered by the return of the agile board.

I think it was only a matter of time for this to happen and can be attributed to the increasing complexity of the project: As I'm adding more and more features, refining existing ones, fixing bugs and addressing feedback from playtests, I need to rely on more than just my memory or handwritten to-do lists. I was dreading the impending re-introduction of the agile board into my workflow, since I now understand very well how it was having a negative impact on the creative vision and design decisions I made in past projects - I was focusing on getting things done, rather than on the things. However, the many Soundgarden-thoughts that accumulated over time were not just filling my to-do lists, they were taking up precious space in the back of my mind, leading to ruminations and an increased worry to simply forget about important - or perhaps even profound - ideas. So the agile board had to come back, but it's new role should be one of a bridge that would bridge the gap between design decisions and active development.

New labels ✨

Rather than using the board as the central point of organizing a project, I now use it as a cache for the concrete tasks that stem from my thoughts and insights. I'm also not using Trello for this anymore, but instead GitHub's built in project board feature, which puts this form of project organization closer to all other forms of reflection hosted in the repository: this journal, the project documentation, and commit messages. In fact, to further reduce any potential negative impact on the design process, this new board now adds another form of reflection in the form of what I call "micro-journals". I'm not yet sure how thorough I will be with this, but I aim to add my evolving thoughts about the tasks in the board to their respective descriptions. As of now, this is takes place in the following way:

1. Each task has a title, a very high level description of the issue or goal at hand.
2. On creating the task, I might have some more concrete thoughts on it, similar to an agile user story. There are no formal requirements, though - whatever works - since it should be quick as to not disrupt my workflow too much. Adding tasks to the board is still something that happens as part of an actual development session!
3. I use the comment feature to add any reflections I might have while working on the task, or even after I'm done with it. Comments are automatically time-stamped, which could help trace and analyze on-the-go design decisions in the future.

During my Trello board times, I would occasionally follow some of the steps I described here. However, I now consciously aim for being more through and mindful with this, even with tasks that may seem to be more on the trivial side of things. Thinking about this, I realize how this is perhaps not only a reflection on my work with Soundgarden but really a product of all my design/project management/working on things experience. I appreciate how a more mindful approach can resurface a long history of previously unconsciously known personal work experience. But then again, this is perhaps true for all aspects of life.

Note to self: Writing this, I realize that I have not actually documented my previous Trello board-based workflow in this journal before. This is a reminder for myself to upload and link my (already existing) report on that here!

Complexity at the push of a button 👇

Thursday, February 13, 12:52 AM I am thinking about the growing complexity of this project. Yesterday, I spent what felt like a whole day on implementing the Odd One Out gameplay.

One week later, I'm looking at this momentary thought after having spent another whole day of (unsuccessfully) implementing what seemed to be a simple button. We need to talk about this.

First, I feel the need to clarify that my lower frequency of journaling as of late is not the agile board's fault. I just had a very clear idea of what I had to do, thanks to a more conscious design approach. In addition to that, as the project progresses, it's complexity increases. New additions may need to be integrated with the many parts that now exist in this project. Some design decisions may pose enhancements of what's already there. Enhancements can be things like usability improvements, more responsive and "juicy" interactions, quality of life additions - which may take considerably more time to implement than initial, proof-of-concept, barebones features.

Take a button, for example. It can be simple. The player looks at it, left clicks on their mouse --> the button is instantly pressed, triggering something. Now, what if I want to have some feedback? I could add a sound, instantly played whenever the button is pressed. But perhaps I want to make it more "fun" to press it. An animation could help, showing a quick inward movement - and this is where complexity starts. First, I have to create the actual animation, either by deciding on the specific positions of the button over time, or parameter-based procedural movement that takes place within a defined positional range. Then, I have to make sure the animation does not conflict with the player's behavior: What if the button is pressed again while it is animating? At this point, we have already added several variables and functions here. Plus, since the state of the game environment is saved in Soundgarden, the state of the button must be saved and also be reconstructed when the game is loaded. Here, we don't just save some Boolean variable, we have to use the information stored to set the button to it's correct pressed/unpressed position. The button may also have some other features. Perhaps it can be activated/deactivated in the sense of being usable/not usable, meaning it could be in either pressed/unpressed position but additionally not trigger a state change when interacted with. This would also pose an additional feature to be stored in a savegame. Also, perhaps the button's state can be influenced by another object, some controller entity that manages a set of buttons as part of some machine or puzzle. And then, of course, the player's interaction with the button itself must be handled. How do we know if the player wants to interact with this specific button? How do we know if the player is looking at it? Do they have to look at it directly, or is there some tolerance? If there is tolerance, how do we handle multiple buttons next to each other? How do we display a signifier that communicates to the player that they can interact with the button?

I want to be clear that the questions above are not meant to address the design of this interaction. Let's assume we got that covered already. All these questions are meant to show the technical reasoning that goes into the implementation, and why implementation of such seemingly trivial things might take some extra time, requiring to divert focus away from design and innovation. The new old agile board helps me to stay on track with things, despite the occasional developmental deep-dive. Commenting my progress and on-the-fly insights through micro-journaling in the respective issues helps sorting my thoughts and keep design reflections alive.

Where's the cool new stuff? 🔍

This journal entry has ended up being more about the project organization and my general development practice for Soundgarden. For the sake of keeping things in focus and to not overburden an already lengthy piece of writing, I will reserve reflections about the cool new stuff for the next post.

Note to self #2: Writing this post, I discovered how to do headings in markdown, making it slightly easier to format the text, and, more importantly, allowing to link back to specific sections. This is a reminder for myself to update my earlier posts with this exciting piece of technology!

Post mortem 💀

I still don't like this term to talk about a finished project. First of all, Soundgarden is not fully finished yet - and even if it was, the result is not something dead but rather something that is very much alive. The way I see it, developing a project does seem to present some parallels to birthing: It may take a long time, it may be painful, but the result is (hopefully) something new, alive, breathing. So I would prefer an alternative term that incorporates that idea. Post natalis would be the closest to my idea. However, post partum sounds more in the vein of post mortem and is actually a commonly used term in the context of birth, even if it may carry a slightly more negative connotation: Postpartum depression, the type of depression that happens after having a baby. But then again, the feeling of emptiness, of falling into a hole when finishing a project or meeting an anticipated deadline may not actually be unfamiliar to some. Also, deadline... another one of these death-themed terms... For now, let's not get caught up in semiotics too much, but perhaps the "undeadening" of design and project management terminology might be something to think about in more depth in the future.

Post partum 👶

As already mentioned, Soundgarden is not fully finished yet, however, it has reached a state where all the essential elements are there and working. Moving on, I would like to leave the project at its current state open to be studied, examined, dissected, with all development resources up to this point being available for anyone to access. I also plan to develop this into a more polished version, where I will further explore and develop some ideas that are already implied in the existing project, finish the remaining empty areas on the sound-led island and lagoon, and perhaps fill in the gaps in between everything else, in addition to technical improvements and overall polish as well as a (cheap) commercial release on online platforms such as Steam. Perhaps "professionalization" would be the appropriate term here? Looks like the baby will be growing up over the next foreseeable period of time. But between the last journal entry (back in February!) and the baby's first steps that will follow, let's shed a light on what has actually happened - a small, or maybe not so small interlude.

The interlude 🎶

A lot has happened in a relatively short time, triggered by the fact that I had been busy with another project but then Soundgarden was about to be presented at Ludodrome, an experimental video games exhibition that took place on May 15. Many additions emerged in the three weeks leading up to it. In that time, there were only two (large) commits, even though the changes, big and small, were countless - I was certainly playing a risky game here, but I was also deep in the flow of adding new things, fixing, finishing, polishing. The accompanying commit messages don't really exude a reflective mind, being merely lists of "new stuff". Still, I wouldn't say that this period was completely devoid of reflection - but let's say, reflection happened more between the lines here... So while perhaps this period of rapid additions and improvements may not have been the most reflection-driven, I realized that sometimes being in the flow of development can be liberating - if used in moderation. Looking back, this phase was like opening a valve, releasing all the ideas that had accumulated in the back of my head (and the project board) over time, while I was busy working on other things. So maybe this was not a period where reflection was absent, but rather that the reflection had happened ahead of this phase - via past rumination and ideation, as can be (partially) witnessed in the previous journal entries and commit messages. Perhaps sometimes it can be healthy to not interrupt "the process" once in a while - spontaneity may also facilitate experimentation in the moment and a closeness to one's inner world, that - while more raw at times - may facilitate a big leap forward, just like these weeks of rapid additions. Still, I wonder if the fact that I re-introduced an agile-style board with high-level task titles had also something to do with the fact that I was journaling less and writing more simple commit messages. The tasks provided a thread to stick to, even though I feel like in the rush of preparing Soundgarden for the exhibition, I might also have just been working from what had already been floating in my head. Most likely, it's a bit of everything - still, I need to treat the task board with caution, to not risk falling into old patterns. But then again, I'm not - otherwise I wouldn't be sitting here writing this journal entry. But let's move from the vague into the concrete and look at the content I had not described previously, as well as what I have added more recently.

Added brushing 🪥

To further build on the idea of embodiment, I have added a system that simulates the player's body brushing against surfaces. It does so by fading in a noise. When the player passes closely along walls or other objects on their side, the noise gets louder based on the players movement speed. To gain inspiration for the system's functionality, I brushed my real shoulders and hands along surfaces. For lack of fitting audio samples and the means (or rather time) to record them, I just used synthesized white noise, with some frequencies filtered out, that is then played back from audio sources placed to the left and right of the player rig. I'm actually surprised how well this simple approach is close to real-life brushing things. I had also hoped that this might make it easier to navigate the dark tunnel, since players would be able to "hear" walls being on the left or right of them, however, the existing wind sounds do seem to mask this. I should still consider simplifying the dark tunnel gameplay by adding clearer auditive c(l)ues, but this is a topic for the future (and yes, there is a future! But more on that later...). Further, I can imagine enhancing the brushing mechanic by adding light controller vibration, although this would only be most effective if we had means to differentiate between vibration coming from the left or right. And this feature is not available in most gamepads, to my knowledge. Still, I should check what's possible here. The PS5 DualSense controller might be able to do something like that. Note to self: Get one of those.

Lagoon 🌊

In lieu of adding a third island, as mentioned in the past, I had decided to make a lagoon that would connect the two existing islands, filling the oceanic gap in between them - and removing the teleport that was meant as a tentative solution anyway. After gathering some inspiration by looking at pictures and videos of real-life lagoons, I sculpted wavy lines of sand that connected the two islands. I then applied the same process I used for the two islands, tracing these lines of sand by copying, pasting, rotating, and scaling of one and the same 3d model of rock along them, to give it more structure and improve the visual look. Very happy how this turned out! I also like the idea of the lagoon because it provides a good excuse of featuring various ocean sounds more prominently, so that the player can hear all this beautiful splishing and splashing all around! (INCLUDE LAGOON PICTURE HERE)

Night sky 💫

I have reworked the day-night cycle, fixing the previously stuttering sun - something I'm not sure I have mentioned before, but yes, the sun was stuttering and now it isn't anymore - and I have also added a starry and colorful night sky, along with a moon. Beyond its aesthetic value, the reason for adding an illuminated night sky and moonlight was to create a reasonable origin for an additional light source during the otherwise pitch black night, which in turn allowed to diegetically avoid complete darkness that made it otherwise difficult to navigate the environment of Soundgarden at night, without having to add countless light sources all over the environment.

Reworked interaction system 👆

While on the surface it may not seem much different from before, I've reimplemented the interaction system that allows the player to pick up rocks and press the buttons in the Odd One Out gameplay area. I have actually transferred this new and more versatile system (one unified system for all interactions) from another project I had been working on at the time - and now I'm happy to have something I can consistently and reliably use across all my projects. Something that also seems bug-free, compared to what I had implemented before. This is also demonstrative of my increased (and still increasing) proficiency with Unreal Engine. With the new system, it's now much easier and quicker to add more interactions in the future, including the one I will describe next.

Sea Organ 🐚

Inspired by the sea organ in Zadar, Croatia, I've created an object that is more akin to a sound installation, albeit in virtual form, located in the lagoon area. This object, at least at the time of writing, might pose one of the visually more distinct objects in Soundgarden's environment, clearly removed from the otherwise natural look of most other elements in its environment. A white, concrete-looking object that emanates layers of sine waves that provide an auditive interplay with the surrounding ocean. Designed to melodically mirror and interact with the sounds of the waves. For comparison, here is a video of what inspired this object:

Zadar Sea Organ

I think if I had more time on my hands, I would've worked out a more complex sound setup, to create something that would sound closer to what can be heard in the video - I like the more "organic" (no pun intended) sound of the Zadar sea organ. Perhaps I can think of something more elaborate in the future. To keep with the "influencing sound" focus of the lagoon, and differentiating my sea organ from its source of inspiration, the player can press one of a set of buttons to change the tonality of the emerging sines, ranging from more harmonic to atonal outcomes, where the pitch of each resulting sine wave will be chosen from the selected predefined scale, originating from a different position across the installation. Pitches are derived from various 12-tone scales - however, as I am writing this, I wonder if I should also introduce at least one microtonal mode here...

Musical marbles ⚽

I had been imagining this idea of large glowing marbles for a long time. Perhaps I just like the glassy look, and the idea of how they would refract the light passing through. Reflecting on this now, perhaps another reason I went with marbles is how they suggest a sense of playfulness: Marbles are meant to be rolled around, pushed, clashed against each other, so their look would invite players to do exactly that. The player can interact with the three marbles by bumping against them, rolling them around. The marbles are located in a pitch in the middle of the lagoon, its walls dampening the sounds of the surrounding water, to allow them to be fully heard. The speed at which the marbles move affect the ensuing pitch. I designed the underlying sound generation such that if the marbles clash with each other or with the boundaries of the play area, the height of the notes they play will wobble, creating an initially fast vibrato, that quickly slows down, musically mirroring the sound that one would hear when clashing marbles with each other. Writing this, I wonder, if the clashing sound should differ, depending on whether the marbles hit each other, or the play area walls.

Fading walls ✨

In the vein of improving the introduction phase in the beginning of Soundgarden and giving a stronger visual feedback that also happens to look more visually appealing, I've developed a shader that fades away the rock barriers in a flash of light, resulting in an (at least for me) impressive visual play. This also helps to more effectively direct the player's attention towards the new paths that open up in the tutorial phase of the game, especially when unlocking the barrier leading around the rim of the first island when the player is on the hill outlook. I have also reused this fading effect in the area featuring the Odd One Out listening puzzle. What I find personally interesting is that while I was developing this effect, I had the Ludodrome exhibition in mind, thinking that a nice effect like that may draw the attention of exhibition visitors to the game. This made me think about how different presentation contexts for a project can inform decisions that have an impact on the overall experience.

Et cetera 🔣

Things I would still like to do and haven't mentioned yet:

• VR support: It would be nice to actually stand on the islands and experience the lagoon from a true first person perspective. I'm also sure this would make for an even more encompassing sound impression and would be more interesting from an embodiment perspective.
• More sophisticated music: I would like to use the insights I have gathered from my electroacoustic explorations over the past months, as well as from the many virtual modular generative sound systems I've built to improve the generative music in Soundgarden that is featured on the ecological island, including dynamic changes of tonality, more variations in timbre, and percussive sounds that integrate better with the overall soundscape and music.
• I still need to finish the northern part of the sound-led island. For the exhibition version, I had closed it off but I would still like to add another sound-led mechanic here. In fall of last year, I had prototyped a rhythm mechanic, where the player would move across floating platforms to one or multiple (synchronized) rhythms, however, I had not implemented that because I was looking for a way to do that in a way that integrated well with the given visual aesthetic of Soundgarden - and because I'm not sure if I'd want to implement gameplay that has been sufficiently explored across many existing rhythm games. Perhaps, I will need to think of something new here, inspired by one of the many sound exercises described by R. Murray Schafer, Pauline Oliveros' concept of Deep Listening, and looking at various (ethno-)musicological studies and musicking practices. These insights will also be helpful - and likely necessary - once I start thinking about a third sound interaction to be added in the lagoon area.

Learnings 🤓

So, what have I learned in the 1+ year of (on and off) working on Soundgarden? First off, Soundgarden was what finally pushed me over to learn Unreal, or rather, to use it productively and learn working with its various aspects over time. I now know various ways of implementing (real-time) audio with the engine: I've deepened my understanding of MetaSounds by transferring my knowledge of modular audio systems and electroacoustic approaches, learning how to implement real-time generative music and dynamic sound for all aspects of the game. I've both implemented sound in more traditional ways by placing static sound sources around the game environment, and developed, as well as implemented my own approaches for dynamic sound sources that would react to changes in the environment, such as time of day, or world geometry and the player's relation to it - for example the sound of the ocean moving around the islands, to be positioned closest to the player to give the impression of the ocean sound being present all around, while avoiding placement of many individual sound sources.

Small side note on moving audio sources and multiplayer: If I ever develop a multiplayer game, I should remember to adapt this "moving sound source" approach so that it happens individually for each player. Even if I have never developed a first person multiplayer game before, just putting this here so this thought will one day magically appear in the back of my head when it will be necessary.

I still need to learn how to apply audio effects like reverberation in Unreal, including making it dynamic and adapt to its immediate environment, to match its sonic features with the world geometry at any given point, i.e., larger closed spaces producing longer reverberation, smaller spaces, producing shorter reverberation - something to keep in mind for the Steam version. I also would like to implement support for more immersive sound setups, including surround sound and Dolby Atmos - it would be nice to hear the parrots' cries actually coming from above you as you walk on the beach below. I have learned how to implement all necessary game logic using UE's blueprint system, and how to develop my own modules in C++, to enhance engine functionality, especially when its more convenient than using visual programming. In that vein, I have learned the downsides of visual programming for certain scenarios, especially dealing with algorithms that require repeated iteration through data structures like arrays. I've also learned how to improve and maintain better performance so that projects run smoother (or at all!) on current and older/less powerful hardware, while still improving graphics quality, and in that vein, how gratifying performance optimization can feel sometimes. I've also learned to use UE's procedural content generation system to build my own tools for level design and creating more stylized environments. While I haven't done the latter in Soundgarden - and I wish I had, had I known how at the time - this will certainly impact how my future projects will look. Overall, these 1.5 years of working on my first released Unreal project has made my transition from Unity complete, meaning that for the time being, UE will be my technology of choice in developing any new games and virtual environments onward.

On a less technical level, this was also the first project throughout which I have consistently employed MDM - the method for design materialization. This was my most reflected upon project and I have come to understand and highly appreciate the power of thorough and constant reflection. I still remember how I had been stumbling through my first reflective journal entries but also how quickly this process became more natural and fluent for me. I find that my design decisions have become far more conscious - or even how I became aware of the in the first place. Where before I would make many decisions from within a flow state, working off vague ideas or repetitions of my own internalized established practices, I feel that now my decisions have become much more informed. This more aware decision making has led to consistent learnings about my own practice, my intentions, about what has already been done, what hasn't, what path I am shaping with my project, consistently checking in with myself and my internal design map - Where have I been? Where am I going? Am I still on the path I envisioned? Am I even taking the right path? What else is there? Journaling and detailed commit messages have helped me shed a light on intentions that perhaps before were buried in the tacit knowledge of my years of practicing game development, working on music and sound, and the theoretical and artistic knowledge I have absorbed both consciously and unconsciously through my work and experiencing the works of others. I find that the process of putting my finger, or rather a searchlight, on my influences and more consciously decide on my path forward has led to decisions that help me avoid running my project into a seeming dead end. I feel that my decisions now still make sense in the long run and help me to consistently improve the quality of and reasoning behind my work - turning me into a better developer, designer, researcher, artist - and perhaps, applying the practice of constant reflection on other areas of life - a better human being. Working on Soundgarden has also helped me to become more confident in the research trajectory that I am taking and helped me streamline my focus. This project was informed by research I had been conducting and informed more research questions in turn. The feedback that I received from players during the Ludodrome exhibition and after talking about Soundgarden as part of my DiGRA 2025 paper presentation pointed out the innovative paths I am taking with the project. People resonated with the idea of consciously engaging with sound in interactive experiences, they appreciated the contemplative nature of Soundgarden and commended its accessibility - both in terms of playability for people who don't usually engage with games, as well as showing potential in presenting game interactions that would work for individuals with visual impairments -the latter would certainly present yet another research trajectory to pursue in the future.

Closing off 🚪

This has become by far the longest journal entry for Soundgarden, though perhaps this makes sense, given the final (or not so final??) nature of this writing. While this is one big entry, I have been writing this one over the course of 2-3 weeks. Perhaps I could've split this into three parts: What I did between winter and Ludodrome, what I did for Ludodrome, and the actual post partum. So yeah, I'm actually not sure yet if this really works as a good post partum/post mortem - perhaps I'll have to look back once again on everything that happened and especially what it means for the project, my practice, and myself. With that said, journaling has been extremely helpful to reflect and improve upon my ideas. The development that I do these days has gained in depth, and - I dare to cautiously say - has improved the quality of the work and research I do. I certainly find it easier to put my thoughts in writing now. It has also tremendously helped enhancing my learnings and their retention. I feel like I've become a better game developer, perhaps even a better scholar, and I'm excited to see where my future ruminations will lead me. At this point, I would also like to extend my sincere gratitude to the Materializing Design project for putting me on the reflective pathways that enabled the making of Soundgarden and helped me improve upon my artistic/design work in general. I am also grateful to my three supervisors who each presented their invaluable perspectives that helped me shape the project and the trajectory of my research in profound ways.

The magic circle 🛞

I was showing Soundgarden to a friend - someone who does play some games from time to time, but who isn't necessarily someone who would identify himself as a gamer. I observed him playing the game and exploring the environment. One moment that was particularly noteworthy was him turning the player view in quick, erratic movements, jerking the mouse across the desk in various motions. This led to not only the view moving so fast it would turn into a blur - it also affected the surrounding sounds, quickly displacing them in the stereo spectrum and affecting their audible frequencies in interesting ways. This made me think about a new interaction for the lagoon area - one that is rooted in the player's movement. I was thinking about a designated spot - the magic circle - that the player can occupy. Once they enter the circle, certain sound objects in the area will react to the player moving the first person camera, i.e., when the player looks around. The velocity and direction of the camera movement could affect the output coming from these sound objects. The idea behind this is to direct the player's attention to the fact that moving their gaze, especially quickly, will affect how sounds around them can be perceived, and how they sound like. Ideally, the player would then not only play around with the interactive sound objects around this magic circle but also be inspired to experiment with such movements anywhere within the Soundgarden archipelago.

A brand new title ✨

In light of Soundgarden's Steam release, I had to think more seriously about branding and legal implications. While Soundgarden does share its name with the eponymous grunge band, it shouldn't be an issue since one is a band and the other a video game. But I don't want to get into trouble with some potentially overzealous trademark lawyers. Therefore a new name had to found for this project. After some rumination, marketing think, and several Google searches, I eventually ended up with: Beyond Sound: Soniferous Isles