today I played around with adding audio convolvers to Mysterious Space.

what are these strange “audio convolver” things? they’re some special algorithms that let you mutate sounds in interesting ways, in real time, to accomplish all kinds of neat effects.

I first encountered them when working on a game in RPG Maker MV. I wanted to add an echo effect to the sounds made by the player, when the player was in a cave. making echo-y versions of the various sound effects (walking, picking up coins, starting fights…), and then choosing which to play based on location, would be time-consuming, error-prone, and difficult to scale up (every new sound added later would require that much more work). further, I was pretty positive that better solutions existed, I just didn’t know what they were! so I started to look around online for how to accomplish the effect I was looking for.

the answer, it turns out, is audio convolution! in particular, a technique where you create an “Impulse Response” by recording a sharp sound (like a clap) made in a space, and then using MAGIC to combine that Impulse Response with whatever sound you want. the result is a new sound that sounds like it’s being played in the location that your Impulse Response was recorded.

you can create an Impulse Response in a large room, a narrow hallway, an echo-y bathroom, wrapped in thick cloth, or whatever you want, and use audio convolution to playback your existing sounds so that they sound like they were recorded in those places/environments.

how this mathematical wizardry works, I don’t know, but happily I don’t HAVE to know (and neither do you!) RPG Maker MV runs on JavaScript, and – to my surprise – vanilla JS has support for audio convolutions built in! many other platforms have support for these as well, including FMOD.

FMOD is a library you can use for playing sounds. I use it to handle all the sound and music in Mysterious Space. MonoGame (the library I use to do graphics drawing, input handling, and tons of other stuff) has audio capabilities as well, however it does not support as many audio file formats, and I could never get it to loop sounds seamlessly (important for background music!) (I’m also skeptical that MonoGame supports audio convolutions, but I haven’t looked into it.)

FMOD takes a bit of work to get set up (it’s a very “low-level” library: it provides advanced features, but rarely wraps them up in convenient-to-use ways), but it offers a lot of power, including the power of audio convolvers.

so I spent a bit of time today figuring out how to get audio convolvers going in Mysterious Space using FMOD, but ran into a problem. not a problem with FMOD, but with the design of Mysterious Space itself!

the effect I want to accomplish with audio convolvers in Mysterious Space is to add an underwater effect to all sounds while your ship is underwater. the problem I’ve run into, however, is that Mysterious Space can be played in local, split-screen co-op mode. up to four screens. why is this a problem? because SOME of those screens might be underwater, while some are not! and I’ve never had a need to play sounds differently for different players before, so Mysterious Space is currently set up to play all sounds through a single, master, sound-player. that has to be changed, or else all sounds played everywhere will have to be subjected to the same audio convolutions.

so I modified the sound-player to have multiple channels – up to five (one per player, plus a player-independent one for music) – and made it require that when you ask it to play a sound, that you also specify which channel to play that sound on! this way, each player can grab their own channel, which can each have different audio convolvers going on.

BUT WAIT: that works really easily for sounds which are clearly triggered by an individual player, like menu cursor bleeps, but even shooting a weapon… what if I shoot a weapon while I’m under water, and you – a co-op player – are nearby, but ABOVE water. how do we play that weapon fire noise through a single speaker system? and maybe it would be good enough to resolve this by saying “if player 1 shot it, then player 1 plays the sound according to whatever rules, and player 2 doesn’t play the sound at all”, but what about for enemies? what if I’m underwater, and you’re above water, and an enemy that’s underwater shoots a weapon, and an enemy that’s above water shoots a weapon…

two possibilities come to mind:

1. whichever player is closest to the source of the sound, THAT player is responsible for playing the sound
2. all players play the sound according to their rules, and we use some additional magic to average the way they each want to play it

I’m thinking #2 is both better, and probably supported by FMOD (even if I don’t quite know how). but I don’t really know. this will require further investigation.

problems like this make me a little sad that I’m not using Unity to make Mysterious Space, where these kinds of problems have already been solved. in Unity, you’d simply attach a “Listener” to each camera, and that’s it! Unity has already figured out what to do when a single sound is heard by multiple “Listeners”. unfortunately, I didn’t know Unity when I started Mysterious Space, and I’m not about to try to rewrite/translate the game into Unity >_> (I am writing Mysterious Space in C# already, which is great, but split-screen logic, input-handling, and SO much core stuff would have to be totally redone…)

and anyway, these are interesting problems to solve. I don’t mind having to figure them out myself, even if it means things take a little longer! 🙂

I haven’t solved the “split-screen audio with convolvers” problem for Mysterious Space yet, and I’m a little tired of working on this particular problem for now, but I’ll get there! and in the meanwhile, there are plenty of other interesting problems to work on solving for Mysterious Space (speech controls??), so I’m going to pivot to one of those 😛