libopenmpt emscripten/wasm into AudioWorkletProcessor

[jllodra]

Hi there,

I have been looking for a way to use libopenmpt with the new Audio Worklet API (more info here: https://developers.google.com/web/updates/2017/12/audio-worklet)

Thing is the Worklet is not able to load the libopenmpt.js file directly.

First I thought that the only way of making it work would be loading the .wasm file in the main thread, instantiate it, and then send to the Worklet with .postMessage.

After having a look at this code, https://github.com/GoogleChromeLabs/web-audio-samples/tree/master/audio-worklet/design-pattern/wasm/,
I tried to load straight away the .js file in the Worklet but I'm getting some errors in the console:

libopenmpt.js:1 ReferenceError: read is not defined
libopenmpt.js:1 failed to asynchronously prepare wasm: RuntimeError: abort(ReferenceError: read is not defined). Build with -s ASSERTIONS=1 for more info.

Before going any more into it, I would love to know if anyone here has done it or want to share knowledge.

Thank you!

[Saga Musix]

I haven't seen anyone using libopenmpt.js that way yet, and I haven't used this new API myself either. Did you start from something like chiptune2.js or similar to see that at least the basics are working?
Note that chiptune2.js hasn't been updated in a while, so you might have to make some changes which I did in my own modified version (for libopenmpt 0.4 and for libopenmpt 0.5).

[jllodra]

I have some good and bad news...

Changed

Code Select Expand


else ifeq ($(EMSCRIPTEN_TARGET),wasm)
# emits native wasm.
CPPFLAGS += -DMPT_BUILD_WASM
CXXFLAGS += -s WASM=1
CFLAGS   += -s WASM=1
LDFLAGS  += -s WASM=1

LDFLAGS += -s ALLOW_MEMORY_GROWTH=1


into

Code Select Expand


else ifeq ($(EMSCRIPTEN_TARGET),wasm)
# emits native wasm.
CPPFLAGS += -DMPT_BUILD_WASM
CXXFLAGS += -s WASM=1 -s BINARYEN_ASYNC_COMPILATION=0 -s=SINGLE_FILE
CFLAGS   += -s WASM=1 -s BINARYEN_ASYNC_COMPILATION=0 -s=SINGLE_FILE
LDFLAGS  += -s WASM=1 -s BINARYEN_ASYNC_COMPILATION=0 -s=SINGLE_FILE

LDFLAGS += -s ALLOW_MEMORY_GROWTH=1


Compiles and works, I can load it from the Audio Worklet Processor. Also had to add an "export default Module;" at the end of the generated js to make it compatible with es6 modules.

Problem is... emscripten relies on performance.now() to get the high-resolution time. The worklet thread has no access to that function.
As a workaround, I set performance.now to be Date.now, less resolution (it would "work" but I have no idea of the impact).

But this problem, I have no idea how to solve it, it seems the libopenmpt need to generate random values (I don't know why), so emscripten uses this code,
using the "window.crypto" built-in function, which again are unavailabe in the Audio Worklet.

Code Select Expand

function getRandomDevice() {
  if (typeof crypto === "object" && typeof crypto["getRandomValues"] === "function") {
    var randomBuffer = new Uint8Array(1);
    return function () {
      crypto.getRandomValues(randomBuffer);
      return randomBuffer[0]
    }
  } else if (ENVIRONMENT_IS_NODE) {
    try {
      var crypto_module = require("crypto");
      return function () {
        return crypto_module["randomBytes"](1)[0]
      }
    } catch (e) {
    }
  }
  return function () {
    abort("randomDevice")
  }
}

At this point I have no idea how to proceed... This is my worklet code. I guess that I could re-implement in JS a random number generator, if that is what emscripten needs.

Code Select Expand


import Module from './libopenmpt.js';

const libopenmpt = Module;

class ModPlayer extends AudioWorkletProcessor {

  maxFramesPerChunk = 4096;

  ptrToFile;
  modulePtr;

  constructor(options) {
    super();
    console.log(window);
    console.log(libopenmpt);
    console.log(libopenmpt._openmpt_get_library_version());
    this.port.onmessage = (event) => {
      console.log(event.data);
      this.ptrToFile = libopenmpt._malloc(event.data.byteLength);
      libopenmpt.HEAPU8.set(event.data, this.ptrToFile);
      this.modulePtr = libopenmpt._openmpt_module_create_from_memory(this.ptrToFile, event.data.byteLength, 0, 0, 0);
      this.leftBufferPtr  = libopenmpt._malloc(4 * this.maxFramesPerChunk);
      this.rightBufferPtr = libopenmpt._malloc(4 * this.maxFramesPerChunk);

    };
  }

  process(inputs, outputs, parameters) {

    return true;
  }
}

registerProcessor('mod-player', ModPlayer);


Regarding your question, yeah, it works with the old api: "createScriptProcessor()" in the main UI thread, etc... that's how I used to do too. Now I'm trying to fill the audio buffers in a separate thread.

[manx]

Emscripten documents WASM_ASYNC_COMPILATION=1 (default) as being required to work in Chrome (https://github.com/emscripten-core/emscripten/blob/master/src/settings.js#L1141), so we cannot use this option by default. Also i fail to see why SINGLE_FILE=1 would make a difference other than simplify loading.

libopenmpt requires random numbers for certain effects (random vibrator/tremolo, and similar for example), and for output dithering (probably irrelevant to your use case if you are not using 16bit integer but instead floating point output sample format). Even though libopenmpt could in theory avoid proper random numbers here and fallback to an internal simple pseudo-random-number-generator, I would rather not implement a hack specifically for this situation in AudioWorklets with emscripten. I think this should get fixed somehow in emscripten, because after all this is a problem for any C++ program using std::random.

In any case, I think your questions are probably better suited for the Emscripten community as we are by no means experts on JavaScript or WASM or AudioWorklets or the emscripten framework. https://github.com/emscripten-core/emscripten/issues/6230 seems related, but you have probably already found that, I guess.

[jllodra]

Got it working ) Just used a plain JS polyfill for web crypto and implemented the missing parts I needed.

Thanks for the reply.

SINGLE_FILE=1 is needed because in the context of an Audio Worklet, no other files can be accessed/loaded, and libopenmpt.js file loads libopenmpt.wasm.
Having both files together in libopenmpt.js makes the loading possible.

Now I should try more modules and see if the playback is OK for all.

[manx]

Also had to add an "export default Module;" at the end of the generated js to make it compatible with es6 modules.

Does -s MODULARIZE=1 -s EXPORT_ES6=1 also work? (see https://github.com/emscripten-core/emscripten/blob/master/src/settings.js#L1024) I for sure do not understand the consequences of MODULARIZE=1, which is why I am asking.

If it does work, a new makefile option EMSCRIPTEN_TARGET=audioworklet-wasm which sets -s WASM_ASYNC_COMPILATION=0 -s MODULARIZE=1 -s EXPORT_ES6=1 -s=SINGLE_FILE=1 would be useful and the only 2 problems left would be clock and random, right?

[jllodra]

I will try the options you mention and let you know if they work.

Btw, I am only testing in latest version of Chrome atm.

I have been trying a couple of mods with a considerable amount of channels and once in a minute I heard
a small dropout/glitch. I think this happens because of the buffer size of the Audio Worklets, which is fixed at 128 samples per block.

128 * 1000 / 48000 ≅ 3 ms.

I am not sure if this is related with 3 ms not being enough time, or not. Also, I am not sure if adding a Ring Buffer in the chain would help or have no impact on the time available for the processing.

I am now testing this:

Code Select Expand


const context = new AudioContext({
  latencyHint: 'playback'
});


https://webaudio.github.io/web-audio-api/#enumdef-audiocontextlatencycategory
and for the moment I haven't heard any playback issue, if I hear any "click" or "pop" I'm sure is because of song samples themselves.
Anyway, the old "createScriptProcessor" way also created dropouts easily since all happened in the same browser script thread.

Regarding the clock and the rng issues, the best thing I could came up was using Date instead of performance, and a crypto polyfill library in plain JS.
A solution that is working surprisingly well.

[jllodra]

Code Select Expand

else ifeq ($(EMSCRIPTEN_TARGET),wasm)
# emits native wasm.
CPPFLAGS += -DMPT_BUILD_WASM
CXXFLAGS += -s WASM=1 -s BINARYEN_ASYNC_COMPILATION=0 -s SINGLE_FILE=1
CFLAGS   += -s WASM=1 -s BINARYEN_ASYNC_COMPILATION=0 -s SINGLE_FILE=1
LDFLAGS  += -s WASM=1 -s BINARYEN_ASYNC_COMPILATION=0 -s SINGLE_FILE=1

LDFLAGS += -s ALLOW_MEMORY_GROWTH=1

else ifeq ($(EMSCRIPTEN_TARGET),wasm2)
# emits native wasm.
CPPFLAGS += -DMPT_BUILD_WASM
CXXFLAGS += -s WASM=1 -s WASM_ASYNC_COMPILATION=0 -s MODULARIZE=1 -s EXPORT_ES6=1 -s SINGLE_FILE=1
CFLAGS   += -s WASM=1 -s WASM_ASYNC_COMPILATION=0 -s MODULARIZE=1 -s EXPORT_ES6=1 -s SINGLE_FILE=1
LDFLAGS  += -s WASM=1 -s WASM_ASYNC_COMPILATION=0 -s MODULARIZE=1 -s EXPORT_ES6=1 -s SINGLE_FILE=1

Both options work. Now I found that

The latter one, it correctly exports the module but it does as a function, so I have to actually call it, eg:

Code Select Expand


import libopenmpt from './libopenmpt.js';

const libopenmpt1 = libopenmpt();


No big deal.

If I'm not wrong BINARYEN_ASYNC_COMPILATION and WASM_ASYNC_COMPILATION are the same but the first is the older name, isn't it?

[manx]

Both options work. Now I found that

The latter one, it correctly exports the module but it does as a function, so I have to actually call it, eg:

Code Select Expand


import libopenmpt from './libopenmpt.js';

const libopenmpt1 = libopenmpt();


Thanks for testing. As this is the way that emscripten supports directly, I think this would be the preferred way to go.

If I'm not wrong BINARYEN_ASYNC_COMPILATION and WASM_ASYNC_COMPILATION are the same but the first is the older name, isn't it?

Yes, that's also how I understand it.

[jllodra]

You're welcome

Yeah, sure.

Maybe we could use the --pre-js flag to add the Date/crypto stuff...

[manx]

Maybe we could use the --pre-js flag to add the Date/crypto stuff...

I would prefer if you could report these issues to emscripten and let the emscripten developers find a solution because this will certainly affect other C++ programs/libraries as well.

[jllodra]

OK, raised an issue, here it is in case you want to check it out:
https://github.com/emscripten-core/emscripten/issues/13224

Regular Web Workers (not Audio Worklets) have access to performance and crypto (through the "self" object iirc),
I think this issue is very specific to audio.

[manx]

Code Select Expand

[quote author=jllodra link=topic=6548.msg47864#msg47864 date=1610282367]
Problem is... emscripten relies on performance.now() to get the high-resolution time. The worklet thread has no access to that function.


Could you check if http://manx.datengang.de/openmpt/temp/avoid-highres-clock-on-emscripten-v3.patch avoids the requirement on performance.now()? This would be a work-around that we could integrate into libopenmpt.

[jllodra]

Sure, I'll try. Just wondering, what is the hires clock used for if it can be omitted?

[manx]

Sure, I'll try. Just wondering, what is the hires clock used for if it can be omitted?

It's used for seeding a pseudo random number generator in case the proper system-provided random_device fails to return data (which sadly is a valid implementation for C++ std::random_device).