Hi there,
I have been looking for a way to use libopenmpt with the new Audio Worklet API (more info here: https://developers.google.com/web/updates/2017/12/audio-worklet)
Thing is the Worklet is not able to load the libopenmpt.js file directly.
First I thought that the only way of making it work would be loading the .wasm file in the main thread, instantiate it, and then send to the Worklet with .postMessage.
After having a look at this code, https://github.com/GoogleChromeLabs/web-audio-samples/tree/master/audio-worklet/design-pattern/wasm/,
I tried to load straight away the .js file in the Worklet but I'm getting some errors in the console:
libopenmpt.js:1 ReferenceError: read is not defined
libopenmpt.js:1 failed to asynchronously prepare wasm: RuntimeError: abort(ReferenceError: read is not defined). Build with -s ASSERTIONS=1 for more info.
Before going any more into it, I would love to know if anyone here has done it or want to share knowledge.
Thank you!
I haven't seen anyone using libopenmpt.js that way yet, and I haven't used this new API myself either. Did you start from something like chiptune2.js or similar to see that at least the basics are working?
Note that chiptune2.js hasn't been updated in a while, so you might have to make some changes which I did in my own modified version (for libopenmpt 0.4 (https://github.com/sagamusix/s3m.it/commit/b69bf732bc9e2a2a0d596f536d4282fe7c07f871) and for libopenmpt 0.5 (https://github.com/sagamusix/s3m.it/commit/a1fe1535ae0963d7b49a57a0f9532b2915801f54#diff-16cedf80ade01c62bdd1ae931d0492330c0b62bf294c08c095ce2fab21a9298d)).
I have some good and bad news...
Changed
else ifeq ($(EMSCRIPTEN_TARGET),wasm)
# emits native wasm.
CPPFLAGS += -DMPT_BUILD_WASM
CXXFLAGS += -s WASM=1
CFLAGS += -s WASM=1
LDFLAGS += -s WASM=1
LDFLAGS += -s ALLOW_MEMORY_GROWTH=1
into
else ifeq ($(EMSCRIPTEN_TARGET),wasm)
# emits native wasm.
CPPFLAGS += -DMPT_BUILD_WASM
CXXFLAGS += -s WASM=1 -s BINARYEN_ASYNC_COMPILATION=0 -s=SINGLE_FILE
CFLAGS += -s WASM=1 -s BINARYEN_ASYNC_COMPILATION=0 -s=SINGLE_FILE
LDFLAGS += -s WASM=1 -s BINARYEN_ASYNC_COMPILATION=0 -s=SINGLE_FILE
LDFLAGS += -s ALLOW_MEMORY_GROWTH=1
Compiles and works, I can load it from the Audio Worklet Processor. Also had to add an "export default Module;" at the end of the generated js to make it compatible with es6 modules.
Problem is... emscripten relies on performance.now() to get the high-resolution time. The worklet thread has no access to that function.
As a workaround, I set performance.now to be Date.now, less resolution (it would "work" but I have no idea of the impact).
But this problem, I have no idea how to solve it, it seems the libopenmpt need to generate random values (I don't know why), so emscripten uses this code,
using the "window.crypto" built-in function, which again are unavailabe in the Audio Worklet.
function getRandomDevice() {
if (typeof crypto === "object" && typeof crypto["getRandomValues"] === "function") {
var randomBuffer = new Uint8Array(1);
return function () {
crypto.getRandomValues(randomBuffer);
return randomBuffer[0]
}
} else if (ENVIRONMENT_IS_NODE) {
try {
var crypto_module = require("crypto");
return function () {
return crypto_module["randomBytes"](1)[0]
}
} catch (e) {
}
}
return function () {
abort("randomDevice")
}
}
At this point I have no idea how to proceed... This is my worklet code. I guess that I could re-implement in JS a random number generator, if that is what emscripten needs.
import Module from './libopenmpt.js';
const libopenmpt = Module;
class ModPlayer extends AudioWorkletProcessor {
maxFramesPerChunk = 4096;
ptrToFile;
modulePtr;
constructor(options) {
super();
console.log(window);
console.log(libopenmpt);
console.log(libopenmpt._openmpt_get_library_version());
this.port.onmessage = (event) => {
console.log(event.data);
this.ptrToFile = libopenmpt._malloc(event.data.byteLength);
libopenmpt.HEAPU8.set(event.data, this.ptrToFile);
this.modulePtr = libopenmpt._openmpt_module_create_from_memory(this.ptrToFile, event.data.byteLength, 0, 0, 0);
this.leftBufferPtr = libopenmpt._malloc(4 * this.maxFramesPerChunk);
this.rightBufferPtr = libopenmpt._malloc(4 * this.maxFramesPerChunk);
};
}
process(inputs, outputs, parameters) {
return true;
}
}
registerProcessor('mod-player', ModPlayer);
Regarding your question, yeah, it works with the old api: "createScriptProcessor()" in the main UI thread, etc... that's how I used to do too. Now I'm trying to fill the audio buffers in a separate thread.
Emscripten documents WASM_ASYNC_COMPILATION=1 (default) as being required to work in Chrome (https://github.com/emscripten-core/emscripten/blob/master/src/settings.js#L1141 (https://github.com/emscripten-core/emscripten/blob/master/src/settings.js#L1141)), so we cannot use this option by default. Also i fail to see why SINGLE_FILE=1 would make a difference other than simplify loading.
libopenmpt requires random numbers for certain effects (random vibrator/tremolo, and similar for example), and for output dithering (probably irrelevant to your use case if you are not using 16bit integer but instead floating point output sample format). Even though libopenmpt could in theory avoid proper random numbers here and fallback to an internal simple pseudo-random-number-generator, I would rather not implement a hack specifically for this situation in AudioWorklets with emscripten. I think this should get fixed somehow in emscripten, because after all this is a problem for any C++ program using std::random.
In any case, I think your questions are probably better suited for the Emscripten community as we are by no means experts on JavaScript or WASM or AudioWorklets or the emscripten framework. https://github.com/emscripten-core/emscripten/issues/6230 (https://github.com/emscripten-core/emscripten/issues/6230) seems related, but you have probably already found that, I guess.
Got it working :))) Just used a plain JS polyfill for web crypto and implemented the missing parts I needed.
Thanks for the reply.
SINGLE_FILE=1 is needed because in the context of an Audio Worklet, no other files can be accessed/loaded, and libopenmpt.js file loads libopenmpt.wasm.
Having both files together in libopenmpt.js makes the loading possible.
Now I should try more modules and see if the playback is OK for all.
Quote from: jllodra on January 10, 2021, 12:39:27
Also had to add an "export default Module;" at the end of the generated js to make it compatible with es6 modules.
Does
-s MODULARIZE=1 -s EXPORT_ES6=1 also work? (see https://github.com/emscripten-core/emscripten/blob/master/src/settings.js#L1024 (https://github.com/emscripten-core/emscripten/blob/master/src/settings.js#L1024)) I for sure do not understand the consequences of
MODULARIZE=1, which is why I am asking.
If it does work, a new makefile option
EMSCRIPTEN_TARGET=audioworklet-wasm which sets
-s WASM_ASYNC_COMPILATION=0 -s MODULARIZE=1 -s EXPORT_ES6=1 -s=SINGLE_FILE=1 would be useful and the only 2 problems left would be clock and random, right?
I will try the options you mention and let you know if they work.
Btw, I am only testing in latest version of Chrome atm.
I have been trying a couple of mods with a considerable amount of channels and once in a minute I heard
a small dropout/glitch. I think this happens because of the buffer size of the Audio Worklets, which is fixed at 128 samples per block.
128 * 1000 / 48000 ≅ 3 ms.
I am not sure if this is related with 3 ms not being enough time, or not. Also, I am not sure if adding a Ring Buffer in the chain would help or have no impact on the time available for the processing.
I am now testing this:
const context = new AudioContext({
latencyHint: 'playback'
});
https://webaudio.github.io/web-audio-api/#enumdef-audiocontextlatencycategory
and for the moment I haven't heard any playback issue, if I hear any "click" or "pop" I'm sure is because of song samples themselves.
Anyway, the old "createScriptProcessor" way also created dropouts easily since all happened in the same browser script thread.
Regarding the clock and the rng issues, the best thing I could came up was using Date instead of performance, and a crypto polyfill library in plain JS.
A solution that is working surprisingly well.
else ifeq ($(EMSCRIPTEN_TARGET),wasm)
# emits native wasm.
CPPFLAGS += -DMPT_BUILD_WASM
CXXFLAGS += -s WASM=1 -s BINARYEN_ASYNC_COMPILATION=0 -s SINGLE_FILE=1
CFLAGS += -s WASM=1 -s BINARYEN_ASYNC_COMPILATION=0 -s SINGLE_FILE=1
LDFLAGS += -s WASM=1 -s BINARYEN_ASYNC_COMPILATION=0 -s SINGLE_FILE=1
LDFLAGS += -s ALLOW_MEMORY_GROWTH=1
else ifeq ($(EMSCRIPTEN_TARGET),wasm2)
# emits native wasm.
CPPFLAGS += -DMPT_BUILD_WASM
CXXFLAGS += -s WASM=1 -s WASM_ASYNC_COMPILATION=0 -s MODULARIZE=1 -s EXPORT_ES6=1 -s SINGLE_FILE=1
CFLAGS += -s WASM=1 -s WASM_ASYNC_COMPILATION=0 -s MODULARIZE=1 -s EXPORT_ES6=1 -s SINGLE_FILE=1
LDFLAGS += -s WASM=1 -s WASM_ASYNC_COMPILATION=0 -s MODULARIZE=1 -s EXPORT_ES6=1 -s SINGLE_FILE=1
Both options work. Now I found that
The latter one, it correctly exports the module but it does as a function, so I have to actually call it, eg:
import libopenmpt from './libopenmpt.js';
const libopenmpt1 = libopenmpt();
No big deal.
If I'm not wrong BINARYEN_ASYNC_COMPILATION and WASM_ASYNC_COMPILATION are the same but the first is the older name, isn't it?
Quote from: jllodra on January 10, 2021, 17:43:01
Both options work. Now I found that
The latter one, it correctly exports the module but it does as a function, so I have to actually call it, eg:
import libopenmpt from './libopenmpt.js';
const libopenmpt1 = libopenmpt();
Thanks for testing. As this is the way that emscripten supports directly, I think this would be the preferred way to go.
Quote from: jllodra on January 10, 2021, 17:43:01
If I'm not wrong BINARYEN_ASYNC_COMPILATION and WASM_ASYNC_COMPILATION are the same but the first is the older name, isn't it?
Yes, that's also how I understand it.
You're welcome
Yeah, sure.
Maybe we could use the --pre-js flag to add the Date/crypto stuff...
Quote from: jllodra on January 10, 2021, 17:57:42
Maybe we could use the --pre-js flag to add the Date/crypto stuff...
I would prefer if you could report these issues to emscripten and let the emscripten developers find a solution because this will certainly affect other C++ programs/libraries as well.
OK, raised an issue, here it is in case you want to check it out:
https://github.com/emscripten-core/emscripten/issues/13224
Regular Web Workers (not Audio Worklets) have access to performance and crypto (through the "self" object iirc),
I think this issue is very specific to audio.
[quote author=jllodra link=topic=6548.msg47864#msg47864 date=1610282367]
Problem is... emscripten relies on performance.now() to get the high-resolution time. The worklet thread has no access to that function.
Could you check if http://manx.datengang.de/openmpt/temp/avoid-highres-clock-on-emscripten-v3.patch (http://manx.datengang.de/openmpt/temp/avoid-highres-clock-on-emscripten-v3.patch) avoids the requirement on performance.now()? This would be a work-around that we could integrate into libopenmpt.
Sure, I'll try. Just wondering, what is the hires clock used for if it can be omitted?
Quote from: jllodra on January 11, 2021, 09:42:33
Sure, I'll try. Just wondering, what is the hires clock used for if it can be omitted?
It's used for seeding a pseudo random number generator in case the proper system-provided random_device fails to return data (which sadly is a valid implementation for C++ std::random_device).
Built with the patch, and it works without the performance.now polyfill.
The thing is that the performance.now code is still there in the libopenmpt.js, but not called (at least not when loading and playing a song, which is what I'm doing).
Edit: had a look at https://github.com/emscripten-core/emscripten/blob/master/src/library.js#L2934 to see the actual code and why performance.now is there. Anyway, I have proved that with patch, _clock_gettime is called and that uses Date.now internally.
Quote from: jllodra on January 11, 2021, 11:55:41
Built with the patch, and it works without the performance.now polyfill.
Thanks for testing.
https://source.openmpt.org/browse/openmpt/?op=revision&rev=14045 (https://source.openmpt.org/browse/openmpt/?op=revision&rev=14045) will be in the next release of every branch.
Nice, thanks.
Regarding the random numbers, I can't see why emscripten can't rely on Math.random() when crypto is now found, do you happen to have an idea?
I understand that it has not enough entropy level from a cryptography point of view, but I wonder if this would have an impact.
https://github.com/emscripten-core/emscripten/issues/542
EDIT: OH, they USED TO fallback on Math.random, but not anymore: https://github.com/emscripten-core/emscripten/blob/master/src/library.js#L2858
I have just changed crypto to Math.random and it works, I cannot hear any difference in the music.
I have added a ticket to our issue tracker for the Makefile changes: https://bugs.openmpt.org/view.php?id=1407 (https://bugs.openmpt.org/view.php?id=1407).
This code at the beginning of the Module
const crypto = {
getRandomValues: (array) => {
for (let i = 0; i < array.length; i++) {
array[i] = (Math.random()*256)|0;
}
}
};
and no need for third party polyfills...
r14059 (libopenmpt 0.6 development) or r14060 (libopenmpt 0.5) might work without requiring any polyfill when using make CONFIG=emscripten EMSCRIPTEN_TARGET=audioworkletprocessor. Could you please check that?
You can either get unreleased sources from svn or git, or from tarballs at https://builds.openmpt.org/builds/auto/libopenmpt/src.makefile/ .
Hi Manx,
will try to have a look during the weekend.
Another problem I had to overcome too is that with the old libopenmpt emscripten version (which I built 3 years ago maybe), I could do things like:
libopenmpt.Pointer_stringify(libopenmpt._openmpt_module_get_metadata(memPtr, keyNameBuffer));
With the current version, I have to do:
UTF8ToString(libopenmpt._openmpt_module_get_metadata(memPtr, keyNameBuffer));
This UTF8ToString function is not being exported using the new built strategy (ES6 module). So I had to export it myself in the generated .js code (not desirable).
Do you know where this UTF8ToString function comes from? Is it from emscripten or is it from libopenmpt?
https://emscripten.org/docs/api_reference/preamble.js.html
Seems to be defined here but unfortunately it is not exposed in the generated .js module.
Answering to my last comment,
probably we'd need to add:
-s "EXPORTED_RUNTIME_METHODS=['UTF8ToString']"
no idea if this will work, will test.
There are several other functions that only some users may need that are not exported by default. Everyone may need a different subset of those, so I think leaving it up to the user to specify them is still a good solution. Maybe it should be documented better, but as you noticed yourself the list of potentially useful stuff also keeps changing with every Emscripten release, so it's difficult to provide a list of options.
Here's what I use on a custom libopenmpt.js build that requires all sorts of metadata extraction, before invoking make:
export CXXFLAGS="-s \"EXTRA_EXPORTED_RUNTIME_METHODS=['lengthBytesUTF8','stackAlloc','stackSave','stackRestore','stringToUTF8','Pointer_stringify','writeArrayToMemory']\""
Quote from: manx on January 20, 2021, 10:28:58
r14059 (libopenmpt 0.6 development) or r14060 (libopenmpt 0.5) might work without requiring any polyfill when using make CONFIG=emscripten EMSCRIPTEN_TARGET=audioworkletprocessor. Could you please check that?
You can either get unreleased sources from svn or git, or from tarballs at https://builds.openmpt.org/builds/auto/libopenmpt/src.makefile/ .
Excellent, it works without polyfills now.
@Saga Musix: Cool, that helped. Thanks!
I think all set with this issue.