Libopenmpt and audio latency

[bass]

I wrote an Android player for mod files and wanted to support more tracker formats now.
Reinventing the wheel and programming everything myself would be extremely time-consuming. I have therefore decided to use your library for all other formats. The integration was relatively simple and worked straight away.
Now I have the question of how to synchronize the UI with the module positions as there are Android devices that sometimes have 200 ms or more latency in the audio output.

In my UI I show the current row, pattern and song positions. I also have a tracker view where the notes, samples and effects are shown for each channel like in OpenMPT.
This should be possible according to the API.

For mod files, I do the synchronization so that I save the information I need for the UI update for each tick and provide it with a timestamp. This way, I can update the UI when the audio is really being played at this position.

But how can I do this with the library API?
As I have seen, there is only the method read. And also the current position in seconds doesn't help me that much.
What do you suggest and how is the latency handled in OpenMPT?
Thanks!

[Saga Musix]

As libopenmpt only decodes audio, there is, by definition, no latency involved at any step as far as libopenmpt itself is concerned. Latency is purely a property of the audio playback API that you use, hence the tools that are needed for synchronized playback must be provided by the audio API - and if they are not available, must be emulated in some way or another. For example, here is a simplified breakdown of what OpenMPT does to synchronize graphics and audio:

In the audio callback, for each rendered block of audio (equivalent to a read call in libopenmpt), store the total number of rendered frames so far, and all the information you are interested in (row position, order position, etc.) in a ring buffer. Note that if the callback asks for a large amount of audio (e.g. 100ms or more) in one go, it makes sense to split up libopenmpt read calls into multiple smaller calls, to increase the resolution of time stamps in the ring buffer.
The size of the ring buffer should large enough to hold enough update notifications for the largest latency you would expect and the smallest update interval you expect. For example, assuming that some worst-case device has a latency of 2 seconds but runs its audio callback every 5ms, you'd need a ring buffer with at least 2000/5=400 elements. OpenMPT's ring buffer contains up to 2000 elements.

In the graphics update thread, retrieve the current audio stream playback position in frames and find the element in the ring buffer closest to the playback position (timestamp in buffer is equal or greater than current position). Update your graphics according to the information contained in this ring bufer element. If the audio API is not able to provide the current playback position, you may have to improvise, but on Android you should be able to use something like AudioTrack.getPlaybackHeadPosition to retrieve the playback position in frames.

A similar mechanism is used for the pattern visualizer in our XMPlay input plugin, which you can find in libopenmpt/xmp-openmpt/xmp-openmpt.cpp. Look how functions update_timeinfos and lookup_timeinfo are used there to get an idea what needs to be done.

[manx]

I wrote an Android player for mod files and wanted to support more tracker formats now.
Reinventing the wheel and programming everything myself would be extremely time-consuming. I have therefore decided to use your library for all other formats. The integration was relatively simple and worked straight away.

Great to hear that!

Now I have the question of how to synchronize the UI with the module positions as there are Android devices that sometimes have 200 ms or more latency in the audio output.

In my UI I show the current row, pattern and song positions. I also have a tracker view where the notes, samples and effects are shown for each channel like in OpenMPT.
This should be possible according to the API.

For mod files, I do the synchronization so that I save the information I need for the UI update for each tick and provide it with a timestamp. This way, I can update the UI when the audio is really being played at this position.

That's the recommended approach. The granularity of such timing information is (currently) limited to the granularity with which you call the read() function. This will be improved in the future, and progress in that area is tracked in https://bugs.openmpt.org/view.php?id=1017. I suggest you monitor that issue and give feedback about API design (if you have some).

But how can I do this with the library API?
As I have seen, there is only the method read. And also the current position in seconds doesn't help me that much.
What do you suggest and how is the latency handled in OpenMPT?

OpenMPT internally does roughly the same, but it has access to the exact tick boundaries, and thus can record the timing information precisely at each tick change (which libopenmpt still does not support, see linked issue).

[bass]

Thanks Saga Musix and manx for the detailed answers!
@Saga Musix, I'm doing the synchronization already nearly exactly the way you described and also using AudioTrack.getPlaybackHeadPosition for this. I only have a list of marker elements where I add and remove elements instead of a ring buffer.

I thought there might be something in the API that I missed, but then I'll make it so that I just render a maximum of 10-20 ms ranges using read (that should look reasonably synchronized, I think) and then manage them as position marker elements for my UI update thread.

I will also check the ticket, you mentioned manx, thank you both!

[bass]

I have now experimented a little with the API. Most things are self-explanatory, I found the module_ext and interactive, interactive2 and interactive3 a bit confusing, probably programmed that way for historical reasons.
What I miss, however, is the possibility of getting a pattern data from a channel with all rows completely as a JSON string for the tracker view. Although there is the "format_pattern_row_channel_command" method, it could be easier to use if you want to display all the information in the tracker view. In my case, it is also not very performant, as I have to execute the commands via the JNI interface, which always involves some additional overhead. I would have over 1000 API-calls for a 4 channel module with 64 rows to get the information for one pattern instead of 4 API-calls. But I will do a performance test, maybe it's not as bad as I think.
Are there any plans to do something about this? OpenMPT can probably access the pattern data directly, which makes things much easier.

[manx]

I have now experimented a little with the API. Most things are self-explanatory, I found the module_ext and interactive, interactive2 and interactive3 a bit confusing, probably programmed that way for historical reasons.

This is actually less so for historic reasons and more so for future reasons. The extension APIs are not meant to be final and in their current form available forever, because they are either not finally designed and incomplete, or because they expose internal implementation details that might change in the future. As these APIs may disappear, you also need to check for interface availability before using them (less of an issue when you are only ever building client code and matching libopenmpt version in one go, which you probably are doing for Android).

What I miss, however, is the possibility of getting a pattern data from a channel with all rows completely as a JSON string for the tracker view. Although there is the "format_pattern_row_channel_command" method, it could be easier to use if you want to display all the information in the tracker view.
In my case, it is also not very performant, as I have to execute the commands via the JNI interface, which always involves some additional overhead. I would have over 1000 API-calls for a 4 channel module with 64 rows to get the information for one pattern instead of 4 API-calls. But I will do a performance test, maybe it's not as bad as I think.

The primary API of libopenmpt is the C++ API, and for C++, the implemented mechanism is actually more performant than serialization to JSON or any other less granular approach would be, because for C++ we can rely on small-string-optimization, which avoids (almost) all allocations in this code path. Memory allocations easily dominate function call overhead. For the C API wrapper, this sadly implies allocating each individual string that gets returned, which has overhead. However we cannot return internal string data because this actually does not (and should not) exist anywhere, so there is not even any other option. We also cannot use fixed-size buffers for the C API, because note names can actually be UTF-8 encoded (when an MPTM module is using custom note names). Highlight handling for UTF-8 data probably needs some work and maybe even an API re-design.

Are there any plans to do something about this?

I really do not see an easy way to provide a fundamentally different API. Batching and serializing multiple cells/rows/channel/whatever into a single call would reduce call overhead and may help for C and other wrapped APIs, so we maybe could consider this. Can you make a ticket about this at https://bugs.openmpt.org/?

OpenMPT can probably access the pattern data directly, which makes things much easier.

In OpenMPT, this is more or less directly integrated into the drawing code, which has its own set of problems. Of course OpenMPT can access pattern data directly, however we really do not want to expose that in libopenmpt. Pattern data alone is not sufficient for correct formatting of the pattern view. You would also at least need instrument information, custom tuning information, and the resulting note names to properly format the note column. The effect columns also need format-specific information to display the correct effect letter. While libopenmpt normalizes a lot of format-specific features, it does not normalize everything, and thus things like formatting require format-specific knowledge. However, in order to keep using libopenmpt simple, we really do not want to expose too much format-specific information and in particular we do not want library users to have to deal with differences between formats (because, frankly, people will get this wrong).

The pattern data is static though, so what you can do is cache the result upfront (or partially), so that you do not have to call the formatting functions repeatedly in your display render loop. This is what xmp-openmpt does, and this is even also what OpenMPT does for its pattern view.

Also, if your JNI wrapper uses the C++ API internally, you can probably do smarter things (like batching at that level) compared to when using the C API.

[bass]

Thanks, I was able to test the performance now and it looks like loading the pattern data is actually quite fast and hardly noticeable. I now do it in such a way that I load all pattern data when loading the module so that I don't have to worry about it later. I have also tested it with modules that have many patterns and many channels and the loading time was only marginally higher.

Now I just have to somehow make sure that I can get the sample data of the individual channels to show my oscilloscopes (I know it's not possible at the moment) :-)