Documenting 228 extensions and the old custom tuning format

cs127 · February 25, 2022, 15:07:01

Hi!

About a month ago, I was curious to know how MPTM features like Custom Tunings are stored in MPTM files.

There was already a section about "228 extensions" on the OpenMPT Format Extensions page on the wiki (https://wiki.openmpt.org/Development:_OpenMPT_Format_Extensions), but it wasn't documented yet.
I checked the history of the page and I saw that this section was added in September 2016, and no one had documented it for about 5 years.

So I decided to read the OpenMPT source code and do it myself.
A few days ago I finished it, and I made this thread on the forum to let you know that I'll be adding it to the wiki soon.

I have already added the old tuning format that was used before OpenMPT r192 (cwtv < 0x088D), and will start to add 228 extensions tomorrow.

Saga Musix · February 25, 2022, 17:31:24

Thanks, nice work!

manx · February 26, 2022, 08:50:55

Yeah, thank you very much for that work.

If you have any questions or remarks about code being confusing, please feel free to ask.

And if you remain unsure about something or the exact semantics of aspects that might not get handled or are not deducible from the code, please mark them as "unclear" or whatever you see fit. That's actually more useful as documentation than just claiming a particular interpretation if things are in fact not 100% clear.

Especially the code in common/serialization_utils is still somewhat confusing, even after progressively cleaning it up significantly over the last couple of years.

cs127 · February 26, 2022, 10:33:02

Hi, I just added 228 extensions to the wiki!

It was a bit long so I did it in a separate page: https://wiki.openmpt.org/Development:_228_Extensions

I also added a link to it under the "228 extensions" section of the Format Extensions page: https://wiki.openmpt.org/Development:_OpenMPT_Format_Extensions#228_Extensions

It might be hard to understand, so I will try to simplify it later.

Quote from: manx on February 26, 2022, 08:50:55

If you have any questions or remarks about code being confusing, please feel free to ask.

And if you remain unsure about something or the exact semantics of aspects that might not get handled or are not deducible from the code, please mark them as "unclear" or whatever you see fit. That's actually more useful as documentation than just claiming a particular interpretation if things are in fact not 100% clear.

The code was a bit confusing, but nothing about the structure of 228 extension chunks were unclear, and I'm pretty sure I wrote everything correctly.

However, I was slightly unsure about a lot of the terminology, because a lot of variable names and comments in the source code were too ambigiuous for me to figure out what each section of a 228 chunk is called.

For example, after what's called a "header byte" in the code, there is a set of bytes called the "header data" (the one that contains the "flag byte"), which is a confusing name. So in the document I called it "additional header data" instead of "header data" which in my opinion makes more sense. I hope that's fine.

If I have made any mistakes, feel free to correct them or let me know about them

Saga Musix · February 26, 2022, 10:43:52

Quote from: cs127 on February 26, 2022, 10:33:02
The code was a bit confusing, but nothing about the structure of 228 extension chunks were unclear, and I'm pretty sure I wrote everything correctly.

If you want to go the extra mile, one way of confirming that would be to write a 228 reader (or writer) just based on your textual description, and see if it can read the extension block as written by OpenMPT (or OpenMPT can read the extension block written by you).

cs127 · February 26, 2022, 10:49:53

Quote from: Saga Musix on February 26, 2022, 10:43:52
If you want to go the extra mile, one way of confirming that would be to write a 228 reader (or writer) just based on your textual description, and see if it can read the extension block as written by OpenMPT (or OpenMPT can read the extension block written by you).

Sure, I'll probably do it whenever I have enough time!

Relabsoluness · February 27, 2022, 23:10:33

While it's embarrassing that someone else than developer had to saw the trouble of writing the specs - sorry about that - it's was nice to read such a well written and thorough documentation. With the binary format and all the bells and whistles there was quite a number of details to document compared to what it would have required had format been something like json.