Impulse tracker format questions

[TheRealByteraver]

So... I'm tackling the .IT format (or the other way around, not decided yet )

I'm going to have a lot of questions I'm afraid. For starters:

- Is there an easy way to detect how many channels the file uses, besides analysing the patterns for presence of data?
- Is there a way to detect if the .IT file is in instrument or sample mode? I guess an .IT file is in sample mode if it has zero instruments? Or is it not that simple?

[Saga Musix]

- Is there an easy way to detect how many channels the file uses, besides analysing the patterns for presence of data?

No. That's exactly what you have to do.

- Is there a way to detect if the .IT file is in instrument or sample mode? I guess an .IT file is in sample mode if it has zero instruments? Or is it not that simple?

Instrument mode is enabled if bit 2 (0x04) of the header flags is set and the number of instruments is > 0. Not setting the flag and having number of instruments > 0 is possible but doesn't make much sense. This is a very basic question that can be answered by reading ITTECH.TXT, so if you haven't looked into that file, I strongly advise to do so.

[TheRealByteraver]

Thanks Saga, I'm sorry I missed the "use instruments" flag in the description, it is indeed hard to miss.

I thought detecting how many channels are used in an .s3m file was not so obvious, .IT takes it to a whole new level

Other question: do you use a specific algorithm to keep track of which logical channel is connected to which virtual channel and vice-versa? In the mixer I mean. Originally I thought to give each logical channel a maximum of 16 virtual channels but that might not work for .IT.

[Saga Musix]

Every channel has a parent channel field (which is just 0 for master channels). The other way around you just have to walk through all virtual channels to find all of them belonging to a specific master channel.

[TheRealByteraver]

Interesting, sort of like a linked list basically. I wanted to avoid them (in general, for performance reasons), but I could not immediately think of a different way to do that.

Thank you for your answers.

[Saga Musix]

Having a pointer between elements doesn't make it a linked list, this is just a 1:1 relation (or 1:N from the perspective of the master channel). The most frequent way (Virtual Channel -> Master Channel) is a simple O(1) lookup with this implementation, and that's the important part. The other way around is indeed slightly more costly (in terms of memory reads) than the most efficent implementation could be, but it's still in the same complexity class O(N), and it doesn't even matter because it is rarely required (only for the relatively rarely used S7x class of commands). Even on a low-power machine this is essentially free compared to the actual sample mixing. The only way you could make it marginally faster is by having a fixed list of virtual channels stored in each master channel, but this list would need to be as big as the maximum number of virtual channels, so it would most likely actually make your code slower (data access will be less localized).

[TheRealByteraver]

I realize now I misunderstood your previous answer at first. Basically every virtual channel points to a logical channel, but the logical channel does not keep track of its own virtual channels. Pretty straightforward.
It is indeed rarely required for a master channel to communicate with its virtual channels now that I come to think of it. Maybe in the congestion management system. Not sure if there is much need for that nowadays.

Thanks for clearing that up.

Regarding instrument versus sample mode: MPT seems to work in a similar way as Impulse Tracker, offering both modes, with an easy way to transition from sample to instrument mode. Is there any penalty from always doing this conversion while loading a sample-type file? My question then is: does playback change if you convert a sample based .IT file (which is more or less an .S3M file I suppose) to an instrument based .IT file? I couldn't hear any differences in playback when I tried that out, nor can I think of any obvious problem with doing things this way, but there might be something that eludes me and I rather know it now than being obliged to do a design change halfway through

[Saga Musix]

Unless I am forgetting some gory details, in practice instrument mode should be a superset of sample mode so you could always assume instrument mode as long as your instrument numbers map to sample numbers 1:1. I guess back in the days of IT, sample mode was mostly a question of memory savings. I did not fully read IT's code but from my understanding of various IT quirks over the years, I think instrument to sample translation is basically one of the earliest steps done when parsing the pattern data, so most of the player is not aware of instruments, just samples.

[TheRealByteraver]

All right, thanks for your confirmation.

And last question for tonight: is there any documentation on sample compression used in .IT files? I was not going to bother originally until I discovered that compressed samples seem to be the norm rather than the exception in .IT files. I also found complete compressed modules, readable even by the older IT v1.06. They simply have the .IT extension (they're not zipped .ITZ files or something) but are compressed.
I realize I can analyze MPT's source code but if there is plain text description of the compression algorithm I would prefer that of course.

Thanks again for your time, you're definitely da real MVP!

[Saga Musix]

I also found complete compressed modules, readable even by the older IT v1.06. They simply have the .IT extension (they're not zipped .ITZ files or something) but are compressed.

Yes, that is the MMCMP module compressor, recognizable by the "ziRCONia" magic bytes at the start of the file. MMCMP was a termine-and-stay-resident tool so it could basically work with any tracker even if it didn't natively implement reading of MMCMP-compressed files.

I realize I can analyze MPT's source code but if there is plain text description of the compression algorithm I would prefer that of course.

There is a textual description written by KB explaining the theory and the bit layout, you can find it for example here:
https://github.com/nicolasgramlich/AndEngineMODPlayerExtension/blob/master/jni/loaders/itsex.c

Note that for stereo samples, you basically do the same thing twice, first for the left channel, then the right channel follows. All (de)compression state must be reset when handling the right channel.

[TheRealByteraver]

Great, thanks again for info!

[TheRealByteraver]

So I had a look at itsex.c (nice filename btw) and while I have now a vague idea of how this works, I don't yet understand why Jeffrey Lim bothered. I did the following test with 2 modules (Creagaia.it, an IT1.06 demo song, and hoffman_and_daytripper_-_professional_tracker.mod):
- removed all patterns, orders, and instrument info, effectively stripping the module of most overhead except the samples
- archived them as .zip and .rar with winrar

The result ("np" = no patterns):

Code Select Expand


Creagaia_np.it       :  266 kb
Creagaia_np.xm       :  269 kb
Creagaia_np_it.zip   :  211 kb
Creagaia_np_xm.zip   :  174 kb

hoffman_pt_np.it     : 1655 kb
hoffman_pt_np.mod    : 1653 kb
hoffman_pt_np_it.rar : 1292 kb
hoffman_pt_np_mod.rar: 1291 kb


Is my test flawed? Or is the sample compression not really worth the trouble?

[Saga Musix]

IT sample compression is actually pretty good, especially considering its age. Depending on the sample data, you can get up to 40-50% reduction (but often it will be less, in particular with 16-bit samples). In particular, it is much better than general-purpose compression algorithms like Deflate (used in ZIP), and IT-compression combined with general-purpose compression still fares better results that either of the two alone.
Note that in OpenMPT, you first have to enable IT sample compression through an advanced setting: Your comparisons indicate that you did not enable it. Here you can read how to enable it: https://forum.openmpt.org/index.php?topic=4961.0

[TheRealByteraver]

Aha, so my test was flawed. I checked the compression ratio on the example songs for the sake of making this post complete:

Code Select Expand


Creagaia_np.it      :  184 kb compressed
hoffman_pt_np.it    : 1276 kb compressed
Creagaia_np_it.rar  :  180 kb compressed & rar'ed
hoffman_pt_np_it.rar: 1267 kb compressed & rar'ed

uncompressed sizes:
Creagaia_np.it      :  266 kb
hoffman_pt_np.it    : 1655 kb

It compresses very well I would say, there does not seem to be a lot more left to do compression-wise for winrar!

However, if I resave the file as .xm (not the same, I know) we get this:

Code Select Expand


CREA_NP.xm          :  271 kb
CREA_NP_xm.rar      :  158 kb rar'ed


So, no surprise there, winrar compresses the same data (more or less) a bit better still.

Does anybody know about how many IT files are out "in the wild" with compressed samples? I'm wondering if it is worth the trouble to implement it. Weird question I know, but maybe someone has a vague idea.

[Saga Musix]

It compresses very well I would say, there does not seem to be a lot more left to do compression-wise for winrar!

RAR is a general-purpose algorithm, which works for a lot of things but not for highly random data like audio. On the other hand, IT's sample compression would for example not work that well for text in return.
However, I think both RAR and ZIP have special audio compression methods which they will use when they recognize audio formats, but not every decompressor supports those compression methods - so you see them very rarely. However, these special compression methods would not apply to module files unless the compressing tool recognizes them as audio files.

However, if I resave the file as .xm (not the same, I know) we get this:

Code Select Expand

CREA_NP.xm          :  271 kb
CREA_NP_xm.rar      :  158 kb rar'ed

So, no surprise there, winrar compresses the same data (more or less) a bit better still.

XM compresses better than uncompressed IT because it uses delta samples, which generally compress better as they reduce the range of the input data (you see the same principle being applied in the IT sample compression description).

Does anybody know about how many IT files are out "in the wild" with compressed samples? I'm wondering if it is worth the trouble to implement it. Weird question I know, but maybe someone has a vague idea.

Any IT file written in Impulse Tracker since 1997 or 1998 will use compressed samples unless the author explicitly disabled sample compression. So yes, that's a whole lot of files and I would consider any IT player without sample decompression to be incomplete and broken.
OpenMPT 1.21 is very old by now, so one of the main reason for not enabling enabling IT-compressed samples by default in OpenMPT is getting less and less relevant. I will consider enabling it by default at least for mono samples in the future, maybe also for stereo samples (given that more players have issues with those as there was no tracker that could save them before OpenMPT 1.21).