Another approach at the MPTM format

[g]

I think XML makes sense, but I'm not sure anymore whether a new file-format for OpenMPT makes sense. Maybe for OpenMPT 2, or for Unnamed Tracker based on OpenMPT, but I think OpenMPT as it is is a bit defined by its legacy: old school tracking. For the next OpenMPT I'd personally like legacy support to be limited to import.

[Saga Musix]

A new file format makes very much sense. ModPlug has raped XM and IT formats, and I want to put this to an end. OpenMPT is two things at the same time, an oldskool and a newskool tracker, but it can't be the latter without its own file format. I won't be the one who adds yet another feature to XM / IT files. As soon as the MPTM format is working, I actually want to get rid of anything that doesn't belong to those two formats. Well, maybe there would still be an "MPT 1.16" edit mode (to use the full potentional of e.g. XMPlay), but it should be easy to create "valid" XM / IT files.

[Relabsoluness]

OpenMPT is two things at the same time, an oldskool and a newskool tracker

I feel this is a really nice thing about OpenMPT. Perhaps even unique; is there any other tracker with decent/good MOD/S3M/XM/IT playback and editing capabilities as well as ability to use more modern tools such as VSTs?

[Saga Musix]

I doubt it. But I didn't say that it's a bad thing. I did say, though, that seperation has to be done, so you can REALLY create "legacy" IT files.

[bvanoudtshoorn]

1. Why have you split up the different parts into different files? To my mind, you'd have a cleaner structure if you had just one or two files, with different sections, rather than distinct files. This way'd actually be slightly easier to work with programatically, too, as you'd only have to load a single file, and then obtain references to the appropriate XML nodes (which you'd do with multiple files, anyway).

I don't really agree that less files would be easier to handle. Especially when I look at the XRNS format - it's terrible to keep everything in one file IMO.

Well, I work a lot of with Jetty webapps that load up XML for various things. I find that it's a lot simpler to load a single XML file in the webapp, and find the correct nodes for each process (or whatever) and use them. When you're programming, you don't want to be dealing with files, you want to be dealing with abstracted XML -- reducing the number of file handles you have to deal with makes your code easier to manage and deal with.

2. In instruments/instrument/envelopes//envelope, I would suggest putting some sort of ID attribute on the nodes, and then referencing this ID from within loop and sustainloop -- an XML document isn't guaranteed to have a particular order, and you don't really want to be linked from the tick number.

Good catch, I actually forgot that loop points have to be linked to a node.

But actually, from what I know, the order in an XML document is important, so I don't see why I would have to use an ID here.

Have a look here: http://www.ibm.com/developerworks/xml/library/x-eleord.html . An XML parser is free to return XML nodes in any order it sees fit, and I don't think that you can assume that parser P is always going to return nodes in the same order. Therefore, I think you should put some actual ordering information on the nodes, so that you can programmatically ensure that they are ordered correctly.

3. Module/Volume -- move the attribute "mixmode" into a subnode.
4. Module/Tempo -- move the attribute "mode" into a subnode.

Why? I consider attributes as "descriptions", and in this case, "mixmode" and "mode" describe how to interpret the child nodes. This is actually one of the few cases where I'd say that attributes make sense.

Fair enough -- I don't think it'll really make much difference either way. For some reason, I just prefer to have them out of attributes, but that's probably just me.

6. Patterns//Pattern -- you need some sort of unique identifier for each distinct pattern, too.

Why's that? Storing empty patterns (which are rare) as <Pattern /> would take less space than using IDs.

See my answer to (2), above. And I think that the file will be able to handle an extra, what, 19-21 bytes of data.

7. How are you referencing the actual sample files? I can't see it...

Not at all ATM, Because I still don't know if I should just take the sample number as a filename or something like "1 - Sample's filename.wav" (where sample's filename would be the <filename> tag)

I think that just using a simple number is a good idea -- this way, you're guaranteed to not have conflicts, which are bad.

9. Patterns need some method of ordering them -- a "seq" attribute, perhaps, going from 0 to n.

I can't really see what this would be useful for, care to explain?

See number (2), above.

[Saga Musix]

Well, I'll simply use Parser P1 that returns the nodes in the correct order then.

[Relabsoluness]

But I didn't say that it's a bad thing.

And I didn't hear you saying so; I just quoted you to point out something I consider one of the most valuable aspects of OpenMPT.

[Rxn]

Seems to be a good idea, it may make development of third party tools easier.

[Saga Musix]

For third-party tools, it is actually worse (as long as the format itself is open) - because they would need to have an xml parser. As long as the format is open, binary would maybe be even better for them, as it's easier to parse. However, I've had a chat with a few people about this at Evoke, and they also agreed that XML is useful but overused A good alternative is a RIFF-Based format.

[bvanoudtshoorn]

For third-party tools, it is actually worse (as long as the format itself is open) - because they would need to have an xml parser. As long as the format is open, binary would maybe be even better for them, as it's easier to parse. However, I've had a chat with a few people about this at Evoke, and they also agreed that XML is useful but overused A good alternative is a RIFF-Based format.

Hmm. I disagree with this, actually. Text-based human-readable formats are, in my opinion at least, much easier to work with than binary formats. But maybe that's just me...

Anyway, I doubt there's a language that's in active use nowadays that doesn't have some kind of XML parser.


And yes, I guess you can use Parser P1 to return nodes in the right order. Bear in mind, though, that your XML manipulation stuff had better have the ability to insert nodes before and after other nodes, or you'll lots of fun trying to make it work correctly...

[Saga Musix]

Hmm. I disagree with this, actually. Smile Text-based human-readable formats are, in my opinion at least, much easier to work with than binary formats. But maybe that's just me...

for a programming, it is easier to have instructions like "length is stored as a DWORD at position 0x123" and it certainly easier to write a loader for such a format.

[g]

for a programming, it is easier to have instructions like "length is stored as a DWORD at position 0x123" and it certainly easier to write a loader for such a format.

On the other hand, isn't this one of the reasons why we need a new file format? If not, just stick with IT and change the suffix, or am I missing something?

[Saga Musix]

I guess you have never looked at the actual IT or MPTM format. At the current state, this is very limited for future expansion. Well, you could do it, but it would be butt-ugly to do. For example, IT headers don't have room for more than 64 channels. Various instrument settings are spread over different places of a file. With a RIFF-Like format and some good ideas, you could actually make a new binary format that's extensible and easy to maintain.

[Relabsoluness]

I guess you have never looked at the actual IT or MPTM format. At the current state, this is very limited for future expansion. Well, you could do it, but it would be butt-ugly to do.

I disagree completely about the extensibility being limited. And whether it is ugly or not is more a matter of taste.

For example, IT headers don't have room for more than 64 channels.

The stuff that doesn't fit there can be saved elsewhere; not a big problem.


What comes to the whole matter, creating a new format would require a considerable amount of design, effort and knowledge - goes out of scope at least for me. Perhaps at some point a XML/XRNS/RIFF/... solution could come to replace that what's currently been used, but since I see no urgent need for a new format, I'm content with the current situation. Of the ideas discussed here, the XRNS idea still sounds intriguing.

[bvanoudtshoorn]

Hmm. I disagree with this, actually. Smile Text-based human-readable formats are, in my opinion at least, much easier to work with than binary formats. But maybe that's just me...

for a programming, it is easier to have instructions like "length is stored as a DWORD at position 0x123" and it certainly easier to write a loader for such a format.

Hmm, maybe I just like the super-clear syntax of

Code Select Expand

xmlDoc.getValueAtPath("/module/samples/sample[id=smpl3]/title")
to get a specific sample's title
or

Code Select Expand

xmlDoc.getElementsAtPath("/module/samples/sample")
to get all of the samples in a track...

Then again, I'm actually in the process of migrating a large part of the data transfer in the system I develop at work to use JSON rather than XML... Maybe we should have a JSON-based format!

And the benefit of a format like XML etc is that it's really, really, easy for non-programmers to see what's going on, and really, really, easy for programmers to write utilities. Think about the possibilities -- scripting changes to the song within OpenMPT easily, just by exposing the XML document (or part thereof) and some simple manipulation routines, like getValueAtPath, setValueAtPath, and so on... Heck, people could write XSLTs for doing cool things to the song! I know I would...