ModPlug Central

Another idea that's worth discussing imho:
Why not use / adapt the XRNS format that's being used in renoise? Basically, XRNS is a zip file which contains the patterns in xml format and the samples as seperate FLAC/MP3/OGG Files. In my opinion, this is a really good and expansible container format.

Sounds like an idea worth keeping in mind. Are the format specs anywhere to be found?

Well, there are some basic hints at http://www.renoise.com/board/index.php?showtopic=11892 and some more stuff at that board, but I'm sure it's easy to read the xml data. There are coming quite a few example songs with can be looked into.

The question, of course, is whether OpenMPT should be compatible with Renoise. The XML+Binary compressed storage thing is used a lot nowadays (for example in OpenOffice and DocX). To be compatible with Renoise, there'd have to be some pretty substantial changes to OpenMPT itself, so perhaps just using the idea of the format is a good one. :) Or is that what you meant in the first place?

I'm pretty sure thats what Jojo ment in the first place.

Well, ModPlug is currently not really "compatible" with Renoise and vice versa, so I just thought about adapting the format - Or else, Renoise people would come up and claim we're raping their format. :D

Its not really possible to fully support the renoise format yet. Simply because renoise offers more things like custon note width and effects column width. The GUI is simply not offering support for that.

Duh, that's what I just said. :P

I think this is a great idea!

Here's my first draft for the MPTM format. I began to write DTDs for the XML files, but I've realized that DTD is too limited for this (e.g. you only have #PCDATA as a data type), so I'd rather go for an XSD schema (but I have my lecture slides about XSD at my study place, so I can't do that right now :P).
Anyway, here are all the xml files needed to describe a module file. Have I forgotten something? A review by someone who knows XML would be appreciated, as I'm not sure if the usage of attributes vs. elements is always logical.

http://sagamusix.de/openmpt/download/mptm_draft.zip

Basic structure:

• MPTM.xml
  Basic information about the module
  A document type definition (DTD) is available.
  
• Samples.xml
  Sample attributes
  
• Instruments.xml
  Instrument attributes
  
• Sequence.xml
  Order list + Restart position
  
• Channels.xml
  Channel attributes
  
• Patterns.xml
  Pattern data
  
• Plugins.xml
  Plugin attributes
  
• Macros.xml
  SFx + Z80-ZFF definitions
  
• MidiMapping.xml
  Stored midi mappings (incomplete)
  
• Tunings.xml
  Stored tunings (incomplete)
  
• Workspace.xml
  Window layout in OpenMPT
  

Well, I work with XML an awful lot of work, so here are my thoughts. :)

1. Why have you split up the different parts into different files? To my mind, you'd have a cleaner structure if you had just one or two files, with different sections, rather than distinct files. This way'd actually be slightly easier to work with programatically, too, as you'd only have to load a single file, and then obtain references to the appropriate XML nodes (which you'd do with multiple files, anyway).

2. In instruments/instrument/envelopes//envelope, I would suggest putting some sort of ID attribute on the nodes, and then referencing this ID from within loop and sustainloop -- an XML document isn't guaranteed to have a particular order, and you don't really want to be linked from the tick number.

3. Module/Volume -- move the attribute "mixmode" into a subnode.

4. Module/Tempo -- move the attribute "mode" into a subnode.

5. This is just me, but I think Module/Flags would be more logically named Module/Options (although yes, they are called 'flags' in the system, but 'options' gives you more semantic scope. :))

6. Patterns//Pattern -- you need some sort of unique identifier for each distinct pattern, too.

7. How are you referencing the actual sample files? I can't see it... Perhaps you need to include a URI node that points to Samples/{{blah.wav}} for each sample. Again, nodes aren't ordered... :)

8. Given that the samples are stored, is it necessary to store redundant information like the sample's bitrate, channels, and length?

9. Patterns need some method of ordering them -- a "seq" attribute, perhaps, going from 0 to n.

10. Perhaps rename Workspace/Window/Tab to ActiveTab for clarity.

Quote1. Why have you split up the different parts into different files? To my mind, you'd have a cleaner structure if you had just one or two files, with different sections, rather than distinct files. This way'd actually be slightly easier to work with programatically, too, as you'd only have to load a single file, and then obtain references to the appropriate XML nodes (which you'd do with multiple files, anyway).

I don't really agree that less files would be easier to handle. Especially when I look at the XRNS format - it's terrible to keep everything in one file IMO.

Quote2. In instruments/instrument/envelopes//envelope, I would suggest putting some sort of ID attribute on the nodes, and then referencing this ID from within loop and sustainloop -- an XML document isn't guaranteed to have a particular order, and you don't really want to be linked from the tick number.

Good catch, I actually forgot that loop points have to be linked to a node.

But actually, from what I know, the order in an XML document is important, so I don't see why I would have to use an ID here.

Quote
3. Module/Volume -- move the attribute "mixmode" into a subnode.
4. Module/Tempo -- move the attribute "mode" into a subnode.

Why? I consider attributes as "descriptions", and in this case, "mixmode" and "mode" describe how to interpret the child nodes. This is actually one of the few cases where I'd say that attributes make sense. :)

Quote
5. This is just me, but I think Module/Flags would be more logically named Module/Options (although yes, they are called 'flags' in the system, but 'options' gives you more semantic scope. Smile)

Good idea.

Quote
6. Patterns//Pattern -- you need some sort of unique identifier for each distinct pattern, too.

Why's that? Storing empty patterns (which are rare) as <Pattern /> would take less space than using IDs. :P

Quote7. How are you referencing the actual sample files? I can't see it...

Not at all ATM, Because I still don't know if I should just take the sample number as a filename or something like "1 - Sample's filename.wav" (where sample's filename would be the <filename> tag)

Quote8. Given that the samples are stored, is it necessary to store redundant information like the sample's bitrate, channels, and length?

Good question. I guess it can be omitted.

Quote
9. Patterns need some method of ordering them -- a "seq" attribute, perhaps, going from 0 to n.

I can't really see what this would be useful for, care to explain? :)

Quote
10. Perhaps rename Workspace/Window/Tab to ActiveTab for clarity.

Done.

Some criticism about the rnsformat using the xml.

http://multimedia.cx/eggs/renoise-xrns/

Quote from: "Rakib"Some criticism about the rnsformat using the xml.

http://multimedia.cx/eggs/renoise-xrns/

I think that's pretty much missing the point...

On a more general note, it would be sweet if there were some cross compatibilty with renoise.

Yes, I don't like the structure of XRNS as well. :P
If noone wants XML, I could also imagine an INI-Like text format, which would still be rather versatile.

I think XML makes sense, but I'm not sure anymore whether a new file-format for OpenMPT makes sense. Maybe for OpenMPT 2, or for Unnamed Tracker based on OpenMPT, but I think OpenMPT as it is is a bit defined by its legacy: old school tracking. For the next OpenMPT I'd personally like legacy support to be limited to import.

A new file format makes very much sense. ModPlug has raped XM and IT formats, and I want to put this to an end. OpenMPT is two things at the same time, an oldskool and a newskool tracker, but it can't be the latter without its own file format. I won't be the one who adds yet another feature to XM / IT files. As soon as the MPTM format is working, I actually want to get rid of anything that doesn't belong to those two formats. Well, maybe there would still be an "MPT 1.16" edit mode (to use the full potentional of e.g. XMPlay), but it should be easy to create "valid" XM / IT files.

Quote from: "Jojo"OpenMPT is two things at the same time, an oldskool and a newskool tracker

I feel this is a really nice thing about OpenMPT. Perhaps even unique; is there any other tracker with decent/good MOD/S3M/XM/IT playback and editing capabilities as well as ability to use more modern tools such as VSTs?

I doubt it. But I didn't say that it's a bad thing. I did say, though, that seperation has to be done, so you can REALLY create "legacy" IT files.

Quote

Quote1. Why have you split up the different parts into different files? To my mind, you'd have a cleaner structure if you had just one or two files, with different sections, rather than distinct files. This way'd actually be slightly easier to work with programatically, too, as you'd only have to load a single file, and then obtain references to the appropriate XML nodes (which you'd do with multiple files, anyway).

I don't really agree that less files would be easier to handle. Especially when I look at the XRNS format - it's terrible to keep everything in one file IMO.

Well, I work a lot of with Jetty webapps that load up XML for various things. I find that it's a lot simpler to load a single XML file in the webapp, and find the correct nodes for each process (or whatever) and use them. When you're programming, you don't want to be dealing with files, you want to be dealing with abstracted XML -- reducing the number of file handles you have to deal with makes your code easier to manage and deal with. :)

Quote

Quote2. In instruments/instrument/envelopes//envelope, I would suggest putting some sort of ID attribute on the nodes, and then referencing this ID from within loop and sustainloop -- an XML document isn't guaranteed to have a particular order, and you don't really want to be linked from the tick number.

Good catch, I actually forgot that loop points have to be linked to a node.

But actually, from what I know, the order in an XML document is important, so I don't see why I would have to use an ID here.

Have a look here: http://www.ibm.com/developerworks/xml/library/x-eleord.html . An XML parser is free to return XML nodes in any order it sees fit, and I don't think that you can assume that parser P is always going to return nodes in the same order. Therefore, I think you should put some actual ordering information on the nodes, so that you can programmatically ensure that they are ordered correctly.

Quote

Quote
3. Module/Volume -- move the attribute "mixmode" into a subnode.
4. Module/Tempo -- move the attribute "mode" into a subnode.

Why? I consider attributes as "descriptions", and in this case, "mixmode" and "mode" describe how to interpret the child nodes. This is actually one of the few cases where I'd say that attributes make sense. :)

Fair enough -- I don't think it'll really make much difference either way. :) For some reason, I just prefer to have them out of attributes, but that's probably just me. :)

Quote

Quote
6. Patterns//Pattern -- you need some sort of unique identifier for each distinct pattern, too.

Why's that? Storing empty patterns (which are rare) as <Pattern /> would take less space than using IDs. :P

See my answer to (2), above. And I think that the file will be able to handle an extra, what, 19-21 bytes of data. :P

Quote

Quote7. How are you referencing the actual sample files? I can't see it...

Not at all ATM, Because I still don't know if I should just take the sample number as a filename or something like "1 - Sample's filename.wav" (where sample's filename would be the <filename> tag)

I think that just using a simple number is a good idea -- this way, you're guaranteed to not have conflicts, which are bad. :)

Quote

Quote
9. Patterns need some method of ordering them -- a "seq" attribute, perhaps, going from 0 to n.

I can't really see what this would be useful for, care to explain? :)

See number (2), above.

Well, I'll simply use Parser P1 that returns the nodes in the correct order then. :P

Quote from: "Jojo"But I didn't say that it's a bad thing.

And I didn't hear you saying so; I just quoted you to point out something I consider one of the most valuable aspects of OpenMPT.

Seems to be a good idea, it may make development of third party tools easier.

For third-party tools, it is actually worse (as long as the format itself is open) - because they would need to have an xml parser. As long as the format is open, binary would maybe be even better for them, as it's easier to parse. However, I've had a chat with a few people about this at Evoke, and they also agreed that XML is useful but overused :) A good alternative is a RIFF-Based format.

Quote from: "Jojo"For third-party tools, it is actually worse (as long as the format itself is open) - because they would need to have an xml parser. As long as the format is open, binary would maybe be even better for them, as it's easier to parse. However, I've had a chat with a few people about this at Evoke, and they also agreed that XML is useful but overused :) A good alternative is a RIFF-Based format.

Hmm. I disagree with this, actually. :) Text-based human-readable formats are, in my opinion at least, much easier to work with than binary formats. But maybe that's just me...

Anyway, I doubt there's a language that's in active use nowadays that doesn't have some kind of XML parser. :P


And yes, I guess you can use Parser P1 to return nodes in the right order. :P Bear in mind, though, that your XML manipulation stuff had better have the ability to insert nodes before and after other nodes, or you'll lots of fun trying to make it work correctly... :)

Quote
Hmm. I disagree with this, actually. Smile Text-based human-readable formats are, in my opinion at least, much easier to work with than binary formats. But maybe that's just me...

for a programming, it is easier to have instructions like "length is stored as a DWORD at position 0x123" and it certainly easier to write a loader for such a format.

Quote from: "Jojo"for a programming, it is easier to have instructions like "length is stored as a DWORD at position 0x123" and it certainly easier to write a loader for such a format.

On the other hand, isn't this one of the reasons why we need a new file format? If not, just stick with IT and change the suffix, or am I missing something?

I guess you have never looked at the actual IT or MPTM format. At the current state, this is very limited for future expansion. Well, you could do it, but it would be butt-ugly to do. For example, IT headers don't have room for more than 64 channels. Various instrument settings are spread over different places of a file. With a RIFF-Like format and some good ideas, you could actually make a new binary format that's extensible and easy to maintain.

Quote from: "Jojo"I guess you have never looked at the actual IT or MPTM format. At the current state, this is very limited for future expansion. Well, you could do it, but it would be butt-ugly to do.

I disagree completely about the extensibility being limited. And whether it is ugly or not is more a matter of taste.

Quote from: "Jojo"For example, IT headers don't have room for more than 64 channels.

The stuff that doesn't fit there can be saved elsewhere; not a big problem.


What comes to the whole matter, creating a new format would require a considerable amount of design, effort and knowledge - goes out of scope at least for me. Perhaps at some point a XML/XRNS/RIFF/... solution could come to replace that what's currently been used, but since I see no urgent need for a new format, I'm content with the current situation. Of the ideas discussed here, the XRNS idea still sounds intriguing.

Quote from: "Jojo"

Quote
Hmm. I disagree with this, actually. Smile Text-based human-readable formats are, in my opinion at least, much easier to work with than binary formats. But maybe that's just me...

for a programming, it is easier to have instructions like "length is stored as a DWORD at position 0x123" and it certainly easier to write a loader for such a format.

Hmm, maybe I just like the super-clear syntax of
xmlDoc.getValueAtPath("/module/samples/sample[id=smpl3]/title")
to get a specific sample's title
or
xmlDoc.getElementsAtPath("/module/samples/sample")
to get all of the samples in a track...

Then again, I'm actually in the process of migrating a large part of the data transfer in the system I develop at work to use JSON rather than XML... :D Maybe we should have a JSON-based format! :P

And the benefit of a format like XML etc is that it's really, really, easy for non-programmers to see what's going on, and really, really, easy for programmers to write utilities. Think about the possibilities -- scripting changes to the song within OpenMPT easily, just by exposing the XML document (or part thereof) and some simple manipulation routines, like getValueAtPath, setValueAtPath, and so on... Heck, people could write XSLTs for doing cool things to the song! :D I know I would...

QuoteI disagree completely; I see nothing restricting future expansion.

Quote
The stuff that doesn't fit there can be saved elsewhere; not a big problem.

Sure, you can always tack stuff at the end of a file, but I can you also see that storing information about an instrument in 4 places in the same file is not a very good idea.

Also, I guess I'll work together on this with xaimus, who has already brought up another possible approach:
http://code.google.com/p/protobuf/

Quoteeally, easy for programmers to write utilities

yes and no. Think of the difficulties to get an easy to use, maintained, working and still light-weight XML library for your weapon of choice (= your favourite programming language). I don't think that this exists in all cases. And getting into XML programming would also be necessary. Once you have this all, it's a piece of cake of course.

Quote from: "bvanoudtshoorn"And the benefit of a format like XML etc is that it's really, really, easy for non-programmers to see what's going on, and really, really, easy for programmers to write utilities.

Perhaps because I don't know about this, I tend to think that this is to some extend similar to "open your source and there'll be bunch of skilled developers working on the program"-hype. Are there easy to understand, concrete examples of such non-programmer made utilities for XRNS/OpenOffice etc.?

Quote from: "Jojo"Sure, you can always tack stuff at the end of a file, but I can you also see that storing information about an instrument in 4 places in the same file is not a very good idea.

Yes, but I think this is more about prettiness than extensibility. And adding stuff at the end of file doesn't imply that the data must be spread. And what is good about this, is that there's no need to write everything from scratch - which is a type of plan that often tends to remain unfinished - but there's existing IT save/load functionality on top of which extensions can be included.

Quote from: "Relabsoluness"

Quote from: "bvanoudtshoorn"And the benefit of a format like XML etc is that it's really, really, easy for non-programmers to see what's going on, and really, really, easy for programmers to write utilities.

Perhaps because I don't know about this, I tend to think that this is to some extend similar to "open your source and there'll be bunch of skilled developers working on the program"-hype. Are there easy to understand, concrete examples of such non-programmer made utilities for XRNS/OpenOffice etc.?

Yep! For example, http://sector-seven.net/software/rpg .

Quote from: "Jojo"yes and no. Think of the difficulties to get an easy to use, maintained, working and still light-weight XML library for your weapon of choice (= your favourite programming language). I don't think that this exists in all cases. And getting into XML programming would also be necessary. Once you have this all, it's a piece of cake of course.

I dunno... Java, C#, C++, Delphi, Javascript, PHP, and lots more all have really good, solid, well-maintained XML management implementations. And to be honest, 'getting into XML programming' is a lot easier than getting into binary-format programming. :)

Think of the awesome possibilities an XML-filetype opens up for end users:

1. Using the HTML-5 <audio> tag, you could actually watch the contents of a file 'play' by syncing pattern switches (or whatever) to key points in the audio (fairly easily, too -- check out some of the examples at http://hacks.mozilla.org/2009/06/exploring-music-audio/ for more on this type of thing)

2. XSLT transformations -- with endless possibilities. For example, transposing the entire track up or down; automatically generating 'echo' channels; interpolating and smoothing effects according to various algorithms, not just linear; scaling effect data; collecting statistical information about the track; creating computational music; generating melodic phrases based on a 'seed'; just about anything! :D Yes, you could do this with a binary format, but you'd have trouble with it.

3. Back- and forward-compatibility. If an earlier version of the software doesn't know about a node, it won't look for it, and Relabs will keep a 'pretty' format. :P

There is, of course, more... You have to understand that the power of XML really shines when it comes to a) working with it in multiple environments, and b) transforming documents using XSLT. For me, (b) is really significant: at work, we use JSON for client/server interaction, but for producing PDF reports, we pull the data back as XML and use FOP+XSLT to transform and render it. You can't do that easily with JSON or a binary format -- with a binary format, you have to write a specific class/program to handle each report. With XML+XSLT, you can write much more generic code.

Think about it -- 'plugins' for OpenMPT could be as simple as loading up an XSLT file, using it transform the current selection/file, and then showing the results. It'd be fairly simple to implement -- much simpler than a 'real' scripting/plugin system.

bvanoudtshoorn:

I don't understand quite everything what you have spoken about here, but it sure touched my heart:)