thoughts on the upcoming new file format

[Snu]

well, i dont know how high priority the new file format is, but it seems that noone has taken the initiative to actually plan much.  i do remember there being a thread on the old forums, but i really dont remember much about it.  i dont know if any of the programmers are interested in the new format, but i for one am, and i would at least like to help it get started.
with that in mind, here are my thoughs on the matter, anyone who is interested, please post ideas, expansions, or arguments.
there have been suggestions of various things that the new format should be able to do, like multiple effects columns, and multi-layered instruments, but my intent for this discussion would be to go into detail on what the format really needs to be.



the current talk of 'open source music has got me thinking, what about creating an open source file format?  something that is as easy to edit as possible in any program, not just mpt.
my thought would be that all the data is stored in plain txt files arranged in directories, the 'project' directory could be opened directly, or stored in a .tar file (renamed to .mpt or something).  

i would think that the main song txt file would be in the main dir, with subdirs of patterns (with each pattern as its own txt file), instruments, and vst.

the instruments could be .iti, .xi, or a text based format (perhaps the sfz format? http://www.rgcaudio.com/sfzformat.htm , or a variation on xml?).
definitely something with the power and flexibility of the sfz format tho.

and yes, i am proposing that vst should be included with the song, but my thought is that the actual .dll (and accompanying files) would need to be loaded into the song, as the user would load a .iti for instance (maybe include a popup warning against commercial vsts, i dunno).

as for the pattern .txt files, whatever the format ends up looking like, the user should have the ability to select the contents of the file, and paste it all into a pattern directly in mpt.


so thats all i have for tonight, ill think about this tomorrow, and post a folow up if i can come up with any more ideas.  in the meantime tho, what is everyone else's suggestions?

[speed-goddamn-focus]

The most important thing, in my opinion, is that the format can be easily extended.

As for including VST's within a module, I doubt it's possible considering that many plugins depend on extra files and need to be installed. Also, just because a plugin is free doesn't mean it's freely distributable.

Not really bout the new file format, but I would like some compatibility options concerning the IT and XM formats, both in respect to the original trackers and to various players. Perhaps this could be user definable, i.e. disable the features you know your player doesn't support.

[rewbs]

I'm definitely thinking of using XML for song/instrument/sample properties & "pattern data" (though I think we should abstract away from the concept of a pattern if possible, and just store events with timing and other properties which would allow them to be mapped to rows/channels). And samples and other resources would go in some directory structure, with the whole lot zipped up in some way. This seems to be the modern way to do things, lots of apps use this structure nowadays. Of course load and save time are sacrificed (and possibly file size too, though we'd be able to provide a flexible set of compression options).

I'm trying to think of the best way of making the spec public as we put it together without too much noise, arguments and flame wars.

Also I'm not very happy with my experiences of coding XML parsing/generating in C++. When it comes to handling XML, I'm used to languages where you can serialize/deserialize objects directly to/from XML with pretty much 1 method call. My (newbie's) understanding is that this isn't so easy in C++ since there's no reflection functionalit. I'm using Xerces - any help/tutoring would be appreciated.

I'm against embedding plugins in the song for the reasons ganja specified (and I feel it breaks the plugin paradigm...).

Regarding compatibility, my objective is to clean up the guts of the player to make it completely agnostic of which format (IT, XM, S3M..) it is handling. All playback differences would then be available as an extensible set of individual options, with presets available for the common combinations. This should make for a pretty flexible system: you could theoretically have XM-style envelope handling with IT nna's, etc.. and accomodate for various playback perculiarities with fine granularity. Of course all this takes time and effort...

[Sam_Zen]

we should abstract away from the concept of a pattern if possible

I'm sure you mean this mattering file-data, because for a composer it is nice to work with entities like patterns.

I have strong doubts about embedding plugins too. Because one cannot be sure, that on the set of someone else, the piece is reproduced as how it was meant to be. Same problem as with Midi-compo's and different soundbanks.
Maybe I'm wrong, but I got the impression that many vst-plugins ask for the presence of a certain version of directx, which would make it even more complicated.

to make it completely agnostic of which format (IT, XM, S3M..) it is handling.

This is a very nice idea. In this way one could get the most potential out of the different characteristics of those formats.

[rewbs]

we should abstract away from the concept of a pattern if possible

I'm sure you mean this mattering file-data, because for a composer it is nice to work with entities like patterns.

Yes, I meant it would be cool if the events are stored in a way that do not couple the format too tightly with the concept of a pattern editor... but maybe that's not such a good idea since people seem to want to be able to edit their track in a text editor (for some reason..).

[Snu]

hmm, i agree that it would be nice to have a more free rhythmic pattern data, but really it needs to be a better system than the note delay thing...
what about having a new column, would be mandatory for all channels, and would show a percent of where in the row the note would take place (00 would be at the beginning, 99 would be at the very end).  this would allow for a more 'ticks' oreinted approach, more unique swings, etc.

also, i think data will still need to be stored in 'patterns' in the files, since the patterns need to be units that can be duplicated and replayed... tho getting away from the columns concept would perhaps be a nice thing.

another point, im thinking that data should be stored in percentages rather than multiples of 2, like panning could be -100 to +100, instead of 1-64...
a much more logical approach.

[shableep]

i totally agree.

oh yah, and something else since i was hearing about multiple effects per channel. you have "hi" "low" and "med" channel views. what if there was a "full" view that had an extreme width of channels and much more detail.

[LPChip]

Why would you do a percentage for panning if you can't have more values than 64?

I'm affraid that because of this change, there's a rounding problem where you can't get the right panning anymore. I don't think that having percentages is a good idea.

[speed-goddamn-focus]

Why would you do a percentage for panning if you can't have more values than 64?

I'm affraid that because of this change, there's a rounding problem where you can't get the right panning anymore. I don't think that having percentages is a good idea.

With a new file format, what reason is there not to change the resolution of the panning?

[shableep]

certain things are still in hex, and that's something that's extremely hard to manage. using the values of up to 64 is not as difficult, but it's an unusual value. it's kind of like having 64cents in a dollar. there's a really good reason why there isn't 64 cents in a dollar.

the whole existance of 'percentage' exists for that same very important reason.

i think we're trying to avoid any of these unusual values so that we have better ease of using them.

[speed-goddamn-focus]

there's a really good reason why there isn't 64 cents in a dollar.

Or 12 inches on a foot. GO METRICS!

[Sam_Zen]

I don't think it's just a matter of number system between hex and decimal.
I may be an exception, but I don't mind at all having to deal with hexadecimal. On the contrary, because it's more efficient in the needed digits fot the number. 2 symbols can mean up to 255 ..

In the meantime I have my doubts about the present situation, where in different columns, with different effect-codes, the max value is still different. Sometimes 40, sometimes FF or something else.

[DustWolf]

As I can read here, it's all thinking in-the-box.

When adopting a format such as XML, you no longer have to think about 100 or 64, you no longer have to think about implementation or anything. In XML, you can have things described in a way which provides the necesary information and then there are links in it's header to other XML files, which describe how the contents of the file is interpreted in diffirent programs (might as well be XSLT files, if you had other tracker / player programs that also used a variety of XML, for example an older version). The only thing you really have to worry about is to have the meaning of tags neatly documentated and that you don't use number formats which are inaccurately rounded up (e.g.: "10.33333333" instead of "10 and 1/3").

XML is not a format for editing on-the-fly, XML files are read into memory, discarded, modified in memory and written back in XML. So in a sense of a format, this would be better suited as an import / export feature, but on second thought there is really no reason for it not to be a format like any other either.

XML has the neat feature of using URIs, which means that you can link togather multiple XML files or even have XML resource files present on the internet to be downloaded when need be.

I think ZIPped (or tar.gzipped) XML files would be a good idea for a new format, since XML files are suitable for backward / forward compatibility. Thinking of the future, if we used pure valid XML, a range of possibilities and technical solutions open up. What pops to mind is storing XML modules in XML-SQL databases to finnaly implement module streaming and yes I have wild ideas sorry hehe.

On a historical note, XM is also some kind of eXtensible Module or something allong those lines, so maybe look there for lessons to learn.

[Randilyn]

I think we need to get away from the lets-stuff-everything-into-a-single-file tradition.  In my opinion, ZIP files are a good idea.

I think it would be best if we started focusing on breaking each part of a track into different files and then grouping them all together with optimal compression.  Not only does it make it easier for the tracker program to work with, it also makes them easier to work with in the absense of a tracker.

You could simply load up the ZIP-format file in a compession program and replace any of the samples at will, extract the TXT or XML format pattern data and edit it at will, extract one or more XML, INI, or CFG files and edit the settings of the instruments at will.  Need I go on?

[rewbs]

On a historical note, XM is also some kind of eXtensible Module or something allong those lines, so maybe look there for lessons to learn.

IIRC XM stood for extendED module.