My "tracker".

[Metro28]

I have in mind a tracker, called "MetroTracker", which will be based on Schism Tracker

I'll leave this project until I at least know how to compile a big project.

The tracker will have an official MM (MetroModule) format, which will have the same limits as OpenMPT; adding some other options that OpenMPT has (plugins, pattern names, channel names, etc.)

I tried to compile a project but never could.

I will try to make this tracker support most possible formats. (Around more than 40 different formats)


Well I hope I get lucky with this thing considered a "tracker" and that it will be called "MetroTracker".

I don't think I will achieve this dream but I will try.

[Exhale]

well it certainly pays to have ambition, but that is nothing without the persistence to make it happen. We all have dreams... I dream of a modplug tracker with visualised samples in the patterns... if I ever learn to program I will try add that to modplug, but I am too much of a dreamer, and I cant focus enough to learn programming. I make art and music, but trying to lean an actual functional programming language always makes me want to die as soon as I get to 'hello world'

[Vojvodinosaurus]

I make art and music, but trying to lean an actual functional programming language always makes me want to die as soon as I get to 'hello world'

that happens to me too :\

[Saga Musix]

I suggest you start small, then the disappointment later will also be smaller. What makes you think that writing a tracker supporting 40 different formats (which is far from "most" possible formats that you could technically implement) and having the same features as MPTM "but just with a different" name would be a good first tracker project to work on? Start with the basics, not a fully- fledged product. OpenMPT is more than 20 years in the making, it's simply completely unreasonable to think that your first personal tracker project should have similar complexity.

[Metro28]

I suggest you start small, then the disappointment later will also be smaller. What makes you think that writing a tracker supporting 40 different formats (which is far from "most" possible formats that you could technically implement) and having the same features as MPTM "but just with a different" name would be a good first tracker project to work on? Start with the basics, not a fully- fledged product. OpenMPT is more than 20 years in the making, it's simply completely unreasonable to think that your first personal tracker project should have similar complexity.

NO NO AH NOO D: *has a heart attack and proceeds to die* The idea is that with the updates, I have big changes, it is obvious that my "project" may take a long time to do and that it does not end as intended.

What I want with this tracker is to create your own format (MM) and compress that format as much as possible.

[manx]

What I want with this tracker is to create your own format (MM) and compress that format as much as possible.

Please DONT.

If you had already the experience of parsing multiple other historic formats, you would not have come to that being the primary design goal of your own format. Having your own format probably only makes sense if your tracker ends up having distinctive features, otherwise it does not.

Designing a file format to be as compressed as possible is a horrible design choice and actually serves no real world purpose in the age of implicit file system compression, implicit network protocol compression, and sophisticated general purpose packers. You will make parsing way more difficult than necessary and thus way more susceptible to semantic parsing problems and security problems.

If your tracker gets popular and does not really have any distinctive features and yet still promotes its own file format, you would have just increased the work for projects like libopenmpt or libxmp, which try to support the most significant module formats out there, for actually no real benefit or reason.

Just DONT. I'm dead serious here. DONT.

Also, DO NOT be tempted to design whatever file format (or re-use an existing one) and establish it being stored as compressed via a general purpose packer. (Why? Because detecting the file's magic bytes in order to detect which application will handle it now requires memory allocation and an actual decompressor run). This is what DefleMask did, and it's stupid and horrible.

[mabersold]

Ah, go easy on him - I had similar ambitions once upon a time. If he's biting off more than he can chew, he will find out on his own soon enough.

[manx]

Ah, go easy on him - I had similar ambitions once upon a time. If he's biting off more than he can chew, he will find out on his own soon enough.

Fair enough if it concerns only ones own projects. However, defining and establishing new file formats concerns EVERYONE ELSE - especially if done without any pressuring need to do so.

And what DefleMask did not only concerns DefleMask or users of DefleMask's file format. It concerns *EVERY APPLICATION* that wants to just detect which program any particular file belongs to. Forever. (or these files will just be detected"zlib stream", which is kind of useless)

[mabersold]

And what DefleMask did not only concerns DefleMask or users of DefleMask's file format. It concerns *EVERY APPLICATION* that wants to just detect which program any particular file belongs to. Forever. (or these files will just be detected"zlib stream", which is kind of useless)

Wouldn't it be easier to just not support that format at all, in this case?

[Metro28]

If you had already the experience of parsing multiple other historic formats, you would not have come to that being the primary design goal of your own format. Having your own format probably only makes sense if your tracker ends up having distinctive features, otherwise it does not.

Designing a file format to be as compressed as possible is a horrible design choice and actually serves no real world purpose in the age of implicit file system compression, implicit network protocol compression, and sophisticated general purpose packers. You will make parsing way more difficult than necessary and thus way more susceptible to semantic parsing problems and security problems.

If your tracker gets popular and does not really have any distinctive features and yet still promotes its own file format, you would have just increased the work for projects like libopenmpt or libxmp, which try to support the most significant module formats out there, for actually no real benefit or reason.

Also, DO NOT be tempted to design whatever file format (or re-use an existing one) and establish it being stored as compressed via a general purpose packer. (Why? Because detecting the file's magic bytes in order to detect which application will handle it now requires memory allocation and an actual decompressor run). This is what DefleMask did, and it's stupid and horrible.

Oh, DefleMask... the worst tracker I have lying around.

The idea with "MM" was to have as much as possible and compress it to save space, but yeah, you're right.

[manx]

And what DefleMask did not only concerns DefleMask or users of DefleMask's file format. It concerns *EVERY APPLICATION* that wants to just detect which program any particular file belongs to. Forever. (or these files will just be detected"zlib stream", which is kind of useless)

Wouldn't it be easier to just not support that format at all, in this case?

Let me try to rephrase the problem here:

In order to determine that a particular file is a deflemask module, you have to first uncompress it with deflemask's compression (which happens to be zlib). In order to determine if you can do X, you first must do a significant part of *THE SAME* X. See the problem here? It's an exceptionally stupid design decision to use an established general purpose compression format as the container of your own file format without adding any identifying magic bytes.

While it is an open standard, uncompressing a zlib stream still involves significant work. A unix utility like file will probably never do that, as it would imply that it has to uncompress EVERY zlib stream in order to just find the deflemask ones.

Why is this a problem? Your file browser now has to potentially uncompress every file (as opposed to just looking at the first 8 bytes or so to find a particular file format magic number) in order to display the file type. This will be awfully slow. (or it can look at the file extension, in which case deflemask just plainly collides with X-Tracker DMF, so even inside the tiny niche of module formats, deflemask failed to find a file extension without colliding with an existing one).

Example file output that will never be improved:

Code Select Expand


manx@quadratus:~/c-home/stuff/DefleMask_Legacy_Windows/songs/modules$ file */Demo\ Songs/*
ARCADE_YM2151_PCM/Demo Songs/airbattle.dmf:                  zlib compressed data
ARCADE_YM2151_PCM/Demo Songs/ambush.dmf:                     zlib compressed data
ARCADE_YM2151_PCM/Demo Songs/dreaming_tomorrow.dmf:          zlib compressed data
ARCADE_YM2151_PCM/Demo Songs/filtration_plant.dmf:           zlib compressed data
ARCADE_YM2151_PCM/Demo Songs/heavy.dmf:                      zlib compressed data
ARCADE_YM2151_PCM/Demo Songs/jingle.dmf:                     zlib compressed data
ARCADE_YM2151_PCM/Demo Songs/jotaro_kujo.dmf:                zlib compressed data
ARCADE_YM2151_PCM/Demo Songs/ken_stage.dmf:                  zlib compressed data
ARCADE_YM2151_PCM/Demo Songs/nectaris_enemyturn.dmf:         zlib compressed data
ARCADE_YM2151_PCM/Demo Songs/rayfuture_2203.dmf:             zlib compressed data
ARCADE_YM2151_PCM/Demo Songs/rising_blue_lightning.dmf:      zlib compressed data
ARCADE_YM2151_PCM/Demo Songs/sidecrawlers_dance.dmf:         zlib compressed data
ARCADE_YM2151_PCM/Demo Songs/silent_colony.dmf:              zlib compressed data
ARCADE_YM2151_PCM/Demo Songs/so_there_can_be_a_tomorrow.dmf: zlib compressed data
ARCADE_YM2151_PCM/Demo Songs/teratoma.dmf:                   zlib compressed data
ARCADE_YM2151_PCM/Demo Songs/un_squadron_cave.dmf:           zlib compressed data
C64/Demo Songs/7of4.dmf:                                     zlib compressed data
C64/Demo Songs/bergertruckung.dmf:                           zlib compressed data
C64/Demo Songs/danmakuofemotions.dmf:                        zlib compressed data
C64/Demo Songs/inthismoment.dmf:                             zlib compressed data
C64/Demo Songs/la_bruja_mala.dmf:                            zlib compressed data
C64/Demo Songs/mistery_bullshit.dmf:                         zlib compressed data
C64/Demo Songs/motherfunksignal.dmf:                         zlib compressed data
C64/Demo Songs/not_the_final_boss.dmf:                       zlib compressed data
C64/Demo Songs/nyan_cat.dmf:                                 zlib compressed data
C64/Demo Songs/plastic_hand.dmf:                             zlib compressed data
C64/Demo Songs/press_z.dmf:                                  zlib compressed data
C64/Demo Songs/rael_adventures.dmf:                          zlib compressed data
C64/Demo Songs/reconciliation.dmf:                           zlib compressed data
C64/Demo Songs/rvas.dmf:                                     zlib compressed data
C64/Demo Songs/ryu.dmf:                                      zlib compressed data
C64/Demo Songs/the_beautiful_power.dmf:                      zlib compressed data
C64/Demo Songs/wizlab.dmf:                                   zlib compressed data
GB/Demo Songs/asterix_challenge_level.dmf:                   zlib compressed data
GB/Demo Songs/asterix_stage.dmf:                             zlib compressed data
GB/Demo Songs/asterix_stage1.dmf:                            zlib compressed data
GB/Demo Songs/asterix_stage2.dmf:                            zlib compressed data
GB/Demo Songs/bergentruckung.dmf:                            zlib compressed data
GB/Demo Songs/bomb_jack.dmf:                                 zlib compressed data
GB/Demo Songs/crystal_caverns.dmf:                           zlib compressed data
GB/Demo Songs/darkman.dmf:                                   zlib compressed data
GB/Demo Songs/die_maus.dmf:                                  zlib compressed data
GB/Demo Songs/dr_wright.dmf:                                 zlib compressed data
GB/Demo Songs/jingle_bells.dmf:                              zlib compressed data
GB/Demo Songs/jurassic_park_bonus.dmf:                       zlib compressed data
GB/Demo Songs/jurassic_park_lvl2.dmf:                        zlib compressed data
GB/Demo Songs/kraken_of_the_sea.dmf:                         zlib compressed data
GB/Demo Songs/ltcc_zoo.dmf:                                  zlib compressed data
GB/Demo Songs/megaman_v_stageselect.dmf:                     zlib compressed data
GB/Demo Songs/mohenjodaro_usas.dmf:                          zlib compressed data
GB/Demo Songs/monster_max.dmf:                               zlib compressed data
GB/Demo Songs/ottifanten_woods.dmf:                          zlib compressed data
GB/Demo Songs/pokemon_route_01.dmf:                          zlib compressed data
GB/Demo Songs/shop_theme.dmf:                                zlib compressed data
GB/Demo Songs/smurf_nightmare_well.dmf:                      zlib compressed data
GB/Demo Songs/soldier_blade_op1.dmf:                         zlib compressed data
GB/Demo Songs/super_star_soldier_area1.dmf:                  zlib compressed data
GB/Demo Songs/super_star_soldier_area2.dmf:                  zlib compressed data
GB/Demo Songs/tomjerry_level3.dmf:                           zlib compressed data
GB/Demo Songs/towntheme.dmf:                                 zlib compressed data
GB/Demo Songs/trouballs_title.dmf:                           zlib compressed data
GB/Demo Songs/undertale_death.dmf:                           zlib compressed data
GB/Demo Songs/usas_jubaruins.dmf:                            zlib compressed data
GB/Demo Songs/v_rally_carselect.dmf:                         zlib compressed data
GENESIS/Demo Songs/algun_dia.dmf:                            zlib compressed data
GENESIS/Demo Songs/aquaticambience.dmf:                      zlib compressed data
GENESIS/Demo Songs/blastermaster_area1.dmf:                  zlib compressed data
GENESIS/Demo Songs/chippin.dmf:                              zlib compressed data
GENESIS/Demo Songs/comixzone_e1page1.dmf:                    zlib compressed data
GENESIS/Demo Songs/core_endofgaem.dmf:                       zlib compressed data
GENESIS/Demo Songs/cruisin'.dmf:                             zlib compressed data
GENESIS/Demo Songs/dead_feelings.dmf:                        zlib compressed data
GENESIS/Demo Songs/delek_dojo.dmf:                           zlib compressed data
GENESIS/Demo Songs/devilish_prairie.dmf:                     zlib compressed data
GENESIS/Demo Songs/dkc_junglehijinx.dmf:                     zlib compressed data
GENESIS/Demo Songs/dubstep_electrosketch.dmf:                zlib compressed data
GENESIS/Demo Songs/ducktales2.dmf:                           zlib compressed data
GENESIS/Demo Songs/elecman.dmf:                              zlib compressed data
GENESIS/Demo Songs/financial_advisor.dmf:                    zlib compressed data
GENESIS/Demo Songs/fire_licks_the_floor.dmf:                 zlib compressed data
GENESIS/Demo Songs/gradius2_equipment_ch3mode.dmf:           zlib compressed data
GENESIS/Demo Songs/heartsaretheweakestshape.dmf:             zlib compressed data
GENESIS/Demo Songs/hes_a_mellow_fellow.dmf:                  zlib compressed data
GENESIS/Demo Songs/japan.dmf:                                zlib compressed data
GENESIS/Demo Songs/lab_hours.dmf:                            zlib compressed data
GENESIS/Demo Songs/ma-da-ra.dmf:                             zlib compressed data
GENESIS/Demo Songs/omake.dmf:                                zlib compressed data
GENESIS/Demo Songs/organic_spring.dmf:                       zlib compressed data
GENESIS/Demo Songs/phasetown.dmf:                            zlib compressed data
GENESIS/Demo Songs/quietwater.dmf:                           zlib compressed data
GENESIS/Demo Songs/shovelknight_mole_knight_stage.dmf:       zlib compressed data
GENESIS/Demo Songs/silversurfer2.dmf:                        zlib compressed data
GENESIS/Demo Songs/sleepy_tracking.dmf:                      zlib compressed data
GENESIS/Demo Songs/smrpg_bewaretheforestmushroom.dmf:        zlib compressed data
GENESIS/Demo Songs/smw2yi_flower_garden.dmf:                 zlib compressed data
GENESIS/Demo Songs/smw_athletic.dmf:                         zlib compressed data
GENESIS/Demo Songs/sonic3clear.dmf:                          zlib compressed data
GENESIS/Demo Songs/sonic_greenhillzone.dmf:                  zlib compressed data
GENESIS/Demo Songs/sonic_hilltopzone.dmf:                    zlib compressed data
GENESIS/Demo Songs/stainfactory.dmf:                         zlib compressed data
GENESIS/Demo Songs/timetrax_track.dmf:                       zlib compressed data
GENESIS/Demo Songs/toejamandearl2_themesong.dmf:             zlib compressed data
GENESIS/Demo Songs/xtunnelscene1.dmf:                        zlib compressed data
NES/Demo Songs/chrono.dmf:                                   zlib compressed data
NES/Demo Songs/comixe1.dmf:                                  zlib compressed data
NES/Demo Songs/feeblemask.dmf:                               zlib compressed data
NES/Demo Songs/megaman.dmf:                                  zlib compressed data
NES/Demo Songs/megaman2_intro.dmf:                           zlib compressed data
NES/Demo Songs/megaman2_wilycastle.dmf:                      zlib compressed data
NES/Demo Songs/ottifanten_die_vorstadt.dmf:                  zlib compressed data
NES/Demo Songs/pictonary.dmf:                                zlib compressed data
NES/Demo Songs/sunsetriders.dmf:                             zlib compressed data
NES/Demo Songs/supermariobros.dmf:                           zlib compressed data
NES/Demo Songs/telefang.dmf:                                 zlib compressed data
NES/Demo Songs/valseducerisier.dmf:                          zlib compressed data
NES/Demo Songs/ygo10.dmf:                                    zlib compressed data
PCE/Demo Songs/adv_claude_vorilhon.dmf:                      zlib compressed data
PCE/Demo Songs/chasing.dmf:                                  zlib compressed data
PCE/Demo Songs/cilius_moon.dmf:                              zlib compressed data
PCE/Demo Songs/doomsday_zone_s3k.dmf:                        zlib compressed data
PCE/Demo Songs/enchanted_lands.dmf:                          zlib compressed data
PCE/Demo Songs/enchanted_lands_gameover.dmf:                 zlib compressed data
PCE/Demo Songs/flying.dmf:                                   zlib compressed data
PCE/Demo Songs/gallantry.dmf:                                zlib compressed data
PCE/Demo Songs/gosus.dmf:                                    zlib compressed data
PCE/Demo Songs/gw_green_house.dmf:                           zlib compressed data
PCE/Demo Songs/hiver.dmf:                                    zlib compressed data
PCE/Demo Songs/lastbattle.dmf:                               zlib compressed data
PCE/Demo Songs/magician_domain.dmf:                          zlib compressed data
PCE/Demo Songs/megamitenseiii.dmf:                           zlib compressed data
PCE/Demo Songs/mohenjodaro_usas.dmf:                         zlib compressed data
PCE/Demo Songs/most_def.dmf:                                 zlib compressed data
PCE/Demo Songs/patrick_duffy_mask_is_a.dmf:                  zlib compressed data
PCE/Demo Songs/purple_sea.dmf:                               zlib compressed data
PCE/Demo Songs/snakerider.dmf:                               zlib compressed data
PCE/Demo Songs/submarine.dmf:                                zlib compressed data
PCE/Demo Songs/supersonic.dmf:                               zlib compressed data
PCE/Demo Songs/the_solemn_abyss.dmf:                         zlib compressed data
PCE/Demo Songs/usas_jubaruins.dmf:                           zlib compressed data
SMS/Demo Songs/animeopening.dmf:                             zlib compressed data
SMS/Demo Songs/bonk_adventure_beginning.dmf:                 zlib compressed data
SMS/Demo Songs/boss_battle.dmf:                              zlib compressed data
SMS/Demo Songs/burning_town.dmf:                             zlib compressed data
SMS/Demo Songs/exception.dmf:                                zlib compressed data
SMS/Demo Songs/forest_frolick.dmf:                           zlib compressed data
SMS/Demo Songs/happy_ending.dmf:                             zlib compressed data
SMS/Demo Songs/hexaprismhall.dmf:                            zlib compressed data
SMS/Demo Songs/ice_fields.dmf:                               zlib compressed data
SMS/Demo Songs/mastertracker.dmf:                            zlib compressed data
SMS/Demo Songs/mohenjodaro_usas.dmf:                         zlib compressed data
SMS/Demo Songs/monday.dmf:                                   zlib compressed data
SMS/Demo Songs/ryu_ending.dmf:                               zlib compressed data
SMS/Demo Songs/short_lived_understanding.dmf:                zlib compressed data
SMS/Demo Songs/syrvanmechanoid.dmf:                          zlib compressed data
SMS/Demo Songs/usas_jubaruins.dmf:                           zlib compressed data
SMS/Demo Songs/yzyx_zashiki_warashi.dmf:                     zlib compressed data


[mabersold]

Right, I see what you're saying - I guess I'm still unsure why you need to even bother with decompression. For example, if you're going by file extension, couldn't you just do something like:

Code Select Expand


if (first four bytes are not "DDMF")
   throw exception (format is not DMF)

proceed with rest of format validation


In other words, if you have established at this point that it's not a valid X-Tracker DMF file, don't you already know enough at this point? Is it important to know that a file is a Deflemask module?

[manx]

In other words, if you have established at this point that it's not a valid X-Tracker DMF file, don't you already know enough at this point?

Calling every .DMF file that is not X-Tracker a DefleMask file only works if you can guarantee that for the eternity of time not even any single other file format or even any single other file ever uses the .DMF extension. i.e. You can't.

Is it important to know that a file is a Deflemask module?

Detecting the file type is *the only* way a multimedia library like ffmpeg is able to decide which decoder library it should use for that particular file. It's is an essential property of any file format, and in particular media file format, to be easily identifiable.

If every program would do what DefleMask has done, no file format ever will be identifiable without decompressing first. It's a disastrous design decision with consequences not only for users of DefleMask, but for everyone else also.

There are also some other formats that do similarly stupid things like using a .zip file structure with specifically layout of contents. This is also bad, but at least it can be identified without decompressing, because the file contents of a zip file can be skimmed through without actually decompressing any data. Guilty formats here are for example Renoise XRNS, and the Open Document formats as used by LibreOffice.

[Saga Musix]

Ah, go easy on him - I had similar ambitions once upon a time. If he's biting off more than he can chew, he will find out on his own soon enough.

True, and I was there as well (everyone who started programming as a teenager probably was). Yet I would still recommend to at least try to come up with somewhat realistic goals. For example, my first tracker was completely MIDI-based, so no sample handling (no mixer to be written), no importing of 40 module formats. That would definitely be a suitable project for a teenager, and then you can still look at other formats and see how they manage to end up with such small file sizes if you are not happy with the result yet. Maybe that tracker would even grow into something bigger eventually. But starting right with the having the same feature set as trackers that have been around for more than a decade or two on your requirements list? You will most likely lose interest in the project before you wrote a single line of code that way (or at least that's how it happened for me).

[Metro28]

My e-mail: *explodes*