Feature suggestion: Create exportet wav as sparse

Started by Waxhead, January 23, 2006, 21:03:58

Previous topic - Next topic

Waxhead

If ModPlug would try to create exported wav files as sparsefiles my life would improve considerably. The air would be fresher, food would taste better, my shoes would feel more confortable and I'm pretty sure I would stop loosing my hair also :)

Matt Hartman

Quote from: "Waxhead"If ModPlug would try to create exported wav files as sparsefiles my life would improve considerably. The air would be fresher, food would taste better, my shoes would feel more confortable and I'm pretty sure I would stop loosing my hair also :)

I do say, HeadWax, what is this, "sparse" you speak of man? I'm quite intrigued to know.

Rioght.
Yeah, sure. Right. Whatever.

speed-goddamn-focus

I think it means that "irrelevant" data isn't written to the file and set to 0 when loading it, thus saving disk space. I'm not quite sure what would be irrelevant in a wave file tho... Then again, I probably misunderstand everything as per usual.

LPChip

Seems that we're not the only one. I'm wondering if Waxhead is the only one that knows what he means. :P

(Finally someone else that can say anything here that no one understands :P)
"Heh, maybe I should've joined the compo only because it would've meant I wouldn't have had to worry about a damn EQ or compressor for a change. " - Atlantis
"yes.. I think in this case it was wishful thinking: MPT is makng my life hard so it must be wrong" - Rewbs

rewbs

Does it mean silence would take up no diskspace?

Waxhead

Oh sorry guys... I guess I was a bit to technical for you except for Rewbs :)

If you save a sample it will be lot's of numbers. Most often there are lot's of zero (silence) in many tracks and using a sparsefile will (very basically) save this data

123000000000000000000000000000000000000000000000004567

like this:

12304567

some sort of 0-compression. ;)

You can create sparsefiles yourself from "dos" with fsutil (native winxp app)

rewbs

I imagine this could be handy when rendering channels/instruments individualy, where some tracks will have long periods of silence. Out of curiosity, do you know any other applications that save wavs as sparse files? I hadn't heard the term before, so I'm guessing its not a very common practice and therefore might have drawbacks (difficult to implement? compatibility issues?)

Matt Hartman

Quote from: "Waxhead"Oh sorry guys... I guess I was a bit to technical for you except for Rewbs :)

If you save a sample it will be lot's of numbers. Most often there are lot's of zero (silence) in many tracks and using a sparsefile will (very basically) save this data

123000000000000000000000000000000000000000000000004567

like this:

12304567

some sort of 0-compression. ;)

This makes me wonder why it isn't like this by default? What's the advantage of having 00000000 when there's no information contained in them? There has to be a specific reason, no?
Yeah, sure. Right. Whatever.

Squirrel Havoc

Well the 0's are part of the audio, just silence im sure. And what he's talking about is RLE (run-length encoding compression). Just zip the wave file :)
Anyone can do anything if they have nothing else to do
-
Most musicians are talented. I'm just determined.

Waxhead

Quote from: "Squirrel Havoc"Well the 0's are part of the audio, just silence im sure. And what he's talking about is RLE (run-length encoding compression). Just zip the wave file :)

Yes or turn on compression on NTFS volumes.

But let me try to be a bit clearer. A sparsefile works in such a way that (total) silence doesn't use loads of precious disk-space. When the file is loaded the packed zero's are regenerated by the filesystem. the advantages are that a saved track takes less time to load and will use less diskspace it should also be quite easy to implement since the encoding is done by the filesystem. However this is a feature of NTFS (not FAT) and if present I can't think of any good reason not to try and save as sparsefile if the filesystem is NTFS.

Waxhead

Quote from: "rewbs"I imagine this could be handy when rendering channels/instruments individualy, where some tracks will have long periods of silence. Out of curiosity, do you know any other applications that save wavs as sparse files? I hadn't heard the term before, so I'm guessing its not a very common practice and therefore might have drawbacks (difficult to implement? compatibility issues?)

http://msdn.microsoft.com/library/default.asp?url=/library/en-us/fileio/fs/sparse_files.asp
http://msdn.microsoft.com/library/default.asp?url=/library/en-us/fileio/fs/fsctl_set_sparse.asp

As you can se the the sparsefilemanagement is done by the filesystem and is transparent to applications. The only potential drawback I can think of is that IF a file have frequent (very) short periods of scilence in it it might need some processing power to load/save... I'm unsure of this tough...

btw: when the new forum is up perhaps it would be an idea to create a "technical" forum for coders to discuss various things ;)

I know I have encountered some apps that creates sparsefiles but I Can't remeber which ones... :(

Sam_Zen

A few aspects are skipped here so far. The matter of Time and the question : do you want to be able to recover the original wav-file with the silences included ?
A gif-file is non-destructive, because it not only skips unnecessary zeroes, but saves the number of them.
So 123000000000000000000000000000000000000000000000004567
would become something like
123 47x0 4567
0.618033988

Waxhead

Time:
 most sparsefiles should load and save faster

Recover:
Sparsefiles are as non-destructive ;)

The only reason I suggested this was that sparsefiles are quite handy for samples. When exporting all tracks as wav and loading them in a multitrack sampleeditor (Audition (coolEdit)  for example) you will probably have a large portion of some tracks as silence. My point is why waste perfectly good disk space when just setting a flag for the NTFS filesystem will solve that problem ;)

Meanwhile let's just use NTFS compression and be happy with it :)