Pop removal that is not part of the sample

[Techno.Jon]

Hey all,

As a note, this is my first post on this MB, though I have been using MODPlug since 2009.  However, this is the first time I've needed a problem solved, so... yeah.

Anyway, I have mostly very complex songs that I do, including vocals, average of 40 channels, loads of instruments, VSTi's, and VST's, but there is a nonfatal but very annoying problem that I keep coming across.  It seems that when my songs get very complex, MODPlug likes to create many pops and clicks for seemingly no reason, although I normalize all my samples and eliminate offsets beforehand and use the standard sample ramping.  These appear for no particular reason randomly throughout the song and are very difficult and annoying to remove.  At the bottom of this post, there are two links to a song I just finished last night, one without pop removal and one after an hour and a half of pop removal.  I've tried everything; increasing the ramping, increasing the buffer, using volume leveling, decreasing the global volume to four as an extreme, but to no avail.  I can only use Audacity's manual "Audio Repair" function, which works on a maximum of 128 samples, making it rather tedious to use on a 7-minute piece.  Is there anything else I can do to repair this automatically?  It adds a significant amount of time onto production.

The links:
no pop removal: https://sites.google.com/site/metal96data/home/Kingdom_v.mp3?attredirects=0&d=1
pop removal: https://sites.google.com/site/metal96data/home/Kingdom.mp3?attredirects=0&d=1
NOTE: the end is chopped off of the pop removal mix because it will segue directly into the next track on the album, which happens to be the last one.
NOTE2: I didn't tell you the name, did I?  It's "Kingdom."

Some stats about the song, in case they are helpful:
Sequence: 69
Patterns: 46
Samples: 26
Instruments: 30
VST's: 13; 3 instruments, 10 fx

I hope that there is a solution to this problem, and thanks for helping (or at least trying)!

[Saga Musix]

Just to clear things up, go through this list step by step...
- Which version of OpenMPT are you using? Does this also happen with older versions?
- Does this also happen while composing, or just during export?
- Do you use Wave Export to render the tune? If not, this might be a drop-out caused by a bad driver, though that would be unlikely because normall a drop-out would result in a zero-volume output.
- Do the pops always occur at the same parts of the tune, or does this happen randomly? It could be a limitation of a demo VST you are using and that you are not aware of. Try disabling the VSTs used one after another, and check if/when the pops cease to appear.
- Does it also happen with "Enable SSE acceleration" disabled?
- Does this also happen when exporting to a different bit rate / sampling rate?
- What are your settings for global volume, VST / sample volume, and which mix mode is set in the song properties?

Nice track btw.

[Techno.Jon]

-1.18, 1.19 is not compatible with some of my plugins, so I don't use it.  Hopefully this will be fixed next version so I can upgrade.
-Yes, this happens during both playback and export.
-Yes.  However, it only happens in some tunes, and the pop usually is just a voltage spike that is rectified later on, a few hundred samples later.  Could this be a static thing?
-Yes, same parts of the song, and it seems to be large amounts of bass.  All VST's are free or bought, so no demostuffs.  I would know because I test all plugins beforehand to check ability and such.
-I don't know, I haven't tried that...  Where is that?
-Yes.
-global = 32, vst = 128, sample = 128, but I use the instrument, sample, and plugin mix volume controls to do volume and mixing adjustments.  Mix mode is polyphase, the default, but I've tried the other ones.

and thanks.

[Saga Musix]

-1.18, 1.19 is not compatible with some of my plugins, so I don't use it.

In what way is it not "compatible"? If you don't tell us that something doesn't work, we cannot fix it. What plugins ceased to work in OpenMPT 1.19?

-I don't know, I haven't tried that...  Where is that?

In the Sound Card options.

-global = 32, vst = 128, sample = 128

Can you please try and see if this still happens if you crank the global volume all the way up, and setting VST / sample volumes to 32 or lower instead? It will probably destroy some reverbs and other effects, but it can help identifying the problem. 128 for VST / sample volume can be much too high with some plugins, so it's possible that there was some overamplification happening, which cannot be undone by the global volume setting if it was way above "normal" volume levels.

Mix mode is polyphase

I was actually talking about the "mix levels" dropdown list which can be found when clicking the "song properties" button on the general tab.

Please keep in mind that I cannot give support on old versions of OpenMPT in my limited spare time, so if the problem does not appear with OpenMPT 1.19, I can not and will not pursue this issue any further.

[LPChip]

Which soundcard driver are you using? (wave, directx, asio)?

And possibly not fixing it, but for the sake of testing: have you increased the latency to see if that helps?

Another thing that I can think of that is causing this: Is it perhaps possible that at some point, your sounds add up in such way that it clips? For all we know, it is your soundcard that has a protection which causes the appearance of clicks and pops.

But it also helps if you can tell us which VST's are in your song. I happen to have had an issue in the past where I had chainer with several instances of Battery in it, and I noticed that due to some weird bug in chainer/battery, if a cpu of 100% was reached inside the plugin (system didn't choke) then it started to randomly cut sounds causing clicks and pops to occur. Also I have seen this hapen if I have too many reverb's with high settings, it really does reach 100% cpu usage and my system chokes and starting to get the same thing.

So for that reason, can you see if you max out a core (not whole CPU, since OpenMPT is not multi-core) when you reach those peaks?

Note, I have a very good system (AMD Athlon 64 Phenom XII 965 4x@3.4ghz) with good other specs, and even I can make my system choke in OpenMPT.

And if you don't check the slow render (or kontact render?) during export, it might still choke inside the VSTi causing it to cause clicks and pops.

[Saga Musix]

Oh boy...

Which soundcard driver are you using? (wave, directx, asio)?
And possibly not fixing it, but for the sake of testing: have you increased the latency to see if that helps?

Both settings are not relevant during wav export, and he said that it happens during both playback and wav export.

Another thing that I can think of that is causing this: Is it perhaps possible that at some point, your sounds add up in such way that it clips?

That is the most likely reason at the moment, since the sample/VST volumes are way too high and the global volume way too low.

For all we know, it is your soundcard that has a protection which causes the appearance of clicks and pops.

Nonsense. A soundcard does not have clipping protection. The values are already clipped in OpenMPT. Or, to be precise, by looking at the waveform they are not clipped but wrapped around (which again strenghtens the point that the VST/sample volumes are too high). Not that the soundcard would be relevant during WAV export, anyway...

And if you don't check the slow render (or kontact render?) during export, it might still choke inside the VSTi causing it to cause clicks and pops.

If a plugin would produce clicks because of that, it would be a very badly designed plugin, and I don't think there are plugins that are as bad as this. The Kontakt render mode won't fix any clicks, it just fixes the problem with direct from disk streaming, which can prevent notes from not being played at all.

[LPChip]

Oh boy...

Oh yes!

Which soundcard driver are you using? (wave, directx, asio)?
And possibly not fixing it, but for the sake of testing: have you increased the latency to see if that helps?

Both settings are not relevant during wav export, and he said that it happens during both playback and wav export.

I know, but it doesn't hurt to test this.

Another thing that I can think of that is causing this: Is it perhaps possible that at some point, your sounds add up in such way that it clips?

That is the most likely reason at the moment, since the sample/VST volumes are way too high and the global volume way too low.

Yay, we agree on something. Can I have a black and white copy of this?

For all we know, it is your soundcard that has a protection which causes the appearance of clicks and pops.

Nonsense. A soundcard does not have clipping protection. The values are already clipped in OpenMPT. Or, to be precise, by looking at the waveform they are not clipped but wrapped around (which again strenghtens the point that the VST/sample volumes are too high). Not that the soundcard would be relevant during WAV export, anyway...

My soundcard has this and does this. And if I make a wave export, and I play that wave file on my soundcard, it does the exact same thing again. I tested this, so I know this is a possibility. I have 2 different protections, one is volume normalizing and the other is to cut extreme sounds like loud noise, and other high volume sounds. It is possible that a plugin causes a certain noise that the soundcard identifies as possible threat. True, my soundcard doesn't produce pops for this, but that doesn't mean it can't happen. (it only pops if the initial sound is much louder, but that is probably not the case in this song)

And if you don't check the slow render (or kontact render?) during export, it might still choke inside the VSTi causing it to cause clicks and pops.

If a plugin would produce clicks because of that, it would be a very badly designed plugin, and I don't think there are plugins that are as bad as this. The Kontakt render mode won't fix any clicks, it just fixes the problem with direct from disk streaming, which can prevent notes from not being played at all.

I can tell you that it happens. For whatever reason, I've had this problem in Battery in my song called Dark December Day (which I entered the modplug compo with) You can hear that at some points in the 2nd drum salvo, the drum often just disappears. I investigated, and Battery gave the message: CPU 100% every time it cut the sounds. This cutting caused other effects in the chain to produce clicks and pops. I was able to get them out, but the cutting wasn't fixable. It is possible that this happens due to being used inside Chainer though.

[Saga Musix]

My soundcard has this and does this. And if I make a wave export, and I play that wave file on my soundcard, it does the exact same thing again. I tested this, so I know this is a possibility. I have 2 different protections, one is volume normalizing and the other is to cut extreme sounds like loud noise, and other high volume sounds. It is possible that a plugin causes a certain noise that the soundcard identifies as possible threat. True, my soundcard doesn't produce pops for this, but that doesn't mean it can't happen. (it only pops if the initial sound is much louder, but that is probably not the case in this song)

That's not "protection", it's an "effect" (namely a limiter). If the soundcard was indeed preventing the signal from clipping, it would get more quiet for a given amount of time, but it would not pop. If the pop would still be there, the signal would be clipped, and how would that be a "clipping protection" in your book, please? But again this is not relevant, because if you had just a brief look in the file posted, you'd have seen that the pops are not caused by clipping, but by a signal that is slightly shifted for a short amount of time, which indicates that the mixing volume was too high. Again this is totally independent of the soundcard since it is not involved during wav export.

I investigated, and Battery gave the message: CPU 100% every time it cut the sounds

So it doesn't even ramp down in such situations? See, that's really bad.

[Techno.Jon]

Huh.  But my vsti volumes are turned down in the master control box or whatever it's called on that gain multiplyer thing, and to allow the saw bass to be heard, they are all turned way down.  This happens to lower the pops, but that is just because of less volume.  The pops seem to mostly occur on the subbass, even when there are no vsti's playing.  Also, what's this with signal wrapping?  Shouldn't it just clip?  Why would the signal wrap around in such a situation? This would cause the pops, but there are some that are only one pop, not pairs like most, and even then most other things clip.  The clipping is designed to be a form of volume normalization that uses a researched understanding of speakers and human hearing to fully maximize the volume of the song.  However, since the pops are fixable, they are not necessary for this to happen.  They seem to happen with too much bass.  Also, this did happen on my old computer, but less, since the songs were less complex and more sample based.  However, I do have an older song made with nearly entirely square waveforms that had this, even though this was done with full global volume and low sample volume.  Idk, is it possibly a thing with too much bass?

And my sound card does have volume normalization, but I choose to turn it off.  And MPT has maxed out a core in playback before, but it slowes down to render the wave, never maxing the core on render, just hitting 95% or so.  It seems that, to avoid problems with maxing, it just slows down the render to less than real-time to compensate.  I once had a 6-min piece that took 10 mins to render to avoid core maxing, and the wave came out sounding much better than real-time.  Also, I don't use slow render, and wave render is software, yes?

I don't know much more I can say...

[Saga Musix]

The clipping is designed to be a form of volume normalization that uses a researched understanding (...)

Uhm, what? Clipping is just like "Hey, I can't increase the volume even more, so we will just stay at the highest possible volume level", there is no "design" or whatever behind it. And clipping can only be applied if the program actually detects that the volume exceeds the allowed range. Since the internal mixing precision is finite, you only have a finite amount of clipping headroom that can be detected as such; if the signal is even louder than this (i.e. if the signal values exceed MPT's internal 32-bit mixing range), you cannot detect the overflow in the mixing routines anymore and the signal will wrap around (i.e. any numbers that are higher than (2^31)-1 will become negative). And that's exactly what your WAV export looks like. If the signal would be clipping, you'd get a straight line at either the top or the bottom of the signal, but in your example the signal is shifted to the other half of the zero line, so there must be some overflow happening. I am not very familiar with the exact internal mixing maths of OpenMPT but I'm rather confident that turning down the gain of your VSTis will not improve this situation, either.

It seems that, to avoid problems with maxing, it just slows down the render to less than real-time to compensate.

OpenMPT doesn't slow down intentionally (unless you choose the "slow render" method) - rendering will take longer if your VST plugins require too much processing time, if you have other applications running or if you change the process priority - that are all factors that OpenMPT itself cannot influence.

It would help if you could just provide an example module where this happens, preferrably with no commercial plugins so we can try this out. I still bet that your VST/sample amplifcation settings are unfortunately chosen, and checking out an actual module would help confirming this.

[LPChip]

I suppose there could be a compressor in the chain or other kind of effect that boosts signals. A compressor/limiter can be at a stage to put things back into perspective, but if the harm is already done, it will not prevent these pops from being heard.

[Saga Musix]

As a side note, this could of course happen with any other kind of plugin that somehow modifies the audio signal, not just compressors. I'd expect the overflow to happen somewhere towards the end of the audio chain, though, since the output of VST plugins is clipped indeed (in this case it's perfectly possible because VST input/output is floating-point, not fixed-point like OpenMPT's internal buffers).

[Techno.Jon]

http://modarchive.org/index.php?request=view_by_moduleid&query=170537

An (aging) module that uses no plugins but yet has some pops here and there.  This was designed to mock my friend who mocked me for making computer music.  This was also my first experience with these pops.

EDIT:  I just halved Kingdom's sample and vsti volume and doubled the global volume.  Fixed.  And I meant that I leave a little bit of clipping to give the maximum amount of volume to a song, I could very easily avoid it.  The amount of clipping to leave was researched, and so I know how to make it lound without percievable distortion.

[Saga Musix]

I cannot hear any pops on that tune... Do you happen to have Bass Expension or any other effects from the upper half of the "Player" tab enabled? If so, they should be disabled - they are cheap effects and especially when composing modules that are supposed to be spread, you cannot assume that anyone else uses the same settings. But yeah, generally it would be a good practice for future projects to keep the global volume high and the VST/sample volume as low as needed. The default settings when creating a new IT file are a good guideline here.

[Techno.Jon]

Yeah.  Thanks!