Is it Possible?

[iamyish]

Before i start down this rabbit hole, is it even possible to port just the audio engine with the instruments and sampler function intact into a vst plugin?

[Saga Musix]

Anything is possible with a bit of code. I guess the main question is what you want to achieve and how much you are willing to bend the code. Do you want to use the OpenMPT engine but provide your own UI to load samples / instruments, or what is it that you need?

[iamyish]

I would like to use the ui as is to start and get a working prototype, then move on to a more tailored ui after.

[Saga Musix]

Putting the entirety of OpenMPT, in partiular the MFC-based UI into a plugin is probably not going to work directly. There's quite a lot of code in OpenMPT that would break if two copies of the UI code were running at the same time (e.g. when loading the plugin twice into your host). There was an attempt at making a basic OpenMPT VSTi in 2007, and that required running a special OpenMPT process that would then communicate with the actual VSTi loaded into the host (probably via shared memory).
On the other hand, the playback engine code is fully portable and can easily be used in a shared library, so starting with your own UI is probably going to present fewer roadblocks.

Note that OpenMPT has no stable "internal" APIs - any communication between engine and UI code can change at any point in time. libopenmpt has a well-defined interface but depending on what you want to do, it might not offer enough functionality. But it could make more sense to add the require functionality to the libopenmpt API instead of directly communicating with the engine, as there would be fewer moving parts when you wanted to update to a newer version of the OpenMPT code.

[iamyish]

Let me do some additional brainstorming to flush my idea out a bit more. Thanks for the replies.