People who are interested in this application should check synplant[0]. It has a ML technology called "Genopatch" which gives you 2 functionality:
1. you can try to describe a sound with some tags and it will try to generate a sound to capture the feeling of these tags
2.you can feed it with a sound sample and it will try to re-synthesize the sound with its synth engine. Though the end result will usually be just a "re-imagined" version of your input sample.
My guess is the underlying model is not a "deep" model. The main benefit is that the end result is not a wave file, but a list of generated parameters that can be synthesized by the synthplant engine. And now it comes the interesting part: you can tweak these parameters to finetune the generated sound. These parameters have actual meanings (FM ratio, reverb etc.)
How far are we from getting a general model that can resynthesize any instrumental audio sound without fiddling with any knobs, so that we can recreate instruments we hear from any song? Seems like it should exist by now?
Fiddling with the knobs is the fun part. https://www.youtube.com/watch?v=la2u4VlGwbQ
For me creating the exact sound is not very interesting from sound designing perspective. You can always sample the real instrument.
Like physical modeling synthesis, the interesting part is to compress the sound to some parameters that you can tweak and generate new sounds
Another approach is VAE, which also you give your some latent embedding, you can tweak the embedding to generate new sound. However the meaning of this embedding is not explicit.
"You can always sample the real instrument."
This doesn't really work on instruments like guitars. Open D sounds way different than fretted D on the E string. Timbre changes with position and it's one of the ways I determine where a player's hands are on the neck when I'm trying to play their song.
That is not something inherent in guitars themselves, it is the norm in steel string guitars and the fan-braced/Spanish guitar but mostly because that is the norm for all those mass produced guitars which make up the bulk of guitars. On steel string you can often greatly decrease this quality just by switching to flatwounds, this is part of the flatwound sound, it shifts the timbrel content into the players technique but if you want much timbrel content with flatwounds you need heavy strings and a high action, and the hand strength and technique that sort of setup requires.
Before the rise of the steel string and the Spanish guitar, guitars tended to be more even across their range and also had less bass which helped even them out, and now that sound is what we are used to. There have always been niches that wanted that more even sound, but for most that just makes it more difficult to play all that music that developed around these quirks, so they remain niches.
I'm not doing fancy AI stuff but I have worked a lot with my own bespoke supercollider system where I record whole fretboards of guitars and then play alternative notes based off of certain rules. For whatever dumb reason though, the most natural sounding thing is really just playing, e.g., any random D4 from its possibilities at any given moment.
Timbral differences also exist depending on force, the manner plucked, the already ringing overtones... It's hard to know what you want, but the most natural thing is always going to be some organic variation in the notes in general.
If you have a good ear, you aren't, I don't think, hearing so much the timbral diff in the individual open or fretted notes as much as the fact that a barre chord and an open chord is a different voicing of the same harmony.
No, I'm going off the timbral differences - same way I identify which pickup position is being used. There's a specific 'thickness' I cue in on to determine pickup and specific note placement.
Huh, got it. That's pretty cool!
SUNO is pretty close. It still has some weird things going on with high frequency artifacts and phase between left and right channels but if you aren't listening on a good system (like a phone) most people probably wont notice.
synplant is a great synth!