Partial answer:
With the additional information "should work both with Pulseaudio, and just with ALSA", I guess you'll have no other choice than to configure which ALSA mixers Pulseaudio changes.
I had a look at that years ago, but I've forgotten all the details. Here's what I remember:
Pulseaudio keeps track of "profiles", which in turn map to a particular ALSA usage, and the files for the mixers are in `/usr/share/pulseaudio/alsa-mixer/paths`. To avoid messing up the default configuration, you need your own profile, and then you need to associate it with the mixers you want.
For writing your own profile, I think I mostly used information from this answer.
I can't remember if I actually needed to setup mixer paths, so there may be a few bumps on the road until that works.
Have fun, this is going to be quite a bit of work.