Why not have the gears inside the bottom bracket? Looking into an IGH it seems the biggest disadvantage is having a huge amount of rotating mass. I've seen one handmade bike where the IGH sits above the bracket, inside a carbon fibre shroud around the place some bikes have rear suspension. There are two chains, one driving the IGH and another to drive the rear wheel. Why not have an internally geared bottom bracket where the cranks drive the input shaft and the crank ring rotates at a ratio selectable by the gear lever? Then the rear wheel could have a single drive wheel driven by a chain or belt and a lightweight hub just solid enough to mount the spokes and disc. Then you could have mud-free gearing while reducing rotating mass even below what you have with a conventional cassette... and given that you have thrown out the bottom bracket the overall weight gain would be less. Quite a job for the home builder to adapt an existing hub I guess but it should be possible.

The lack of popularity is partially because it's a solution in need of a problem. The effect of rotating mass in general is generally overstated when considering the total energy that goes into cycling, and mass near the axle has even less of an effect. You can work out the math here.

The main advantage comes when considering unsprung mass on full-suspension bikes, where having weight on the rider's side of the suspension can improve the handling and response of the bike. This is why inroads have been made in downhill circles.