From the way you describe the problem it seems less likely to be one of the common mechanical causes, like a stiff chain link or bent chainring. I have two guesses as to the cause. One is a subtle mechanical problem that requires the right conditions to set it off. The other is about technique.
1. The rear derailleur may not be taking up enough slack when you're in the small chainring. (This is not always obvious.) If you happen to shift around the time you hit a little bump or dip, or while you throw your weight forward to stand, or while you adjust your cadence, the slack chain gives a bounce and off it comes.
2. You may be shifting improperly on these rare occasions. If you shift to the innermost chainring while you're in the outermost cog, and the chain line is angled too sharply, off it goes. Also, to reduce the odds of dropping the chain, a front shift ought to be done like this.