The global reputation recalc is responsible for your rep loss, this happened globally across the network. Previously, reputation was not real-time or anywhere close, so things that negatively affected reputation were not applied and your _actual_ value vs the highly denormalized value you see got way off sync.
This round of changes aimed to fix that, we not keep your reputation in sync, in most cases instantaneously, in other complicated cases, within 5 minutes.
Downvotes _are_ instantly reflected on your reputation, though the confusion here may be that questions do not penalize you 1 rep any more, only downvotes on answers do this.