The danger of this is your rim may begin to rub the brake pad – especially when you put in hard out-of-the-saddle efforts.
In theory, if your caliper has become off-centre then one pad will have come closer to the rim and the other will have come further away. You should simply be able to push and rotate the caliper by hand back into a central position without loosening (or tightening) the retaining nut. Pull the brake lever a couple of times and if the caliper does not return to a central position, your brake requires balancing i.e. one pad is moving more than the other towards the rim and causing them not to touch the rim at the same time. The other effect of this is it may cause the caliper to rest off-centre.
There is a small balancing (tensioning) screw on top of the caliper. Turn this left or right until both pads move the same amount to the rim.