To use IPSec both sides needs some tools for key exchange. You _can_ do key exchange manually but nobody does it. Protocol named ISAKMP/IKE used for key exchange. It uses `udp/500`. On *BSD systems racoon daemon is used for it. Since key exchange takes place at first connection and then keys are updated periodically you need to accept incoming `udp/500` for IPSec to work.
If you close `udp/500` IPSec may detect it and use NAT traversal (< (`udp/4500`) which allows client not to accept incomming connections. This could be your case.