Math is hard for everyone. It takes long hours of practice to become competent. I think that's what people want to hear; and fortunately, it's true.
This reply has the advantage of steering the conversation in a somewhat more positive direction---in my experience, usually towards whatever it is that the person _does_ like to do and has practiced long hours at. This is usually more interesting than having a conversation about poorly taught math courses.