I looked into this, and it does seem like `remove_id` was never implemented for `usb-serial`. Should be able to take the work in `drivers/usb/core/driver.c` and implement `remove_id` in `drivers/usb/serial/bus.c`.
Sorry for not having an easy answer.