`C-h` is bound to `backward-delete-char` by default. If you don't like that, bind it to a different command in your `.inputrc`, e.g.
"\C-h": nop
to make it do nothing.
Note that if your terminal sends C-h for the `BackSpace` key¹, and `C-h` is not explicitly bound to an existing command (so not `nop`), then `C-h` will execute `backward-delete-char`.
¹ More precisely, if your terminal's configuration tell applications that this is so. This is the `erase` character in the stty settings. You can display stty settings by running `stty -a`.