No. Feeling this way is not letting him down. Feeling this way and throwing a hissy fit, demanding he cancel would be letting him down. But you were never going to do that ....