Hmm. I guess I would feel genital is worse, but is it really?
I will say I was married many, many years and I only vaguely knew I couldn't kiss or give oral if a sore was present. I just had no idea how potentially serious my fever blisters were. My husband never got it, at least never had a breakout. Your boyfriend was probably like that (uninformed and unaware.) And since it sounds like his sores pop up rarely, it just may have never occurred to him.
You're very kind. My partner was furious with me and unleashed his anger full force for a few weeks. I can't blame him, but it hurt.