Do I need to break up with him? How do I fix this? Is this all just silly?

Well what is your question? It dosnt seem like there is a way to fix it, either you guys break up or you dont. And you said you already plan to break it off. So i guess just learn to deal with it as best as you can