I love this that you wrote @unbreakable :
You describe your tough situation so well. If I were in your shoes, I would insist on "setting him free". If he really does want you and love you all the past complications notwithstanding, he will come and get you. Then you'll KNOW that he wants you for you not because he HAS to be with you. I would need that reassurance for myself. That way, when he does come back looking at you like he would give the world just to have you, you will be able to say to yourself, "Yes this is the awesome dude who LOVES me and WANTS me" Guilt solved and you can go forward as equal partners to build a great future.