We've just talked about it a little on the phone and I admitted that I wasn't ok with them spending time together as a family and that it wasn't fair on anyone. He agreed with me. I still feel guilty but at least I was honest