Paula's right. You just have to tell him that you love him and you want him to know that, and to know that you want to be there for him whenever you can and if he's willing you'd like to build a relationship with him. Tell him that you don't want to take him away from his grandparents, but you love him too and would love to spend some time with him.