You got the blue badge partly for your IBS, didn’t you? Are you saying that you’ve never had to find a loo quickly, then? My MH disabilities are hidden to the extent a student psychiatrist thought I was a member of staff when I was in a MH hospital!
Honestly? I doubt people are taking any time out of their lives to frown at you from walking away from a disabled bay - I strongly suspect that’s just your (wrong) perception of it. The only person who doesn’t believe you need those support structures in place is you.
