No, I do not feel guilty. I can't follow everybody that follows me. I follow someone if I know them, want to read their content and feel following them benefits me.
I agree, there's a few people following me (for some reason) that I don't know, and haven't made any worthwhile posts for me to bother with following them back