You have no need to feel guilty! In fact you have every right to also be acknowledged. I know the 'invisibility' comes with the territory. Everyone always asks about the one being cared for...
I had many occasions where people ask after my mother, don't ask me how I'm doing, then to add insult to injury don't even wait for my reply about my mother!
I read once that the person behind the wheelchair is invisible, people only see the person in the wheelchair. I thought that strange until it actually happened to me.. and then I understood and it actually made me mad. From then on I always greet, acknowledge, smile at the person who is doing the caring, then I speak to the one being cared for. It's a strange little distinction I'm sure but somehow it makes me feel that I'm doing something about the crime of not acknowledging the caregiver.
You are not selfish or uncaring! The very fact you are doing what you are doing means you are not a selfish person. So you give yourself a pat on the back and try by all means to make some times for yourself, even if its just 20 minutes. it really does help.