I agree! The good visits are really precious. I know for a fact I would never be able to visit my mom everyday (even if I physically could) because emotionally it takes huge amount out of me each time. I arrive with high hopes and in 2min my hopes are dashed and I know its going to be a tough visit. Don't feel bad at all about not visiting everyday, I certainly couldn't and I know of many others who also can't manage it, try by all means to make each visit as pleasant as you can, and sometime the old phrase 'absence makes the heart grow fonder' really is true.
I often left earlier than I intended because it just gets too much. I don't blame myself for that anymore, if I need to leave to 'preserve my sanity' then that's what I should do. I still keep hoping the next visit will be better. So I reckon if you're feeling 'cornered' or a bit bullied in anyway just make tracks... you can even go back a bit later if you want to. Having the knowledge that you can excuse yourself is calming in itself.
I truly hope that the visits improve and you can also take the time to take care of yourself - something very important!