I haven't read much on here lately - but I just happen to take a quick glance and saw your post.
I think you are correct that both your sister and BIL need counselling - but if neither party is willing to take that step then the ONLY thing you can do is suggest.
I am sorry that your Mother is getting to the point that she is fearful to leave the house. Although different - my Mother has always been fearful of leaving the house - not so much because she doesn't know where she is - but has always believed that someone is out of get her. It's very hard to watch someone like that.
I guess the one thing that surprised me - is that this is really getting to your BIL. Especially since they cared for HIS Father for a year before he passed away - so care-giving is not a "new" and "unfamiliar" road.
Maybe you could suggest a mini vacation for them - maybe even a long weekend - where you & your other sister & your Mom's aide could care for your Mom - where they can just get away together and have some alone time with no responsibilities.
I do hope it all works out.