You do have the right to withdraw yourself as an agent for the POA. It would entail drawing up a new POA with the designated persons who would be the agents, then having it signed by your father and witnesses in front of a notary public.
Before you do something like that Vick, consider this.
What if your other siblings should fall ill or die. What if your father's health declines to a point where he cannot make a reasonable decision and your siblings decide to "take over" and really mismanage everything. Are you willing to let that happen? Should your father get to a stage where he could no longer sign legal documents and you are no longer in a position to make your voice heard, you would be in a real jam.
Take time to think about this. Talk to your father about it. Ask him what he wants. After all, the biggest decision about all of this would be affecting his life more than anyone else's.
Good luck. It's a difficult decision to make. I feel for you, kiddo. Really.