Patty Cakes, it's OK to have two or several emotions at once, and it's perfectly OK to have emotions that represent two ends of the spectrum. I love that you have fully admitted that you don't want to be his caretaker.
Nichole Sachs (TMS therapist/author) has a story about feeling two emotions, especially toward someone vulnerable. She was disliking a family member intensely because they trigger her people pleasing and goodism and were very judgy to her (and that triggered self-judgement). When having to face that person, it struck her that she also felt some famillial love, some connection and at the same time was angry - and that it was totally fine to have the two emotions together. The same thing happened when she wrote her first journal entry that she both hated and loved her children at the same time. She knew she didn't really hate the children - she hated some parts of her live surrounding them: the isolation when they were small, the feeling of overwhelming responsibility, a changing relationship with her spouse, a change in her identity etc. The lack of control, the unknown etc.
What your deep inner psyche feels is probably not what your rational adult mind feels and that creates a little internal tug. It's perfectly normal.