I don't want to upset you with my post, but just want to be firm and honest.
I can see that you have a bad relationship with your brother and perhaps one which may be too far gone to salvage.
You're post resonates with me, because, I, too have had people whom I thought didn't care and everytime I saw them my natural reaction was to be upset.
I think now you have come to this hard, self realisation, that he is and always be the same, you need to accept him for how he is. I know it's not easy, but as you shift your focuss from upset, anger,
Sadness into a more compassionate space ie, accepting what is and letting go you will feel a reduction in anger and your bodys energy will restore to a place where healing is possible.