He is totally opposed to me or opposed of me.

[ROBINDESBOIS]

I meant opposite to me or opposite of me.

[Rusty]

He is my opposite. (We are nothing alike.)
We are (like) night and day. We are total opposites.

"He is totally opposed to me," means that you and he don't agree on a particular point. Another way to say that is "He is at odds with me." Neither of these mean that you are total opposites.

[Jubilee]

we're like chalk and cheese

[chileno]

Maybe "he is the opposite of me"?