He is totally opposed to me or opposed of me.

He is totally opposed to me or opposed of me.
 

I meant opposite to me or opposite of me.

He is my opposite. (We are nothing alike.)
We are (like) night and day. We are total opposites.

"He is totally opposed to me," means that you and he don't agree on a particular point. Another way to say that is "He is at odds with me." Neither of these mean that you are total opposites.

we're like chalk and cheese

Maybe "he is the opposite of me"?