This is a set phrase. It is rendered as pasárselo in Spain, by the way.
Just use the phrase as you hear it used, with the meaning as explained.
Lexical pronouns carry a pragmatic meaning instead of representing an antecedent (the direct object pronoun doesn't represent a feminine person/thing or, in Spain, a masculine person/thing/idea).
|