[Tomisimo]

Estaban sentados. = They were seated. / They were sitting.
Se estaban sentando. = They were sitting down. (They were in the act of sitting down; implies motion even)

Note to non-native English speakers:

Estaban sentados. = They were seated. (sounds quite formal or even slightly archaic, depending on the situation and the speaker)
Estaban sentados. = They were sitting. (informal, extremely common usage, can be used even in writing, including relatively formal writing)

I would translate Se estaban sentando as They were sitting down or even They were in the act of sitting down.

Another example:
They were sitting around the fire. = Estaban sentados junto a la fogata.