Compositing in X is taking all the X windows and turning them into a single window so that you can apply special effects to them. These pages discuss compositing in Metacity.
Metacity/How we do compositing (rather techy)
Metacity/libcm (what it is, why it matters)
You might also be interested to read: