GNOME Shell uses its own compoziting window manager called Mutter. According to Owen Taylor's mail it is not possible to use an alternative window manager.

To try and make GNOME Shell integrate
with multiple window managers would
either greatly constrain the user
interface vision or greatly increase
the amount of work involved. The power
of the GNOME shell approach is that we
are working within the desktop scene
graph of the window
manager/compositor.

Using Compiz to create a GNOME desktop
using GNOME applications, the GNOME
control-center, and so forth will of
course remain possible. We have no
current plans to create hard
dependencies on GNOME Shell within the
GNOME desktop (just as there are no
hard dependencies on gnome-panel now.)

http://www.mail-archive.com/[email protected]/msg15587.html