I think what's happening is that the inner xdotool command is reporting multiple window IDs. So the outer command sees something like xdotool windowactivate 12345678 21043361 and doesn't know what to make of the second number.

If you don't mind which open Nautilus window you activate, you can select the first one:

xdotool windowactivate $(xdotool search --onlyvisible --class nautilus | head -n 1)

If you do mind, you might want to select further with --title or --maxdepth.

To exclude the root window, which is provided by Nautilus, this should work:

xdotool windowactivate $(xdotool search --onlyvisible --class nautilus |

                         grep -vxF $(xwininfo -int -size -root |

                                     sed -n 's/.*Window id: *([0-9]+).*/1/p') |

                         head -n 1)