This is an old question, but I found it looking for a solution to the same problem and never really found a complete answer. I came up with a simple(-ish) way of doing it myself:

gm convert $(find . -type f -printf '%p0' | sort -z | sed 's/x00/ /g') file.pdf

It won't work if there are spaces or new lines in any of the original files' paths, though.

This method has to execute convert once for every input file. It takes much (much much) longer, especially if there are a large number of original images, but it won't get tripped up by file names:

find . -type f -printf '%p0' | sort -z | xargs -0 -I {} gm convert -adjoin file.pdf {} file.pdf