rsync is generally perceived to be the fastest of these options, the main reason being that the data is split in chunks and only the new chunks are actually transferred (for example, in a large file of which only small fragment has been modified).

Rsync can also compress data on the fly (just like scp does it); moreover, rsync uses ssh behind the scenes (yes, it can also use ftp: for example, mount the ftp server locally over fuse and use rsync to synchronize the data), so with the exception of compression and the above algorithm, rsync will not be greatly faster than scp.

Unfortunately, that all depends on how duplicity is using rsync. Duplicity files are compressed, so compression will not change much.

If I were you, I would test the different combinations myself before deciding upon one of them.