I have 4 hard disks and want to find out which files on this four harddisks (including sub directories) are duplicates. It should be checked not only within a harddisk but against all the others as well.
The hard disks are large (3TB) therefor it has to be efficient (first filename than checksum checks or so)
I used the FSlint project and find to get the thing done.
My process to get all of this sorted out on multiple disks with the requirement to run everything via CLI & screen
sudo apt-get install fslintfind path1/2/3 -type f -empty -delete & find path1/2/3 -type d -empty -delete (to get rid of alle empty or not completly copied stuff)/usr/share/fslint/fslint/findsn path1 path2 path3 (delete everything which is stored on the same directory with the same size on different disks)/usr/share/fslint/fslint/findup path1 path2 path3 (delete all duplicate files)find path1/2/3 -type d -empty -delete (to get rid of the directories which are empty after findup)after that I was able to mount all disks as a combined drive with mhddfs again without having duplicates wasting diskspace again
