Grimoire-
Command
.es

GNU+Linux command memo

Find duplicate files (to remove them and save space)

Trouver des fichiers dupliqués pour les supprimer et libérer du stockage.

Modern Rust solution : fclones

$ fclones group --cache /tmp > /tmp/dupes.txt (1)
$ fclones link < /tmp/dupes.txt (2)
1 Creates the list of duplicated files
2 Replaces clones by links, other operations are : move, remove, dedupe (via native filesystem deduplication capabilities)

A lot of include, exclude and filtering options exists to focus only on some files.

Traditional C solution : jdupes

$ jdupes --delete --immediate --recurse /mnt/backup (1)
$ jdupes /srv/content --recurse: /mnt/backup (2)
1 jdupes can show a lost-space summary or automatically delete dup files. Files are found by sizes, then MD5 signatures (of a heading file portion, or full file), followed by a byte-by-byte comparison (which is very slow but gives exact results)
2 will follow subdirectories under /mnt/backup, but not those under /srv/content.