jdupes Deduplication Explained: CoW, Symlinks, Hard Links, and Deletion

jdupes Once you find duplicate files, you can deal with them in four completely different ways:

• -B --dedupe: Let duplicate files share underlying data blocks;
• -l --link-soft: Replace duplicate copies with relative symbolic links;
• -L --link-hard: Replace duplicate copies with hard links;
• -d --delete: Interactively selects files to keep and deletes remaining copies.

These four options can all reduce duplicate data, but they cannot be simply understood as four “delete duplicate files” written in different ways.

They change objects at different levels: data blocks, inodes, path references, or directory entries.

When making a wrong choice, the most troublesome thing is often not that an error is reported on the spot, but that a few months later a program modifies, moves, or deletes a file, only to discover that other paths are also affected.

This article starts from the file system specifications and behavior to explain the real differences, applicable scenarios, restrictions and verification methods of the four modes.

Special reminder: jdupes -d is an operation that will delete files. Do not write the same directory repeatedly to the command line, and do not use -d with -s or --symlinks without fully understanding symbolic link traversal behavior.

Quick comparison: the core differences between the four methods

If you only remember one sentence, you can judge it like this:

• Want each file to remain independent and the file system supports CoW: give priority to -B;
• There is an explicit need for multiple filenames that collectively represent the same file: consider -L;
• Explicitly want some paths to be just references to another file: consider -l;
• Confirm that the redundant path has no preservation value: only use -d.

How jdupes verifies that two files are identical

Although the processing methods are different, the previous duplicate file identification process is the same.

According to the jdupes official manual, the default matching process includes:

1. Compare file sizes;
2. Compare partial file hashes;
3. Compare complete file hashes;
4. Finally, a byte-by-byte comparison is performed.

Only files that pass these stages will enter the same duplicate file collection.

Therefore, the default mode is not “if the file names are the same, even if they are duplicates”, nor is it “if the hashes are the same, perform the operation immediately”.

Without any action options, jdupes only prints duplicate file sets by default:

1

jdupes -r /srv/data

Different sets are separated by blank lines.

Before actually using -B, -l, -L or -d, you should run a read-only scan to confirm that the path range and matching results are as expected.

Don’t bypass final verification in the pursuit of speed

-Q --quick skips byte-by-byte confirmation and relies only on the hash result.

The official manual clearly flags it as a data loss risk.

-T --partial-only is riskier because it only matches a partial hash at the beginning of the file.

When performing deletion, linking, or CoW deduplication, these two options should not be randomly included in order to reduce scan time.

The subsequent examples in this article all use the default full confirmation process.

-B --dedupe: Keep independent files and share data blocks

The basic commands are as follows:

1

jdupes -r -B /srv/data

-B will request the file system to perform low-level deduplication on duplicate files.

The official manual refers to this process as copy-on-write, CoW, cloning or reflink deduplication.

From the directory level, the original file names are still there.

From the inode level, they are still separate files.

From the data block level, the same content can reference the same set of physical blocks, thereby reducing the actual space occupied.

CoW structure after deduplication

Suppose there are two identical files:

1
2

/srv/data/a.iso -> inode 1001 -> 数据块 A、B、C
/srv/data/b.iso -> inode 2002 -> 数据块 A、B、C

The two paths correspond to different inodes, but the same extent shares the underlying data block.

If part of b.iso is later modified, a CoW-enabled file system will allocate new blocks for the changed sections:

1
2

/srv/data/a.iso -> inode 1001 -> 数据块 A、B、C
/srv/data/b.iso -> inode 2002 -> 数据块 A、X、C

Unmodified segments remain shared, while writes are detached.

This is the most critical difference between -B and hard links.

Why it’s suitable for files that may still be modified

When using hard links, multiple paths are actually the same inode.

The program writes content in place through any path, and the content seen by other paths will also change.

When using CoW deduplication, files remain independent.

As long as the file system implements CoW correctly, modifying one file will not change another file simultaneously.

Therefore, the following scenarios are generally more suitable for -B:

• Multiple virtual machine images;
• Backup directories for multiple versions;
• Copies of photos or video footage;
• Duplicate installation packages in software repositories;
• Files with the same content but different lifecycles.

File system requirements for -B

-B It is not jdupes that recompresses or moves the file contents itself.

It relies on the same segment deduplication or cloning interface provided by the operating system and file system.

Support objects listed in the official manual include:

• Btrfs;
• XFS with reflink feature enabled;
• Apple APFS.

Whether it works depends on the kernel, mounting environment, jdupes compilation features, and file system formatting parameters.

For XFS, it is not enough to see that the file system type is XFS; the reflink capability must be enabled when the file system is created.

You can check the file system type first:

1

findmnt -T /srv/data

XFS can further view file system information:

1

xfs_info /srv/data

Do not assume that all duplicate data has been released just because the command does not report an obvious error.

How to verify CoW deduplication results

First make sure the files still have different inodes:

1
2
3

stat -c '%n inode=%i size=%s blocks=%b' \
  /srv/data/a.iso \
  /srv/data/b.iso

This is expected if the two inodes are different.

However, the number of blocks displayed by stat may not directly and accurately express the shared section.

Different file systems may count shared blocks differently.

You can also compare the file system free space before and after deduplication:

1

df -h /srv/data

For Btrfs, you can use file system-specific tools to observe space allocation:

1

btrfs filesystem usage /srv/data

Testing should use discardable samples, modify one of the files, and then verify that the hash of the other file has not changed.

Limitations of -B

CoW deduplication does not equal “zero cost”.

Shared sections require the file system to maintain reference relationships.

Subsequent writes trigger new block allocations and may generate additional fragmentation.

For high-frequency randomly written data, database files, or continuously changing virtual disks, the impact of write amplification and fragmentation should be evaluated first.

Snapshots can also make spatial statistics more complex.

After deleting a file in a directory, the shared block is not released immediately if it is still referenced by another file or snapshot.

If your focus is on Btrfs and NAS scenarios, you can continue to refer to the Btrfs + jdupes CoW deduplication practice on the site.

-L --link-hard: Let multiple paths become the same file

The basic commands are as follows:

1

jdupes -r -L /srv/data

-L replaces the other copies in each set of duplicate files with hard links pointing to the first file inode.

After processing, the multiple path names still exist, but they are no longer separate files.

The structure after hard-linking

Before processing:

1
2

/srv/data/a.bin -> inode 1001 -> 数据块 A、B、C
/srv/data/b.bin -> inode 2002 -> 数据块 A、B、C

After processing:

1
2
3

/srv/data/a.bin --+
                 +-> inode 1001 -> 数据块 A、B、C
/srv/data/b.bin --+

At this time, a.bin and b.bin are two directory entries of the same inode.

They are not just “shared content”, but the same file with two names.

What happens when you modify any hard link

If the program directly opens b.bin and overwrites the bytes in it, the content read by a.bin will also change.

Because both paths ultimately access the same inode.

Inode metadata such as file permissions, owners, and timestamps are no longer independent of each other.

But there is one confusing exception:

Some editors do not modify the file in place, but create a temporary file and then replace the original path with a rename.

This “write new file and then replace” saving method will cause the replaced path to obtain a new inode, thereby releasing the hard link relationship.

So you can’t infer that all applications will behave the same based on just one edit test.

Hard links cannot cross file systems

Hard links can only be created within the same file system.

Even if both mount points use ext4, have different device or file system instances, hard links cannot be made across the boundary.

You can use the following command to view the device number and inode:

1
2
3

stat -c '%n device=%d inode=%i links=%h' \
  /srv/data/a.bin \
  /srv/data/b.bin

After successful processing, the device number and inode should be the same for both paths, and the link count will usually increase.

Also available:

1

ls -li /srv/data/a.bin /srv/data/b.bin

Deleting a hard link path does not immediately delete the content

Deleting b.bin simply removes one of the directory entries.

As long as a.bin still points to that inode, the file data still exists.

The file system will reclaim the corresponding objects and data blocks only when the last hard link is deleted and no process continues to open the file.

This is completely different from the symbolic link “leaving a broken link after the target path disappears”.

When to use -L

Hard links are suitable for scenarios where the premise is clear:

• The files are located in the same file system;
• Multiple paths can indeed be considered the same object in business terms;
• The file is expected to be read-only or the content will no longer be modified in place;
• Backup, synchronization, indexing, and rights management software handle hard links correctly;
• There is no need for separate inode metadata for each path.

Typical examples include read-only package caching, immutable archive copies, and controlled media repositories.

When not to use -L

Use with caution or avoidance in the following situations:

• A copy may be modified in-place by the application;
• Files belong to different users or permission policies;
• Backup software may expand hard links into multiple copies of data;
• Synchronization software may not preserve hard link relationships;
• Applications rely on inode uniqueness to identify files;
• Files span multiple file systems or network mount points.

If two paths have the same content now but different responsibilities in the future, having the same content does not mean they should become the same inode.

-l --link-soft: Replace the copy with a relative symbolic link

The basic commands are as follows:

1

jdupes -r -l /srv/data

-l keeps the first file in each group and replaces other duplicate files with relative symbolic links pointing to it.

Assume that before processing there is:

1
2

/srv/data/original/report.pdf
/srv/data/archive/report-copy.pdf

After processing the second path may behave like a similar relationship:

1

/srv/data/archive/report-copy.pdf -> ../original/report.pdf

The actual relative path is determined by the directory location.

What a relative symbolic link means?

Symbolic links hold the target path text, not the additional name of the target inode.

The starting point for parsing relative links is the directory where the symbolic link is located, not the current working directory when running the command.

Therefore, when moving a directory tree whose internal structure remains unchanged, relative symbolic links will usually still work.

However, moving just the link, or just the target, can invalidate the relationship.

Differences in behavior between symbolic links and hard links

Symbolic links can span filesystems.

Hard links generally cannot span file systems.

Symbolic links have their own file type and inode.

A hard link is just another name for the same normal file inode.

After the symbolic link target is deleted or renamed, the link may become broken.

When you delete a hard link name, the content remains accessible as long as there are other hard links.

Symbolic links can point to directories; the jdupes actions discussed in this article target duplicate file sets.

A symbolic link changes the file type

Before executing -l, both paths were ordinary files.

After execution, one of them is still an ordinary file, and the other paths become symbolic links.

This affects some applications:

• Security policies may deny following symbolic links;
• The container or sandbox may not see the link target;
• The web service may deny access to the path to which the link points;
• Backup software may only back up the link itself;
• File monitoring programs may generate different events for links and targets;
• Permissions checks ultimately fall on the target path and its parent directory.

Therefore, “the file can still be opened” does not mean that the application compatibility has been verified.

How to verify symbolic links

View the target text saved by the link:

1

readlink /srv/data/archive/report-copy.pdf

Analyze the final goal:

1

realpath /srv/data/archive/report-copy.pdf

Check for broken links in the directory tree:

1

find /srv/data -xtype l -print

find -xtype l will find symbolic links that the final target cannot resolve.

Before official use, operations such as moving target files, moving upper-level directories, backup and recovery, and application reading should also be tested.

When to use -l

Symbolic links suit scenarios where the reference relationship is inherently appropriate:

• Want to explicitly specify an authoritative copy;
• Other paths are inherently just entry or compatibility paths;
• Requires cross-file system references;
• Applications explicitly support symbolic links;
• Able to guarantee the life cycle of the target path.

Symbolic links are generally unsuitable if the directory tree is frequently split, synchronized separately, or packaged separately.

-d --delete: Select items to keep, delete remaining copies

The basic commands are as follows:

1

jdupes -r -d /srv/data

-d prompts the user for each set of duplicate files to select which paths to keep, and then deletes the remaining paths.

This is the most intuitive of the four methods and the most irreversible one.

-B, -L and -l will retain the original path entry in a certain form.

-d actually removes the directory entries of unreserved files.

Removing duplicate content does not mean there is no loss of information

The fact that the contents of the two files are exactly the same does not mean that the two paths have no value.

Different paths may express different things:

• directory classification;
• File name semantics;
• Affiliated projects;
• access control context;
• Backup retention policy;
• application indexing relationships;
• User workflow.

jdupes determines whether the file content is duplicated, and will not determine which directory entry has operational significance for you.

Therefore, both the content and the path need to be reviewed before deletion.

Critical risk 1: do not specify the same directory more than once

Don’t run it like this:

1

jdupes -d /srv/data /srv/data

Also avoid passing the same directory tree twice using different writing methods, for example:

1
2

cd /srv
jdupes -d data /srv/data

Official documentation warns: When the same directory is specified multiple times, the same file may appear in the collection as a “duplicate of itself”.

If a user retains one display item in this confusing collection but deletes another display item that represents the same real file, data may be lost.

All input paths should be normalized before execution and confirmed that they do not have duplicates, aliases, or unexpected overlaps.

You can check the real path first:

1
2

realpath /srv/data
realpath ./data

Also keep an eye on bind mounts, symlink directories, and container mappings to allow the same content to be traversed from multiple entries.

Critical risk 2: combining -d with -s

-s or --symlinks will follow the symbolic link directory.

When this is used together with -d, the interactive list may show both the symlink-related path and the real file being pointed to.

The official manual states that users may mistakenly retain a symbolic link but delete the file it points to.

The result is that a path appears to be preserved, but in fact only links are left that cannot reach the target.

Do not use: unless you have fully mapped and verified symbolic link relationships:

1

jdupes -r -s -d /srv/data

A safer approach is not to follow symbolic links first, but to check the real directory and link structure separately.

-N --no-prompt will turn deletion into an automatic operation

-N, when used with --delete, automatically retains the first file of each group and deletes the remaining files.

For example:

1

jdupes -r -d -N /srv/data

This is not an ordinary “skip confirmation”, but directly turns “who is the first item” into a retention strategy.

If you really need automation, you should first understand:

• -o name sorts by file name by default;
• -o time can be sorted by modification time;
• -i will reverse the sorting;
• -O --param-order will give priority to preserving the impact of command line parameter order on the result set.

Before automatic deletion, a read-only scan should be performed with the same path, sorting, and recursion options and the results should be saved for auditing.

For data without reliable backup, it is not recommended to use -d -N directly.

The impact of four modes on permissions and metadata

By default, jdupes’ core goal is to detect duplicate content.

But files with the same content may have different owners, groups, or permission bits.

-p --permissions can require that files with different owners, groups, and permissions not be treated as duplicates:

1

jdupes -r -p /srv/data

This is especially important for -L.

After hard linking, multiple paths share inode metadata, and it is impossible to maintain their different owners and permission bits.

For -d, which path is chosen to keep will also determine which file metadata is ultimately left.

For -l, access to content is ultimately controlled by target file and path traversal permissions.

For -B, the file inodes remain independent, so their respective permissions and most metadata can continue to exist independently.

If you also rely on ACLs, extended attributes, SELinux tags, or file capabilities, additional verification should be done using the corresponding tools.

Just comparing traditional permission bits does not mean that all extended metadata has been incorporated into business judgments.

Choosing across filesystems

When scanning multiple mount points, you can join:

1

jdupes -r -1 /srv/data /mnt/archive

-1 --one-file-system prevents matching across file systems or devices.

This can reduce the probability that subsequent actions will encounter the boundary of capabilities.

The differences between the four modes across file systems are as follows:

• -B relies on the file system interface and shared segment capabilities, usually requiring the target to be within the compatibility range;
• -L Unable to create cross-file system hard link;
• -l can save path references across file systems, but the link is unavailable when the target mount is missing;
• -d You can delete copies on different file systems, but you must confirm that the storage where the retained items are located is available for the long term.

For example, deleting the local copy and retaining only the copy on a removable hard drive or temporary network mount may be technically successful, but business-wise it is very dangerous.

A safe execution workflow

Do not add action options directly to the production directory.

It is recommended to execute in the following order.

Step 1: Confirm that the backup or snapshot is recoverable

The existence of a snapshot does not mean that it can be restored.

Confirm at least:

• The snapshot covers all directories scanned this time;
• The snapshot creation time is earlier than the deduplication operation;
• Snapshots do not share the same fault domain as the production directory;
• Known ways to recover individual files and entire directories;
• Recovery operations are subject to spot testing.

Hard links and symbolic links also require confirmation that the backup tool preserves link semantics.

Step 2: Confirm that the input path is not repeated

List each directory to be scanned:

1
2

realpath /srv/data/project-a
realpath /srv/data/project-b

Check if they:

• Point to the same real directory;
• One directory completely contains another directory;
• Enter the same directory tree again via a symbolic link;
• Aliases appear through bind mount or network mount;
• Duplicate due to shell wildcard expansion.

Especially when using -d, any unclear overlap should be eliminated first.

Step 3: Do a read-only scan first

1

jdupes -r /srv/data/project-a /srv/data/project-b

Save the output to an audit file:

1
2

jdupes -r /srv/data/project-a /srv/data/project-b \
  > /tmp/jdupes-review.txt

The action option is not used here, so only the set of matches is output.

Check group by group which paths should preserve independent semantics and which are just redundant copies.

Step 4: Verify behavior in a small test directory

Create a discardable test directory:

1
2
3
4

test_dir="$(mktemp -d)"
mkdir -p "$test_dir/a" "$test_dir/b"
printf 'jdupes test data\n' > "$test_dir/a/sample.txt"
cp "$test_dir/a/sample.txt" "$test_dir/b/sample.txt"

First look at the inode and hash:

1
2

stat -c '%n inode=%i links=%h' "$test_dir"/*/sample.txt
sha256sum "$test_dir"/*/sample.txt

Then only test one action at a time.

Test hard links:

1
2

jdupes -L "$test_dir/a" "$test_dir/b"
stat -c '%n inode=%i links=%h' "$test_dir"/*/sample.txt

To test other actions, re-create a clean sample and do not continue overlaying tests on already hard-linked files.

Step 5: Perform one action at a time

Don’t pile four processing modes into the same command.

Split batches based on directory responsibilities, for example:

1

jdupes -r -B /srv/data/mutable-archive

1

jdupes -r -L /srv/data/immutable-cache

1

jdupes -r -l /srv/data/compat-links

1

jdupes -r -d /srv/data/manual-review

This makes it easier to interpret results, verify changes, and perform recovery.

Step 6: Post-operation verification

Whichever method you choose, perform at least these checks:

1

jdupes -r /srv/data

1

find /srv/data -xtype l -print

1

df -h /srv/data

Then add verification based on the model:

• -B: Confirm that the inode is independent, and after sampling and modifying one copy, the other remains unchanged;
• -L: Confirm that the inodes are the same and the link count is correct;
• -l: Use readlink and realpath to confirm the target and check for broken links;
• -d: Confirm that the reserved path still exists according to the audit list.

Finally, have the application that actually uses the files perform a read, index, backup, or sync test.

Choosing by use case

NAS media library and photo archive

If the underlying layer is Btrfs that supports deduplication, and the file may be modified by a photo management or media management program, evaluate -B first.

It can retain independent file semantics and reduce the risk of linked modifications of hard links.

-L may also work if the directory is completely read-only and the application handles hard links correctly.

Package caching and build artifacts

-L is usually simple and efficient when the immutable build artifacts are on the same file system.

However, hard links should be avoided if the build tool overwrites cached files in-place.

For caches that can be regenerated at any time, you can also use -d directly, but you still have to make sure that deletion will not destroy the index.

Multi-version backup directories

Multi-version catalogs require each version to maintain a separate view.

When CoW is supported, -B generally conforms to this semantics better than -L.

Hard links are also commonly used for incremental backups, but they must be systematically managed by the backup program and cannot be temporarily replaced based on “the current content is the same”.

Maintaining compatibility with old paths

If an authoritative file needs to be accessed from multiple legacy paths and the software supports symbolic links, you can choose -l.

This is equivalent to explicitly telling the system that other paths are just references.

The destination path must be stable and covered by backup, container, and permission policies.

One-time cleanup of a download directory

To confirm that duplicate copies have no directory semantics and backups are available, use interactive -d.

Read-only scan first, then select reserved items group by group.

Don’t jump right into -N just to save a few keystrokes.

Common misconceptions

Misconception 1: All four modes save exactly the same amount of space

uncertain.

-d, -L, and -l end up with just a plain file, but still with different numbers and types of directory entries or link objects.

Actual savings from -B depend on successfully shared extents, block sizes, alignments, snapshots, and subsequent writes to the file system.

Misconception 2: A hard link is simply a more reliable symbolic link

The two have different semantics and cannot be simply sorted by “reliability”.

Hard links bind multiple names to the same inode.

Symbolic links store a resolvable target path.

The former does not have the problem of “the link is broken when the target path is renamed”, while the latter can cross file systems and clearly express the reference relationship.

Misconception 3: Files deduplicated with CoW become hard links

no.

The files after CoW deduplication still have independent inodes, but share some or all data blocks.

A hard link is the same file from the inode level.

This difference can be observed with ls -li or stat.

Misconception 4: If the content is identical, either copy can be deleted safely

The same content only proves that the bytes are the same.

Paths, file names, permissions, labels, business ownership and backup policies may still differ.

The selection of -d must be based on path semantics, not just file content.

Misconception 5: Relative symbolic links are always more portable

Relative links are only portable if the link and target maintain relative positions.

If you copy only one of the subdirectories, the link may immediately become invalid.

Packaging, synchronization and container mounting may change the original directory relationship.

Misconception 6: df must show more free space immediately after deletion

uncertain.

The file may still be open by the process, or it may be referenced by a snapshot, other hard link, or CoW share.

File system delayed reclamation and space accounting methods also affect the observed results.

References

• jdupes official project repository
• jdupes official man page source code

Final recommendations

If the file system supports CoW and you want the file to be able to be independently modified later, -B --dedupe is usually the most least intrusive choice at the semantic level of the four options.

If you explicitly need multiple paths to the same immutable file, all on the same file system, use -L --link-hard.

If you need to establish a clear path reference relationship and can accept the risk of broken links after the target moves, use -l --link-soft.

If the redundant path itself has no preservation value and there is already a reliable recovery method, use -d --delete.

No matter which one you choose, you should follow the same principles:

First perform a read-only scan, check the path, test the application behavior, then execute the action, and finally verify the file system and operational results.

Again, for -d: do not specify the same directory twice, and do not combine it with -s or --symlinks without fully checking the symbolic link relationship.

The saved storage space can be repurchased, but the lost directory semantics and unique copies may not be able to be reconstructed.