Two Btrfs Zoned Failures on an HC620: Deadlock, Read-Only Remount, and Kernel Troubleshooting

This is a real fault review. The environment is a Western Digital HC620 Host-Managed SMR hard disk, using Btrfs zoned, mounted on /mnt/disk1, the system is Ubuntu 26.04 LTS, and the kernel is:

1

Linux data-server 7.0.0-28-generic

The fault was not an ordinary hard disk speed decrease, but two serious abnormalities occurred one after another:

1. btrfs-cleaner and btrfs-transaction enter the uninterruptible D state, and the file system cannot be unmounted;
2. The Btrfs transaction is aborted with -11, and the kernel actively switches the file system to read-only.

Both failures occurred around multi-terabyte deletions, data copies, and zoned space reclamation. The system was not reinstalled, and btrfs check --repair was never run. This article preserves the original logs, reasoning, commands, and unresolved questions for anyone investigating a similar failure.

Note: This does not prove that every Btrfs deployment on an HC620 will fail, and a single call stack cannot identify a specific kernel fix. The kernel and Ubuntu package versions reflect the system as of July 24, 2026. Check the current repositories before following the version-specific steps.

Environment and triggering process

This disk mainly stores backup and cold data. The operations before the first failure are roughly as follows:

1
2
3
4

删除约 3TB 旧数据
→ 从另一块盘复制约 3TB 新数据
→ Btrfs 后台清理与新写入重叠
→ btrfs-cleaner 锁死

The HC620 is a Host-Managed SMR. Disks are managed according to zones. Each zone is displayed on site as:

1

Device zone size: 256.00MiB

On these devices, a written zone cannot be overwritten freely as it can on a conventional CMR drive. Deleting a file only removes references to its data blocks. The invalidated space cannot be reused until Btrfs reclaims the block group, moves any live data, and resets the zone. Btrfs reports this space as:

1

Device zone unusable

So “delete 3TB and immediately write 3TB” is not just a delete plus a copy. Possibly happening simultaneously in the background:

1
2
3
4
5
6
7

提交删除事务
删除空 block group
回收旧 chunk
finish/reset zone
分配新 chunk
激活新 zone
持续写入新数据

This becomes an important background for subsequent analysis of the call stack.

First failure: btrfs-cleaner blocked for a long time

The initial log contains:

1
2
3

INFO: task btrfs-cleaner:1488 blocked for more than 368 seconds.
Not tainted 7.0.0-28-generic #28-Ubuntu
task:btrfs-cleaner state:D pid:1488

The key call chain is:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12

__mutex_lock_slowpath
mutex_lock
btrfs_chunk_alloc
btrfs_inc_block_group_ro
do_zone_finish
btrfs_zone_finish_one_bg
btrfs_zoned_activate_one_bg
reserve_chunk_space
check_system_chunk
btrfs_remove_chunk
btrfs_delete_unused_bgs
cleaner_kthread

Read it from bottom to top, the process is:

1
2
3
4
5
6
7

btrfs-cleaner 清理未使用的 block group
→ 删除旧 chunk
→ 检查并预留 system chunk 空间
→ 激活 zoned block group
→ finish 一个 zone
→ 再次进入 chunk 分配
→ 等待 mutex

The log also indicates that the mutex is likely held by the same btrfs-cleaner thread. Combined with the call chain, the on-site judgment is that the cleaning thread enters the lock wait when block-group deletion, zone finish and new chunk allocation overlap. Subsequently, btrfs-transaction and kworker waiting for Btrfs flush were also blocked.

This is an explanation of the on-site call stack, which does not mean that the only corresponding upstream bug has been found. What is certain is:

• Blocking occurs in the block-group/chunk cleanup path of Btrfs zoned;
• The thread has entered the D state in the kernel state;
• Ordinary kill -9 cannot terminate the kernel thread;
• Continuing operations that depend on transaction commit may get stuck together.

Initial diagnostic checks

First stop rsnapshot, rsync, Docker container or other disk writing tasks, and then check the uninterruptible process:

1
2

ps -eo state,pid,ppid,comm,wchan:35,args | \
awk 'NR==1 || $1 ~ /^D/'

Save the kernel log of this startup:

1
2
3
4
5

sudo journalctl -k -b \
  > /root/btrfs-hang-$(date +%F-%H%M).log

sudo dmesg -T | \
grep -iE 'btrfs|hung task|I/O error|ata|blk_update'

Save the relevant thread stack:

1
2

sudo cat /proc/1488/stack
sudo cat /proc/1489/stack

Add a timeout when querying the file system to prevent the diagnostic command itself from permanently blocking:

1
2

sudo timeout 15s btrfs device stats /mnt/disk1
sudo timeout 15s btrfs filesystem usage /mnt/disk1

Then try a normal uninstall:

1
2

cd /
sudo timeout 60s umount /mnt/disk1

turn out:

1

umount: /mnt/disk1: target is busy.

target is busy is not the same thing as kernel lockup

target is busy usually means that the following reference is still available:

• The current working directory of the process is located at /mnt/disk1;
• The file is still open;
• The backup or copy program is still running;
• There are sub-mount points under /mnt/disk1;
• The user process has entered the D state due to file system lock.

First ensure that all terminals exit the mounting directory:

1
2

cd /
pwd

Check submounts and occupiers:

1
2
3
4

sudo findmnt -R /mnt/disk1
sudo timeout 15s fuser -vmM /mnt/disk1

pgrep -af 'rsnapshot|rsync|borg|restic|tar|cp|mv|docker'

Ordinary user processes can be terminated normally before considering forced termination:

1
2
3

sudo kill -TERM 1234 5678
sleep 5
sudo kill -KILL 1234 5678

If rsnapshot or rsync:

1
2
3
4
5

sudo pkill -TERM -x rsnapshot
sudo pkill -TERM -x rsync
sleep 5
sudo pkill -KILL -x rsnapshot
sudo pkill -KILL -x rsync

But don’t try to kill:

1
2

btrfs-cleaner
btrfs-transaction

They are kernel threads. On-site inspection shows that the main blocker is the Btrfs kernel thread, and continuing kill, umount or global sync cannot solve the root cause.

Why lazy unmount or forced unmount is not used?

It was not implemented at that time:

1
2

sudo umount -l /mnt/disk1
sudo umount -f /mnt/disk1

umount -l just removes the mount point from the directory tree first, and the actual file system reference still has to wait until it is no longer busy before it can be cleaned up. When a Btrfs transaction is locked, it may only hide the mount point and does not mean that the data has been committed or that the file system has been unmounted.

umount -f is also not a reliable means of relieving local Btrfs kernel deadlocks. It is more commonly used on parts of the network or in user-mode file systems.

Also not running:

1
2
3
4
5
6

sync
btrfs balance start /mnt/disk1
btrfs scrub start /mnt/disk1
btrfs filesystem defragment -r /mnt/disk1
mount -o remount,ro /mnt/disk1
btrfs check --repair /dev/sda

These operations may wait for the blocked transaction or re-enter the problematic block-group reclamation path.

Add noauto before restarting

In order to avoid using the same kernel to automatically read and write this disk immediately after the machine restarts, first check /etc/fstab:

1

grep -n '/mnt/disk1' /etc/fstab

Backup configuration:

1
2
3
4

sudo cp -a /etc/fstab \
  /etc/fstab.bak-$(date +%F-%H%M%S)

sudoedit /etc/fstab

For example it turned out to be:

1

UUID=xxxx /mnt/disk1 btrfs defaults 0 0

Temporarily changed to:

1

UUID=xxxx /mnt/disk1 btrfs defaults,noauto 0 0

noauto means only that it will not be automatically mounted when booting, not that it will disable the file system. After the system is started, it can still be manually mounted read-only or read-write.

From normal restart to last resort

A normal restart was performed on site:

1

sudo systemctl reboot

This time it was completed normally, and the file system was able to be remounted after restarting. If the normal restart is stuck due to Btrfs, the level of enforcement will be increased step by step according to the risk:

1

sudo systemctl reboot --force

The last resort is:

1

sudo systemctl reboot --force --force

Two --force no longer stop the service and unmount the file system normally, the effect is close to a hard reset, and uncommitted data may be lost. It can only be used when the system cannot be terminated normally and cannot be used as a normal restart command.

Check after restart

First confirm the actual booted kernel and disk:

1
2
3

uname -a
lsblk -o NAME,SIZE,MODEL,FSTYPE,MOUNTPOINTS
sudo smartctl -x /dev/sda

Confirm there is no automatic mounting:

1

findmnt /mnt/disk1

Mount read-only for the first time:

1

sudo mount -o ro /dev/sda /mnt/disk1

In the actual environment, UUID or /dev/disk/by-id/ should be used first, and the default device should not always be /dev/sda.

Check the kernel log, Btrfs device count, and space status:

1
2
3
4
5

sudo dmesg -T | \
grep -iE 'btrfs|ata|I/O error|medium error|uncorrect|reset'

sudo btrfs device stats /mnt/disk1
sudo btrfs filesystem usage /mnt/disk1

Live btrfs device stats all 0:

1
2
3
4
5

[/dev/sda].write_io_errs    0
[/dev/sda].read_io_errs     0
[/dev/sda].flush_io_errs    0
[/dev/sda].corruption_errs  0
[/dev/sda].generation_errs  0

This shows that Btrfs does not record device read and write, flush, verification or generation errors, which is a good sign. But it does not prove that the kernel lockup has been fixed, nor does it rule out the problem of unread data.

What scrub does and doesn’t do

A scrub is scheduled after the system is stable:

1
2

sudo btrfs scrub start /mnt/disk1
sudo btrfs scrub status /mnt/disk1

If you want the front desk to wait for results:

1

sudo btrfs scrub start -B /mnt/disk1

Cancel task:

1

sudo btrfs scrub cancel /mnt/disk1

Scrub reads the stored data and metadata and checks the checksum saved by Btrfs. It is used to find read errors, data or metadata validation errors, not:

1
2
3
4
5
6

碎片整理
空间回收
balance
坏道修复
完整 fsck
内核死锁修复

If the result is:

1

Error summary: no errors found

It can only show that no exception was found in this reading and verification, but it cannot prove that the zoned cleanup path of btrfs-cleaner will not be locked again in the future.

Save results:

1
2

sudo btrfs scrub status /mnt/disk1 | \
sudo tee /root/btrfs-scrub-$(date +%F).log

Do not run rsync, rsnapshot, balance or perform mass deletions at the same time during Scrub. For this type of backup disk, it can be performed every one to three months based on actual usage intensity.

Why balance is not run for “reclaiming space”

Appeared at the scene:

1

Data,single: 98.67%

This number only represents the usage of the allocated Data block group, and does not mean that only 1.33% of the entire disk is left. At that time, there were still several TiB Device unallocated, and there was no shortage of space to continue to allocate.

So there is no execution:

1
2

sudo btrfs balance start /mnt/disk1
sudo btrfs balance start -dusage=0 /mnt/disk1

Balance will relocate the block group where the first failure is:

1
2
3
4

btrfs_delete_unused_bgs
btrfs_remove_chunk
btrfs_zoned_activate_one_bg
do_zone_finish

Running balance on the affected kernel may re-enter similar block-group deletion and zone reclamation paths.

How to handle multi-terabyte deletes and writes

The most effective workflow adjustment is to copy and verify new data before deleting old data when there is enough space.

1
2
3
4

先复制新数据
→ 校验复制结果
→ 暂停新写入并观察
→ 分批删除旧数据

copy:

1
2
3

rsync -aHAX --info=progress2 \
  /来源目录/ \
  /mnt/disk1/目标目录/

Compare capacity:

1

du -sh /来源目录 /mnt/disk1/目标目录

Practice using the content verification method without modifying the target:

1
2
3

rsync -aHAXnc \
  /来源目录/ \
  /mnt/disk1/目标目录/

When deletion is necessary first, do not delete several TB at once and then write immediately. It can be divided into batches of 200~500GiB by date, subdirectory or capacity:

1
2

rm -rf /mnt/disk1/旧数据/第一批
sudo btrfs filesystem sync /mnt/disk1

btrfs filesystem sync will commit the file system transaction and wake up related cleanup work, but the command return does not mean that all block groups and zone reclaim have been completed.

If you are deleting a Btrfs subvolume or snapshot, you can also execute:

1

sudo btrfs subvolume sync /mnt/disk1

It waits for the subvolume for which the deletion request has been submitted to complete cleanup and does not apply to normal rm -rf deletions.

Check after each batch:

1
2
3
4
5
6
7
8

sudo btrfs filesystem usage /mnt/disk1
sudo btrfs device stats /mnt/disk1

ps -eo state,pid,comm,wchan:35,args | \
awk 'NR==1 || $1 ~ /^D/'

sudo journalctl -k --since "30 minutes ago" | \
grep -iE 'btrfs.*(blocked|hung|error|warning)|I/O error'

Meet the following conditions before continuing:

• btrfs filesystem sync returns normally;
• No Btrfs threads are in D state;
• There are no new blocked, hung, abort or Btrfs errors in the kernel log;
• Disk activity has dropped significantly;
• Device zone unusable is not growing rapidly.

Device zone unusable does not need to fall to zero. It may remain at a few dozen GiB until the next reclamation threshold is reached; that alone does not mean the file system is still busy or stuck.

Writes can also be batched, with priority and bandwidth limited:

1
2
3
4

sudo ionice -c2 -n7 nice -n 15 \
rsync -aHAX --info=progress2 --bwlimit=100M \
  /来源目录/ \
  /mnt/disk1/目标目录/

--bwlimit=100M is just a conservative starting point. Rate limiting cannot fix kernel bugs, but it can reduce the probability of cleaner, transaction commit, writeback, and zone activation being under high load at the same time. Don’t run multiple large rsyncs concurrently.

Monitor zoned reclamation metrics

First get the file system UUID:

1
2
3
4

FSID=$(sudo btrfs filesystem show /mnt/disk1 | \
       awk '/uuid:/ {print $NF; exit}')

echo "$FSID"

Check the space and recovery indicators of data, metadata and system:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16

for type in data metadata system; do
    echo "===== $type ====="
    base="/sys/fs/btrfs/$FSID/allocations/$type"

    grep -H . \
      "$base/bytes_pinned" \
      "$base/bytes_reserved" \
      "$base/bytes_may_use" \
      "$base/bytes_zone_unusable" \
      "$base/reclaim_count" \
      "$base/reclaim_bytes" \
      "$base/reclaim_errors" \
      "$base/periodic_reclaim" \
      "$base/dynamic_reclaim" \
      "$base/bg_reclaim_threshold" 2>/dev/null
done

in:

• bytes_pinned: Space that usually cannot be released until the current transaction is committed;
• bytes_zone_unusable: zoned space that has expired but cannot be directly reused;
• reclaim_count: cumulative number of reclamation attempts;
• reclaim_bytes: cumulative number of reclaimed bytes;
• reclaim_errors: cumulative number of reclamation errors.

You can also view transaction commit statistics:

1

cat "/sys/fs/btrfs/$FSID/commit_stats"

If max_commit_ms lasts for tens or hundreds of seconds, or the log appears again:

1
2

btrfs-cleaner blocked
btrfs-transaction blocked

You should stop the backup task and save the log, do not continue to pour data and wait for it to recover on its own.

Disable backup tasks from overlapping

All rsnapshot cycles use the same lock:

1
2
3
4

/usr/bin/flock -n /run/lock/rsnapshot.lock \
  /usr/bin/ionice -c2 -n7 \
  /usr/bin/nice -n 15 \
  /usr/bin/rsnapshot daily

weekly and monthly also use:

1

/run/lock/rsnapshot.lock

This can avoid daily, weekly, monthly and a large number of rsync being performed at the same time. Scrubs, mass deletes, and large copies should also be scheduled for different time periods.

Second failure: transaction aborted and forced read-only

After the first failed restart, the file system can be mounted normally and there are no errors in device stats. But when writing a large amount of data again later, a more explicit transaction failure occurred:

1
2
3
4
5
6

BTRFS error (device sda): error while writing out transaction: -11
BTRFS warning (device sda): Skipping commit of aborted transaction.
BTRFS: Transaction aborted (error -11)
BTRFS: error (device sda state A) in cleanup_transaction:2060:
errno=-11 unknown
BTRFS info (device sda state EA): forced readonly

The operating environment is still:

1
2

CPU: 0 PID: 1678 Comm: btrfs-transacti
Not tainted 7.0.0-28-generic #28-Ubuntu

This time it is not “temporarily stuck”, but the transaction has been aborted, and the kernel actively switches to read-only to protect the file system. -11 corresponds to EAGAIN, but the specific code that triggers it cannot be determined based on errno alone.

The device statistics at that time were still all 0:

1
2
3
4
5

write_io_errs    0
read_io_errs     0
flush_io_errs    0
corruption_errs  0
generation_errs  0

The space status is not full:

1
2
3
4
5

Device size:            12.73TiB
Device allocated:        4.95TiB
Device unallocated:      7.78TiB
Device zone unusable:   53.58GiB
Used:                    4.88TiB

This set of evidence favors zoned Btrfs internal write, chunk allocation, or transaction logic anomalies rather than ordinary media I/O errors or disk-wide ENOSPC. However, without a complete analysis of the first error and the corresponding source code, the cause should not be asserted to be a certain patch.

Correct handling after forced readonly

Don’t try:

1
2
3
4

sudo mount -o remount,rw /mnt/disk1
sudo btrfs balance start /mnt/disk1
sudo btrfs filesystem sync /mnt/disk1
sudo btrfs check --repair /dev/sda

Stop the writing task first:

1
2
3
4

sudo pkill -TERM -x rsync
sudo pkill -TERM -x rsnapshot

pgrep -af 'rsync|rsnapshot|btrfs balance|btrfs scrub'

Save complete logs before and after the failure. The first error that actually causes the transaction to abort may appear dozens of lines before forced readonly:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10

sudo journalctl -k -b \
  --since "2026-07-24 17:20:00" \
  --until "2026-07-24 17:35:00" \
  > /root/btrfs-abort-2026-07-24.log

sudo dmesg -T \
  > /root/dmesg-btrfs-abort-2026-07-24.log

sudo smartctl -x /dev/sda \
  > /root/smart-sda-2026-07-24.log

Confirm mount options:

1

findmnt -no TARGET,SOURCE,FSTYPE,OPTIONS /mnt/disk1

ro should appear there. Then uninstall and restart normally:

1
2
3

cd /
sudo umount /mnt/disk1
sudo reboot

If it prompts busy:

1

sudo fuser -vmM /mnt/disk1

Stop the listed normal user processes before uninstalling.

Check for an incomplete copy

Part of the data that has been written before the transaction is aborted cannot be directly regarded as a complete backup. May appear:

• The file has not been created yet;
• The file length is incomplete;
• The data is written, but the directory entry or timestamp is not committed;
• Changes in the last transaction are rolled back.

Therefore the source disk must be retained. After switching to a stable environment and re-reading and writing the mount, use the original rsync to complete:

1
2
3
4

sudo ionice -c2 -n7 nice -n 15 \
rsync -aHAX --info=progress2 \
  /来源目录/ \
  /mnt/disk1/目标目录/

Then use the verification mode to verify:

1
2
3

rsync -aHAXnc \
  /来源目录/ \
  /mnt/disk1/目标目录/

No output means that rsync has not found anything that needs to be updated.

Kernel solution: first distinguish between testing and long-term use

The same machine, the same 7.0.0-28-generic, and the same HC620 have appeared continuously:

1. btrfs-cleaner is locked with the transaction thread;
2. error -11, transaction aborted, forced readonly.

Therefore, we no longer continue to use this kernel to read and write terabytes of data to this disk. Three routes were discussed:

The most important thing here is not to pursue the largest version number, but:

• Keep an old kernel that can be booted;
• Only do a one-time GRUB boot for the first time;
• Mount read-only first;
• Test step by step from 100 to 200GiB;
• “Write 3TB immediately after deleting 3TB” is no longer directly reproduced.

Confirmed on Ubuntu 24.04 6.8 GA

Do not mix Ubuntu 24.04 packages with existing Ubuntu 26.04. A safer way is to install Ubuntu Server 24.04 on another SSD and then mount the HC620 data disk for testing.

Install and keep the 6.8 GA metapackage:

1
2
3

sudo apt update
sudo apt install --install-recommends linux-generic-6.8
sudo reboot

Confirm system and kernel:

1
2
3
4
5

cat /etc/os-release
uname -r

dpkg -l | \
grep -E '^ii +linux-(generic|image-generic|headers-generic)(-6\.8)? '

The key is that uname -r starts with 6.8., and do not hard-code a certain patch version permanently.

Query the current repository instead of copying the historical version of this article:

1
2
3
4

apt-cache policy \
  linux-generic-6.8 \
  linux-generic \
  linux-generic-hwe-24.04

If testing targets GA 6.8, don’t let linux-generic-hwe-24.04 take the system back to another HWE core line.

Short term test on Ubuntu 24.04 6.17

First check whether the current repositories still provide and maintain the package:

1
2
3
4
5

sudo apt update

apt-cache policy \
  linux-generic-6.17 \
  linux-image-generic-6.17

After confirming that the package is still available, install it explicitly:

1
2

sudo apt install --install-recommends linux-generic-6.17
sudo update-grub

Confirmation document:

1

ls -lh /boot/vmlinuz-6.17.0-*-generic

Only start once for the first time. The full version below must be replaced with the version actually installed on your machine:

1
2
3
4
5

sudo grub-reboot \
  'Advanced options for Ubuntu>Ubuntu, with Linux 6.17.0-xx-generic'

sudo grub-editenv - list
sudo reboot

Check after restarting:

1

uname -r

The 24.04 Noble repository should not be added to Ubuntu 26.04 to force-install this kernel, otherwise it may introduce firmware, DKMS, initramfs and subsequent apt maintenance issues.

Boundaries when testing 7.1 Mainline

The Mainline package is suitable for verifying “whether the new upstream kernel changes the fault behavior” and is not equivalent to the official Ubuntu production kernel. Must check before installation:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10

. /etc/os-release
echo "$PRETTY_NAME  codename=$VERSION_CODENAME"

dpkg --print-architecture
df -h /boot /

sudo apt update
sudo apt install -y mokutil
mokutil --sb-state
dkms status

Testing risk increases significantly if Secure Boot is enabled, critical DKMS modules are present, or the server does not have an out-of-band console. The specific version, file name and check value should be re-obtained from the current Ubuntu Mainline Kernel Archive, and expired download addresses should not be copied.

Keep the current bootable kernel and boot the new kernel all at once via GRUB. Check after startup:

1
2
3
4
5
6
7
8

uname -r
systemctl --failed
ip addr
ip route
dkms status

sudo dmesg -T | \
grep -iE 'btrfs|hung task|blocked for more|I/O error|ata|medium error|uncorrect'

First mounting and testing of the new kernel

Whether testing 6.8, 6.17 or updating the kernel, first keep noauto in /etc/fstab and mount it read-only after startup:

1
2
3
4
5
6
7

sudo mount -o ro /dev/sda /mnt/disk1

sudo journalctl -k -b | \
grep -iE 'btrfs|error|warning|abort|readonly|hung|blocked|I/O error'

sudo btrfs device stats /mnt/disk1
sudo btrfs filesystem usage /mnt/disk1

After no new errors:

1
2
3
4

sudo umount /mnt/disk1
sudo mount /dev/sda /mnt/disk1

findmnt -no SOURCE,TARGET,FSTYPE,OPTIONS /mnt/disk1

Step by step test:

1
2
3
4
5
6

写入 100～200GiB
→ btrfs filesystem sync
→ 检查日志和 D 状态
→ 删除 100～200GiB
→ 等待并再次检查
→ 再写入下一批

Just because a small batch passed doesn’t immediately mean it’s “fixed” with terabytes of pressure. At the very least, multiple rounds, large amounts of data, and long run times are required to improve confidence.

Whether to reformat Btrfs

Reformatting can result in a clean file system, but it cannot fix kernel zoned path problems alone. Both exceptions occurred in the block-group, zone and transaction path at runtime, and the device stats had no media errors.

Rebuilding the file system only makes sense when:

• btrfs check --readonly An explicit structural error was found;
• scrub continues to have irreparable verification errors;
• Unable to mount normally or generation errors continue to occur;
• It has been decided to adjust the data/metadata profile;
• Prepare to replace the file system.

Even if you want to rebuild, complete another verifiable backup first rather than treating formatting as a trial-and-error command.

How to choose between Btrfs and F2FS

F2FS can bypass the Btrfs-specific calling path that appears this time:

1
2
3
4
5

btrfs_delete_unused_bgs
btrfs_remove_chunk
btrfs_zone_finish_one_bg
Btrfs chunk allocation
Btrfs transaction commit

But it’s not a “problem-free” solution. F2FS also has risks in GC, segment migration, zone reset, power failure recovery and file system repair. Long foreground GC delays may still occur with large writes immediately following large deletes.

The main trade-offs between the two are:

If the HC620 is only one of multiple backups, F2FS plus external parity is acceptable. Neither using single-disk Btrfs nor F2FS is safe enough if it’s the only copy.

You can save the SHA256 list to another disk when using F2FS:

1
2
3
4
5
6

cd /mnt/disk1

find backup_data -type f -print0 | \
sort -z | \
xargs -0 sha256sum \
  > /其他磁盘/backup_data.sha256

Verify later:

1
2

cd /mnt/disk1
sha256sum -c /其他磁盘/backup_data.sha256

The drive’s own ECC, bad sector remapping, and SATA CRC will still work, but they won’t cover the complete end-to-end link between applications, memory, file systems, controllers, and disks. Btrfs checksum and hard disk ECC are complementary, not duplicate functions.

Practical rules from this incident

This troubleshooting resulted in the following practical rules:

1. No longer use 7.0.0-28-generic to do large-scale reading and writing of this HC620;
2. Retain the source data and at least one independent copy;
3. Copy and verify first, then delete in batches;
4. Divide several TB operations into batches of 200~500GiB;
5. rsync, rsnapshot, scrub, mass deletion and balance do not overlap;
6. Stop writing immediately after the D status or hung-task log appears;
7. Do not force remount,rw after forced readonly;
8. Do not use btrfs check --repair to make blind attempts;
9. The test kernel must retain the rollback items and start with a read-only mount;
10. Long-term reliability comes from multiple copies first and file system selection second.

The entire fault chain can be summarized as:

1
2
3
4
5
6
7

数 TB 删除
→ zoned 无效空间和后台回收增加
→ 数 TB 新写入与 chunk/zone 管理交叠
→ 第一次：btrfs-cleaner 锁等待，事务线程阻塞
→ 重启后暂时恢复，scrub 与设备计数未见错误
→ 再次大量写入
→ 第二次：transaction -11，中止并 forced readonly

This set of results is enough to show that the current “HC620 + Btrfs zoned + 7.0.0-28 + CLOS” combination is not suitable for continuing large-scale unattended tasks, but it is not enough to prove that all HC620s, all Btrfs zoned, or all 7.0 kernels have the same problem.

References

• Btrfs zoned mode
• Btrfs scrub
• Btrfs balance
• Btrfs sysfs interface
• Linux kernel Btrfs documentation
• Ubuntu Mainline Kernel Archive
• Site: Misunderstandings and correct usage of Western Digital HC620 shingled disk