When using Btrfs zoned on HC620, if the file system is close to 100% and then suddenly can only be read but cannot be written, the most common clues are indeed ENOSPC, zone reclaim does not have enough working space, or the kernel forces the switch to read-only after the transaction fails to commit.
However, “full” and “read-only” describe only the symptoms observed at the time of failure. They do not prove that the drive itself is healthy. Use this order instead:
- Stop writing tasks immediately;
- Save the kernel log when the failure occurs;
- Confirm the device, mounting status and zoned space distribution;
- Distinguish ordinary space exhaustion, blocked reclaim, transaction aborts, and lower-level I/O errors;
- Only when the evidence supports pure ENOSPC, try to mount it again in read-write mode and free up space in batches;
- Once space is available again, perform only low-risk cleanup and complete the recovery checks.
Do not run
mount -o remount,rw, an unfiltered balance, orbtrfs check --repairjust because the filesystem is read-only. Forced read-only is a protective response. Find the first error that triggered it.
This article assumes that the mount point is /mnt/hc620. /dev/sdX is only a placeholder; replace it with the actual device or partition before running a command. Never copy the placeholder device path literally.
Why is it more difficult to free up space on HC620 than a regular hard drive after it is full?
The Western Digital Ultrastar DC HC620 is a Host-Managed SMR, that is, a shingled disk where the write area is managed by the host.
Linux usually exposes it as a zoned block device:
|
|
Expect to see something like:
|
|
The model, displayed capacity, and device name may differ. Use the values reported by your own system.
Regular CMR hard drives can overwrite allocated sectors. Host-Managed SMR’s sequential write zone must continue writing from the current write pointer; to reuse a written zone, it is usually necessary to move the still valid data first, and then reset the entire zone.
Btrfs supports zoned mode starting from Linux 5.12. It uses COW. After a file is overwritten, metadata is updated or deleted, the old data block may no longer be referenced by the file system, but it will not immediately become a re-writable empty zone.
These spaces will appear in btrfs filesystem usage as:
|
|
It does not mean “permanently damaged hard disk capacity”, but space that has been written in the past and is no longer referenced, but still needs to go through block group reclaim and zone reset before it can be reused.
The recycling process requires moving the data that is still valid in the zone to a new location. If there are not enough new zones to receive the data, a transient ENOSPC may occur.
So, on the HC620:
|
|
Not necessarily immediately equal to:
|
|
This is also the reason why zoned Btrfs is more difficult to process on site than a normal disk after it is actually written to the limit.
Which tasks should be stopped immediately?
After discovering that the mount point is read-only, do not retry the original copy task repeatedly.
Start by stopping services that may continue to make I/O or log noise, such as:
rsync,rcloneor download tasks;- containers and virtual machines;
- Media library scan;
- Scheduled snapshots and backups;
- scrub, balance or mass delete tasks.
First confirm whether there is balance or scrub:
|
|
Do not automatically cancel the task in order to execute the instructions in this article. Record the status first; if balance or scrub is running and the system is still responding, you should combine the logs to determine whether it is part of the fault site.
Check which processes are still occupying the mount point:
|
|
This command is only for observation and will not automatically terminate the process.
Save the fault log first, then uninstall or restart
The most valuable thing in dmesg is often not the last line forced readonly, but the earliest error before it.
First save the complete kernel log to another normal disk, such as the system disk:
|
|
Then extract the relevant lines:
|
|
If the machine has been restarted, check the last startup:
|
|
If the system does not have a persistent journal, the log from the last startup may not exist. Therefore, the fault site should try to export the log first and then restart.
Also save the base environment:
|
|
Among them, if OPTIONS of findmnt contains ro, it can only confirm that the current mount is read-only, but cannot explain why it became read-only.
Confirm the identity of the device. It is prohibited to fill in /dev/sdX based on your feelings.
First check the source device from the mount point:
|
|
Check the devices in the file system again:
|
|
Confirm stable device link:
|
|
If the file system is built on a partition, subsequent mounts must use the partition instead of the entire disk. For example, if the actual source device is /dev/sda1, it cannot be written as /dev/sda.
When recording actual results as variables, manual verification is also required:
|
|
Do not execute the Chinese placeholders in the example as they are.
Read Btrfs spatial distribution
Execute while the file system is still mounted:
|
|
Key notes:
|
|
A typical high-risk portfolio looks like:
|
|
Don’t just look at df -h. The data, metadata, system block group and device unallocated space of Btrfs are at different levels; COW also needs new space to complete transactions.
Device zone unusable cannot be used as a fault conclusion alone:
- A non-zero value is a normal phenomenon after zoned COW works;
- If the value is very large and
Device unallocatedis close to 0, it means that the reclaimable space and the immediately allocable space are seriously misaligned; - The value may change during background reclaim;
- Even if it is 0, this does not rule out metadata exhaustion or underlying I/O errors.
Check persistent device error count
implement:
|
|
Normal output example:
|
|
The meaning of each item:
write_io_errs: The lower block device failed to complete the write request;read_io_errs: The lower block device failed to complete the read request;flush_io_errs: The write with FLUSH failed, related to the order of transaction placement;corruption_errs: Checksum mismatch or damaged metadata header found;generation_errs: The block’s generation is inconsistent with the parent node’s expectations.
All 0s are a good sign, but not proof that the drive is healthy. Also combine kernel logs, SMART, HBA and link error judgment.
Don’t use it yet:
|
|
-z will clear the count after printing. If the fault evidence is cleared before being saved, important baselines will be lost.
If you really want to observe whether the errors continue to increase, save the current output first, and then record the time:
|
|
Compare again after completing controlled testing rather than just looking at a single snapshot.
Use logs to classify faults into four categories
Category 1: Pure space exhaustion or temporary ENOSPC
Further evidence supporting this category includes:
|
|
Satisfy both:
btrfs device statshas no non-zero errors;- There is no
I/O errorin the log; - No checksum, tree-checker, corrupt leaf or parent transid errors;
- The file system is indeed close to full;
Device unallocatedis rare, or zoned recycling is significantly limited.
This situation is suitable for entering the later “controlled read-write mount and free up space” process.
Category 2: Zone or block group reclaim is blocked
The log may appear:
|
|
The problem at this point is not necessarily just a capacity number zeroing out, but may also involve recycling paths, long lock waits, or zoned active zone limits under a specific kernel.
Do not add a full balance at this time. First keep the call stack, kernel version and complete logs, and then refer to the actual fault review on the site:
[Review of two Btrfs zoned failures on HC620: lockup, read-only and kernel troubleshooting] (/en/2026/07/24/hc620-btrfs-zoned-deadlock-readonly-troubleshooting/)
Category 3: Transaction aborted, but the reason has not yet been determined
The following lines only describe the results:
|
|
The real reason is usually up front. You need to look up dozens to hundreds of lines to find the first error, return code and call stack.
All transaction aborted cannot be classified as full. Low-level write failures, metadata validation failures, and kernel bugs can also abort transactions.
Category 4: I/O, checksum, or structure errors
If any of the following clues appear, you should stop the process of “delete a few files and it will be fixed”:
|
|
At this time, priority is given to protecting data copies and checking disks, SAS/SATA HBAs, cables, power supplies, and kernel zoned errors. Don’t simply interpret a non-zero device count as “just a full disk”.
Do read-only mounting and data rescue first
If the current mount can still be read, copy the most important data that has no other copies first.
If you need to uninstall and then remount, exit the shell occupying the directory first, and then execute:
|
|
Confirm that it has been uninstalled:
|
|
No output means that the mount point is not in the current mount table.
First mount in read-only mode:
|
|
Then copy important files to another healthy disk. Do not put the target directory back into the HC620 itself during a read-only rescue.
If ordinary read-only mounting fails, do not randomly superimpose the rescue= parameter. Different errors require different rescue options. Error parameters may cover up the log or change the reading results. The decision should be made based on the specific error report.
When is it allowed to try to read and write the mount again?
A controlled attempt may be considered when the following conditions are met simultaneously:
- Fault log saved;
- Important data has other copies, or read-only rescue has been completed first;
- The first reason in the log clearly points to
ENOSPC; - No I/O, checksum, tree-checker or zone write pointer errors;
btrfs device statshas no exception count;- No stuck old kernel tasks;
- The file system can be cleanly unmounted.
Uninstall the read-only mount first:
|
|
Perform one fresh mount attempt instead of using remount,rw to force the current mount back to read-write:
|
|
Verify now:
|
|
If the mount option still contains ro, or the transaction abort appears again in the log, stop trying and do not cycle unmount and mount.
If rw is displayed successfully, do not restore replication service immediately. The first task of the read-write window is to release known deletable data in batches.
How to safely free up space after read and write recovery
Give priority to directories that have been confirmed to have other copies, can be rebuilt, and are large in size.
First check the size of the first-level directory:
|
|
Don’t copy directly:
|
|
The real target should be printed first to confirm that there are no variable expansion errors:
|
|
Make sure the target is correct before deleting:
|
|
For multi-terabyte data, it is recommended to delete in separate subdirectories or batches and not to perform large-scale writes at the same time.
Check after each batch:
|
|
btrfs filesystem sync It may take a while. Before the command returns, do not mistake “directory disappeared” as space recovery has been completed.
Aim to free up hundreds of GiB first, rather than just deleting a few GiB and then filling it up immediately. The specific safety margin depends on the workload, zone size, metadata and recycling efficiency, and cannot be given an absolute ratio for all HC620.
In long-term use, it is not recommended to maintain zoned Btrfs at 99% ~ 100%. To reserve both:
- Daily new writing space;
- COW transaction space;
- Metadata growth space;
- workspace for zone reclaim to relocate still-valid data;
- Emergency deletion and maintenance windows.
Why the space may not come back immediately after deletion
Deleting files primarily dereferences and updates metadata. For zoned devices, there may be valid extents in the relevant zones, and Btrfs must move them away before resetting the zone.
So you might see:
- The file has been deleted;
Usedhas declined;Device zone unusabletemporarily rises;Device unallocatedis not increased synchronously according to the deletion amount;- The background cleaner continues to generate I/O.
This does not necessarily mean that the deletion failed. Observe a complete controlled recycling cycle and judge based on the trend of logs and btrfs filesystem usage -T.
If the value does not change for a long time, and cleaner blocking, transaction errors, or D-state tasks occur at the same time, it should be handled according to the recovery path failure instead of continuing to delete a large number of data.
balance can only be executed incrementally after space is restored
balance will move the block group. Most balance operations need to first create a new block group as a workspace, so running balance without a filter directly when the disk is full may trigger ENOSPC again.
Don’t execute it right away:
|
|
First confirm that there is currently no balance:
|
|
On the premise that the log is stable, the space has been released, and the device error count has not increased, you can first process the completely unused block group:
|
|
usage=0 does not require additional workspace and is the low-risk starting point for ENOSPC given by the official Btrfs documentation.
Check after completion:
|
|
The low threshold is only considered when both space and logs are stable:
|
|
After observing again, it is possible to improve to:
|
|
These numbers are not a fixed prescription that must be implemented step by step. The higher the threshold, the more data may be moved and the more work space and I/O required.
If btrfs-cleaner, block group deletion or zone finish path has been stuck before, do not regard balance as automatic repair. It may re-enter similar code paths, and the kernel issue should be investigated first.
Do not perform operations on a faulty disk
Don’t force remount to be read-write
avoid:
|
|
If the kernel has just forced the file system to be read-only due to an error, direct remount will skip the “save log, unmount, and re-evaluate” boundary, and usually cannot eliminate the root cause of the read-only file system.
Don’t blindly execute a full balance
avoid:
|
|
A full disk balance needs to move data, which may require more temporary space than the original failure.
Do not run btrfs check --repair
avoid:
|
|
The official Btrfs documentation clearly warns: Do not use --repair unless advised of a specific error by the developer or an experienced person.
If you really need offline structure checking, you must uninstall it first and use read-only mode by default:
|
|
But on a 14TB or 15TB file system, this check can take a long time, consume a lot of memory, and generate continuous reads. This is generally not required as a first step when there is only pure ENOSPC evidence.
Do not clear device error statistics first
Avoid executing before saving evidence:
|
|
Zeroing will not repair the drive, cables, or HBA, it will simply erase the comparison baseline.
Don’t continue to delete and write at the same time
The overlap of multi-terabyte deletes, zone reclaims, and multi-terabyte new writes on a Host-Managed SMR can significantly expand the failure surface.
A more reliable process is:
|
|
Validation after space recovery
Restoring rw is just the beginning and cannot be used as the final success criterion.
Verify mount status
|
|
confirm:
- The source device is correct;
- The file system is
btrfs; - Mount options include
rw; - There are no wrong subvolumes or partitions mounted.
Verify spatial distribution
|
|
Don’t just go for Device zone unusable zero. What’s more:
- There is already enough available space;
Device unallocatedor allocable space restored;- Metadata is no longer pushed to its limits;
- There are no new errors in the recycling process.
Validation error count not growing
|
|
Compare to baseline saved after failure. Stop resuming large writes if write_io_errs, flush_io_errs, corruption_errs, or generation_errs grows.
Verify that the kernel is not forcing read-only again
|
|
Pay attention to the newly generated logs, and do not misjudge the old error a few hours ago as a recurrence.
Do small-scale write tests
First write to a clear test directory:
|
|
Delete the test files and submit again:
|
|
Then check the logs, space and device statistics again.
Passing the 1GiB test can only prove that small-scale writes are successful, but cannot prove that continuous writes of several TB are necessarily stable. Formal tasks should be resumed in small batches and monitored.
Schedule scrub after having redundancy or backup
Scrub reads all allocated data and verifies checksums, which can take a long time. Do not run balance, copy, and delete immediately on a newly restored full disk at the same time.
After confirming that the file system is stable and there are copies of important data, schedule an independent maintenance window:
|
|
Check it out when finished:
|
|
The single disk single data does not have another mirror for Btrfs to automatically repair. Just because scrub can find verification problems does not necessarily mean they can be fixed.
Ways to avoid filling up again in the long term
Set capacity stop line for write tasks
Don’t let the backup script just look at the command exit code. Record both before and after writing:
|
|
When the available space falls below your maintenance threshold, stop receiving new data and alert you.
The threshold should not only be calculated based on “how much space can fit in the next file”, but also include space for COW, metadata, and zone reclaim.
Stagger large deletes and large writes
recommend:
|
|
Do not schedule rsync, snapshot removal, scrub, and balance to compete for I/O during the same maintenance window.
Maintain an external inventory of archived data
You can generate a check list for important directories and copy the list to another disk:
|
|
The above text manifest format may still be inconvenient to handle when file names contain newlines; if the data source produces special file names, a dedicated manifest process that supports NUL delimitation should be used.
Do not treat a single disk file system as a backup
Btrfs checksums can detect silent corruption, but single-disk data profiles usually don’t have a second copy of the data to recover from.
The data on the HC620 should also include at least:
- Another local disk copy;
- Another machine or off-site copy;
- Periodically verifiable object storage or offline replicas.
FAQ
Device zone unusable is very large. Does it mean the disk is broken?
uncertain. It represents space that has been written in the past and is no longer referenced due to COW, but has not yet been reclaimed and zone reset.
You should also look at Device unallocated, data and metadata usage, kernel logs, recycling trends, and device error counts.
Can the same capacity be restored immediately after deleting a file?
Not guaranteed. Deletion first changes the file system reference. The underlying zone may also need to move valid data and reset.
If the normal remount is successful, will it continue to be full?
cannot. You should first make significant margin, confirm that logs and error counts are stable, then do small-scale testing, and finally resume business in batches.
write_io_errs is 0, is the hard disk problem ruled out?
It cannot be completely ruled out. It only shows that there are no write failures for this type of persistence count recorded by Btrfs; also check the kernel block layer logs, SMART, HBA, wires and power supply.
Can I use mount -o recovery?
Don’t think of recovery from the old tutorial as a universal repair switch. The current Btrfs rescue mount option has clear applicable errors, and usually requires a read-only mount, which should be selected based on the actual log.
Why does df report ENOSPC when it shows there is still space?
Btrfs’ COW, data and metadata block group, transaction reservation and zoned reclamation all require space. The file-level available estimates presented by df do not fully express these internal constraints.
Should I change the metadata from DUP to single?
Don’t make temporary profile changes just to squeeze out space. Profile conversion itself requires moving block groups, which requires work space and a lot of I/O; profiles supported by zoned mode are also subject to kernel and tool versions.
Why is usage=0 balance relatively safe?
Because it only scans and reclaims completely unused block groups, the official documentation states that this filter requires no additional workspace.
“Relatively safe” still does not mean suitable for all failures. If the log shows that the recycling path is stuck or has a structural error, stop and analyze it first.
Is it possible to directly cut off the power and restart it?
If the system is still responding, you should save the log, stop the task, wait for necessary transactions, and uninstall normally. Forced power outages increase the uncertainty of uncommitted writes and failure analysis.
If the kernel task is in the uninterruptible D state for a long time and the machine cannot shut down normally, you should save as many logs as possible through another machine or remote console before assessing the risk of restarting.
An actionable fault list
Execute in order, do not jump to balance or repair:
|
|
What output needs to be collected to continue judging?
If you still can’t confirm which category you belong to, hide sensitive information such as serial numbers when collecting the following output, but don’t remove the error context:
|
|
Based on these outputs, the next step can usually be converged to:
- Simply fill it up and you can make space under control;
- zone reclaim has no workspace, or the reclaim path is blocked;
- The Btrfs transaction is aborted and the first error needs to be traced;
- Real I/O failure on HC620, HBA, cable, or power supply;
- There is a verification or structure error in the file system, and the data needs to be rescued first and then handled by experienced personnel.
References
- Western Digital Ultrastar DC HC620 Data Sheet
- Btrfs official document: Zoned mode
- Btrfs official document: btrfs-filesystem
- Btrfs official document: btrfs-balance
- Btrfs official document: btrfs-device
- Btrfs official document: btrfs-check
When an HC620 becomes read-only after filling up, the priority is not to make it display rw again as quickly as possible. First determine why the kernel switched it to protective read-only mode. Only when the evidence clearly supports ordinary ENOSPC should you preserve the logs and data copies, then free space in a controlled manner. If I/O, checksum, structural, or zone write-pointer errors appear, stop writing and shift the effort to data recovery and hardware-path diagnostics.