macOS APFS disk utilization

Channels DVR Server Troubleshooting
#1

There seems to be a problem with macOS APFS drives, in part because of the weird games Apple play with maintaining snapshots on these drives.
For example right now I have a 1TB drive I use for Channels.

  • Channels claims there is only 1.52G available, and refuses to record
  • Finder says there is
  • 193.88GB available, 192.27GB purgeable
  • 998,170,664,960 bytes used (but as I say, 190G of that is basically purgeable)

My analysis is that there is plenty of space available, but Channels is not querying the file system for the 'correct" version of available space, and needs to be updated to handle this, uh, quirk, of APFS. Either query the "real" amount of available space, or, when the amount of available space looks like it's low, force a purge, then re-query how much space is available.

No space left on device
#2

Could you submit diagnostics from the DVR so we can look into this further?

#3

Sure. How do you want me to provide them?

Here's the status right now (after I cleaned as much as I could last night):

Screen Shot 2022-07-07 at 11.41.07 AM
Screen Shot 2022-07-07 at 11.41.07 AM1532×710 92.1 KB

Screen Shot 2022-07-07 at 11.41.48 AM
Screen Shot 2022-07-07 at 11.41.48 AM534×734 274 KB

#4

You can see that I managed to free up around 80G (I think mainly be deleting APFS snapshots), but there's still the larger problem that ChannelsDVR is unaware of purgeable space, and so will refuse to record even though (as I understand APFS) if it just went ahead and asked for the space to record, purgeable space would be freed.

#5

Logs are submitted via the Support menu under the Troubleshooting option.

What exactly was the error message shown when it refused to record?

#6

Screen Shot 2022-07-07 at 11.47.05 AM
Screen Shot 2022-07-07 at 11.47.05 AM1904×758 95.1 KB

#7

The logs are being gathered and submitted right now.

#8

That message comes directly from the operating system. We are trying to write to the disk and macOS errors out.

AFAIK, we don't have any visibility into or control over how purgable space works at the OS layer.

#9

Crikey! That's really depressing when what the OS claims is so DRAMATICALLY at variance with what's actually available :frowning:
OK, thanks. I assumed a cushion of 100G or so (as visible in Finder) was sufficient, but apparently not.

#10

Note that the error is coming from mkdir, which on other filesystems can suggest inode exhaustion. I don't know enough about APFS to know if uses or has max inodes.

#11

If you run df in Terminal.app, you will see two columns %Capacity and %iused

#12

I'm definitely not close to inode exhaustion.
The real problem is that Apple reports like five very different different values of "free space" in five different places, and it's not at all clear which of these is the one to trust for any particular purpose. I assumed the "purgeable" space would automatically free up when required, because common sense!!!, but apparently not.
There's apparently (as far as the internet knows) no understanding of what "purgeable" means, or how to forcibly free it.

But none of this is your fault, you're just caught in the cross hairs of some Apple dumbness. Well, let's see what Ventura brings us. Every year there's a new hope, every year perhaps 15% of the issues are fixed :frowning:

#13

If this disk is backed up with time machine, there are cases where time machine snapshots don’t get purged properly. sorry I don’t have details handy, but “sudo tmutil listlocalsnapshots /System/Volumes/whatever” will give you a clue - iirc that list should be short or empty after time machine completes a backup.

-j

#14

You've run to near fullness on that disk. It is purgeable, but that doesn't mean it will be purged IIRC. I had to wipe a disk with similar issues to get it working again, some kind of corruption prevented easy cleanup. I wound up tossing that one and not reusing it after the wipe because it still had some issues. Maybe yours is warning you of a pending hardware issue as well?

#15

I think this exactly describes what happened! The on-disk snapshots should be purged under various conditions (at the very least when a Time Machine backup is performed, ideally also when space runs so low that otherwise the file system would return an error).

Well, every new OS version we hope the bugs are fixed. And occasionally we even get lucky!

#16

It's certainly possible (and I've had disks behave badly in ways I ascribed to the file system until the hardware error become really obvious) but I don't think so in this case. The drive is a fairly new SSD, and DriveDX/SMART don't believe there's an issue.
I think it's more likely that the snapshot mechanism (and specifically when snapshots should be released) still has a few bugs in it.

#17

If you want to delete all of the time machine snapshots on your "working" disks with terminal commands:-

for s in $(tmutil listlocalsnapshotdates | grep "-"); do sudo tmutil deletelocalsnapshots $s; done

As expected, it does not delete snapshots from your time machine backup disk.

You can also use the Monterey "Disk Utility.app": Select your Channels volume, it should display its snapshots (might take a while); click on one, then [command][A] to select them all; then [control]click and choose Delete... from the menu.

#18

Sure, one can do this.

But it's not ideal in that every time one wants to release a large amount of space ASAP it requires

(a) move/delete the files

(b) delete the snapshots

And one has to remember to perform step (b)...

Like I said this has to be a bug on the part of Apple -- it makes no sense to fail an app now on the CHANCE that data that I deliberately chose to destroy might possibly prove more important than the data I now want to store on the disk!

But this has wandered far from Channels. All we can do is wait to see if/when Apple fixes it.

#19

Yes, we could use a third party app (I wouldn't) or hope that Apple gives us an official tool...