compact sparsebundle

Now, I want to know what the difference is between the 100GB and 200GB sparsebundles.

So, the first thing to understand here is what's going on behind the scenes. By design, Disk Images and the file system are largely "invisible" to each other. APFS thinks it's running on a standard mass storage device, and the Disk image system has no idea what file system is running on it. With that context, there are two other critical details:

1. On the DiskImage side, what the "sparsebundle" format actually does is track what block the file system has written to, only allocate storage when the file system actually asks for it, and using its own mapping table to track what blocks have been written to.

2. On the vast majority of file systems (including APFS), deleting a file doesn't actually modify ANY of that file’s data blocks. All delete actually does is destroy the catalog records that told the file system "what the file was" and mark data blocks the file used as "available" for reuse. Those blocks will eventually be overwritten, but they aren't actually modified.

With that context, the real mystery here is actually this:

I noticed that the 100GB test1.sparsebundle automatically reclaimed the space,

and how this works at all:

hdiutil compact test2.sparsebundle

There’s an answer to that in the man page as well as a hint to why it didn't work:

 compact image [options]
			scans the bands of a sparse (SPARSE or SPARSEBUNDLE) disk
			image containing an APFS or HFS+ filesystem, removing those
			parts of the image which are no longer being used by the
			filesystem.  Depending on the location of files in the hosted
			filesystem, compact may or may not shrink the image.  For
			SPARSEBUNDLE images, completely unused band files are simply
			removed.

What's actually going here is that hdiutil is asking the file system to return a structure that describes its unused data runs. That also explains why it would fail— the file system isn't maintaining a block-by-block accounting of every individual data block, it's using its own structure to track usage. hdiutil can only reclaim that APFS the blocks APFS SAYS are free, which isn't necessarily the same as what's actually being used/not used.

and how I can reclaim the space in the 200GB test2.sparsebundle.

In terms of the volume you're actively using, I'm not sure you can. I suspect the space will free up automatically under normal usage, but I don't think there is any way to force it to clear without any other activity.

__
Kevin Elliott
DTS Engineer, CoreOS/Hardware

@DTS Engineer

Thank you for your answer.

I roughly understand the underlying mechanism now. Essentially, the reason the space cannot be reclaimed is that APFS doesn’t actually delete the content but merely marks those blocks as unused through its own data structures. As a result, hdiutil doesn’t realize those blocks are no longer in use.

This reminded me of the TRIM functionality in SSDs, but unfortunately, I found that the mounted SPARSEBUNDLE isn’t recognized as an SSD.

However, I did find a workaround: using hdiutil resize to shrink the image’s capacity. For example, if the actual usage is 20GB, I resized the capacity to 100GB and mounted it. At this point, APFS seems to start actually deleting the data because it detects that the container’s free space has reached a certain threshold. This might also explain why 100GB could be automatically reclaimed while 200GB couldn’t. I wonder if my guess is correct.

If it is correct, then the key to reclaiming space is to make APFS truly delete the data it has marked as unused. Are there any tools or methods available to achieve this?

I conducted more tests and made the following discoveries:

1. The disk does shrink, but it always retains around 20GB of occupied space. For example, I created a 1TB sparsebundle, filled it with 200GB of data, then deleted 100GB of it—the actual occupied space shrank to around 120GB. After deleting another 50GB, the disk shrank further to about 70GB. Finally, when I deleted everything, it shrank to around 20GB. Then, I used the resize command to adjust the disk size to 100GB, and it almost fully reclaimed the space, leaving only about 200MB occupied.

2. The space reduction always happens some time after remounting. Moreover, the shrinking only occurs when the sparsebundle is opened by double-clicking in Finder. If I mount it using the hdidutil attach command, no shrinking happens.

So, what’s the difference between opening a sparsebundle in Finder and mounting it with the hdidutil attach command? Does Finder perform some additional operations?

I hit the character limit, so you get a two parter...

Moving this to the top, as I think it's the most critical point to address.

Are there any tools or methods available to achieve this?

I think my question here is "what are you actually trying to do". If your goal here is to create fixed media of minimal size, then that's a very different conversation and it's possible that the better option might even by to use HFS+ (particularly for readonly images).

However, is this is a dynamic image and you're just trying to force a one time "shrink", then resizing the volume is a pretty reasonable option. There may be other tools/tricks that will trigger this, but there isn't any specific API for this.

I roughly understand the underlying mechanism now. Essentially, the reason the space cannot be reclaimed is that APFS doesn’t actually delete the content but merely marks those blocks as unused through its own data structures.

Yes, though this basic behavior is true of all* file systems, not just APFS.

*I'm always hesitant to say "all", as the world is a vast and complicated place, but the behavior is certainly common across very "major" file system and I'm not aware of any "oddball" exceptions.

As a result, hdiutil doesn’t realize those blocks are no longer in use.

So, the term "use" here is tricky. A file system is basically a collection of large data structures, it's of which is independently managing it's own storage usage. For performance and management reasons, that storage is managed in much large units than the device block size and each of those structures has it's mechanisms for tracking the storage it's actually using. There generally isn't THAT much storage "lost" to that, however:

This reminded me of the TRIM functionality in SSDs, but unfortunately, I found that the mounted SPARSEBUNDLE isn’t recognized as an SSD.

Something sort of analogous to TRIM actually does happen in APFS, because APFS actually has two implementation layers, not one. The volume layer handles the mapping of "data" (files/directories/etc) into the file systems logical structure while the container handles that process mapping that logical structures to physical media. The process largely works the way you'd expect- the volume asks the container for storage when it needs and returns that storage when it's "done". However, the volume may choose to "hold" storage it isn't immediately using as a straightforward optimization.

However, I did find a workaround: using hdiutil resize to shrink the image’s capacity. For example, if the actual usage is 20GB, I resized the capacity to 100GB and mounted it.

Yes, that would work at least some of the time (I don't think it would work under all circumstances) and I should have mentioned it yesterday.

__
Kevin Elliott
DTS Engineer, CoreOS/Hardware

Part 2....

At this point, APFS seems to start actually deleting the data because it detects that the container’s free space has reached a certain threshold. This might also explain why 100GB could be automatically reclaimed while 200GB couldn’t. I wonder if my guess is correct.

Sort of. What's actually going on here is that the container can also "ask" the volume to "give back" storage it isn't using. That process isn't an entirely passive process, as returning data to the container also means that the volumes own data structures need to be modified.

As an aside here, when you run "hdiutil compact":

1. Is the volume mounted?

2. Is the apfs volume encrypted?

Both of those issue could constrain hdiutil is actually doing.

If it is correct, then the key to reclaiming space is to make APFS truly delete the data it has marked as unused.

Again, I don't like the term "delete" in these conversation, as it implies that the actual data on disk is being overwritten, when that's not actually happening.

That leads to here:

The disk does shrink, but it always retains around 20GB of occupied space. For example, I created a 1TB sparsebundle, filled it with 200GB of data, then deleted 100GB of it—the actual occupied space shrank to around 120GB. After deleting another 50GB, the disk shrank further to about 70GB. Finally, when I deleted everything, it shrank to around 20GB.

One thing to understand here is that in the context of physical media the behavior above is basically "right". A 1 TB drive with 200GB of data on it is 4/5ths empty and returning storage to the container that the container doesn't need doesn't provide any real benefit. The volume can return it if/when the container "needs" it back. This is also why there isn't any general "shrink APFS" command/tool. The ONLY contexts doing this makes any sense are when resizing containers/partitions and when manipulating disk images.

Then, I used the resize command to adjust the disk size to 100GB, and it almost fully reclaimed the space, leaving only about 200MB occupied.

...which is exactly what happens when a resize occurs.

The space reduction always happens some time after remounting. Moreover, the shrinking only occurs when the sparsebundle is opened by double-clicking in Finder. If I mount it using the hdidutil attach command, no shrinking happens.

Conceptually, "attach" and "mount" are best understood as entirely separate operations, though the hdiutil command muddles them together a bit. What they actually (should) mean are:

• "attach"-> Build out the physical device tree in IOKit for a particular device. The final result of this is the /dev nodes the provide block level access to the media.

• "mount"-> Read those /dev nodes to find a corresponding file system driver, then use that driver to create a new directory hierarchy within the existing logical hierarchy.

This division is actually how ALL block storage devices are handled (not just disk images). What actually happens when you plug in a new device is:

1. IOKit detects that a device has been attached and a long chain of drivers are loaded. The "top" of that driver chain reads the partition map of the device and creates a series of IOMediaBSDClients, each of which create a corresponding /dev/disk

2. diskarbitrationd monitors IOKit for new clients and starts the process of mounting them.

Note that step #2 here is basically just the "default" behavior. It's possible to stop diskarbitrationd from one and if you do so what will happen is... volume will stop showing up one the desktop when you plug devices in. Similarly, DiskArbitration is the public API for this and one of it's primary functions is allowing apps to block device from mounting.

With that background context:

Does Finder perform some additional operations?

No. As in many other cases, people tend to think of the Finder as "doing stuff" because it's the front end user interface that you actually "see". However, the reality is that it's role is generally much more passive than that, basically acting as a "viewer" for what other parts of the system are "doing". In the case of volumes, what it's actually doing is monitoring for volume mounts (using DiskArb) and then displaying them based on what it was "told" to do.

So, what’s the difference between opening a sparsebundle in Finder and mounting it with the hdidutil attach command?

I'm not sure sure exactly what attach command you ran, but the answer is that the standard "attach" operation above doesn't actually interpret the volume or perform I/O. The partition map of the device is read by IOKit and part of the APFS implementation (the APFS container architecture is largely implemented in IOKit and happens at "attach"), but there is minimal if ANY writing occurring until the volume actually mounts.

Related to that, on this point:

The space reduction always happens some time after remounting

This is a complicated side effect of the evolution of file system design. The model in most peoples head is that you "do something" and "the file system changes", but file system's have been moving away from that for a long time. Simplifying greatly, the more "modern" architecture is the the file system records what it's "going" to do and then "completes" that work over time. That also mean that it can (and does) defer "maintenance work", with "immediately after mount" being an obvious point where it avoids doing anything it doesn't "have" to do.

__
Kevin Elliott
DTS Engineer, CoreOS/Hardware

Thank you for your explanation.

Actually, I am using sparsebundles as an encrypted file storage area, so I need it to be readable and writable, which is why I can't use a dmg. The problem I'm encountering now is that when files are deleted from the sparsebundles, the corresponding space isn't immediately freed up. I'm currently looking for a method to reclaim the extra occupied space (around 20GB) after deleting files from the sparsebundles.

Thank you for your explanation.

Actually, I am using sparsebundles as an encrypted file storage area, so I need it to be readable and writable, which is why I can't use a dmg.

OK, so encryption was actually the critical point here:

As an aside here, when you run "hdiutil compact": Is the volume mounted? Is the APFS volume encrypted?

The issue here is that an encrypted APFS volume is a "black box" until the volume has been unlocked (which generally corresponds to mounting). The container “knows" what blocks have been granted to the volume but that's ALL it knows, making it impossible to reclaim storage.

One question here:

The problem I'm encountering now is that when files are deleted from the sparsebundles, the corresponding space isn't immediately freed up.

Just to clarify the overall behavior, what should be happening here is that when new data is added to the volume, that data should be stored on the blocks the volume already "owns". Making this concrete, if you go back to the situation you are at here:

I noticed that the 100GB test1.sparsebundle automatically reclaimed the space, while the 200GB test2.sparsebundle still retained 5.4GB of usage.

If you copy, say, 3GB into test2.sparsebundle, the overall DiskImage size should not really change, as the data should simply "fill up" that 5.4GB of usage. There may be cases where you want to make the image as small as possible, but most of the time you should just be able to ignore the issue and let it self-correct.

I'm currently looking for a method to reclaim the extra occupied space (around 20GB) after deleting files from the sparsebundles.

I think your best option "within" the volume is to resize the volume down and then back up again.

__
Kevin Elliott
DTS Engineer, CoreOS/Hardware