Forum

[Solved] "System" is using lots of Disk Space on Mojave and High Sierra  

   RSS

0

Like @johnfbraun mentioned in a recent episode, I've now started seeing "System" using over 100GB on my APFS-formatted High Sierra and Mojave systems. I've been running all kinds of utilities to find the answer, to little/no avail.

Daves System Storage Usage

This morning, with DaisyDisk, I realized that my Drive Genius logs were taking up ~20GB. Those were mostly in ~/Library/Logs/com_prosofteng_logs/DriveGenius/dp. I don't launch Drive Genius often these days, so I assume these are from its Drive Pulse app (which I'm not keen on anyway). I opened that folder and saw a few very large .tlg files in there. Before just manually deleting them, I launched Drive Genius... and watched it wipe them away.

Perhaps Drive Genius is built to "process" these, but only after launch?

Anyway, that's one small dent in my System Storage usage (I still have 152GB used there). What have you found that helps?

Might want to check mail logging - may have massive useless connection log files.
⁨Library⁩ ▸ ⁨Containers⁩ ▸ ⁨com.apple.mail⁩ ▸ ⁨Data⁩ ▸ ⁨Library⁩ ▸ ⁨Logs⁩

Turn off mail connection logging: https://apple.stackexchange.com/questions/223390/huge-apple-mail-logs-connection-logging-enabled

I just free'd up over 650Gb of space by removing the log files and turning off connection logging. Not sure why it was on in the first place!

Oh wow, @smuskal. This is brilliant. I'm cueing it up to share on the show, too. Great stuff!

Thanks - I frustrated for days on this. After literally sifting through my whole system, I came across the large files, so thought I would share the love. CleanMyMac was great as well, but didn't help me find these files - perhaps because I didn't grant it access to email. Anyway, thanks for passing it along. I imagine several others will benefit.

13 Answers
1

Ok. I finished some testing here.

DaisyDisk is definitely the simplest one to run in terms of seeing everything. It alerts you if it needs Full Disk Access, and can scan all your files with "Scan as Administrator" without having to use the Terminal/sudo trick.

Omni Disk Sweeper will warn you if you need to add it to Full Disk Access, but it won't scan files belonging to other users on its own. You can use the Terminal/sudo trick, but if you do that then ODS doesn't know to warn you because it's running "inside" Terminal, and requires you to add Terminal to Full Disk Access in Mojave.

In the end, with both configured properly, they'll both see everything, though.

1

@davehamilton,

Yes. you do "need" a larger SSD. Just because you are a power user/Dave The Nerd 🤓 and can work around these issues, I just know that having dealt with these things in the past you are making a lot more work for yourself in the future by using working around the issue versus just getting more internal storage for your files.

Yes. you can manually direct iTunes, Photos, Mail and other apps to use other drives for their storage but soon you will come up against a wall that can not be worked around easily or at all as not all applications are well built and many I have had to deal with on both macOS and Windows assume your main partition is where your files live. Now I know we are taking macOS here but still the same lazy devs exist that make those assumptions and will continue to do so.

Plus, I assume the time you are spending now and have spent in the past working around the limitations has a cost that is more than the cost of the storage. For my clients I have been putting in 1TB SSD's to replace their 1TB HDD's even if they are not yet using the storage as I know that I can always take those drives out in the future when the system I upgraded gets retired and either put the drive in an external enclosure or put it in to a newer system later on.

I am glad you got a clear picture of what is going on with your storage as it is important that no matter what you ultimately decide to do now and the future you need to know if your space issues are because of extra files you want to keep around or cruft that has been generated over the years either by the system or because of extra software was installed that you do not need or can do without.

Good to know that DD is the better option as long as you give it the correct permissions.

Yes, but think of all the things Dave is learning and they passing along to us, and other users that are storage restricted, especially current Macbook Air/Pro users that cannot add storage to their boot disk.

1

@datafornothinandbitsforfree,

Yes. While all of that is true. It is also true for people who have laptops like MacBook Air/Pro computers, they will (in many cases) need to still think about purchasing as much internal local storage as they need or better yet, just use the built in optimize storage options with macOS and iCloud to not worry about it as using external storage for your main files with a laptop is normally a non starter as it restricts the portability of the device as every time you want to take it anyplace, you have to bring your drive(s) with you. That can become a pain to deal with.

All well and good if the user does think about this before they buy.  But after the fact they have to deal with what they have.

Optimized iCloud storage may be one after the fact work around.  And in 5 years when it is time for the next Mac, they may think about more up-front storage.

But until then, Dave is doing them a service.

Personally, my Macbook Pro is 1TB in size.  It has been growing over the years.  Chances are the next one will be 1.5 to 2TB in size as I do not want to use iCloud storage optimization and I do want to use use external storage for anything except backups and more backups or for 24/7 services that a traveling laptop cannot provide (such as my personal home web server, and network backup server via Carbon Copy Cloner).

But I'm an exception, in that I've been professionally playing with computers since the early '70's writing software for computer manufactures.  Rather Dave's storage research is helping all the first time Mac owners, and those that have been blissfully ignorant, but suddenly find they are out of disk space.

And it is possible this research will end up being a well written "The Mac Observer" article that Google searches may find for those users that are not MGG listeners.

0

@davehamilton,

I would run though what this blog post says and see if it helps:

How to Clean Your Mac When “System” Storage Takes Too Much Space

0

Thanks, @brianmonroe. I had done quite a few of those things, but not all.

I really think DaisyDisk is better for this than OmniDiskSweeper, as much as I'm surprised to see myself type that. When doing "Scan as Administrator" DaisyDisk actually showed me numbers that match what Apple shows: nothing is hidden or missing. I'm a little sad about that because I thought maybe there was 100GB just waiting to be deleted but, alas, no... I'm really filling up my drive.

Things like iCloud Photo Library's cache at 25GB (which shouldn't cache since I store my photo library on an external drive!), Chrome's "File System" at 15GB, and my 50GB Downloads folder (oops!) are taking up space, for sure.

This post was modified 7 months ago by Dave Hamilton
0

@davehamilton,

Glad to heat that the mystery has been solved. It sounds like you need a much larger internal/boot SSD and to change your default downloads folder for you various browsers you use.along with your Mail Downloads Folder:

https://www.macworld.com/article/2150769/solving-the-mysteries-of-the-mail-downloads-folder.html

As I have found many people forget that folder exists and it does get cluttered up with all sorts of junk that normally can be deleted.

0

@brianmonroe — I wish it were just my Mail Downloads folder. That one is pretty small... it's my ~30-year email archive that eats most of that space! Maybe it's time to consider MailSteward, but I really am concerned about long-term access when moving my mail out of my "Mail", if that makes sense.

Interestingly, here are two screenshots, one from DaisyDisk and the other from OmniDiskSweeper. Same drive, completely different results. DD shows 348GB for "Users" while OmniDiskSweeper shows 247GB. And it's not a different user account, digging in on both shows my user account as 347.1GB and 246.8GB, respectively. Insane.

Both were run in "Administrator" mode so should "see" everything and yet... the proof is in the pictures.

DaisyDisk
OmniDiskSweeper

This post was modified 7 months ago by Dave Hamilton

I'm thinking several ways for a disconnect between macOS, Daisy Disk, and OmniDiskSweeper.

If the are compressed files, the file allocation vs byte length would be different.  Byte length would ignore sparse holes (does APFS support sparse files?), and compression, but allocation would take that into consideration.  If one utility used byte length vs allocation that would account for some differences.

If APFS snapshots are sharing storage, maybe OmniDiskSweeper (which has not really been updated recently) might not.

If there are any file system hardlinks where there are 2 directory entries for the same file, such as they way iPhoto's was duplicated into the Photos library.  If a directory tree is just walked, and the file id is not checked to see if it was already seen, then hardlinked files could be counted twice.

@davehamilton,

So it looks like I was right in the fact that you do need to upgrade your boot drive to something larger. The good news is that storage prices keep dropping for SSD's so that should not be that big of an expense. Yes, you could use an app like MailSteward to archive off your old e-mails but that does add complexity to your configuration that will cause you problems down the line as you upgrade your Mac to newer model with more space on the boot drive.

As for the various applications that you are using to check on disk usage (DaisyDisk, OmniDiskSweeper) it looks like if they are just putting a front end on the UNIX du command that it may all just be an estimate according to the wiki:

https://en.wikipedia.org/wiki/Du_(Unix)

Also, according to the Wikipedia, one could be using the du and the other could be using the df command. Also, they could be reporting files that have been deleted but the blocks have not yet been freed up.

I would be more inclined to believe what the du command has to say than any 3rd party app is reporting as without having knowledge of what API's they are using to come up with their data, it is hard to say exactly what they are reporting on and how they handle errors for data that they do not have permissions to view. Most likely they just skip the files/folders that they do not have permissions to see even if they should have admin privileges as that is not always the same as su on macOS from what I have seen.

0

In looking a little deeper at ODS vs. DD (albeit still quickly), the issue seems to be a discrepancy between them in the size of my ~/Library folder. And, looking in there, seems ODS doesn't see my ~/Library/Mail folder at all. I have both apps added to Mojave's "Full Disk Access" list but obviously something's missing.

My guess is that they're using different engines/methods of determining usage, perhaps both with du albeit different permissions. Regardless, seems like even running ODS in sudo mode and with Full Disk Access enabled still doesn't help it keep from missing a lot of stuff.

Honestly, my guess is that du would also fail at this. So I did some testing.

du -sh ~/Library/Mail results in "du: Mail: Operation not permitted", so I tried:

sudo du -sh Mail and entered my password.

Same rejection, so I added Terminal to the Full Disk Access list and then it worked. Seems whatever ODS is doing isn't actually taking advantage of Full Disk Access, even though the app is there. ¯\_(ツ)_/¯ Actually, maybe now that I've added Terminal to the FDA list, maybe ODS will work properly. Running a test. I'll report back.

In watching my Terminal session that controls ODS, seems that lsbom is another tool it uses. Haven't seen du show up, but it might be happening too quickly to see.

Will you update The Mac Observer

http://www.macobserver.com/tmo/article/how_to_recover_missing_hard_drive_space

article to include Mojave Full Disk Access instructions?

0

@davehamilton,

That's exactly what I was seeing as well when I ran the du command on my user/home directory and I am also in agreement that there is a good chance that the developers of DD and ODS have decided to not present error messages when they do not have permissions to access files/folders as I am sure that they did not want to get lots of tech support calls dealing with users who would not understand.

I am sure that both DD and ODS are using different methods as your tests have proven or they would both line up as they are both looking at the same source for their data.

I am curious if the free DiskInventory X would give you a more accurate number assuming you gave it full disk access?

While this is fun to get sorted out, I still think the smart move at this point is to pickup a larger SSD to swap out your internal drive with as 500GB is just not enough for power users like you. Depending on how long you are planning on keeping your system you could look at a 1TB or a 2TB SSD as they both have come down in price a lot.

Regarding me "needing" a larger SSD, perhaps. But, as a power user, I'm also more than capable of relocating my larger files to "permanently"-attached external drives. I've already done this with my iTunes and Photos libraries, I could do this with my Downloads folder easily enough, and could also do it with Mail (though I won't, because I like having my Mail on my SSD). But still, with that, it's very much possible to manage my storage well-under the 500GB "limit". In the studio that machine still has only 250GB on the boot disk, and happily supports two users, albeit just for business tasks, audio record, and obviously podcasting.

My issue here was that I couldn't get a clear picture fo what was taking up my storage, so I didn't know what to move. If I had larger storage, well, none of this discussion would've been necessary, so... 🙂

0

@brianmonroe – You highlight an important difference here. What works for "me" is rarely the same thing I would recommend for my customers (or yours) who are paying for advice and troubleshooting. If someone's paying $150-$300/hour for troubleshooting, none of the above is worthwhile time spent. You are absolutely right in recommending 1TB SSDs for them 100% of the time.

@datafornothinandbitsforfree and now you've highlighted the main reason we moved everything to these forums instead of our previous Facebook group: the future "findability" of this content. If it turns into a TMO article, awesome. If it doesn't, it's still here... and still findable. 🙂

Based on the number of "Where has my Storage Gone" posts in the http://discussions.apple.com forums, I think a well written The Mac Observer article would be well worth it.

The discussion here has theories and such, but an article would consolidate all this discussion.  And there is most likely more advertising revenue to a TMO article 😀

If some sort of definitive explanation(s) is found then if Apple's reporting of this ghost space usage doesn't quickly improve there will be the opportunity for a smart developer (eg. Eclectic Light) to produce an app that gives a better explanation so us support people can at least "point at" something for our clients!

0

@datafornothinandbitsforfree,

Yes. They do have to deal with what they have. While it may not be ideal to have to use iCloud storage optimization, it may be the only option until they can purchase a new system.

Yes. @davehamilton is providing valuable information by getting to the bottom of this storage issue as many people are going to run in to this problem for the near term on macOS just like we did on iOS when Apple was shipping 8GB/16GB iPhone's. The same is happening now with these macOS systems that come with smaller SSD's built in and in many cases that can not be upgraded/replaced so because of the cost people are just not buying enough storage for their near term future needs.

Yes. You, me, @davehamilton are all exceptions to the rule as we have been using computers and technology for decades so we can live with the limitations and workarounds and understand the tradeoffs. The average user does not.

Yes. This research should become an article for the TMO website so others can easily find it and not have to search in the forums.

0

@davehamilton,

I am glad you agree with me that time is money and it is better spent on not wasting it on trying to hack around storage issues that could easily be solved by upgrading for not that much cost. The thing is that while there are times that I may want to hack around to find a solution for myself, more often than not, I would just rather buy the storage I need and move on. Life is too short to mess around with trying to force a square peg in to a round hole.

0

Graham,

I agree. Apple does need to do a much better job with their storage reporting tools in macOS as users should not need to rely on 3rd party tools to figure these things out as they are baked in to the UNIX system so Apple should be able to get a better GUI interface on that information including what is going on with your iCloud and Mail storage usage as well.

Yes., if Apple does not offer a 1st party solution then it will be up to the 3rd party developers like Eletric Light to come out with something better.

Share: