[Solved] "System" is using lots of Disk Space on Mojave and High Sierra
Like @johnfbraun mentioned in a recent episode, I've now started seeing "System" using over 100GB on my APFS-formatted High Sierra and Mojave systems. I've been running all kinds of utilities to find the answer, to little/no avail.
This morning, with DaisyDisk, I realized that my Drive Genius logs were taking up ~20GB. Those were mostly in ~/Library/Logs/com_prosofteng_logs/DriveGenius/dp. I don't launch Drive Genius often these days, so I assume these are from its Drive Pulse app (which I'm not keen on anyway). I opened that folder and saw a few very large .tlg files in there. Before just manually deleting them, I launched Drive Genius... and watched it wipe them away.
Perhaps Drive Genius is built to "process" these, but only after launch?
Anyway, that's one small dent in my System Storage usage (I still have 152GB used there). What have you found that helps?
Yes. you do "need" a larger SSD. Just because you are a power user/Dave The Nerd 🤓 and can work around these issues, I just know that having dealt with these things in the past you are making a lot more work for yourself in the future by using working around the issue versus just getting more internal storage for your files.
Yes. you can manually direct iTunes, Photos, Mail and other apps to use other drives for their storage but soon you will come up against a wall that can not be worked around easily or at all as not all applications are well built and many I have had to deal with on both macOS and Windows assume your main partition is where your files live. Now I know we are taking macOS here but still the same lazy devs exist that make those assumptions and will continue to do so.
Plus, I assume the time you are spending now and have spent in the past working around the limitations has a cost that is more than the cost of the storage. For my clients I have been putting in 1TB SSD's to replace their 1TB HDD's even if they are not yet using the storage as I know that I can always take those drives out in the future when the system I upgraded gets retired and either put the drive in an external enclosure or put it in to a newer system later on.
I am glad you got a clear picture of what is going on with your storage as it is important that no matter what you ultimately decide to do now and the future you need to know if your space issues are because of extra files you want to keep around or cruft that has been generated over the years either by the system or because of extra software was installed that you do not need or can do without.
Good to know that DD is the better option as long as you give it the correct permissions.
Yes. While all of that is true. It is also true for people who have laptops like MacBook Air/Pro computers, they will (in many cases) need to still think about purchasing as much internal local storage as they need or better yet, just use the built in optimize storage options with macOS and iCloud to not worry about it as using external storage for your main files with a laptop is normally a non starter as it restricts the portability of the device as every time you want to take it anyplace, you have to bring your drive(s) with you. That can become a pain to deal with.
I would run though what this blog post says and see if it helps:
Thanks, @brianmonroe. I had done quite a few of those things, but not all.
I really think DaisyDisk is better for this than OmniDiskSweeper, as much as I'm surprised to see myself type that. When doing "Scan as Administrator" DaisyDisk actually showed me numbers that match what Apple shows: nothing is hidden or missing. I'm a little sad about that because I thought maybe there was 100GB just waiting to be deleted but, alas, no... I'm really filling up my drive.
Things like iCloud Photo Library's cache at 25GB (which shouldn't cache since I store my photo library on an external drive!), Chrome's "File System" at 15GB, and my 50GB Downloads folder (oops!) are taking up space, for sure.
Glad to heat that the mystery has been solved. It sounds like you need a much larger internal/boot SSD and to change your default downloads folder for you various browsers you use.along with your Mail Downloads Folder:
As I have found many people forget that folder exists and it does get cluttered up with all sorts of junk that normally can be deleted.
@brianmonroe — I wish it were just my Mail Downloads folder. That one is pretty small... it's my ~30-year email archive that eats most of that space! Maybe it's time to consider MailSteward, but I really am concerned about long-term access when moving my mail out of my "Mail", if that makes sense.
Interestingly, here are two screenshots, one from DaisyDisk and the other from OmniDiskSweeper. Same drive, completely different results. DD shows 348GB for "Users" while OmniDiskSweeper shows 247GB. And it's not a different user account, digging in on both shows my user account as 347.1GB and 246.8GB, respectively. Insane.
Both were run in "Administrator" mode so should "see" everything and yet... the proof is in the pictures.
In looking a little deeper at ODS vs. DD (albeit still quickly), the issue seems to be a discrepancy between them in the size of my ~/Library folder. And, looking in there, seems ODS doesn't see my ~/Library/Mail folder at all. I have both apps added to Mojave's "Full Disk Access" list but obviously something's missing.
My guess is that they're using different engines/methods of determining usage, perhaps both with du albeit different permissions. Regardless, seems like even running ODS in sudo mode and with Full Disk Access enabled still doesn't help it keep from missing a lot of stuff.
Honestly, my guess is that du would also fail at this. So I did some testing.
du -sh ~/Library/Mail results in "du: Mail: Operation not permitted", so I tried:
sudo du -sh Mail and entered my password.
Same rejection, so I added Terminal to the Full Disk Access list and then it worked. Seems whatever ODS is doing isn't actually taking advantage of Full Disk Access, even though the app is there. ¯\_(ツ)_/¯ Actually, maybe now that I've added Terminal to the FDA list, maybe ODS will work properly. Running a test. I'll report back.
In watching my Terminal session that controls ODS, seems that lsbom is another tool it uses. Haven't seen du show up, but it might be happening too quickly to see.
That's exactly what I was seeing as well when I ran the du command on my user/home directory and I am also in agreement that there is a good chance that the developers of DD and ODS have decided to not present error messages when they do not have permissions to access files/folders as I am sure that they did not want to get lots of tech support calls dealing with users who would not understand.
I am sure that both DD and ODS are using different methods as your tests have proven or they would both line up as they are both looking at the same source for their data.
I am curious if the free DiskInventory X would give you a more accurate number assuming you gave it full disk access?
While this is fun to get sorted out, I still think the smart move at this point is to pickup a larger SSD to swap out your internal drive with as 500GB is just not enough for power users like you. Depending on how long you are planning on keeping your system you could look at a 1TB or a 2TB SSD as they both have come down in price a lot.
@brianmonroe – You highlight an important difference here. What works for "me" is rarely the same thing I would recommend for my customers (or yours) who are paying for advice and troubleshooting. If someone's paying $150-$300/hour for troubleshooting, none of the above is worthwhile time spent. You are absolutely right in recommending 1TB SSDs for them 100% of the time.
@datafornothinandbitsforfree and now you've highlighted the main reason we moved everything to these forums instead of our previous Facebook group: the future "findability" of this content. If it turns into a TMO article, awesome. If it doesn't, it's still here... and still findable. 🙂
Yes. They do have to deal with what they have. While it may not be ideal to have to use iCloud storage optimization, it may be the only option until they can purchase a new system.
Yes. @davehamilton is providing valuable information by getting to the bottom of this storage issue as many people are going to run in to this problem for the near term on macOS just like we did on iOS when Apple was shipping 8GB/16GB iPhone's. The same is happening now with these macOS systems that come with smaller SSD's built in and in many cases that can not be upgraded/replaced so because of the cost people are just not buying enough storage for their near term future needs.
Yes. You, me, @davehamilton are all exceptions to the rule as we have been using computers and technology for decades so we can live with the limitations and workarounds and understand the tradeoffs. The average user does not.
Yes. This research should become an article for the TMO website so others can easily find it and not have to search in the forums.
I am glad you agree with me that time is money and it is better spent on not wasting it on trying to hack around storage issues that could easily be solved by upgrading for not that much cost. The thing is that while there are times that I may want to hack around to find a solution for myself, more often than not, I would just rather buy the storage I need and move on. Life is too short to mess around with trying to force a square peg in to a round hole.
I agree. Apple does need to do a much better job with their storage reporting tools in macOS as users should not need to rely on 3rd party tools to figure these things out as they are baked in to the UNIX system so Apple should be able to get a better GUI interface on that information including what is going on with your iCloud and Mail storage usage as well.
Yes., if Apple does not offer a 1st party solution then it will be up to the 3rd party developers like Eletric Light to come out with something better.