The Job of the Filesystem

It should do more so you can do less

Originally published at Hacker Noon

Apple just released iOS 10.3, which for the first time puts one of their production platforms on their new filesystem, APFS. I’m pretty happy that they are finally replacing such an aging, obsolete piece of core technology, but I’m nowhere near as excited about it as John Siracusa is. All APFS seems to do is finally meet the basic needs of a modern operating system, and not even all of those. It doesn’t come anywhere near doing what I care about most in the filesystem, which is owning metadata so applications don’t.

As anyone following the discussion of federal surveillance knows, metadata has become at least as important to our lives as our data. It’s not just the phone calls, it’s who you called, when, and for how long. It’s not just the phone, it’s which ones are your favorites and how they relate to each other. This metadata is as important to us personally as it is to the NSA, because it’s what brings simplicity, usefulness, and most importantly accessibility to the reams of information we create and rely on.

The reason I’m disappointed in APFS is that it missed the opportunity to promote metadata into something as important as the data, and as a result I’ve largely lost hope that computers will ever be as useful and open as they can be.

I’ve been trying for months to convert from Apple’s Photos to Adobe’s Lightroom for photo management, and the biggest barriers to doing so are metadata. It’s easy to get all of the photos - just drag the Masters folder into Lightroom. Oh, you organized your photos into albums? Yeah, that’s all gone. You had thousands of photos categorized as favorites so you could easily find the best? Oops, we ignore that.

Google Photos has the exact same problem. I can’t imagine how anyone who cares even a snit about photos could migrate to it, because there’s literally no way to do so that preserves any metadata at all. You get to keep the photos, but not how you organized them, and your only option is to rely on what they think matters to you.

Can you imagine switching email providers but losing all of the information about who the email is from or when it was sent? That’s as idiotic as what’s happening here.

If my filesystem owned this metadata, on the other hand, no conversion would be necessary. Any application could be written to read and write it directly, or I could trivially use any scripting language I wanted to do so. In fact, I could easily switch back and forth between different apps for different purposes, because they would not be able to lock me into their proprietary databases.

This shift is hard to understand, as we’re so used to thinking of applications and their metadata as somehow intertwined. For those in the programming world, there’s a clean analogy that helps to explain the horror of the modern filesystem:

For most of the history of software, applications have been written to directly access databases. They needed to understand the structure of tables and rows, and they had to write and manage their own queries for interacting with that data. Because of the complexity of duplicating this ability, applications tended to own their data entirely, and no other apps could get access.

We’ve recently instead seen a trend towards building very simple data services on top of databases, so that no app directly accesses them. This data service provides an API that understands how all of the data works, and also provides all of the key operations that can be performed on the data. This way, any application can access the data - for both reading and writing - without worrying that its use of the data is somehow incompatible with another consumer. You get the added benefit that any optimizations in the data service help every consumer, not just the one or two apps you remember to change.

Those data services are exactly what have enabled modern applications to scale to the level they do. The application developer can focus on their goals for the data, relying on the data service to provide consistency, portable operations, and everything else.

In the land of the operating system, though, we have no such luck. Everyone who talks to the filesystem gets about the thinnest interface imaginable: Get me a blob of data. Yes, there’s basic metadata (who owns it, when it was created, etc.), but none of it is or can be custom to the application, which makes it useless. Can you imagine how unpleasant it would be if we had to shoehorn both the subject of an email and the subject of a photo into the same keyword?

As a result, everyone turns that blob that the filesystem is trusted with into a database, and that’s where they put all of the metadata. Some even put the data there, even if it means huge database files and entirely non-portable systems. Really great app developers then go to the effort of building a full read and write API on top of this database that allows you to manage and port everything in the database. That’s enough work that most people skip it, though.

I would never have seen this missed opportunity in the filesystem if I hadn’t used BeOS. I went to Reed College, where Steve Jobs famously attended briefly (I made the unfortunate mistake of graduating, unlike him). This meant I was a Mac user all through school, which were the bad days of Quadras and MacOS 8 and 9. shudder I switched to BeOS because, well, I could, and it’s what really turned me into an OS junkie. There were many awesome things about it (many of which are now in OS X), but my favorite became how much responsibility the OS took for metadata. Applications became smaller, simpler, and in many cases, you realized your whole concept of an application was a way of organizing metadata, which meant you could rely on the OS entirely, and needed to write no code.

The greatest irony is that the author of Be’s awesome filesystem, Dominic Giampaolo, is now a lead engineer on APFS. Through the work of him and the rest of the team at Be, I came away with a clear idea of the value of my metadata, and especially the benefit of the operating system putting it on center stage.

So while I’m happy that my phone is no longer running the oldest, crappiest filesystem around, and I’m looking forward to the days when my computers aren’t either, I’m not excited. Until it was released, I could always dream that Apple saw the same future I did.

Now that the new filesystem is here, that hope is lost.