New File Storage backend

You may notice that the Files screen loads quite a lot faster now, particularly if you have many folders.

The File Storage backend has been in need of an overhaul for a long time.  In order to have reliable cache expiry, it was quite single threaded.  We have been throwing around various ideas about how to make those changes for a while, and today I finally set a full day aside, ignoring all other issues, and tested a few things.

One major thing has changed since I first wrote the VFS code 6 years ago.  We’re not constrained by memory for metadata any more.  The VFS had small limits set to avoid blowing out the memory on an individual web process.  Well.  Our smallest web server has 24Gb RAM now.  The newer ones have 48.  The smallest DB server has 64Gb RAM.  There’s no point in caching hot data to disk, because it will be in memory anyway.

So the eventual change was to throw away all the caching layers except one very temporary in-memory one.  There were 1 disk (tmpfs) and two in-memory layers of caching before, so it probably actually saves memory anyway.

The code was also very general, which is fine - but a couple of carefully thought out queries later, I could make one DB fetch to get the full directory tree, plus metadata, and pre-populate the in-memory cache with the fields it was about to ask for.  This, again, is much more efficient than pulling the data from a local cache and checking it for staleness.

The end result - faster response, simpler code, and a few bugs (particularly with long-lived FTP connections) cleared up.

I also backported all the changes to the oldweb interface, so attachments on the compose screen still work, and the Files screen there still works.

The take-home lesson from all this, keep it simple stupid.  The caching complexity isn’t needed any more, if it ever was, and the simpler architecture will help.  I didn’t even have to make any DB schema changes (except dropping a couple of no-longer-used Cache management tables)

There should be no user-visible changes from any of this.  The APIs are all identical for our application layers: webdav, ftp, websites and the Files screen are all the same.