IRC channel logs
2026-06-26.log
back to list of logs
<alganet>VERSION might have been a stretch (I could be wrong, would love to hear the rationale behind it). I like the idea of the fs <matrix_bridge><Jeremiah Orians> The idea is to make it cheap to determine the latest version of a file/folder and possible to identify and access specific previous versions of the file. If you have bugs in your implementation of this file system, it might allow you to recover a previous version of a file.(It is also a feature that existed on ITS and RSX-11 operating systems (Files-11 file system) that seemed worth keeping alive) <xentrac>yeah, I grew up on VMS, which also did this <xentrac>basically the full VMS filename syntax was DEVICE:[USER.SUBDIR.SUBSUBDIR]FILENAME.EXT;53 where 53 was the version <xentrac>but most of those elements were optional; usually you could use just the FILENAME part, which defaulted to the current directory, some contextually appropriate extension (file type), and the latest version <xentrac>there was a PURGE command which deleted all but the latest version of the file <xentrac>I forget if there was also an auto-purge limit you could set which would delete the oldest version of the file if too many of them existed. Emacs's numbered backups feature, obviously inspired by having this feature on ITS, does do this <xentrac>IIRC when you listed a directory you would always see all the versions of the file. and they did not share space, so having 10 almost identical versions of a file would use up 10 times as much of your disk quota <xentrac>however, the filesystem versioning scheme I liked better was NetApp's WAFL implementation of "snapshots". NetApp builds NFS servers <xentrac>in every directory on a WAFL filesystem you can always "cd .snapshot", although there is no ".snapshot" directory visible. and there you would see subdirectories hourly.0, hourly.1, hourly.2, ... daily.0, daily.1, ... weekly.0, weekly.1, etc. <xentrac>and inside those subdirectories you would find all your files exactly as they had been at the time of the snapshot in question --- but sharing space with the current version of the files, unless they had been changed. <xentrac>in the very common case where you deleted or overwrote a file by accident and realized it immediately, you could basically always recover it from .snapshot/hourly.0 <xentrac>only losing whatever other changes had happened in the last hour <xentrac>unlike VMS's approach, the WAFL approach also worked with database tablespace files, and even preserved all the within-datat <xentrac>unlike VMS's approach, the WAFL approach also worked with database tablespace files, and even preserved all the within-database consistency propertes that might involve multiple tables in multiple tablespaces, as long as they were in the same filesystem <xentrac>and the consistent snapshot also permitted you to back the database up with regular file copying facilities <xentrac>the filesystem was designed to always write modified filesystem data into a new, unused location on the disk, never overwriting in place. NetApp was driven to this approach by, among other things, their choice of RAID4, which has a terrible parity-disk hotspot problem for small writes, so they had to group together a large group of writes to go in the same place. but once you're doing that, <xentrac>providing consistent snapshots and preserving them for future use is not that hard <matrix_bridge><Jeremiah Orians> Well, you could tie the new version behavior to an fsync and get the database sort of behavior <matrix_bridge><Jeremiah Orians> I am guessing that the hourly snapshots were a solution to if no fsync occurs <xentrac>you mean when you're trying to back up database tablespaces? <matrix_bridge><Jeremiah Orians> Well database tablespaces are just files; if the database does an fsync when all is to be committed. Then you can be certain that it is in a good state at that moment (while the database waits for the fsync to get back) <matrix_bridge><Jeremiah Orians> Moving old versions into a .snapshot directory seems relatively straightforward. But that would be more work that could just be done by user space code and not by the file system itself <xentrac>so the thing about databases is that they have to be able to recover not just from the point where they maybe called fsync() <xentrac>but also from intermediate points between fsync()s <xentrac>at which point some unknown subset of the post-fsync() writes may have been committed to disk at the time of power failure <matrix_bridge><Jeremiah Orians> Well yes, that is why they have journal files <xentrac>fsync() isn't for consistency. it's for durability. you have to *always* be in a consistent state <xentrac>so you might reasonably think you can just back up an active database with `cp` on any Unix system, if the database on disk is always in a consistent state? <xentrac>but that turns out not to be true, and the reason is that `cp` reads different parts of the file at different times, typically starting from the beginning of the file <xentrac>there might be many transactions during the time that `cp` is reading the tablespace, with many fsync()s <matrix_bridge><Jeremiah Orians> But with copy on write handles, the file can’t change on you while copying <xentrac>yes, and WAFL snapshots are copy-on-write "handles" to the entire filesystem <matrix_bridge><Jeremiah Orians> That is what the old directory and old file handles are <xentrac>which means that even transactions that involve multiple tablespace files are guaranteed to be in a consistent state, at least if the database was ever capable of recovering from a power failure reliably <xentrac>Linux has a thing called "reflinks" which creates a copy-on-write snapshot of a single file, although it isn't supported on ext4fs <xentrac>so the advantage of hourly.0, hourly.1, hourly.2, ... daily.0, daily.1, ... over the VMS approach is twofold: <xentrac>2. Usually, if a file is changing many times per hour, it is more valuable to have five hourly snapshots of the file than to have the last five versions of the file all within the last hour <xentrac>I say "the VMS approach" because that's where I experienced it; I never used ITS or RSX-11 <xentrac>there might be differences I'm not aware of <matrix_bridge><Jeremiah Orians> RSX-11 was just a more primitive version of what VMS ultimately had (files-11) <matrix_bridge><Jeremiah Orians> ITS was much more full featured from what I can tell but most of the advanced stuff was not in the file system but manipulating it from user space to save disk space and customization of the versioning behavior <xentrac>yeah, VMS had APIs with horrible bletcherous names for manipulating the versions <xentrac>also at the most basic level you could DELETE a particular version of the file, including the latest one <xentrac>everything about VMS kind of felt like some kind of industrial machinery you'd expect to find in a factory, covered in soot and dripping oil <matrix_bridge><Jeremiah Orians> Well DCL was the spiritual father of PowerShell and all the classic vms +1 jokes about WNT <matrix_bridge><Jeremiah Orians> Although NTFS does have a horrible track record with corrupting databases (it is why PostgreSQL recommends Linux and Oracle recommends giving them their own proprietary file system) <xentrac>it was in many ways a very capable system, but it was much better at making hard things possible than at making easy things easy <xentrac>and it was not aesthetically pleasing at all <xentrac>the VMS authors had abominable taste <matrix_bridge><Jeremiah Orians> Well I did like their PRISM cpu design that ultimately got turned into Alpha and ripped out half the registers and removed the byte instructions <xentrac>yeah, but PRISM wasn't done by the VMS designers <xentrac>it was barely even the same company. wasn't PRISM from DECWRL? <matrix_bridge><Jeremiah Orians> It was part of the whole Crystal themed project that ultimately was doomed and resulted in Windows NT when the OS designers booked it to Microsoft <matrix_bridge><Jeremiah Orians> Which definitely suffered from second system effect <matrix_bridge><Jeremiah Orians> But ultimately, I know that I can’t do simpler than builder-hex0 as that is an append only file system <xentrac>probably you can't do anything simpler in this context <xentrac>but Forth systems did do something simpler <xentrac>what I mean is that bootOS's filesystem has named files, but they're of a single fixed size, 512 bytes <xentrac>Forth systems have "blocks" which they use sort of like source files. they're 1024 bytes, and for text-file use, they are divided into 16 lines of 64 characters, space-padded <xentrac>they go one step further than bootOS in reducing functionality by not naming them, just numbering them <matrix_bridge><Jeremiah Orians> Well that makes perfect sense when you have to write every file in hex unless you create a text editor like SET in 512 bytes <xentrac>I think the F-83 text editor is about three 1024-byte blocks, but that's in source-code form