> When do you split the files? How do you track which data resides in which files? Does one file represent one kind of data (table)? Does it reflect data within a given time range? Do you sometimes need to retrieve data that crosses file boundaries?
Not really. Splitting anywhere from per-day to per-year is probably fine! Or split arbitrarily and merge the files at runtime. Make it configurable! Add tools to split or merge files, it's really not that hard, a far cry from a database engine.
> You quickly lose the simplicity inherent in saving to just a single file.
No, you really don't.
> Which is where Sqlite shines. It's a single file. But with a full, user defined schema. And can update it and query it incrementally, without having to read and write the entire thing frequently. It handles all of that complexity for you.
That you need a particular tool or library to interact with.
I'm not going to try and sell you on the benefits of using plaintext tools because you've already clearly made up your mind, but there are reasons even if you can't see them. SQLite has like 160k lines of code complexity that isn't necessary and is inherently less composable.
> When do you split the files? How do you track which data resides in which files? Does one file represent one kind of data (table)? Does it reflect data within a given time range? Do you sometimes need to retrieve data that crosses file boundaries?
Not really. Splitting anywhere from per-day to per-year is probably fine! Or split arbitrarily and merge the files at runtime. Make it configurable! Add tools to split or merge files, it's really not that hard, a far cry from a database engine.
> You quickly lose the simplicity inherent in saving to just a single file.
No, you really don't.
> Which is where Sqlite shines. It's a single file. But with a full, user defined schema. And can update it and query it incrementally, without having to read and write the entire thing frequently. It handles all of that complexity for you.
That you need a particular tool or library to interact with.
I'm not going to try and sell you on the benefits of using plaintext tools because you've already clearly made up your mind, but there are reasons even if you can't see them. SQLite has like 160k lines of code complexity that isn't necessary and is inherently less composable.