I have been working with databases for quite a while and have found my sweet spot is in storing the data collected by a shell script, within the shell script itself. Sometimes it'll be single line logs for tracking data in quick projects but other times it will be JSON objects written there (which I find loads quicker than an unopened .json file).
Here's an example of what it looks like for Python
# ====================================================
# Don't write below this line, live database below.
# ====================================================
# "scores":{"Adam":2,"John":6,"Steve":1,"Tim":35}
# "teams":{"shockwave":{"wins":5,"losses":2},"jellyfish":{"wins":3,"losses":4}}
With this method I've found nothing but benefits from keeping it one file for security, preformance, ease of use, ease of transportation and backup, minimal chance of database collisions, and just overall simplicity. I also know that this is the architecture sqllite uses to stay easy to implement for devs.
I do however care about writing good code and would like to know any downfalls to this method.
When the database gets big, will tons of data in comments weigh the program down even if it's not being looked at durring times when other data is being processed?
Is there a limit to the size a file can get before it becomes unstable?
Why isn't this method used more often?
Is there a source where I can learn to implement this better?
Is there anything I'm not thinking of?
Before you answer though, I know having a single 10TB file would be an impractical piece of work to have one process dealing with. I like making clusters and neurons that work both together and individually, and typically use this method for smaller implementations like that. So please do keep that in mind when answering because I know there's better large scale solutions for industry usage.