Like you I use single documents for things, and the pile of documents has snowballed into thousands of documents over the decades. I have been using OpenOffice (Writer and Calc mostly), but it is not a good solution for large storage. The upper bound on a Calc spreadsheet is around 30000 rows, so for some applications I keep having to split lists (spreadsheets) into smaller lists.
One of the things to be concerned about long-term is what formats to use for storage. OpenOffice still properly supports ancient StarOffice formats, so I have not had to do any major conversions for many years. Regardless of who makes the product(s) you use, you don't know for a fact if support for your existing formats will be there five minutes from now, let alone five years from now.
What works also depends on the context. The element14 community has a forum for people who do electronic hardware development, and a lot of the most prolific participants create posts to document their efforts and they comment on the posts of others in the various forums. For that type of work, I find that a very effective way to document what you do, and I have documented a few of my efforts in the past. But there is a specific focus in the forums that limits what is covered, so it would not be suitable for more general use.
That raises another question, namely, are you trying to document what you are doing for yourself, or are you trying to make the information available to others? Some who have already given answers are obviously trying to track things strictly for their own benefit, but others are trying to make their efforts available to other folks, so the means and format makes a big difference.