Memory management system #5

New issue

Closed

opened 2022-12-07 14:53:05 +01:00 by Jef Roosens · 1 comment

Jef Roosens commented

2022-12-07 14:53:05 +01:00

Owner

I just had a rather cool idea.

First of all, I'd like to switch to my own binary format for data storage, where the trie just stores random data as void pointers. The application using the trie would then be responsible for casting the pointers, with the trie not caring about the contents at all. The literal bytes could then be stored inside the data file.

Now, the second idea is to allow configuring the maximum memory that the trie is allowed to use. The trie would keep track of its own internal memory usage, including all allocations etc. The main part where this setting would come into play is when storing the actual data in the trie. As long as the trie has space left, the data will be stored in memory. If there's no space left, only the file path or something of the entry is stored, with it being read from disk when requested.

This idea still needs some fletching out, but it could be cool. The important part here is that this memory management idea could hopefully be managed completely when adding & removing items, and not when searching, meaning search will stay just as fast.

I just had a rather cool idea. First of all, I'd like to switch to my own binary format for data storage, where the trie just stores random data as void pointers. The application using the trie would then be responsible for casting the pointers, with the trie not caring about the contents at all. The literal bytes could then be stored inside the data file. Now, the second idea is to allow configuring the maximum memory that the trie is allowed to use. The trie would keep track of its own internal memory usage, including all allocations etc. The main part where this setting would come into play is when storing the actual data in the trie. As long as the trie has space left, the data will be stored in memory. If there's no space left, only the file path or something of the entry is stored, with it being read from disk when requested. This idea still needs some fletching out, but it could be cool. The important part here is that this memory management idea could hopefully be managed completely when adding & removing items, and not when searching, meaning search will stay just as fast.