Continuation from some of things touched on in #84
What is wrong
Using py-trie with an asyncio codebase is problematic. Trie operations can be heavy and in an asyncio environment you typically don't want that stuff blocking the main event loop.
How can it be fixed.
I don't really know....
There are two things that I think this can try to address.
- The computationally heavy components (like all the trie node hashing)
- The database components, actually writing to and reading from the base database.
The database part feels easy enough. An AsyncTrie class where get/set/delete are awaitables, or extend the existing class to have coro_get/coro_set/coro_delete methods that are awaitable. I lean towards the AsyncTrie approach but I could be convinced either way.
The heavy computation offloading feels harder. I think this is because there isn't a one true way to offload computationally heavy stuff because the right approach varies in different contexts. That leads me to believe that we might need to build an API that would enable users of the library to hook into the right places where offloading should/could occur and leave it up to the developer to do the actual offloading. We could provide some built-in mechanisms that operate using this API for things like using the asyncio.Executor patterns.
Continuation from some of things touched on in #84
What is wrong
Using
py-triewith an asyncio codebase is problematic. Trie operations can be heavy and in an asyncio environment you typically don't want that stuff blocking the main event loop.How can it be fixed.
I don't really know....
There are two things that I think this can try to address.
The database part feels easy enough. An
AsyncTrieclass whereget/set/deleteare awaitables, or extend the existing class to havecoro_get/coro_set/coro_deletemethods that are awaitable. I lean towards theAsyncTrieapproach but I could be convinced either way.The heavy computation offloading feels harder. I think this is because there isn't a one true way to offload computationally heavy stuff because the right approach varies in different contexts. That leads me to believe that we might need to build an API that would enable users of the library to hook into the right places where offloading should/could occur and leave it up to the developer to do the actual offloading. We could provide some built-in mechanisms that operate using this API for things like using the
asyncio.Executorpatterns.