Generalize task management #17

New issue

Open

opened 2024-07-24 14:54:42 +02:00 by Jef Roosens · 0 comments

Jef Roosens commented

2024-07-24 14:54:42 +02:00

Owner

Generalizing this system would also allow hooking e.g. an admin information feed into the task system

Requirements

support both sync and async functions
Be able to define dependencies that spawn on success (this would mostly allow defining data flows between sync and async workers)
Pass some form of state to the dependent tasks (e.g. pass path and id of parsed package)

Rieter already uses some form of specialized worker pool that tasks can be scheduled on to process uploaded packages. The mirroring system also adds a few, so it might be better to generalize the system so that later data processing can be better integrated in the existing system. Generalizing this system would also allow hooking e.g. an admin information feed into the task system ## Requirements * support both sync and async functions * Be able to define dependencies that spawn on success (this would mostly allow defining data flows between sync and async workers) * Pass some form of state to the dependent tasks (e.g. pass path and id of parsed package)