Generalize task management #17

New Issue

Jef Roosens · 2024-07-24T14:54:42+02:00

Jef Roosens commented

2024-07-24 14:54:42 +02:00

Rieter already uses some form of specialized worker pool that tasks can be scheduled on to process uploaded packages. The mirroring system also adds a few, so it might be better to generalize the system so that later data processing can be better integrated in the existing system.

Generalizing this system would also allow hooking e.g. an admin information feed into the task system

Requirements

support both sync and async functions
Be able to define dependencies that spawn on success (this would mostly allow defining data flows between sync and async workers)
Pass some form of state to the dependent tasks (e.g. pass path and id of parsed package)

Rieter already uses some form of specialized worker pool that tasks can be scheduled on to process uploaded packages. The mirroring system also adds a few, so it might be better to generalize the system so that later data processing can be better integrated in the existing system. Generalizing this system would also allow hooking e.g. an admin information feed into the task system ## Requirements * support both sync and async functions * Be able to define dependencies that spawn on success (this would mostly allow defining data flows between sync and async workers) * Pass some form of state to the dependent tasks (e.g. pass path and id of parsed package)

Jef Roosens added the

enhancement

label 2024-07-24 14:54:42 +02:00

Sign in to join this conversation.