Job queue datastructure #2

New Issue

Jef Roosens · 2023-01-18T16:26:16+01:00

Jef Roosens commented

2023-01-18 16:26:16 +01:00

Build job queue will consist of a red-black tree that stores all jobs currently in the system. This tree will allow fast access to both specific jobs, or the entire list of jobs.

Each job will be in one of a few states (these can change throughout development). It'll start in scheduled, and once the job is set to execute (defined by its cron schedule), it'll move to the ready state. Jobs in this state can then be picked up by agents. In each state, a job can be temporarily pop'ed from a queue to perform the respective processing. During this time, a bool flag indicating this should be set to true. Finally, once an agent has finished processing the job, it'll be removed from the entire system. If the job is a recurring build (most are), it'll be re-added as a new one.

The queueing system for these various states will be provided using heaps. Jobs will moved between these heaps as needed.

Interesting information to store as well would be when a job moved into each state, so we can monitor how smoothly everything flows through the system.

Potential uses of job states

With this state system, we could first move a job to the "check sources" state or something like that. The server would then first of all check whether the sources have been updated. This would prevent a lot of unnecessary builds. Now that I think of it, this should probably be handled by the agents as well 🤔

Another idea that just popped into my head is notifications, or just post-processing in general. After a job has finished, it could be moved to a post-processing section where e.g. notifications could be sent to an ntfy server, or any other ideas that I might have.

Build job queue will consist of a red-black tree that stores all jobs currently in the system. This tree will allow fast access to both specific jobs, or the entire list of jobs. Each job will be in one of a few states (these can change throughout development). It'll start in `scheduled`, and once the job is set to execute (defined by its cron schedule), it'll move to the `ready` state. Jobs in this state can then be picked up by agents. In each state, a job can be temporarily pop'ed from a queue to perform the respective processing. During this time, a bool flag indicating this should be set to true. Finally, once an agent has finished processing the job, it'll be removed from the entire system. If the job is a recurring build (most are), it'll be re-added as a new one. The queueing system for these various states will be provided using heaps. Jobs will moved between these heaps as needed. Interesting information to store as well would be when a job moved into each state, so we can monitor how smoothly everything flows through the system. ### Potential uses of job states With this state system, we could first move a job to the "check sources" state or something like that. The server would then first of all check whether the sources have been updated. This would prevent a lot of unnecessary builds. Now that I think of it, this should probably be handled by the agents as well 🤔 Another idea that just popped into my head is notifications, or just post-processing in general. After a job has finished, it could be moved to a post-processing section where e.g. notifications could be sent to an ntfy server, or any other ideas that I might have.

Jef Roosens added the

Idea

label 2023-01-18 16:26:16 +01:00

Sign in to join this conversation.