Server-agent model #24

New Issue

Jef Roosens · 2022-01-14T16:58:19+01:00

Jef Roosens commented

2022-01-14 16:58:19 +01:00

The goal should be to distribute the work as much as possible. For that, I currently have the following idea:

The main repository server handles periodically restarting builds, e.g. every hour (configurable)
Every iteration, the repository server splits the list of provided Git repositories into n parts (n being the number of agents), and sends each of these lists to a respective agent.
Each agent reads/generates the pkgver for each received repo (can be done by generating a source-only package). They then send a request back to the repository to check whether this version is already present. If not, they build the package & publish it to the repository.
An agent should always send a response to the repository server, both when having to re-build & when not. This allows the server to keep track of when a package was last updated. After the build, the agent should send a response to the repository server indicating whether the build failed or not.

Agent Registration

An agent registers itself with the repository server when starting up. All agents & the server will require a copy of a secret to authenticate requests. After registration, the agent will stay on stand-by until the repository server sends a new request. The repository server will also periodically send a heartbeat request to make sure the agent is still ready to accept connections. If this heartbeat check failed, the repository server will remove the agent from its list of agents until it reregisters.

The goal should be to distribute the work as much as possible. For that, I currently have the following idea: * The main repository server handles periodically restarting builds, e.g. every hour (configurable) * Every iteration, the repository server splits the list of provided Git repositories into n parts (n being the number of agents), and sends each of these lists to a respective agent. * Each agent reads/generates the pkgver for each received repo (can be done by generating a source-only package). They then send a request back to the repository to check whether this version is already present. If not, they build the package & publish it to the repository. * An agent should always send a response to the repository server, both when having to re-build & when not. This allows the server to keep track of when a package was last updated. After the build, the agent should send a response to the repository server indicating whether the build failed or not. ## Agent Registration An agent registers itself with the repository server when starting up. All agents & the server will require a copy of a secret to authenticate requests. After registration, the agent will stay on stand-by until the repository server sends a new request. The repository server will also periodically send a heartbeat request to make sure the agent is still ready to accept connections. If this heartbeat check failed, the repository server will remove the agent from its list of agents until it reregisters.

Jef Roosens added the

idea

label 2022-01-14 16:58:19 +01:00

Jef Roosens commented

2022-01-29 13:24:18 +01:00

Poster

This idea is too complicated for a first version of the project. It would be better to just build the packages on the host of the vieter server, without any additional components. This would allow me to get this project up and running a lot sooner.