Chewing_Bever
/
site
1
0
Fork 0
Code Issues 3 Pull Requests Releases Activity

rieter: add devlog-1 post
ci/woodpecker/push/woodpecker Pipeline was successful Details

main
Jef Roosens 2024-06-08 21:08:06 +02:00
parent 4190604cbb
commit 5c7675dfbe
Signed by: Jef Roosens
GPG Key ID: B75D4F293C7052DB
1 changed files with 78 additions and 0 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

78
content/dev/rieter/devlog-1.md 100644
View File

@ -0,0 +1,78 @@
---
title: "Rethinking the Vieter project"
date: 2024-06-08
---
I've been meaning to recreate my Vieter project for a while. The codebase is
full of technical debt, and I've grown dissatisfied with the language it was
originally written in. That's where the Rieter project comes in: a full
reimagining and reimplementation of the core ideas of the project, in Rust. I
am however following a different mindset this time around.
My plan is to develop the project in two stages. The first stage involves
creating a well-designed general-purpose repository server. This includes
serving and storing packages, as well as providing a REST API and web UI to
interact with the repository packages. In this stage I'll also add mirroring
functionality to allow a Rieter server to automatically maintain a local copy
of another repository. This could be used to easily create another mirror for a
distribution's servers, or perhaps to create a local mirror for faster
downloads.
Once the first stage is finished, we have a solid foundation on which we can
build the second stage: the build system. This will involve redesigning the
agent-server architecture that's currently used in Vieter, with the goal of
completely replacing Vieter in due time.
This post is the first in a hopefully plentiful series of devlogs for this
project where I'll document my progress along the way.
## Current progress
The implementation of the repository server itself is almost done. A user can
publish, request and remove packages for any number of repositories and
architectures. Repositories are then further grouped into distributions,
allowing a single server to be used for multiple distributions if need be (e.g.
I would for example have `arch` and `endeavouros` as distributions on my
personal server). A package's information is added to the database, and this
data is then exposed via a paginated REST API.
The only real hurdle left for a first release is concurrency, which brings with
it a couple of problems. With the current implementation, it's possible for
concurrent uploads of packages to corrupt the repository. The generation of the
package archives happens inside the request handler for each upload, meaning
multiple requests basically do duplicate work and can cause CPU usage spikes.
The parsing of packages is also done inside the request handler, which once
again causes the server to spike in CPU usage if multiple packages are uploaded
in parallel. These things combined make concurrent uploads of packages a rather
painful problem to deal with.
My solution for these problems consists of two parts. First I want to add a
queueing system for new packages. Instead of parsing the packages directly in
the request handler, they would get added to a queue, with the server then
responding with a [`202
Accepted`](https://developer.mozilla.org/en-US/docs/Web/HTTP/Status/202). The
actual parsing of the packages would be done asynchronously by a configurable
number of worker threads that parse the packages.
The second part involves serializing and stalling the generation of the package
archives until needed. Instead of actually generating the package archives for
each uploaded package, we simply notify some central worker thread that the
repository has been altered. This worker would then generate the package
archives, after ensuring the queue is empty and no new packages have arrived in
the last `n` seconds. This pattern accounts for groups of packages being
uploaded at once without needlessly stressing the server.
By implementing these features, the server should be able to handle a large
number of package uploads without using excessive resources, ensuring Rieter
can scale to proper sizes.
## First release
Once this is implemented, the codebase should be ready for a 0.1.0 release!
This version will already be useable as a fully-fletched repository server on
which I can then build the other parts of the first stage.
For the 1.0 release, I'll be adding a web UI, as this was something that I was
sorely missing from Vieter. Perhaps most exciting of all, automatic mirroring
will also be added which I'm definitely looking forward to! I hope to publish
another post here soon, but until then, thanks for reading.