Chewing_Bever/site
1
0
Fork 0
Code Issues 3 Pull requests Releases Activity

Moving so much stuff around

Browse source
This commit is contained in:
Jef Roosens 2024-06-05 21:43:47 +02:00
parent b1719e8d24
commit cd10df9d32
Signed by: Jef Roosens
GPG key ID: B75D4F293C7052DB
13 changed files with 34 additions and 22 deletions
Download patch file Download diff file Expand all files Collapse all files

29
content/dev/vieter/0.2.0.md Normal file
View file

@ -0,0 +1,29 @@
---
title: "Vieter 0.2.0"
date: 2022-04-11
---
When this post gets published, I'll have successfully released version 0.2.0 of
[Vieter](https://git.rustybever.be/Chewing_Bever/vieter)! For the uninitiated,
Vieter is currently my biggest passion project. It's an implementation of an
Arch repository server, paired with a build system for automatically building
packages from the AUR & other sources.
This release brings a lot of goodies; the changelog & release binaries can be
found
[here](https://git.rustybever.be/Chewing_Bever/vieter/releases/tag/0.2.0). The
biggest changes are that Vieter now supports multiple repositories with support
for packages for multiple architectures! Besides that, there's some bug fixing,
improvements to the CLI & an added setting for the build system that allows for
building on other architectures. The [docs](https://rustybever.be/docs/vieter/)
have also been updated to reflect this new update.
Of course, development won't just stop now, I have too many ideas for that ;p
[0.3.0](https://git.rustybever.be/Chewing_Bever/vieter/milestone/27) will bring
with it some big improvements to the builder system, allowing for more
flexibility & configuration.
If you're interested in the project, join me over at
[#vieter:rustybever.be](https://matrix.to/#/#vieter:rustybever.be) on Matrix!
Cheers <3

69
content/dev/vieter/0.3.0.md Normal file
View file

@ -0,0 +1,69 @@
---
title: "Vieter 0.3.0"
date: 2022-06-13
---
When this post is live, Vieter 0.3.0 will have been released! This release
really does come with a lot of new features, including more reliable builds and
a new cron implementation!
This release ended up taking me over two months, but I'm quite proud of it :)
It not only adds a lot of useful features, but also paves the way for a lot
more cool features down the road!
## What is Vieter?
Vieter consists of two independents parts, namely an implementation of an Arch
(Pacman) repository, & a build system to populate said repository. The goal is
to remove the need for an AUR helper & move all builds to a remote server. Not
only does this greatly reduce update times on lower-end systems, it also
prevents AUR packages from being built multiple times on different systems.
The repository can also be used independently, providing a convenient server
for publishing Arch packages from CI builds for example.
While I specifically mention Arch & the AUR, Vieter is compatible with any
Pacman-based distro & can build PKGBUILDs provided from any Git source.
## What's changed?
### New cron daemon
Perhaps the most important feature in this release is the implementation of a
cron daemon. While 0.2.0 still relied on crond to periodically start builds,
0.3.0 can schedule builds completely independently.
The daemon understands a subset of the cron expression syntax. The build
schedule can be either configured globally or on a per-repo basis. This allows
the user to fine-tune certain packages, e.g. if they want them to be updated
more regularly than all the rest.
### More robust builds
Often, a build would fail with exit code 8. This error indicates that makepkg
wasn't able to install all dependencies, caused by the builder image not being
up to date enough. Due to this, each build now runs `pacman -Syu` before
running the actual build.
Builds can now also use dependencies that are part of the target repository.
This allows building packages with AUR dependencies, as long as all
dependencies are also being built for said repository.
### Build logs
The main server now stores the logs of each build, including the exit code.
This makes it a lot easier to debug why builds fail.
### Improved documentation
The [Vieter documentation](https://rustybever.be/docs/vieter/) has had a pretty
major re-write to get it up to date with this new release. Now there's also
[HTTP API docs](https://rustybever.be/docs/vieter/api/#introduction) & [man
pages](https://rustybever.be/man/vieter/vieter.1.html).
## Interested?
If you're interested in Vieter, considering joining
[#vieter:rustybever.be](https://matrix.to/#/#vieter:rustybever.be) on Matrix!
The source code can be found on my personal
[Gitea](https://git.rustybever.be/vieter/vieter).

69
content/dev/vieter/0.4.0.md Normal file
View file

@ -0,0 +1,69 @@
---
title: "Vieter 0.4.0"
date: 2022-10-01
---
Right at the start of October, I managed to release another version of Vieter!
## What is Vieter?
Vieter consists of two independents parts, namely an implementation of an Arch
(Pacman) repository, & a build system to populate said repository. The goal is
to remove the need for an AUR helper & move all builds to a remote server. Not
only does this greatly reduce update times on lower-end systems, it also
prevents AUR packages from being built multiple times on different systems.
The repository can also be used independently, providing a convenient server
for publishing Arch packages from CI builds for example.
While I specifically mention Arch & the AUR, Vieter is compatible with any
Pacman-based distro & can build PKGBUILDs provided from any Git source.
## What's changed?
### Renaming `repos` to `targets`
Before 0.4.0, "repos" was the term used to describe the list of Git
repositories that periodically get built on your Vieter instance. This term
however was rather confusing, as the Vieter server itself also hosts Pacman
repositories, making it difficult to correctly talk about the features. That's
why I've made the decision to rename this to "targets". All CLI commands
previously found under `vieter repos` can now be used via `vieter targets`
instead. API routes have also been renamed.
Along with this, a new kind of target can now be added which specifies the link
to a PKGBUILD file, instead of a Git repository. This can for example be used
to link a PKGBUILD that's contained inside some other Git repository that's not
specifically used for that PKGBUILD.
### Refactored web framework
The underlying web framework has seen a proper refactor to better accomodate
the rest of the codebase. All API routes can now be found under a versioned
`/api/v1` prefix.
The repository endpoints now support `DELETE` requests to remove packages,
arch-repos & repos. All routes serving files (e.g. the repository routes) now
support HTTP byte range requests, which not only allows Pacman to resume
downloads after failure, but also allows tools such as
[`axel`](https://github.com/axel-download-accelerator/axel) to work properly
using a Vieter server.
Endpoints creating new entries on the server now return the ID of the newly
created object (e.g. a target or a build log).
### CLI UX
The CLI has seen some useful changes. There's now a `-r` flag that makes
Vieter's output better for scripting. Besides that, a small tool has been added
to interact with the AUR and add AUR packages directly to your list of targets!
`vieter targets add` and `vieter logs add` now return the ID of the newly
created entry.
## Interested?
If you're interested in Vieter, considering joining
[#vieter:rustybever.be](https://matrix.to/#/#vieter:rustybever.be) on Matrix!
The source code can be found on my personal
[Gitea](https://git.rustybever.be/vieter/vieter).

99
content/dev/vieter/0.5.0.md Normal file
View file

@ -0,0 +1,99 @@
---
title: "Vieter 0.5.0"
date: 2022-12-29
---
As 2022 comes to a close, and in the middle of exams, I'm ready to release
another version of Vieter!
## What's Vieter?
As usual, a small refresher on what Vieter actually is for the new readers.
Vieter provides two main services: a Pacman repository server and a build
system for publishing Arch packages to said server. The goal is to fully
replace any need for an AUR helper, with the AUR packages being built "in the
cloud" instead. Of course, one can also publish their own packages to this
server, allowing you to create your very own customized Arch repositories!
## What's changed?
### Server-agent architecture
The biggest change this version introduces is the migration to a polling-based
server-agent architecture.
Previous versions relied on a "cron daemon". This daemon needed to be deployed
on every architecture the user wanted to build packages for, with the daemon
periodically polling the server for the list of targets to build. All
scheduling was done on the node performing the builds.
While this system served me well for a while, it did limit possibilities for
improvement. Building on multiple servers wasn't possible, as the cron daemons
had no way of synchronizing with each other, meaning they'd all run all builds.
There was also no way for the server (or the user for that matter) to control
these daemons; they had a fixed build schedule that could only be changed by
changing a target's configuration.
Due to these limitations, I've decided to revamp the build system & convert it
to a server-agent architecture! With this new system, the main server handles
all scheduling. On each server running builds, a "build agent" is deployed
which periodically polls the main server for new jobs. This allows a Vieter
instance to run builds on an arbitrary number of build nodes! Thanks to this,
I'm able to run 137 builds in under 40 minutes, whereas before, I needed this
time to process less than half of that :) As a bit of a stress test for my
instance, I've started replicating the EndeavourOS repository for fun.
With the main server now being in control of scheduling, I've also been able to
implement manual scheduling of builds on the server. If a package needs to be
rebuilt, you can simply send an HTTP request (or use the accompanying CLI
command) to schedule a build job.
### Quality-of-life improvements
This release was mostly the build system redesign, but I've also added some QoL
improvements. Noteable additions are the option to periodically remove logs, as
an active Vieter instance can collect thousands over time. The CLI tool should
now also be a lot more stable, and will correctly display HTTP errors if
needed. There's also the option to specify what subdirectory of a Git
repository to use for a build. This is for example useful when building
packages using a Git repository [containing multiple
PKGBUILDs](https://github.com/endeavouros-team/PKGBUILDS).
## What's to come?
My brain never stops, so I still have a lot of cool ideas I wish to implement
in the coming months.
For starters: better build awareness. The build system right now does not track
the progress of jobs. Once a job is dispatched to an agent, the main server
forgets about it and moves on. If a build fails due to unknown means causing
the logs to never be uploaded, it'll be as if the build never happened. I'd
like to change this. I want the main server to be aware of what jobs are
running, have failed, or have perhaps timed out. This information could then be
made available through the API, providing the user with valuable insight into
the workings of their Vieter instance.
Building on this idea, I wish to know what specific command caused the build to
fail. Was it makepkg, an HTTP error? And if it was makepkg, what error code?
can the build system respond to it by itself? The main goal is to provide a
deeper understanding into the workings of builds and the build system as a
whole.
Another big one to tackle is API access to the repositories. These are
currently only accessible through Pacman, but having this information available
as a convenient REST API, useable from the CLI tool, sounds like a valuable
asset. This would pave the way towards repository-level configuration.
Of course there's a lot more ideas, but the list would be too long to put here
;p
## Conclusion
As you can see, I still have *a lot* of ideas for Vieter. As usual however, I
can't predict when any of these features will get implemented. It all depends
on whether the uni life leaves me some time :)
If you're interested in Vieter, considering joining
[#vieter:rustybever.be](https://matrix.to/#/#vieter:rustybever.be) on Matrix!
The source code can be found on my personal
[Gitea](https://git.rustybever.be/vieter-v/vieter).

62
content/dev/vieter/0.6.0.md Normal file
View file

@ -0,0 +1,62 @@
---
title: "Vieter 0.6.0"
date: 2023-07-20
---
It's been a while since I've released a new version of Vieter. A busy semester
combined with a lack of interest in the project has definitely slowed down
development. Nonetheless, I've got a new release ready for you!
## What's Vieter?
Vieter consists of two central components: an Arch Linux (well, Pacman
actually) repository server that supports uploading package archives, combined
with a build system to populate this server by periodically building select
packages. The goal of this is to completely remove the need for an AUR helper,
and moving all build times to the cloud, allowing for smooth updates across as
many machines as required.
## What's changed?
This is a rather small update, and it mostly contains a few quality-of-life
improvements.
For one, there's now a Prometheus metrics endpoint so you can integrate Vieter
into your existing stack. Currently the metrics are limited to API request
timings, but this could be expanded upon in the future.
The API can now filter the list of targets, allowing you to more easily search
for specific targets. This functionality has also been added to the CLI.
Builds can now be configured with a timeout, with build containers being
automatically killed if this timeout is reached.
Behind the scenes, the codebase has been updated to a compiler commit after the
0.3.3 release, and the cron logic has been rewritten in C using the
[libvieter](https://git.rustybever.be/vieter-v/libvieter) library. Agents now
use worker threads, meaning they will not spawn a new thread anymore for each
new build. Package uploads now properly fail if the TCP connection was closed
before all bytes of a file were received. Lastly, the deprecated cron daemon
has been removed.
### What's next?
Throughout the last couple of months, I've grown more and more tired of the V
programming language, and the codebase in general. There's a lot of technical
debt present, and due to the limitations of the language and existing
frameworks, I've had to resort to questionable practices for a lot of the
features (e.g. POST request data as query parameters). Due to this, I've
decided to restart this project in Rust, under the name
[rieter](https://git.rustybever.be/Chewing_Bever/rieter). With this, I hope to
move away from this technical debt, and build a new solid foundation on which I
can further expand this project. I'm not going to be making any promises on
when this will be ready to replace Vieter, but I hope to get there soon.
## Conclusion
Just picture a very creative ending of this post here ;)
If you're interested in Vieter, consider joining
[#vieter:rustybever.be](https://matrix.to/#/#vieter:rustybever.be) on Matrix!
The source code can be found on my personal
[Gitea](https://git.rustybever.be/vieter-v/vieter).

1
content/dev/vieter/_index.md
View file

@ -1,6 +1,7 @@
---
title: "Vieter"
summary: "Arch Linux repository server & build system, written in V"
type: "project"
---
lmao what