From 4ecf6a11c4518f01854fbace08ce33db382a732b Mon Sep 17 00:00:00 2001 From: Chewing_Bever Date: Wed, 8 Jun 2022 22:16:46 +0200 Subject: [PATCH 1/7] docs: rewrote installation page --- README.md | 37 +++++----- docs/config.toml | 7 +- docs/content/_index.md | 15 ++-- docs/content/installation.md | 131 +++++++++++++++++++---------------- 4 files changed, 97 insertions(+), 93 deletions(-) diff --git a/README.md b/README.md index 34e6fd3..892396a 100644 --- a/README.md +++ b/README.md @@ -22,13 +22,6 @@ a while now. I wanted a fast language that I could code while relaxing, without having to exert too much mental effort & V seemed like the right choice for that. -### Compiler - -Vieter compiles with the standard Vlang compiler. However, I do maintain a -[mirror](https://git.rustybever.be/Chewing_Bever/v). This is to ensure my CI -does not break without reason, as I control when & how frequently the mirror is -updated to reflect the official repository. - ## Features * Arch repository server @@ -41,20 +34,24 @@ updated to reflect the official repository. ## Building -In order to build Vieter, you'll need a couple of libraries: +Besides a V installer, Vieter also requires the following libraries to work: -* An installation of V * gc * libarchive * openssl +* sqlite3 -**NOTE**: if you encounter any issues compiling Vieter using the absolute -latest version of V, it might be because my mirror is missing a specific commit -that causes issues. For this reason, the `make v` command exists which will -clone my compiler in the `v` directory & build it. Afterwards, you can use this -compiler with make by prepending all make commands with `V_PATH=v/v`. If you do -encounter this issue, please let me know so I can update my mirror & the -codebase to fix it! +### Compiler + +Vieter compiles with the standard Vlang compiler. However, I do maintain a +[mirror](https://git.rustybever.be/vieter/v). This is to ensure my CI does not +break without reason, as I control when & how frequently the mirror is updated +to reflect the official repository. + +If you encounter issues using the latest V compiler, try using my mirror +instead. `make v` will clone the repository & build the mirror. Afterwards, +prepending any make command with `V_PATH=v/v` tells make to use the locally +compiled mirror instead. ## Contributing @@ -70,7 +67,7 @@ If you wish to contribute to the project, please take note of the following: The `docs` directory contains a Hugo site consisting of all user & administrator documentation. `docs/api` on the other hand is a -[slate](https://github.com/slatedocs/slate) project describing the HTTP web +[Slate](https://github.com/slatedocs/slate) project describing the HTTP web API. To modify the Hugo documentation, you'll need to install Hugo. Afterwards, you @@ -81,8 +78,8 @@ can use the following commands inside the `docs` directory: hugo # Host an auto-refreshing web server with the documentation. Important to note -is that the files will be at `http://localhost:1313/docs/vieter` instead of -just `http://localhost:1313/docs/vieter` +# is that the files will be at `http://localhost:1313/docs/vieter` instead of +# just `http://localhost:1313/` hugo server ``` @@ -96,6 +93,6 @@ docker run \ -v $(pwd)/docs/api/source:/srv/slate/source slatedocs/slate serve ``` -This'll make the slate docs available at http://localhost:4567. Sadly, this +This will make the Slate docs available at http://localhost:4567. Sadly, this server doesn't auto-refresh, so you'll have to manually refresh your browser every time you make a change. diff --git a/docs/config.toml b/docs/config.toml index 5b0f20f..61aa3ae 100644 --- a/docs/config.toml +++ b/docs/config.toml @@ -6,7 +6,8 @@ theme = 'hugo-book' # Book configuration disablePathToLower = true -enableGitInfo = true +# Doesn't work with docs as subdir +enableGitInfo = false # Needed for mermaid/katex shortcodes [markup] @@ -29,7 +30,7 @@ enableGitInfo = true [menu] [[menu.after]] name = "API Documentation" - url = "https://rustybever.be/docs/vieter/api" + url = "https://rustybever.be/docs/vieter/api/" weight = 10 [[menu.after]] name = "Man Pages" @@ -69,7 +70,7 @@ enableGitInfo = true # Set source repository location. # Used for 'Last Modified' and 'Edit this page' links. - BookRepo = 'https://git.rustybever.be/Chewing_Bever/docs' + BookRepo = 'https://git.rustybever.be/vieter/vieter/docs' # (Optional, default 'commit') Specifies commit portion of the link to the page's last modified # commit hash for 'doc' page type. diff --git a/docs/content/_index.md b/docs/content/_index.md index 3a1144b..efcdf6d 100644 --- a/docs/content/_index.md +++ b/docs/content/_index.md @@ -9,12 +9,9 @@ documentation might not be relevant anymore for the latest release. ## Overview -Vieter has a few main features: - -* It's a simple & lightweight implementation of an Arch repository server -* It allows for uploading of built package archives -* It supports a basic build system to periodically re-build packages & upload - them to the server +Vieter consists of two main parts, namely an implementation of an Arch +repository server & a scheduling system to periodically build Pacman packages & +publish them to a repository. {{< hint info >}} **Note** @@ -26,12 +23,12 @@ well. ### Why? -Vieter is my personal solution for a problem I've been facing for months: +Vieter is my personal solution to a problem I've been facing for months: extremely long AUR package build times. I run EndeavourOS on both my laptops, one of which being a rather old MacBook Air. I really like being a beta-tester for projects & run development builds for multiple packages (nheko, -newsflash...). The issue with this is that I have to regularly re-build these -packages in order to stay up to date with development & these builds can take a +newsflash...). Because of this, I have to regularly re-build these packages in +order to stay up to date with development. However, these builds can take a really long time on the old MacBook. This project is a solution to that problem: instead of building the packages locally, I can build them automatically in the cloud & just download them whenever I update my system! diff --git a/docs/content/installation.md b/docs/content/installation.md index 17d3874..5cc7a0e 100644 --- a/docs/content/installation.md +++ b/docs/content/installation.md @@ -3,76 +3,85 @@ weight: 10 --- # Installation +Vieter consists of a single binary, akin to busybox. The binary's behavior is +determined by its CLI arguments, e.g. `vieter server` starts the repository +server. + +All installation solutions can be configured the same way, +as described [here](/configuration). + ## Docker -Docker is the recommended way to install vieter. The images can be pulled from -[`chewingbever/vieter`](https://hub.docker.com/r/chewingbever/vieter). You can -either pull a release tag (e.g. `chewingbever/vieter:0.1.0-rc1`), or pull the -`chewingbever/vieter:dev` tag. The latter is updated every time a new commit is -pushed to the development branch. This branch will be the most up to date, but -does not give any guarantees about stability, so beware! +Docker images are published to the +[`chewingbever/vieter`](https://hub.docker.com/r/chewingbever/vieter) Docker +Hub repository. You can either pull a release tag (e.g. +`chewingbever/vieter:0.1.0-rc1`), or pull the `chewingbever/vieter:dev` tag. +The latter is updated every time a new commit is pushed to the development +branch. This branch will be the most up to date, but does not give any +guarantees about stability, so beware! -The simplest way to run the Docker image is using a plain Docker command: +Due to the single-binary design of Vieter, this image can be used both for the +repository server & the cron daemon. -```sh -docker run \ - --rm \ - -d \ - -v /path/to/data:/data \ - -e VIETER_API_KEY=changeme \ - -e VIETER_DEFAULT_ARCH=x86_64 \ - -p 8000:8000 \ - chewingbever/vieter:dev +Below is an example compose file to set up both the repository server & the +cron daemon: + +```yaml +version: '3' + +services: + server: + image: 'chewingbever/vieter:dev' + restart: 'always' + + environment: + - 'VIETER_API_KEY=secret' + - 'VIETER_DEFAULT_ARCH=x86_64' + volumes: + - 'data:/data' + + cron: + image: 'chewingbever/vieter:dev' + restart: 'always' + user: root + command: 'vieter cron' + + environment: + - 'VIETER_API_KEY=secret' + # MUST be public URL of Vieter repository + - 'VIETER_ADDRESS=https://example.com' + - 'VIETER_DEFAULT_ARCH=x86_64' + - 'VIETER_MAX_CONCURRENT_BUILDS=2' + - 'VIETER_GLOBAL_SCHEDULE=0 3' + +volumes: + data: ``` -Here, you should change `/path/to/data` to the path on your host where you want -vieter to store its files. - -The default configuration will store everything inside the `/data` directory. - -Inside the container, the Vieter server runs on port 8000. This port should be -exposed to the public accordingely. - -For an overview of how to configure vieter & which environment variables can be -used, see the [Configuration](/configuration) page. - ## Binary -On the [releases](https://git.rustybever.be/Chewing_Bever/vieter/releases) -page, you can find statically compiled binaries for all released versions. You -can download the binary for your host's architecture & run it that way. +On the +[releases](https://git.rustybever.be/vieter/vieter/releases) +page, you can find statically compiled binaries for all +released versions. This is the same binary as used inside +the Docker images. -For more information about configuring the binary, check out the -[Configuration](/configuration) page. +## Arch + +I publish both development & release versions of Vieter to my personal +repository, https://arch.r8r.be. Packages are available for `x86_64` & +`aarch64`. To use the repository, add the following to your `pacman.conf`: + +``` +[vieter] +Server = https://arch.r8r.be/$repo/$arch +SigLevel = Optional +``` + +Afterwards, you can update your system & install the `vieter` package for the +latest official release or `vieter-git` for the latest development release. ## Building from source -Because the project is still in heavy development, it might be useful to build -from source instead. Luckily, this process is very easy. You'll need make, -libarchive & openssl; all of which should be present on an every-day Arch -install. Then, after cloning the repository, you can use the following commands: - -```sh -# Builds the compiler; should usually only be ran once. Vieter compiles using -# the default compiler, but I maintain my own mirror to ensure nothing breaks -# without me knowing. -make v - -# Build vieter -# Alternatively, use `make prod` to build the production build. -make -``` -{{< hint info >}} -**Note** -My version of the V compiler is also available on my Vieter instance, -https://arch.r8r.be. It's in the `vieter` repository, with the package being -named `vieter-v`. The compiler is available for both x86_64 & aarch64. -{{< /hint >}} - -## My Vieter instance - -Besides uploading development Docker images, my CI also publishes x86_64 & -aarch64 packages to my personal Vieter instance, https://arch.r8r.be. If you'd -like, you can use this repository as well by adding it to your Pacman -configuration as described [here](/usage#configuring-pacman). Both the -repository & the package are called `vieter`. +The project [README](https://git.rustybever.be/vieter/vieter#building) contains +instructions for building Vieter from source. From d7d77afe09f193582b47168447bec40f8666f78e Mon Sep 17 00:00:00 2001 From: Chewing_Bever Date: Thu, 9 Jun 2022 09:32:41 +0200 Subject: [PATCH 2/7] docs: wrote part of new usage section --- docs/config.toml | 12 +++++----- docs/content/installation.md | 15 ++++++++++++ docs/content/usage/_index.md | 4 ++++ docs/content/usage/repository.md | 38 +++++++++++++++++++++++++++++++ docs/content/{ => usage}/usage.md | 0 5 files changed, 63 insertions(+), 6 deletions(-) create mode 100644 docs/content/usage/_index.md create mode 100644 docs/content/usage/repository.md rename docs/content/{ => usage}/usage.md (100%) diff --git a/docs/config.toml b/docs/config.toml index 61aa3ae..d0eecb2 100644 --- a/docs/config.toml +++ b/docs/config.toml @@ -7,7 +7,7 @@ theme = 'hugo-book' # Book configuration disablePathToLower = true # Doesn't work with docs as subdir -enableGitInfo = false +enableGitInfo = true # Needed for mermaid/katex shortcodes [markup] @@ -29,7 +29,7 @@ enableGitInfo = false [menu] [[menu.after]] - name = "API Documentation" + name = "HTTP API Docs" url = "https://rustybever.be/docs/vieter/api/" weight = 10 [[menu.after]] @@ -37,8 +37,8 @@ enableGitInfo = false url = "https://rustybever.be/man/vieter/vieter.1.html" weight = 20 [[menu.after]] - name = "Source" - url = "https://git.rustybever.be/Chewing_Bever/docs" + name = "Git" + url = "https://git.rustybever.be/vieter/vieter" weight = 30 [[menu.after]] name = "Hugo Theme" @@ -70,14 +70,14 @@ enableGitInfo = false # Set source repository location. # Used for 'Last Modified' and 'Edit this page' links. - BookRepo = 'https://git.rustybever.be/vieter/vieter/docs' + BookRepo = 'https://git.rustybever.be/vieter/vieter' # (Optional, default 'commit') Specifies commit portion of the link to the page's last modified # commit hash for 'doc' page type. # Requires 'BookRepo' param. # Value used to construct a URL consisting of BookRepo/BookCommitPath/ # Github uses 'commit', Bitbucket uses 'commits' - # BookCommitPath = 'commit' + BookCommitPath = 'src/commit' # Enable "Edit this page" links for 'doc' page type. # Disabled by default. Uncomment to enable. Requires 'BookRepo' param. diff --git a/docs/content/installation.md b/docs/content/installation.md index 5cc7a0e..2cf57e7 100644 --- a/docs/content/installation.md +++ b/docs/content/installation.md @@ -53,11 +53,26 @@ services: - 'VIETER_DEFAULT_ARCH=x86_64' - 'VIETER_MAX_CONCURRENT_BUILDS=2' - 'VIETER_GLOBAL_SCHEDULE=0 3' + volumes: + - '/var/run/docker.sock:/var/run/docker.sock' volumes: data: ``` +If you do not require the build system, the repository server can be used +independently as well. + +{{< hint info >}} +**Note** +Builds are executed on the cron daemon's system using the host's Docker daemon. +A cron daemon on a specific architecture will only build packages for that +specific architecture. Therefore, if you wish to build packages for both +`x86_64` & `aarch64`, you'll have to deploy two cron daemons, one on each +architecture. Afterwards, any Git repositories enabled for those two +architectures will build on both. +{{< /hint >}} + ## Binary On the diff --git a/docs/content/usage/_index.md b/docs/content/usage/_index.md new file mode 100644 index 0000000..bb0376f --- /dev/null +++ b/docs/content/usage/_index.md @@ -0,0 +1,4 @@ +# Usage + +This section describes how to use the various parts of Vieter. It assumes you +have a Vieter repository server up and running. diff --git a/docs/content/usage/repository.md b/docs/content/usage/repository.md new file mode 100644 index 0000000..c30c8ca --- /dev/null +++ b/docs/content/usage/repository.md @@ -0,0 +1,38 @@ +# Pacman repository + +The part of Vieter that users will interact with the most is the Pacman +repository aka `vieter server`. + +## Design overview + +A Vieter repository server has support for multiple repositories, with each +repository containing packages for multiple architectures. + +If you wish to use these repositories on your system, add the following to +`/etc/pacman.conf` for each repository you wish to use: + +``` +[repo-name] +Server = https://example.com/$repo/$arch +SigLevel = Optional +``` + +Here, `$repo` & `$arch` are not variables you have to fill in yourself. Rather, +Pacman will substitute these when reading the config file. `$repo` is replaced +by the name between the square brackets (in this case `repo-name`), & `$arch` +is replaced by your system's architecture, e.g. `x86_64`. Of course, you can +also fill in these values manually yourself, e.g. if you wish to use a +different name inside the square brackets. + +Important to note is that, when two repositories contain a package with the +same name, Pacman will choose the one from the repository that's highest up in +the `pacman.conf` file. Therefore, if you know your repository has packages +with the same name as ones from the official repositories, it might be better +to place the repository below the official repositories to avoid overwriting +official packages. + +## Publishing packages + +Packages can be easily published using a single HTTP POST request. Check out +the [HTTP API docs](https://rustybever.be/docs/vieter/api/) for more info on +these routes, including example cURL commands. diff --git a/docs/content/usage.md b/docs/content/usage/usage.md similarity index 100% rename from docs/content/usage.md rename to docs/content/usage/usage.md From 0ab39a334db6f4b278608c28f6239a94df2aa358 Mon Sep 17 00:00:00 2001 From: Chewing_Bever Date: Thu, 9 Jun 2022 10:25:36 +0200 Subject: [PATCH 3/7] docs: started new builds usage page --- docs/content/other/_index.md | 3 ++ docs/content/other/builds-in-depth.md | 4 +++ docs/content/usage/_index.md | 7 ++--- docs/content/usage/builds.md | 40 +++++++++++++++++++++++++++ docs/content/usage/repository.md | 3 ++ 5 files changed, 53 insertions(+), 4 deletions(-) create mode 100644 docs/content/other/_index.md create mode 100644 docs/content/other/builds-in-depth.md create mode 100644 docs/content/usage/builds.md diff --git a/docs/content/other/_index.md b/docs/content/other/_index.md new file mode 100644 index 0000000..394456b --- /dev/null +++ b/docs/content/other/_index.md @@ -0,0 +1,3 @@ +--- +weight: 100 +--- diff --git a/docs/content/other/builds-in-depth.md b/docs/content/other/builds-in-depth.md new file mode 100644 index 0000000..a6f0c21 --- /dev/null +++ b/docs/content/other/builds-in-depth.md @@ -0,0 +1,4 @@ +# Builds In-depth + +For those interested, this page describes how the build system works +internally. diff --git a/docs/content/usage/_index.md b/docs/content/usage/_index.md index bb0376f..1518e5e 100644 --- a/docs/content/usage/_index.md +++ b/docs/content/usage/_index.md @@ -1,4 +1,3 @@ -# Usage - -This section describes how to use the various parts of Vieter. It assumes you -have a Vieter repository server up and running. +--- +weight: 30 +--- diff --git a/docs/content/usage/builds.md b/docs/content/usage/builds.md new file mode 100644 index 0000000..db11bec --- /dev/null +++ b/docs/content/usage/builds.md @@ -0,0 +1,40 @@ +--- +weight: 20 +--- +# Building packages + +The automatic build system is what makes Vieter very useful as a replacement +for an AUR helper. It can perodically build packages & publish them to your +personal Vieter repository server, removing the need to build the packages +locally. + +## Adding builds + +Before the cron system can start building your package, you need to add its +info to the system. The Vieter repository server exposes an HTTP API for this +(see the [HTTP API Docs](https://rustybever.be/docs/vieter/api/) for more +info). For ease of use, the Vieter binary contains a CLI interface for +interacting with this API (see [Configuration](/configuration) for +configuration details). The [man +pages](https://rustybever.be/man/vieter/vieter-repos.1.html) describe this in +greater detail, but the basic usage is as follows: + +``` +vieter repos add some-url some-branch some-repository +``` + +Here, `some-url` is the URL of the Git repository containing the PKGBUILD. This +URL is passed to `git clone`, so the repository should be public. Vieter +expects the same format as an AUR Git repository, so you can directly use AUR +URLs here. + +`some-branch` is the branch of the Git repository the build should check out. +If you're using an AUR package, this should be `master`. + +Finally, `some-repo` is the repository to which the built package archives +should be published. + +The above command intentionally leaves out a few parameters to make the CLI +more useable. For information on how to modify all parameters using the CLI, +see +[vieter-repos-edit(1)](https://rustybever.be/man/vieter/vieter-repos-edit.1.html). diff --git a/docs/content/usage/repository.md b/docs/content/usage/repository.md index c30c8ca..1dc6bc6 100644 --- a/docs/content/usage/repository.md +++ b/docs/content/usage/repository.md @@ -1,3 +1,6 @@ +--- +weight: 10 +--- # Pacman repository The part of Vieter that users will interact with the most is the Pacman From a9ddfd8ec8d9d76ea2f45e15bec643538ed13c76 Mon Sep 17 00:00:00 2001 From: Chewing_Bever Date: Thu, 9 Jun 2022 16:26:53 +0200 Subject: [PATCH 4/7] docs: too many changes to count --- docs/config.toml | 2 +- docs/content/configuration.md | 78 ++++++++++++++----- .../usage/{builds.md => builds/_index.md} | 2 +- docs/content/usage/builds/schedule.md | 42 ++++++++++ docs/content/usage/repository.md | 12 +-- 5 files changed, 109 insertions(+), 27 deletions(-) rename docs/content/usage/{builds.md => builds/_index.md} (95%) create mode 100644 docs/content/usage/builds/schedule.md diff --git a/docs/config.toml b/docs/config.toml index d0eecb2..5c82044 100644 --- a/docs/config.toml +++ b/docs/config.toml @@ -1,7 +1,7 @@ # hugo server --minify --themesDir ... --baseURL=http://0.0.0.0:1313/theme/hugo-book/ baseURL = 'https://rustybever.be/docs/vieter/' -title = 'The Rusty Bever - Docs' +title = 'Vieter - Docs' theme = 'hugo-book' # Book configuration diff --git a/docs/content/configuration.md b/docs/content/configuration.md index ded40cb..27eb213 100644 --- a/docs/content/configuration.md +++ b/docs/content/configuration.md @@ -3,7 +3,7 @@ weight: 20 --- # Configuration -All vieter operations by default try to read in the TOML file `~/.vieterrc` for +By default, all vieter commands try to read in the TOML file `~/.vieterrc` for configuration. The location of this file can be changed by using the `-f` flag. If the above file doesn't exist or you wish to override some of its settings, @@ -19,38 +19,78 @@ the value in the environment variable is used. {{< hint info >}} **Note** All environment variables can also be provided from a file by appending them -with `_FILE`. This for example allows you to provide the API key from a docker +with `_FILE`. This for example allows you to provide the API key from a Docker secrets file. {{< /hint >}} -## Modes +## Commands -The vieter binary can run in several "modes", indicated by the first argument -passed to them. Each mode requires a different configuration. +The first argument passed to Vieter determines which command you wish to use. +Each of these can contain subcommands (e.g. `vieter repos list`), but all +subcommands will use the same configuration. Below you can find the +configuration variable required for each command. -### Server +### `vieter server` -* `log_level`: defines how much logs to show. Valid values are one of `FATAL`, - `ERROR`, `WARN`, `INFO` or `DEBUG`. Defaults to `WARN` -* `log_file`: log file to write logs to. Defaults to `vieter.log` in the - current directory. +* `log_level`: log verbosity level. Value should be one of `FATAL`, `ERROR`, + `WARN`, `INFO` or `DEBUG`. + * Default: `WARN` +* `log_file`: log file to write logs to. + * Default: `vieter.log` (in the current directory) * `pkg_dir`: where Vieter should store the actual package archives. * `data_dir`: where Vieter stores the repositories, log file & database. * `api_key`: the API key to use when authenticating requests. -* `default_arch`: architecture to always add packages of arch `any` to. +* `default_arch`: this setting serves two main purposes: + * Packages with architecture `any` are always added to this architecture. + This prevents the server from being confused when an `any` package is + published as the very first package for a repository. + * Git repositories added without an `arch` value use this value instead. -### Builder + +### `vieter repos` * `api_key`: the API key to use when authenticating requests. * `address`: Base your URL of your Vieter instance, e.g. https://example.com -* `base_image`: image to use when building a package. It should be an Archlinux - image. The default if not configured is `archlinux:base-devel`, but this - image only supports arm64. If you require aarch64 support as well, consider - using - [`menci/archlinuxarm:base-devel`](https://hub.docker.com/r/menci/archlinuxarm) - ([GH](https://github.com/Menci/docker-archlinuxarm)) +* `base_image`: image to use when building a package using `vieter repos + build`. + * Default: `archlinux:base-devel` -### Repos +### `vieter cron` + +* `log_level`: log verbosity level. Value should be one of `FATAL`, `ERROR`, + `WARN`, `INFO` or `DEBUG`. + * Default: `WARN` +* `log_file`: log file to write logs to. + * Default: `vieter.log` (in `data_dir`) +* `address`: *public* URL of the Vieter repository server to build for. From + this server the list of Git repositories is retrieved. All built packages are + published to this server. +* `api_key`: API key of the above server. +* `data_dir`: directory to store log file in. +* `base_image`: Docker image to use when building a package. Any Pacman-based + distro image should work, as long as `/etc/pacman.conf` is used & + `base-devel` exists in the repositories. Make sure that the image supports + the architecture of your cron daemon. + * Default: `archlinux:base-devel` (only works on `x86_64`). If you require + `aarch64` support, consider using + [`menci/archlinuxarm:base-devel`](https://hub.docker.com/r/menci/archlinuxarm) + ([GitHub](https://github.com/Menci/docker-archlinuxarm)). This is the image + used for the Vieter CI builds. +* `max_concurrent_builds`: how many builds to run at the same time. + * Default: `1` +* `api_update_frequency`: how frequently (in minutes) to poll the Vieter + repository server for a new list of Git repositories to build. + * Default: `15` +* `image_rebuild_frequency`: Vieter periodically builds a builder image using + the configured base image. This makes sure build containers do not have to + download a lot of packages when updating their system. This setting defines + how frequently (in minutes) to rebuild this builder image. + * Default: `1440` (every 24 hours) +* `global_schedule`: build schedule for any Git repository that does not have a + schedule defined. + * Default: `0 3` (3AM every night) + +### `vieter logs` * `api_key`: the API key to use when authenticating requests. * `address`: Base your URL of your Vieter instance, e.g. https://example.com diff --git a/docs/content/usage/builds.md b/docs/content/usage/builds/_index.md similarity index 95% rename from docs/content/usage/builds.md rename to docs/content/usage/builds/_index.md index db11bec..c6f0603 100644 --- a/docs/content/usage/builds.md +++ b/docs/content/usage/builds/_index.md @@ -24,7 +24,7 @@ vieter repos add some-url some-branch some-repository ``` Here, `some-url` is the URL of the Git repository containing the PKGBUILD. This -URL is passed to `git clone`, so the repository should be public. Vieter +URL is passed to `git clone`, meaning the repository should be public. Vieter expects the same format as an AUR Git repository, so you can directly use AUR URLs here. diff --git a/docs/content/usage/builds/schedule.md b/docs/content/usage/builds/schedule.md new file mode 100644 index 0000000..38f76a4 --- /dev/null +++ b/docs/content/usage/builds/schedule.md @@ -0,0 +1,42 @@ +# Cron schedule syntax + +The Vieter cron daemon uses a subset of the cron expression syntax to schedule +builds. + +## Format + +`a b c d` + +* `a`: minutes +* `b`: hours +* `c`: days +* `d`: months + +An expression consists of two to four sections. If less than four sections are +provided, the parser will append `*` until there are four sections. This means +that `0 3` is the same as `0 3 * *`. + +Each section consists of one or more parts, separated by a comma. Each of these +parts, in turn, can be one of the following (any letters are integers): + +* `*`: allow all possible values. +* `a`: only this value is allowed. +* `*/n`: allow every n-th value. +* `a/n`: allow every n-th value, starting at a in the list. +* `a-b`: allow every value between a and b, bounds included. +* `a-b/n`: allow every n-th value inside the list of values between a and b, + bounds included. + +Each section can consist of as many of these parts as necessary. + +## Examples + +* `0 3`: every day at 03:00AM. +* `0 0 */7`: every 7th day of the month, at midnight. + +## CLI tool + +The Vieter binary contains a command that shows you the next matching times for +a given expression. This can be useful to understand the syntax. For more +information, see +[vieter-schedule(1)](https://rustybever.be/man/vieter/vieter-schedule.1.html). diff --git a/docs/content/usage/repository.md b/docs/content/usage/repository.md index 1dc6bc6..3ddd2fc 100644 --- a/docs/content/usage/repository.md +++ b/docs/content/usage/repository.md @@ -20,12 +20,12 @@ Server = https://example.com/$repo/$arch SigLevel = Optional ``` -Here, `$repo` & `$arch` are not variables you have to fill in yourself. Rather, -Pacman will substitute these when reading the config file. `$repo` is replaced -by the name between the square brackets (in this case `repo-name`), & `$arch` -is replaced by your system's architecture, e.g. `x86_64`. Of course, you can -also fill in these values manually yourself, e.g. if you wish to use a -different name inside the square brackets. +Here, `$repo` and `$arch` are not variables you have to fill in yourself. +Rather, Pacman will substitute these when reading the config file. `$repo` is +replaced by the name between the square brackets (in this case `repo-name`), +and `$arch` is replaced by your system's architecture, e.g. `x86_64`. Of +course, you can also fill in these values manually yourself, e.g. if you wish +to use a different name inside the square brackets. Important to note is that, when two repositories contain a package with the same name, Pacman will choose the one from the repository that's highest up in From c341d7a0243526a215d6698d75479d2c1bb20574 Mon Sep 17 00:00:00 2001 From: Chewing_Bever Date: Thu, 9 Jun 2022 17:24:16 +0200 Subject: [PATCH 5/7] docs: some changed; removed some old files --- docs/api/source/includes/_git.md | 2 +- docs/content/CLI.md | 27 --------------- docs/content/configuration.md | 29 +++++----------- docs/content/usage/builds/_index.md | 11 ++++++ docs/content/usage/usage.md | 54 ----------------------------- 5 files changed, 20 insertions(+), 103 deletions(-) delete mode 100644 docs/content/CLI.md delete mode 100644 docs/content/usage/usage.md diff --git a/docs/api/source/includes/_git.md b/docs/api/source/includes/_git.md index f41d28d..8458834 100644 --- a/docs/api/source/includes/_git.md +++ b/docs/api/source/includes/_git.md @@ -112,7 +112,7 @@ Parameter | Description url | URL of the Git repository. branch | Branch of the Git repository. repo | Vieter repository to publish built packages to. -schedule | Cron build schedule +schedule | Cron build schedule (syntax explained [here](https://rustybever.be/docs/vieter/usage/builds/schedule/)) arch | Comma-separated list of architectures to build package on. ## Modify a repo diff --git a/docs/content/CLI.md b/docs/content/CLI.md deleted file mode 100644 index 32bb6f8..0000000 --- a/docs/content/CLI.md +++ /dev/null @@ -1,27 +0,0 @@ -# Vieter CLI - -I provide a simple CLI tool that currently only allows changing the Git -repository API. Its usage is quite simple. - -First, you need to create a file in your home directory called `.vieterrc` with -the following content: - -```toml -address = "https://example.com" -api_key = "your-api-key" -``` - -You can also use a different file or use environment variables, as described in -[Configuration](/configuration). - -Now you're ready to use the CLI tool. - -## Usage - -* `vieter repos list` returns all repositories currently stored in the API. -* `vieter repos add url branch repo arch...` adds the repository with the given - URL, branch, repo & arch to the API. -* `vieter repos remove id` removes the repository with the given ID prefix. - -You can always check `vieter -help` or `vieter repos -help` for more -information about the commands. diff --git a/docs/content/configuration.md b/docs/content/configuration.md index 27eb213..135c1ff 100644 --- a/docs/content/configuration.md +++ b/docs/content/configuration.md @@ -47,14 +47,6 @@ configuration variable required for each command. * Git repositories added without an `arch` value use this value instead. -### `vieter repos` - -* `api_key`: the API key to use when authenticating requests. -* `address`: Base your URL of your Vieter instance, e.g. https://example.com -* `base_image`: image to use when building a package using `vieter repos - build`. - * Default: `archlinux:base-devel` - ### `vieter cron` * `log_level`: log verbosity level. Value should be one of `FATAL`, `ERROR`, @@ -87,7 +79,8 @@ configuration variable required for each command. how frequently (in minutes) to rebuild this builder image. * Default: `1440` (every 24 hours) * `global_schedule`: build schedule for any Git repository that does not have a - schedule defined. + schedule defined. For information about this syntax, see + [here](/usage/builds/schedule). * Default: `0 3` (3AM every night) ### `vieter logs` @@ -95,17 +88,11 @@ configuration variable required for each command. * `api_key`: the API key to use when authenticating requests. * `address`: Base your URL of your Vieter instance, e.g. https://example.com -### Cron +### `vieter repos` -* `log_level`: defines how much logs to show. Valid values are one of `FATAL`, - `ERROR`, `WARN`, `INFO` or `DEBUG`. Defaults to `WARN` * `api_key`: the API key to use when authenticating requests. -* `address`: Base your URL of your Vieter instance, e.g. https://example.com. - This *must* be the publicly facing URL of your Vieter instance. -* `data_dir`: where Vieter stores the log file. -* `base_image`: Docker image from which to create the builder images. -* `max_concurrent_builds`: amount of builds to run at once. -* `api_update_frequency`: how frequenty to check for changes in the repo list. -* `image_rebuild+frequency`: how frequently to rebuild the builder image -* `global_schedule`: cron schedule to use for any repo without an individual - schedule +* `address`: Base your URL of your Vieter instance, e.g. https://example.com +* `base_image`: image to use when building a package using `vieter repos + build`. + * Default: `archlinux:base-devel` + diff --git a/docs/content/usage/builds/_index.md b/docs/content/usage/builds/_index.md index c6f0603..cd463a4 100644 --- a/docs/content/usage/builds/_index.md +++ b/docs/content/usage/builds/_index.md @@ -38,3 +38,14 @@ The above command intentionally leaves out a few parameters to make the CLI more useable. For information on how to modify all parameters using the CLI, see [vieter-repos-edit(1)](https://rustybever.be/man/vieter/vieter-repos-edit.1.html). + +## Reading logs + +The logs of each build are uploaded to the Vieter repository server, along with +information about the exit code of the build container, when the build +started/ended etc. These logs can then be accessed using the [HTTP +API](https://rustybever.be/docs/vieter/api/). + +For ease of use, the logs are also available using some CLI commands; see +[vieter-logs(1)](https://rustybever.be/man/vieter/vieter-logs.1.html) for more +information. diff --git a/docs/content/usage/usage.md b/docs/content/usage/usage.md deleted file mode 100644 index 06671b4..0000000 --- a/docs/content/usage/usage.md +++ /dev/null @@ -1,54 +0,0 @@ ---- -weight: 30 ---- -# Usage - -## Starting the server - -To start a server, either install it using Docker (see -[Installation](/installation)) or run it locally by executing `vieter -server`. See [Configuration](/configuration) for more information about -configuring the binary. - -## Multiple repositories - -Vieter works with multiple repositories. This means that a single Vieter server -can serve multiple repositories in Pacman. It also automatically divides files -with specific architectures among arch-repos. Arch-repos are the actual -repositories you add to your `/etc/pacman.conf` file. See [Configuring -Pacman](/usage#configuring-pacman) below for more info. - -## Adding packages - -Using Vieter is currently very simple. If you wish to add a package to Vieter, -build it using makepkg & POST that file to the `//publish` endpoint of -your server. This will add the package to the repository. Authentification -requires you to add the API key as the `X-Api-Key` header. - -All of this can be combined into a simple cURL call: - -``` -curl -XPOST -H "X-API-KEY: your-key" -T some-package.pkg.tar.zst https://example.com/somerepo/publish -``` - -`somerepo` is automatically created if it doesn't exist yet. - -## Configuring Pacman - -Configuring Pacman to use a Vieter instance is very simple. In your -`/etc/pacman.conf` file, add the following lines: - -``` -[vieter] -Server = https://example.com/$repo/$arch -SigLevel = Optional -``` - -Here, you see two important placeholder variables. `$repo` is replaced by the -name within the square brackets, which in this case would be `vieter`. `$arch` -is replaced by the output of `uname -m`. Because Vieter supports multiple -repositories & architectures per repository, using this notation makes sure you -always use the correct endpoint for fetching files. - -I recommend placing this below all other repository entries, as the order -decides which repository should be used if there's ever a naming conflict. From c15f4a482faa31d2746731a3ce2caafbfdd55bed Mon Sep 17 00:00:00 2001 From: Jef Roosens Date: Thu, 9 Jun 2022 21:08:13 +0200 Subject: [PATCH 6/7] docs: added in-depth build explanation --- docs/config.toml | 2 +- docs/content/other/builds-in-depth.md | 77 +++++++++++++++++++++++++++ 2 files changed, 78 insertions(+), 1 deletion(-) diff --git a/docs/config.toml b/docs/config.toml index 5c82044..f8e23cd 100644 --- a/docs/config.toml +++ b/docs/config.toml @@ -37,7 +37,7 @@ enableGitInfo = true url = "https://rustybever.be/man/vieter/vieter.1.html" weight = 20 [[menu.after]] - name = "Git" + name = "Vieter" url = "https://git.rustybever.be/vieter/vieter" weight = 30 [[menu.after]] diff --git a/docs/content/other/builds-in-depth.md b/docs/content/other/builds-in-depth.md index a6f0c21..a04acc4 100644 --- a/docs/content/other/builds-in-depth.md +++ b/docs/content/other/builds-in-depth.md @@ -2,3 +2,80 @@ For those interested, this page describes how the build system works internally. + +## Builder image + +Every cron daemon perodically creates a builder image that is then used as a +base for all builds. This is done to prevent build containers having to pull +down a bunch of updates when they update their system. + +The build container is created by running the following commands inside a +container started from the image defined in `base_image`: + +```sh +# Update repos & install required packages +pacman -Syu --needed --noconfirm base-devel git +# Add a non-root user to run makepkg +groupadd -g 1000 builder +useradd -mg builder builder +# Make sure they can use sudo without a password +echo 'builder ALL=(ALL) NOPASSWD: ALL' >> /etc/sudoers +# Create the directory for the builds & make it writeable for the +# build user +mkdir /build +chown -R builder:builder /build +``` + +This script updates the packages to their latest versions & creates a non-root +user to use when running `makepkg`. + +This script is base64-encoded & passed to the container as an environment +variable. The container's entrypoint is set to `/bin/sh -c` & its command +argument to `echo $BUILD_SCRIPT | base64 -d | /bin/sh -e`, with the +`BUILD_SCRIPT` environment variable containing the base64-encoded script. + +Once the container exits, a new Docker image is created from it. This image is +then used as the base for any builds. + +## Running builds + +Each build has its own Docker container, using the builder image as its base. +The same base64-based technique as above is used, just with a different script. +To make the build logs more clear, each command is appended by an echo command +printing the next command to stdout. + +Given the Git repository URL is `https://examplerepo.com` with branch `main`, +the URL of the Vieter server is `https://example.com` and `vieter` is the +repository we wish to publish to, we get the following script: + +```sh +echo -e '+ echo -e '\''[vieter]\\nServer = https://example.com/$repo/$arch\\nSigLevel = Optional'\'' >> /etc/pacman.conf' +echo -e '[vieter]\nServer = https://example.com/$repo/$arch\nSigLevel = Optional' >> /etc/pacman.conf +echo -e '+ pacman -Syu --needed --noconfirm' +pacman -Syu --needed --noconfirm +echo -e '+ su builder' +su builder +echo -e '+ git clone --single-branch --depth 1 --branch main https://examplerepo.com repo' +git clone --single-branch --depth 1 --branch main https://examplerepo.com repo +echo -e '+ cd repo' +cd repo +echo -e '+ makepkg --nobuild --syncdeps --needed --noconfirm' +makepkg --nobuild --syncdeps --needed --noconfirm +echo -e '+ source PKGBUILD' +source PKGBUILD +echo -e '+ curl -s --head --fail https://example.com/vieter/x86_64/$pkgname-$pkgver-$pkgrel && exit 0' +curl -s --head --fail https://example.com/vieter/x86_64/$pkgname-$pkgver-$pkgrel && exit 0 +echo -e '+ [ "$(id -u)" == 0 ] && exit 0' +[ "$(id -u)" == 0 ] && exit 0 +echo -e '+ MAKEFLAGS="-j$(nproc)" makepkg -s --noconfirm --needed && for pkg in $(ls -1 *.pkg*); do curl -XPOST -T "$pkg" -H "X-API-KEY: $API_KEY" https://example.com/vieter/publish; done' +MAKEFLAGS="-j$(nproc)" makepkg -s --noconfirm --needed && for pkg in $(ls -1 *.pkg*); do curl -XPOST -T "$pkg" -H "X-API-KEY: $API_KEY" https://example.com/vieter/publish; done +``` + +This script: + +1. Adds the target repository as a repository in the build container +2. Updates mirrors & packages +3. Clones the Git repository +4. Runs `makepkg` without building to calculate `pkgver` +5. Checks whether the package version is already present on the server +6. If not, run makepkg & publish any generated package archives to the server From ddccceb336aa0bb5b16b41baa1935e5b54323d18 Mon Sep 17 00:00:00 2001 From: Jef Roosens Date: Thu, 9 Jun 2022 21:46:39 +0200 Subject: [PATCH 7/7] docs: removed another old file & read over some parts --- docs/content/builder.md | 56 --------------------------- docs/content/configuration.md | 4 +- docs/content/installation.md | 4 +- docs/content/other/builds-in-depth.md | 2 +- 4 files changed, 5 insertions(+), 61 deletions(-) delete mode 100644 docs/content/builder.md diff --git a/docs/content/builder.md b/docs/content/builder.md deleted file mode 100644 index 659717d..0000000 --- a/docs/content/builder.md +++ /dev/null @@ -1,56 +0,0 @@ -# Builder - -Vieter supports a basic build system that allows you to build the packages -defined using the Git repositories API by running `vieter build`. For -configuration, see [here](/configuration#builder). - -## How it works - -The build system works in two stages. First it pulls down the -`archlinux:latest` image from Docker Hub, runs `pacman -Syu` & configures a -non-root build user. It then creates a new Docker image from this container. -This is to prevent each build having to fully update the container's -repositories. After the image has been created, each repository returned by -`/api/repos` is built sequentially by starting up a new container with the -previously created image as a base. Each container goes through the following steps: - -1. The repository is cloned -2. `makepkg --nobuild --syncdeps --needed --noconfirm` is ran to update the `pkgver` variable inside - the `PKGBUILD` file -3. A HEAD request is sent to the Vieter server to check whether the specific - version of the package is already present. If it is, the container exits. -4. `makepkg` is ran with `MAKEFLAGS="-j\$(nproc)` -5. Each produced package archive is uploaded to the Vieter instance's - repository, as defined in the API for that specific Git repo. - -## Cron image - -The Vieter Docker image contains crond & a cron config that runs `vieter build` -every night at 3AM. This value is currently hardcoded, but I wish to change -that down the line (work is in progress). There's also some other caveats you -should be aware of, namely that the image should be run as root & that the -healthcheck will always fail, so you might have to disable it. This boils down -to the following docker-compose file: - -```yaml -version: '3' - -services: - cron: - image: 'chewingbever/vieter:dev' - command: crond -f - user: root - - healthcheck: - disable: true - - environment: - - 'VIETER_API_KEY=some-key' - - 'VIETER_ADDRESS=https://example.com' - volumes: - - '/var/run/docker.sock:/var/run/docker.sock' -``` - -Important to note is that the container also requires the host's Docker socket -to be mounted as this is how it spawns the necessary containers, as well as a -change to the container's command. diff --git a/docs/content/configuration.md b/docs/content/configuration.md index 135c1ff..600c6f3 100644 --- a/docs/content/configuration.md +++ b/docs/content/configuration.md @@ -86,12 +86,12 @@ configuration variable required for each command. ### `vieter logs` * `api_key`: the API key to use when authenticating requests. -* `address`: Base your URL of your Vieter instance, e.g. https://example.com +* `address`: Base URL of your Vieter instance, e.g. https://example.com ### `vieter repos` * `api_key`: the API key to use when authenticating requests. -* `address`: Base your URL of your Vieter instance, e.g. https://example.com +* `address`: Base URL of your Vieter instance, e.g. https://example.com * `base_image`: image to use when building a package using `vieter repos build`. * Default: `archlinux:base-devel` diff --git a/docs/content/installation.md b/docs/content/installation.md index 2cf57e7..b5bdbaf 100644 --- a/docs/content/installation.md +++ b/docs/content/installation.md @@ -20,8 +20,8 @@ The latter is updated every time a new commit is pushed to the development branch. This branch will be the most up to date, but does not give any guarantees about stability, so beware! -Due to the single-binary design of Vieter, this image can be used both for the -repository server & the cron daemon. +Thanks to the single-binary design of Vieter, this image can be used both for +the repository server & the cron daemon. Below is an example compose file to set up both the repository server & the cron daemon: diff --git a/docs/content/other/builds-in-depth.md b/docs/content/other/builds-in-depth.md index a04acc4..d8df6ec 100644 --- a/docs/content/other/builds-in-depth.md +++ b/docs/content/other/builds-in-depth.md @@ -78,4 +78,4 @@ This script: 3. Clones the Git repository 4. Runs `makepkg` without building to calculate `pkgver` 5. Checks whether the package version is already present on the server -6. If not, run makepkg & publish any generated package archives to the server +6. If not, run `makepkg` & publish any generated package archives to the server