hilde/ROADMAP.md

# Roadmap

This file describes a broad roadmap of what's currently planned with the
project.

## Readable repository

The first goal is to make the server a fully functional Debian package
repository. This includes:

### Setting up a database

Hilde stores an index of the stored deb files in a PostgreSQL database. Any
data that can be calculated beforehand and/or stored efficiently inside a
database (e.g. hashes, package names & versions, descriptions etc.) will be
stored there. Any request that isn't a deb file will be queried and/or
generated from this database. For example, the `Packages` file can be
generated on the fly instead of storing it on disk. The goal is for the data
storage to only include deb files.

### Indexing the directory on startup

On startup, Hilde will scan the packages directory for deb files. These will
then all be parsed, and synced with the database.

### Functioning as a repository

Using this database backend, the server should fully mimic a Debian repository
server ([an example](http://ftp.debian.org/debian/)). As there's no real reason
not to, we should implement the
[repository spec](https://wiki.debian.org/DebianRepository) as complete as
possible.

## The ideas

Now that we have a repository that we can use, it's time to add some extra
functionality.

### Uploading versions

This is the initial reason I wanted to start this project. My goal is to add a
way to send POST requests to the server containing new versions of a new or
existing package. These packages can be from any distribution, architecture, or
name, and the server will just dynamically create the directories n stuff when
needed. This feature can then be used in CI pipelines to simplify uploading
versions to a repository.

### User management & JWT tokens

Considering we can't just make an upload endpoint publicly available, we'll
need a system to log in. There will be an admin user that can do everything,
but I want to add a detailed user management system. This system will have
fine-grained controls over what user can do what, and each user is able to
create JWT tokens that further narrows down the functionality. For example, a
user could have read & write access to the stable distribution's updates
section. Then they can create a token that only allows to write a certain
package, and use it in their CI pipeline for that particular package.

### Federation

I think the idea of being able to spread a service across multiple hosts is
really cool, so I've thought about adding this to Hilde as well. I have never
created anything like this before, but I think the idea is really cool. My
current approach would be as follows:

The nodes communicate using a common secret key. Each node only requires the
location of a single other node. When a new node connects to the network, it
asks the node it's been given about which nodes it knows. These nodes are then
added to the node's list of known nodes. These new nodes are then also
notified, and this process is repeated until all nodes are discovered. Each
node also keeps an index of which packages can be found on which node. This
makes it so that every node can be used as an entrypoint to the network, making
it easy to load balance.

Each node could maybe send a notification to all the other nodes when it
discovers/loses a node, or receives a new package.

When uploading a new package, we could query the system to see which node has
the most space left, as to more efficiently distribute the packages, if
possible.

### Mirroring repositories

It might be useful to add functionality to mirror an existing repo. This could
for example make a home setup fully self-sufficient, by mirroring the official
Debian repositories.