Package download system #7

New issue

Open

opened 2024-07-01 22:25:05 +02:00 by Jef Roosens · 0 comments

Jef Roosens commented

2024-07-01 22:25:05 +02:00

Owner

The mirroring system needs to download lots of packages, so we need some sort of queueing system.

For performing the downloads, we can use Hyper, as we already import it for Axum anyways. The download queue will therefore run inside the asynchronous runtime.

The straightforward solution would be to use a channel. The download queue could contain thousands of packages at once for a busy server, and these downloads can take a while to complete. Therefore it might be better to back up the queueing system with the database. A simple table containing (repo id, file path) tuples should suffice. Each download agent would take one row at a time (synchronised using a mutex probably), download the package, and place it in the parse queue. We could perhaps use a Notify construct to wake up the agents, so they don't need to poll the database. Once awoken, the agents will keep pulling rows from the database until there are none before going back to sleep.

The amount of concurrent downloads should be configurable using a simple config option that specifies how many download actors to spawn.

The mirroring system needs to download lots of packages, so we need some sort of queueing system. For performing the downloads, we can use [Hyper](https://docs.rs/hyper/latest/hyper/), as we already import it for Axum anyways. The download queue will therefore run inside the asynchronous runtime. The straightforward solution would be to use a channel. The download queue could contain thousands of packages at once for a busy server, and these downloads can take a while to complete. Therefore it might be better to back up the queueing system with the database. A simple table containing `(repo id, file path)` tuples should suffice. Each download agent would take one row at a time (synchronised using a mutex probably), download the package, and place it in the parse queue. We could perhaps use a `Notify` construct to wake up the agents, so they don't need to poll the database. Once awoken, the agents will keep pulling rows from the database until there are none before going back to sleep. The amount of concurrent downloads should be configurable using a simple config option that specifies how many download actors to spawn.