otter: devlog-1

content/dev/otter/_index.md

@@ -0,0 +1,9 @@
+---
+title: "Otter"
+summary: "Lightweight implementation of the Gpodder API"
+type: "project"
+params:
+    links:
+        - name: Source
+          url: 'https://git.rustybever.be/Chewing_Bever/otter'
+---

content/dev/otter/devlog-1.md

@@ -0,0 +1,236 @@
+---
+title: "Gpodder, Rust and Domain Driven Design"
+date: 2025-03-15T12:55:31+01:00
+---
+
+Recently I've gotten into listening to podcasts more, notably the
+[Severed](https://open.spotify.com/show/3yJnUXzFvX4p4QlsfVPuyZ) podcast going
+in depth on every Severance episode. I use Kasts on my computers and AntennaPod
+on my phone to listen, and I looked into how I could sync my subscriptions
+between my devices. This search ended with [Gpodder.net](https://gpodder.net/),
+a free and open-source web service that's supported by both AntennaPod and
+Kasts! The server is also self-hostable, but it's quite the behemoth written in
+Python, and I felt like starting another project, so I decided to implement my
+own version of the API instead. That project is Otter.
+
+## The Gpodder API
+
+The API documentation for Gpodder is available on [Read The
+Docs](https://gpoddernet.readthedocs.io/en/latest/). The documentation is okay,
+but it lacks a lot of important information about how the API should behave. My
+development process for this project consisted of trying to implement routes
+via the documentation before testing it using Kasts. This took a lot of trial
+and error, but I ended up getting most of the routes working the way most
+clients expect (I hope).
+
+Currently Otter supports all routes required for clients to synchronize
+subscriptions and play information for episodes. I've based some of my work on
+[Opodsync](https://github.com/kd2org/opodsync), another implementation of the
+same idea, written in PHP.
+
+## Domain Driven Design
+
+Usually the way I develop projects is very chaotic. I try out different things
+to see what sticks and often end up with very interconnected and difficult to
+separate components that join together into a big mess. Usually this does clean
+up after some time, but I still have a tendency to develop too tightly
+connected components. One example of this would be having ORM logic in web
+route handlers, or jumping through hoops to be able to use the same structs for
+the database operations and the web representation. For Otter, I took a step
+back for once.
+
+I took some inspiration from domain driven design and clean architecture to
+structure my codebase. The idea (at least, the way I understand/interpret it)
+is to work with abstractions that cleanly separate various components. The
+first step was designing said abstraction. I introduced the concept of a
+"Gpodder repository", which is an object that provides all the necessary
+methods required to function as the backend data store for a Gpodder server.
+This repository is defined as a collection of traits:
+
+```rust
+pub trait EpisodeActionRepository {
+    /// Insert the given episode actions into the datastore.
+    fn add_episode_actions(&self, user: &User, actions: Vec<EpisodeAction>)
+        -> Result<i64, AuthErr>;
+
+    /// Retrieve the list of episode actions for the given user.
+    fn episode_actions_for_user(
+        &self,
+        user: &User,
+        since: Option<i64>,
+        podcast: Option<String>,
+        device: Option<String>,
+        aggregated: bool,
+    ) -> Result<(i64, Vec<EpisodeAction>), AuthErr>;
+}
+```
+
+For example, the `EpisodeActionRepository` provides the methods required to
+serve the Episode Actions API section of the API documentation.
+
+```rust
+async fn get_episode_actions(
+    State(ctx): State<Context>,
+    Path(username): Path<StringWithFormat>,
+    Extension(user): Extension<gpodder::User>,
+    Query(filter): Query<FilterQuery>,
+) -> AppResult<Json<EpisodeActionsResponse>> {
+    if username.format != Format::Json {
+        return Err(AppError::NotFound);
+    }
+
+    if *username != user.username {
+        return Err(AppError::BadRequest);
+    }
+
+    Ok(tokio::task::spawn_blocking(move || {
+        ctx.repo.episode_actions_for_user(
+            &user,
+            filter.since,
+            filter.podcast,
+            filter.device,
+            filter.aggregated,
+        )
+    })
+    .await
+    .unwrap()
+    .map(|(timestamp, actions)| Json(EpisodeActionsResponse { timestamp, actions }))?)
+}
+```
+
+In our web layer (here I'm using Axum), all that needs to be done is to call
+the respective function on the repository and convert it to the correct
+representation format, as required by the API.
+
+This complete separation of concerns makes development much less mentally
+taxing in my opinion, as each component can be viewed on its own. The
+repository abstraction defines models and methods required for a representation
+layer (in this case, the web server) to access the various parts of the
+repository without having to know anything about the internal workings. The
+current implementation of the repository works using Sqlite, but thanks to the
+abstraction, I could add a Postgres implementation as well by simply
+reimplementing the abstraction. Rust's type system adds to the strength of this
+design pattern by providing strong abstraction primitives via traits.
+
+## Configuration
+
+One other fun thing I'd like to mention is how Otter handles configuration
+variables. In my eyes, the ideal application supports configuration as a
+combination of a config file, environment variables and CLI arguments, with
+environment variables overwriting config file variables, and CLI arguments
+taking precedence above all. Using [Serde](https://serde.rs/),
+[Clap](https://github.com/clap-rs/clap) and
+[Figment](https://github.com/SergioBenitez/Figment), this can be done in a very
+clean way.
+
+First, we define our configuration as a struct. This is both the expected
+format of the configuration file, and the final configuration struct that we
+wish to end up with:
+
+```rust
+#[derive(Deserialize)]
+pub struct Config {
+    #[serde(default = "default_data_dir")]
+    pub data_dir: PathBuf,
+    #[serde(default = "default_domain")]
+    pub domain: String,
+    #[serde(default = "default_port")]
+    pub port: u16,
+}
+
+fn default_data_dir() -> PathBuf {
+    PathBuf::from("./data")
+}
+
+fn default_domain() -> String {
+    "127.0.0.1".to_string()
+}
+
+fn default_port() -> u16 {
+    8080
+}
+```
+
+Important here is the `Deserialize` derive. The Figment library heavily relies
+on Serde to function, and a `Deserialize` implementation is required for our
+final configuration to be deserialized into this struct. We also define
+sensible defaults where applicable.
+
+The second step is defining our CLI arguments using Clap:
+
+```rust
+#[derive(Serialize, Args, Clone)]
+pub struct ClapConfig {
+    #[arg(
+        short,
+        long = "config",
+        env = "OTTER_CONFIG_FILE",
+        value_name = "CONFIG_FILE"
+    )]
+    config_file: Option<PathBuf>,
+
+    #[serde(skip_serializing_if = "Option::is_none")]
+    #[arg(long = "data", value_name = "DATA_DIR")]
+    data_dir: Option<PathBuf>,
+
+    #[serde(skip_serializing_if = "Option::is_none")]
+    #[arg(short, long, value_name = "DOMAIN")]
+    domain: Option<String>,
+
+    #[serde(skip_serializing_if = "Option::is_none")]
+    #[arg(short, long, value_name = "PORT")]
+    port: Option<u16>,
+}
+```
+
+Here we see two things of note. First, the `ClapConfig` implements `Serialize`,
+not `Deserialize`. This is because we'll be using the `Serialize`
+implementation later when combining the configuration sources. We also add a
+`#[serde(skip_serializing_if = "Option::is_none")]` attribute to each variable
+that can be overwritten. This tells Serde to ignore the variable altogether if
+its value is `None` as a workaround for Figment not being able to ignore `None`
+values. Important here is how the variables names in the `ClapConfig` struct
+are the same as those in the `Config` struct. This is required for Figment to
+be able to combine these configuration sources.
+
+Finally, we combine everything using Figment:
+
+```rust
+let mut figment = Figment::new();
+
+if let Some(config_path) = &self.config.config_file {
+    figment = figment.merge(Toml::file(config_path));
+}
+
+let config: crate::config::Config = figment
+    .merge(Env::prefixed("OTTER_"))
+    .merge(Serialized::defaults(self.config.clone()))
+    .extract().unwrap();
+```
+
+We start by defining an empty figment. In the Figment library, a "figment" is a
+combination of configuration providers that can be deserialized using Serde. A
+provider is an object that can provide configuration variables from some
+source, be it a TOML file or environment variables. Figments can be combined
+with providers in various ways. Here, we make use of "merge", which overwrites
+any existing configuration variables with the values received from the new
+provider.
+
+We merge the empty figment with the `Toml` provider, which reads a TOML file
+for configuration variables. Then, we merge again, first with the
+`Env::prefixed("OTTER_")` provider, and then with the
+`Serialized::defaults(self.config.clone())` provider. The `Env` provider reads
+variables from environment variables, while `Serialized` provides variables by
+serializing a struct, in this case our `ClapConfig` struct (hence why we need a
+`Serialize` implementation). The result is a figment consisting of variables
+from the config file, environment variables and CLI arguments. This can be
+deserialized into our `Config` struct using `extract`.
+
+In my opinion, all of this combines into a very clean configuration system that
+allows configuration variables from all convenient sources, without sacrificing
+flexibility or resorting to boilerplate code.
+
+I hope to deploy a first build of Otter on my own servers in the near future to
+test it and work out any kinks. Afterwards, I'll announce a `0.1` release!
+
+Thanks for reading.