Ivago: scraper to populate search cache #27

New issue

Open

opened 2021-04-16 09:36:55 +02:00 by Jef Roosens · 1 comment

Jef Roosens commented

2021-04-16 09:36:55 +02:00

Owner

Before we can switch to the database for requests, we need to write a helper program that can populate said database.

Convert Street struct to Diesel-compatible struct
Add necessary tables to migrations
Figure out how to remove outdated entries from the database (#34)
Idea: Diesel keeps track of when an entry was last updated, so maybe we can use this to clear out any outdated entries after updating the database

Before we can switch to the database for requests, we need to write a helper program that can populate said database. - [x] Convert `Street` struct to Diesel-compatible struct - [x] Add necessary tables to migrations - [ ] Figure out how to remove outdated entries from the database (#34) Idea: Diesel keeps track of when an entry was last updated, so maybe we can use this to clear out any outdated entries after updating the database

Jef Roosens added this to the Fej 1.2.0 - Curse of the Forge milestone 2021-04-16 09:36:55 +02:00

Jef Roosens added the

enhancement

label 2021-04-16 09:36:56 +02:00

Jef Roosens self-assigned this 2021-04-16 09:36:56 +02:00

Jef Roosens added this to the Fej 1.2 - Curse of the Forge project 2021-04-16 09:36:56 +02:00

Jef Roosens added a new dependency 2021-04-16 09:37:58 +02:00

#28 Ivago: make Ivago lib database-compatible

Jef Roosens modified the milestone from Fej 1.2.0 - Curse of the Forge to Fej 1.1.0 - The Calm before the Storm

2021-04-17 09:42:28 +02:00

Jef Roosens modified the project from Fej 1.2 - Curse of the Forge to Fej 1.1.0

2021-04-17 11:05:36 +02:00

Jef Roosens removed a dependency 2021-04-17 23:22:12 +02:00

#28 Ivago: make Ivago lib database-compatible

Jef Roosens commented

2021-04-22 12:46:36 +02:00

Author

Owner

The best method will probably be to keep track of the last time a street name was inserted/update:

Auto-refresh the timestamp column when inserting/updating the table using a trigger
Each time the scraper is run, we delete any entries that are too old after the refresh

The best method will probably be to keep track of the last time a street name was inserted/update: * Auto-refresh the timestamp column when inserting/updating the table using a trigger * Each time the scraper is run, we delete any entries that are too old after the refresh

Jef Roosens added a new dependency 2021-04-22 12:48:59 +02:00

#34 Ivago: make populate_ivago remove deleted entries

Jef Roosens changed title from ~~Ivago: scraper to populate cache~~ to Ivago: scraper to populate search cache

2021-04-22 12:49:11 +02:00

Jef Roosens added

Parent

and removed

enhancement

labels 2021-04-22 12:51:42 +02:00

Jef Roosens added the

project/backend

label 2021-04-29 09:28:52 +02:00