Chewing_Bever/fej
Archived
1
0
Fork 0
Code Issues 28 Pull requests 10 Projects 3 Releases 4 Wiki Activity

Ivago: scraper to populate search cache #27

Open
opened 2021-04-16 09:36:55 +02:00 by Jef Roosens · 1 comment
Jef Roosens commented 2021-04-16 09:36:55 +02:00
Owner
Copy link

Before we can switch to the database for requests, we need to write a helper program that can populate said database.

  • Convert Street struct to Diesel-compatible struct
  • Add necessary tables to migrations
  • Figure out how to remove outdated entries from the database (#34)
    Idea: Diesel keeps track of when an entry was last updated, so maybe we can use this to clear out any outdated entries after updating the database
Before we can switch to the database for requests, we need to write a helper program that can populate said database. - [x] Convert `Street` struct to Diesel-compatible struct - [x] Add necessary tables to migrations - [ ] Figure out how to remove outdated entries from the database (#34) Idea: Diesel keeps track of when an entry was last updated, so maybe we can use this to clear out any outdated entries after updating the database
Jef Roosens commented 2021-04-22 12:46:36 +02:00
Author
Owner
Copy link

The best method will probably be to keep track of the last time a street name was inserted/update:

  • Auto-refresh the timestamp column when inserting/updating the table using a trigger
  • Each time the scraper is run, we delete any entries that are too old after the refresh
The best method will probably be to keep track of the last time a street name was inserted/update: * Auto-refresh the timestamp column when inserting/updating the table using a trigger * Each time the scraper is run, we delete any entries that are too old after the refresh
Jef Roosens changed title from Ivago: scraper to populate cache to Ivago: scraper to populate search cache 2021-04-22 12:49:11 +02:00
Commenting is not possible because the repository is archived.
No Branch/Tag specified
Branches Tags
develop
renovate/reqwest-0.x
renovate/serde-1.x
renovate/eslint-plugin-vue-8.x
renovate/vitejs-plugin-vue-2.x
renovate/typescript-eslint-parser-5.x
renovate/typescript-eslint-eslint-plugin-5.x
renovate/vue-tsc-0.x
renovate/chrono-tz-0.x
renovate/eslint-plugin-promise-6.x
renovate/alpine-3.x
68-clippy
master
v1.0.2
v1.0.1
v1.0.0
v0.1.0
Labels
Clear labels   
Brainstorm
Place to brainstorm problems or possible ideas

  
Parent
Description of larger problem with multiple issues

  
bug
Something is not working

  
duplicate
This issue or pull request already exists

  
enhancement
New feature

  
help wanted
Need some help

  
idea
A possible idea for a feature

  
invalid
Something is wrong

  
project/backend
An issue related to the backend

  
project/frontend
An issue related to the frontend

  
question
More information is needed

  
wontfix
This won't be fixed

No labels
Brainstorm
Parent
bug
duplicate
enhancement
help wanted
idea
invalid
project/backend
project/frontend
question
wontfix
Milestone
Clear milestone
No items
Projects
Clear projects
No items
No project
Fej 1.1.0
Assignees
Clear assignees
No assignees
Jef Roosens
1 participant
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Depends on
#34 Ivago: make populate_ivago remove deleted entries
Chewing_Bever/fej
Reference
Chewing_Bever/fej#27
No description provided.