Chewing_Bever/fej
Archived
1
0
Fork 0
Code Issues 28 Pull requests 10 Projects 3 Releases 4 Wiki Activity

Upgrading the Ivago scraper #41

New issue
Open
opened 2021-04-22 12:42:51 +02:00 by Jef Roosens · 0 comments
Jef Roosens commented 2021-04-22 12:42:51 +02:00
Owner
Copy link

General improvements

  • Switch to async functions
  • Run requests to search api concurrently

Support for scraping calendar

  • Command line argument

The scraper currently always scrapes the Search API, but we need support for scraping the calendar API as well. To fix this, we can add a command line argument that specifies which scraping job it should perform, e.g. ivago_scrape search or ivago_scrape calendar. This can then be used in the cronjobs to specify what task to perform.

## General improvements - [ ] Switch to async functions - [ ] Run requests to search api concurrently ## Support for scraping calendar - [ ] Command line argument The scraper currently always scrapes the Search API, but we need support for scraping the calendar API as well. To fix this, we can add a command line argument that specifies which scraping job it should perform, e.g. `ivago_scrape search` or `ivago_scrape calendar`. This can then be used in the cronjobs to specify what task to perform.
Jef Roosens added this to the Fej 1.1.0 - The Calm before the Storm milestone 2021-04-22 12:42:51 +02:00
Jef Roosens added the
enhancement
 label 2021-04-22 12:42:51 +02:00
Jef Roosens self-assigned this 2021-04-22 12:42:51 +02:00
Jef Roosens added this to the Fej 1.1.0 project 2021-04-22 12:42:51 +02:00
Jef Roosens added the
project/backend
 label 2021-04-29 09:28:52 +02:00
This repository is archived. You cannot comment on issues.
No Branch/Tag specified
Branches Tags
develop
renovate/reqwest-0.x
renovate/serde-1.x
renovate/eslint-plugin-vue-8.x
renovate/vitejs-plugin-vue-2.x
renovate/typescript-eslint-parser-5.x
renovate/typescript-eslint-eslint-plugin-5.x
renovate/vue-tsc-0.x
renovate/chrono-tz-0.x
renovate/eslint-plugin-promise-6.x
renovate/alpine-3.x
68-clippy
master
v1.0.2
v1.0.1
v1.0.0
v0.1.0
Labels
Clear labels   
Brainstorm

Place to brainstorm problems or possible ideas

  
Parent

Description of larger problem with multiple issues

  
bug

Something is not working

  
duplicate

This issue or pull request already exists

  
enhancement

New feature

  
help wanted

Need some help

  
idea

A possible idea for a feature

  
invalid

Something is wrong

  
project/backend

An issue related to the backend

  
project/frontend

An issue related to the frontend

  
question

More information is needed

  
wontfix

This won't be fixed

No labels
Brainstorm
Parent
bug
duplicate
enhancement
help wanted
idea
invalid
project/backend
project/frontend
question
wontfix
Milestone
Clear milestone
No items
No milestone
Fej 1.1.0 - The Calm before the Storm
Projects
Clear projects
No items
No project
Fej 1.1.0
Assignees
Clear assignees
No assignees
Jef Roosens
1 participant
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference: Chewing_Bever/fej#41
No description provided.
Delete branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?