Upgrading the Ivago scraper #41

New Issue

Jef Roosens · 2021-04-22T12:42:51+02:00

Jef Roosens commented

2021-04-22 12:42:51 +02:00

General improvements

Switch to async functions
Run requests to search api concurrently

Support for scraping calendar

Command line argument

The scraper currently always scrapes the Search API, but we need support for scraping the calendar API as well. To fix this, we can add a command line argument that specifies which scraping job it should perform, e.g. ivago_scrape search or ivago_scrape calendar. This can then be used in the cronjobs to specify what task to perform.

## General improvements - [ ] Switch to async functions - [ ] Run requests to search api concurrently ## Support for scraping calendar - [ ] Command line argument The scraper currently always scrapes the Search API, but we need support for scraping the calendar API as well. To fix this, we can add a command line argument that specifies which scraping job it should perform, e.g. `ivago_scrape search` or `ivago_scrape calendar`. This can then be used in the cronjobs to specify what task to perform.

Jef Roosens added this to the Fej 1.1.0 - The Calm before the Storm milestone 2021-04-22 12:42:51 +02:00

Jef Roosens added the

enhancement

label 2021-04-22 12:42:51 +02:00

Jef Roosens self-assigned this 2021-04-22 12:42:51 +02:00

Jef Roosens added this to the Fej 1.1.0 project 2021-04-22 12:42:51 +02:00

Jef Roosens added the

project/backend

label 2021-04-29 09:28:52 +02:00

Sign in to join this conversation.