Chewing_Bever
/
fej
1
0
Fork 0
Code Issues 28 Pull Requests 10 Projects 3 Releases 4 Wiki Activity

Curse: scraping curseforge.com #38

New Issue
Open
opened 2021-04-22 12:08:25 +02:00 by Jef Roosens · 0 comments
Jef Roosens commented 2021-04-22 12:08:25 +02:00
Owner

In order to scrape their site, we'll need to get past the Cloudflare protection. We can either find a way to solve the challenges, or we can use something like Selenium to mimic a browser.

Selenium-related packages:

In order to scrape their site, we'll need to get past the Cloudflare protection. We can either find a way to solve the challenges, or we can use something like Selenium to mimic a browser. Selenium-related packages: * [Fantoccini](https://github.com/jonhoo/fantoccini) * [thirtyfour](https://crates.io/crates/thirtyfour)
Jef Roosens added this to the Fej 1.1.1 milestone 2021-04-22 12:08:25 +02:00
Jef Roosens added the
Brainstorm
 label 2021-04-22 12:08:25 +02:00
Jef Roosens self-assigned this 2021-04-22 12:08:25 +02:00
Jef Roosens added this to the Fej 1.1.1 project 2021-04-22 12:09:54 +02:00
Jef Roosens added the
project/backend
 label 2021-04-29 09:28:52 +02:00
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
renovate/reqwest-0.x
renovate/serde-1.x
renovate/eslint-plugin-vue-8.x
renovate/vitejs-plugin-vue-2.x
renovate/typescript-eslint-parser-5.x
renovate/typescript-eslint-eslint-plugin-5.x
renovate/vue-tsc-0.x
renovate/chrono-tz-0.x
develop
renovate/eslint-plugin-promise-6.x
renovate/alpine-3.x
68-clippy
master
v1.0.2
v1.0.1
v1.0.0
v0.1.0
Labels
Clear labels   
Brainstorm

Place to brainstorm problems or possible ideas   
Parent

Description of larger problem with multiple issues   
bug

Something is not working   
duplicate

This issue or pull request already exists   
enhancement

New feature   
help wanted

Need some help   
idea

A possible idea for a feature   
invalid

Something is wrong   
project/backend

An issue related to the backend   
project/frontend

An issue related to the frontend   
question

More information is needed   
wontfix

This won't be fixed
No Label
Brainstorm
Parent
bug
duplicate
enhancement
help wanted
idea
invalid
project/backend
project/frontend
question
wontfix
Milestone
Clear milestone
No items
No Milestone
Fej 1.1.1
Projects
Clear projects
No project
Fej 1.1.1
Assignees
Clear assignees
No Assignees
Jef Roosens
1 Participants
Notifications
Due Date
The due date is invalid or out of range. Please use the format 'yyyy-mm-dd'.

No due date set.

Dependencies

No dependencies set.

Reference: Chewing_Bever/fej#38
There is no content yet.
Delete Branch "%!s(<nil>)"

Deleting a branch is permanent. Although the deleted branch may exist for a short time before cleaning up, in most cases it CANNOT be undone. Continue?