Scraper example using bible.com as the source.
Steve Thielemann
9371e904d8
Working walker. Change to struct to clean it up.
|
11 hónapja | |
|---|---|---|
| src | 11 hónapja | |
| .gitignore | 11 hónapja | |
| Cargo.lock | 11 hónapja | |
| Cargo.toml | 11 hónapja | |
| README.md | 11 hónapja | |
| fetch1.html | 11 hónapja | |
| initial.config | 11 hónapja |
README.md
Rust - Reqwest and Scraper example
This is an example web scraper project. It downloads and extracts the Bible from https://www.bible.com.
User-Agent
The program will update the user-agent string it uses via https://www.mozilla.org/en-US/firefox/releases/ because I can.
-a or --agent-update
Bible versions
- ESV
- NIV
- YLT98
- KJV
- NASB2020
- MKJV
-v NIV
Relative/absolute URLs
See relative_to_absolute(url: &str, href: &str) -> Result to properly handle those.
Using
- Copy initial.config to app.config.
- Make a bible directory.
- Run the fetch command.
- Run the extract command. Try the --output option.
- Run the verse command.