Startsida Utforska Hjälp
Logga in
stevet
/
rust-scraper1
1
0
Fork 0
Filer Ärenden 0 Pull-förfrågningar 0 Wiki

Scraper example using bible.com as the source.

Träd: 9371e904d8
Grenar Taggar
rust-scraper1
ZIP TAR.GZ
Steve Thielemann 9371e904d8 Working walker. Change to struct to clean it up. 11 månader sedan
src 9371e904d8 Working walker. Change to struct to clean it up. 11 månader sedan
.gitignore f6790cbbbc Making a single request to httpbin.org. 11 månader sedan
Cargo.lock 507c90360f Added user agent updater. 11 månader sedan
Cargo.toml 507c90360f Added user agent updater. 11 månader sedan
README.md 736758e4bd Added extract --output for JSON output. 11 månader sedan
fetch1.html 060db0ff68 Initial working parser for verses. 11 månader sedan
initial.config 507c90360f Added user agent updater. 11 månader sedan

README.md

Rust - Reqwest and Scraper example

This is an example web scraper project. It downloads and extracts the Bible from https://www.bible.com.

User-Agent

The program will update the user-agent string it uses via https://www.mozilla.org/en-US/firefox/releases/ because I can.

-a or --agent-update

Bible versions

  • ESV
  • NIV
  • YLT98
  • KJV
  • NASB2020
  • MKJV

-v NIV

Relative/absolute URLs

See relative_to_absolute(url: &str, href: &str) -> Result to properly handle those.

Using

  • Copy initial.config to app.config.
  • Make a bible directory.
  • Run the fetch command.
  • Run the extract command. Try the --output option.
  • Run the verse command.