Startsida Utforska Hjälp
Logga in
stevet
/
rust-scraper1
1
0
Fork 0
Filer Ärenden 0 Pull-förfrågningar 0 Wiki

Scraper example using bible.com as the source.

Träd: 15fee3b1fc
Grenar Taggar
rust-scraper1
ZIP TAR.GZ
Steve Thielemann 15fee3b1fc Fixed relative absolute url call. 11 månader sedan
src 15fee3b1fc Fixed relative absolute url call. 11 månader sedan
.gitignore f6790cbbbc Making a single request to httpbin.org. 1 år sedan
Cargo.lock 507c90360f Added user agent updater. 11 månader sedan
Cargo.toml 507c90360f Added user agent updater. 11 månader sedan
README.md 736758e4bd Added extract --output for JSON output. 11 månader sedan
fetch1.html 060db0ff68 Initial working parser for verses. 1 år sedan
initial.config 507c90360f Added user agent updater. 11 månader sedan

README.md

Rust - Reqwest and Scraper example

This is an example web scraper project. It downloads and extracts the Bible from https://www.bible.com.

User-Agent

The program will update the user-agent string it uses via https://www.mozilla.org/en-US/firefox/releases/ because I can.

-a or --agent-update

Bible versions

  • ESV
  • NIV
  • YLT98
  • KJV
  • NASB2020
  • MKJV

-v NIV

Relative/absolute URLs

See relative_to_absolute(url: &str, href: &str) -> Result to properly handle those.

Using

  • Copy initial.config to app.config.
  • Make a bible directory.
  • Run the fetch command.
  • Run the extract command. Try the --output option.
  • Run the verse command.