Back to Home

Textharvester

A threaded Python crawler for building NLP text corpora. Depth-first from seed URLs, domain whitelisting, dumps URLs and page text to disk.

PythonBeautifulSoupMulti-threadingNLPPyPI

Overview

Textharvester is a two-stage crawler I wrote in 2020 when I wanted a text corpus and didn't like the shape of what was on PyPI.

Stage one collects URLs from a set of seeds using a depth-first walk: pull the links on the seed pages, pick a random subset of those, repeat. Stage two takes the URL list and downloads page text.

Config knobs

  • starturls - one seed or many
  • forcedomains + alloweddomainslist - domain whitelist toggle
  • limittotal - cap on how many URLs the crawl phase collects
  • threads - I ran 50 for the crawler and 25 for the downloader

Output is two files: one URL per line, one page's text per entry.

crawler = th.TextHarvester(starturls="https://startsite.com", limittotal=80)
crawler.harvest(threads=50, write="outfile_urls.txt")

Notes

  • Shipped on PyPI. Windows needs the crawler initialized inside the main loop; that was a threading model quirk I never got around to fixing cleanly.
  • Depth-first + random link sampling was a deliberate choice over BFS: I wanted variety in the corpus rather than a shallow slice of one site.

Links