JobFinder — Developer Guide

Local development setup

The repository uses a src/ layout, so an editable install is recommended — it puts the jobfinder package on your path so module commands work from a fresh clone.

# create an environment (conda shown; a venv works too)
conda create -n JobFinder python=3.14 -y
conda activate JobFinder
python -m pip install --upgrade pip

# install the package and the dev tools
python -m pip install -e .
python -m pip install -r requirements-dev.txt

# create your local config
cp .env.example .env

After the editable install, these console scripts are available:

jobfinder-pipeline --help
jobfinder-scrape --help
jobfinder-evaluate --help

For CV PDF generation you also need LaTeX (latexmk + xelatex). On Ubuntu: sudo apt-get install -y latexmk texlive-xetex texlive-latex-extra. On macOS, install MacTeX.

Project structure

The package lives under src/jobfinder; thin root scripts keep older commands working.

JobFinder/
├── .github/workflows/      # ci.yml (tests/lint) + jobs.yml (production pipeline)
├── configs/                # filters.json + example keywords
├── cv/                     # example LaTeX CV (your real CV stays private)
├── prompts/                # example evaluator prompt (private in real use)
├── scripts/                # thin compatibility wrappers
├── src/jobfinder/          # the package
│   ├── core/               # cross-cutting helpers (logging)
│   ├── providers/          # board adapters, Apify client, registry
│   ├── scraper/            # search, filters, exports, run history
│   ├── dedupe/             # deterministic duplicate matching + merge
│   ├── evaluator/          # OpenAI evaluation, parsing, storage, PDFs
│   ├── spreadsheet/        # shared column contracts
│   ├── pipeline/           # multi-step CLI and preflight
│   ├── operations/         # CI report helpers
│   └── integrations/google/# Google credentials, Sheets, Drive
├── tests/                  # pytest suite (no live network calls)
├── job_fit_evaluator.py    # wrapper → jobfinder.evaluator.cli
├── linkedin_job_scraper.py # wrapper → jobfinder.scraper.cli
└── run_job_pipeline.py     # wrapper → jobfinder.pipeline.cli

Each package documents itself in its own README.md. Start with src/jobfinder/README.md.

Architecture overview

JobFinder is a data pipeline split into focused modules with clear ownership boundaries.

Module	Responsibility
`providers/`	Build actor payloads, normalize board output, register provider adapters.
`scraper/`	Build searches, run Apify concurrently, apply final filters, export, track run history.
`dedupe/`	Deterministic cross-board duplicate matching and canonical merging (no AI).
`evaluator/`	Build prompts, call OpenAI, parse responses, save results, generate CV PDFs.
`spreadsheet/`	The shared column contract used by scraper and evaluator.
`pipeline/`	Run scraper then evaluator in one command, plus preflight checks.
`integrations/google/`	OAuth credentials, Google Sheets, and Google Drive adapters.

For the full flow (scraping → dedupe → history → evaluation) read How it works, and for design direction read the Architecture notes.

Testing & checks

The suite avoids real Apify, Google and OpenAI calls by using fakes and monkeypatching. Run the same checks as CI before you commit:

python -m ruff check .
python -m ruff format --check .
python -m mypy src
python -m pytest

Useful focused runs:

python -m pytest tests/test_scraper_search.py
python -m pytest tests/test_dedupe_matching.py
python -m pytest tests/test_evaluator_storage.py

Extending & contributing

Common changes and where they belong:

Change	Start here
Add a job board	`providers/`, `scraper/search.py`, `scraper/settings.py`, provider tests
Change output columns	`spreadsheet/schema.py`, exporters, evaluator parsing/storage, docs, tests
Tune dedupe identity	`dedupe/normalize.py`, `dedupe/scoring.py`, `dedupe/matching.py`
Change evaluator parsing	`evaluator/parsing.py`, `evaluator/models.py`
Change production scheduling	`.github/workflows/jobs.yml`

New providers register a ProviderAdapter in providers/registry.py. Column changes start in spreadsheet/schema.py and flow into exporters, evaluator, tests and docs together.

Read the contributing notes and the LICENSE (contributions are accepted under its terms) before opening a pull request.

Open the full developer guide →