Download the PHP package wiserwebsolutions/laravel-boarddocs-scraper without Composer

Laravel BoardDocs Scraper

AI-authored notice The initial commit of this project was made by an AI assistant that was asked to convert specific components of a private Python script into a publishable Laravel package.

A Laravel package that scans public BoardDocs sites for meeting agendas and attachments, exposes a fluent, Laravel-flavored "unofficial API," and exports self-contained meeting PDFs — with remote BoardDocs links rewritten into in-document anchors — plus a JSONL search index that plugs into the Laravel AI SDK.

It is a PHP/Laravel port of the Python AbandonBoard exporter, so a district can archive its agendas before losing BoardDocs access. Public data only — the private/login features of the original are intentionally omitted.

Requirements

• PHP 8.2+
• Laravel 11, 12, or 13
• tecnickcom/tcpdf + setasign/fpdi (installed automatically) for PDF assembly

Installation

Set your default site in .env:

The fluent API

The BoardDocs() helper (or the BoardDocs facade) is the entry point. Committee and meeting lists are returned as Laravel Collections and are cached, so ->first(), ->firstWhere(), ->map() etc. all work as expected.

The one-liner from the top works too:

The low-level HTTP client (the raw "unofficial API") is available if you need it:

Self-contained meeting PDFs

With self_contained enabled (default), each meeting PDF contains the printed agenda followed by every PDF attachment merged inline, one bookmark per attachment, and non-PDF attachments embedded as file attachments. When remap_links is on, any remote BoardDocs link in the agenda is rewritten into an internal PDF anchor that jumps to the merged attachment's page — so the document stays fully navigable offline, even if a district's BoardDocs subscription ends.

Two rendering engines are available (configure boarddocs.pdf.engine):

Scanning & the search index

boarddocs:scan is boarddocs:prefetch followed by boarddocs:build, run as one command for the common case where nothing is currently blocking you:

Starting over

Two commands wipe previously fetched/built data so the next run starts from scratch instead of skipping what's already there (e.g. after changing the PDF template):

Both default to the configured site and ask for confirmation; pass --all to wipe every site instead of just one, and --force to skip the prompt.

Two on-disk trees: what BoardDocs gave us vs. what we produce

Everything is split across two directories (on the local disk by default, i.e. under storage/app/private, since that's already the local disk's root in Laravel 11+):

The names describe provenance, not web visibility — both live under Laravel's private local disk by default; point output.disk/archive.disk at a public disk yourself if you want the generated PDFs served directly.

Surviving BoardDocs blocking your server mid-scan

BoardDocs will eventually 403 a server that scans it too aggressively. boarddocs:prefetch and boarddocs:build exist specifically to make that survivable, by splitting "talk to BoardDocs" from "produce the PDF" into two independent steps:

• boarddocs:prefetch fetches the committee list, each committee's meeting list, and every not-yet-exported meeting's agenda HTML + attachment files into the archive — nothing else. It stops immediately on the first 403 it hits, since every request after that is expected to fail the same way.
• boarddocs:build never makes a BoardDocs request. It builds PDFs straight from whatever is already archived. A meeting whose archive is complete builds fine even while BoardDocs is actively blocking you; a meeting whose archive is missing something is skipped with a clear message (run prefetch again to fill the gap) instead of silently reaching out live.

Run prefetch on its own — e.g. on a schedule, or as your own retry after a 403 — to warm the archive ahead of time, then build (or boarddocs:scan, later, whenever) picks up from there:

If it's your deployed server's IP getting blocked

Running boarddocs:prefetch on the same server that's already blocked doesn't help — it hits the same wall. What actually helps is running it from a different network, against the same archive the deployed server's boarddocs:build reads from. Point archive.disk (and output.disk, if you want) at a shared disk instead of local — any Flysystem-backed disk works, e.g. S3:

Configure an s3 disk in config/filesystems.php with the same bucket/credentials on both the deployed server and whatever machine you'll run prefetch from (your laptop, a different cloud box, anywhere with a clean IP). Then:

No package code cares which disk archive.disk points to — Archive only ever calls exists()/get()/put() on it, so any Laravel filesystem disk works without further changes.

index.jsonl has the same shape as the original project (one meeting per line):

Meetings whose PDF already exists are skipped, except those within scan.refresh_recent_days (default 7) which are re-exported.

Laravel AI SDK integration

The package ships ready-to-use AI SDK tools so an agent can search and consume the exported meeting PDFs/index. Install the SDK to use them:

Use the bundled agent directly:

...or from the command line:

Pass --show-tool-calls to see which tool calls the agent made along the way, or --json for the raw text + tool calls as JSON. --provider=/--model= override the defaults from your ai config. This is the AI-agent counterpart to boarddocs:search, which only ever does local keyword matching and never calls a model.

…or register the individual tools on your own agent:

• SearchAgendasTool — keyword search over agenda text + attachment titles; returns meetings with snippets and the path to fetch full details.
• GetMeetingTool — full indexed record (agenda text, attachments + pages) for one path.
• ListCommitteesTool — live committee list for a site.

Vector store search (optional)

By default BoardDocsAgent searches the local index.jsonl via SearchAgendasTool (free, deterministic keyword search). You can instead delegate search to a vector store so the model performs semantic retrieval over the actual meeting PDFs — useful when relevant content lives in scanned attachments or phrasing that keyword search misses.

1. Create a store once (OpenAI or Gemini only) and wire up its ID:

   This creates the store, then writes BOARDDOCS_AI_SEARCH_DRIVER=vector and BOARDDOCS_VECTOR_STORE_ID into .env for you. Omit --write-env to just print the values and add them yourself; pass --provider=openai (or gemini) to pin a provider other than your app's default. Equivalent by hand:

2. Run php artisan boarddocs:scan (or prefetch + build) as usual, then sync the store as its own step:

   This uploads, for every meeting boarddocs:build has already produced: the merged agenda PDF, the raw print-agenda HTML, and every raw attachment file — each tagged with path/district/committee/date/kind metadata (plus bookmark/title/ item_unique/file_unique/href for attachments) so a FileSearch citation on any of them can be traced back to the exact meeting and, for attachments, the exact file. It's a separate command (rather than running inline during build) so a slow or failing upload to the vector store's API never interrupts PDF generation. It's idempotent — each document's id is recorded on the meeting's index entry, and re-runs only touch files that don't have one yet, unless you pass --force.

With the driver set to vector, BoardDocsAgent registers Laravel\Ai\Providers\Tools\FileSearch against that store as its only tool, instead of SearchAgendasTool/GetMeetingTool/ListCommitteesTool (falling back to that trio automatically if vector_store.id isn't set). GetMeetingTool and ListCommitteesTool are dropped under the vector driver because Gemini rejects a request that combines its built-in FileSearch tool with regular function-calling tools unless tool_config.include_server_side_tool_invocations is set — something laravel/ai's Gemini gateway doesn't currently send. boarddocs:sync-vector's per-file metadata (kind, committee, date, bookmark/title for attachments) is what lets the model still identify which meeting and file a retrieved chunk came from, without a follow-up tool call.

Configuration

See config/boarddocs.php. Highlights:

PDF templates

pdf.template (default default) controls the fonts, colors, and spacing applied to the agenda in the generated PDF. BoardDocsScraper\Pdf\Templates\DefaultTemplate is a clean, modern, minimal-color design: neutral grays for body text with a single muted-blue accent for headings and links.

To build a custom template, implement BoardDocsScraper\Pdf\Templates\PdfTemplate:

Register it under pdf.templates in config/boarddocs.php and point pdf.template at its key, or set pdf.template directly to the class name:

Troubleshooting

BoardDocs will start returning 403 responses if a scan hits it too fast or too long. Enable request/response logging to see exactly when that starts happening:

Every outbound request and response (method, URL, status, elapsed time, headers, and a truncated body) is logged, and any 403 is additionally logged as a warning tagged boarddocs.blocked. Logs go to a boarddocs log channel if your app defines one in config/logging.php, otherwise to the app's default channel. If you're seeing 403s, raise http.request_delay (BOARDDOCS_REQUEST_DELAY), scope the scan down with --committee/--limit/--since, or run boarddocs:prefetch during a window when BoardDocs is reachable so boarddocs:scan can build PDFs from the archive afterward — see Surviving BoardDocs blocking your server mid-scan.

Testing

laravel/ai is a dev dependency (composer install pulls it in) so the tests/Feature/Ai suite — BoardDocsAgent, the AI SDK tool classes, VectorStoreSync, and boarddocs:vector-store:create — is fully exercised against the SDK's fakes (Stores::fake()) by default. If laravel/ai isn't present (e.g. composer install --no-dev), that suite skips itself with a clear message instead of failing.

License

MIT