Download the PHP package mytree/index-providers without Composer

MyTree Index Providers

Standalone PHP package for acquiring genealogical index data from:

• Geneteka — paginated JSON from the internal getAct.php endpoint,
• Metryki-Wołyń — HTML search results acquired parish-by-parish and year-by-year,
• BASIA — bounded, low-rate search through the public advanced-search form plus indexed-catalog discovery.

Providers accept a standard PSR-18 HTTP client, so MyTree or another host application can inject its own implementation without adapting to a MyTree-specific HTTP interface. The standalone CLI uses a default Guzzle 7 composition.

Requirements

• PHP 8.2+
• Composer 2
• PHP DOM extension (ext-dom)
• dependencies from composer.json (guzzlehttp/guzzle and PSR-18/PSR-7/PSR-17 contracts)

Install dependencies before using the library or CLI:

HTTP and integration boundary

The public HTTP-client boundary is Psr\Http\Client\ClientInterface. Requests and responses use PSR-7, and providers may also receive a custom Psr\Http\Message\RequestFactoryInterface and, when a request body is needed, a Psr\Http\Message\StreamFactoryInterface.

The default standalone client can be created with:

NativeHttpClient remains temporarily as a deprecated compatibility shim. It implements PSR-18 and delegates to the default Guzzle-based stack.

A host application may inject any compatible PSR-18 client and, optionally, its own PSR-17 factories:

The default composition retries transport failures, HTTP 429, and 5xx responses. Redirects for GET/HEAD are handled explicitly. Retry does not include POST by default; a provider performing a replayable read-only POST must opt in deliberately. The BASIA CLI does so because BASIA search is a read-only POST with a reproducible body.

Rate limiting remains a provider responsibility. BASIA defaults to a minimum interval of 5000 ms and a 200-second timeout; Geneteka and Metryki-Wołyń keep the existing defaults of 2000 ms and 60 seconds.

See docs/LARAVEL_INTEGRATION.md for host-application composition.

Quick start

Geneteka — Imbramowice, Małopolskie

The input type is MyTree's canonical RecordType, independent of provider-specific transport codes:

• birth,
• marriage,
• death.

One Geneteka acquisition represents one provider form state and one record type. Combining several record types, partitioning work into intervals, and scheduling subsequent queries belong to a higher orchestration layer.

Geneteka fluent API

A query is built as an immutable configuration rather than by extending the acquire(...) argument list:

Verified form fields have typed convenience methods such as person, secondPerson, years, exact, and excludeParents. Provider-specific fields that do not yet have a dedicated method can be added through the controlled escape hatch:

Transport parameters bdm, w, rid, length, and start are reserved and cannot be overridden through formParameter().

Geneteka availability and per-type identifiers

The Geneteka UI publishes year coverage that may be discontinuous, and the Birth/Marriage/Death links for one logical parish may use different rid values. The provider exposes this metadata separately from acquisition:

Each GenetekaRecordAvailability contains its recordType, the corresponding providerParishId, and YearRange[]. The package does not automatically turn those ranges into an acquisition plan; a later Acquisition Manager or Laravel layer may do so.

Diagnostic CLI:

See docs/GENETEKA_ACQUISITION.md.

BASIA — bounded search

BASIA is intentionally not supported as a crawler or full-database mirror. One provider operation represents one deliberately bounded state of the public advanced-search form:

At least one of --surname, --name, or --place is required. Supported canonical types are:

• birth,
• marriage,
• death,
• banns,
• other.

banns is distinct from marriage. other means only the provider's explicit other/uncategorized category. A future BASIA category without an accepted canonical mapping is preserved as provider:basia:<token> rather than silently becoming other.

Additional CLI filters include --place, --distance-km, --similarity, --sex, --relation, and --unit-type. BASIA form codes remain private to the provider adapter; callers use canonical values only.

Fluent API:

BASIA can be slow for broad queries. HTTP 200 is not automatically treated as a successful search response: the parser requires the upstream search-complete marker. A response without that marker is not promoted to the reusable raw cache and does not create a completion checkpoint. Narrow the query deliberately and retry. A complete search with zero results is a valid success.

A scan URL returned by BASIA is preserved as a locator/acquisition lead. This Index Provider does not download the scan.

See docs/BASIA_ACQUISITION.md.

BASIA — indexed catalog

The BASIA indexed-content catalog (content-all.php?lang=pl) is broader than a parish list. One locality may expose, for example, a Catholic parish, a Protestant parish, a civil registry office, and other units at the same time. Catalog discovery therefore uses the additive IndexCatalogUnit contract instead of AvailableParish.

CLI:

PHP API:

BasiaProvider implements the optional IndexCatalogDiscoveryInterface. Each unit serializes as mytree.index-catalog-unit.v1 and keeps locality/place separate from the indexed records-holding unit. Initial mappings are:

The Polish labels above are source values published by BASIA and are intentionally preserved. An unknown future unit type remains provider:basia:<token>. An entry with no unit label is preserved as provider:basia:unlabeled. The provider does not manufacture parish semantics for non-parish units.

Availability stores YearRange[] per record type so gaps remain explicit. The catalog also preserves raw labels, county, locality-wide entry totals, indexers, and raw-response provenance.

Catalog cache validation is structural: every parsed locality must contain the expected BASIA locality-completion metadata (Razem wpisów and Indeksujący) and parsing must succeed before the response is promoted to reusable cache. BASIA does not expose a known page-level completion marker analogous to bounded search's Czas wyszukiwania, so the package does not claim to prove global completeness of an otherwise structurally valid provider snapshot.

Metryki-Wołyń — Szumsk

Metryki-Wołyń acquisition proceeds year by year. Known sections map to:

• birth,
• marriage,
• death,
• parish_census.

A future/unrecognized section is not discarded. Each row is emitted with a deterministic provider-qualified record_type such as provider:wolyn-metryki:<token>, while the exact section title, headers, cells, cell HTML, links, and request provenance are preserved. The provider does not guess column semantics or an event year for an unknown section.

Discovery capabilities

The package exposes two deliberately separate discovery contracts.

Legacy parish discovery

Geneteka and Metryki-Wołyń retain AvailableParish / mytree.available-parish.v1 and the --list-parishes CLI mode. Their public return type is not changed merely because a more general catalog contract now exists.

Geneteka — one region

For Geneteka, provider_parish_id is the provider rid, for example:

The list is read from the current Geneteka search form. If the form structure changes, the provider has a fallback using the parishes field returned by getAct.php.

All regions can also be discovered:

This performs several requests, one per region, so --delay-ms still applies. Prefer discovery for a specific region when that is sufficient.

Metryki-Wołyń

The provider uses the portal's Zawartość page. In addition to the parish name it retains the portal's published ranges for entries and complete indexes covering births, marriages, deaths, and parish censuses.

Example table view:

See docs/PARISH_DISCOVERY.md.

Generalized catalog discovery

The optional IndexCatalogDiscoveryInterface returns IndexCatalogUnit[]. BASIA implements this capability in P1. AvailableParish is neither removed nor changed.

Representative BASIA result:

Discovery output formats and cache

Discovery output defaults to a table. JSON and JSONL are also supported:

Output can be saved to a file:

Representative legacy parish-discovery record:

For Metryki-Wołyń, provider_parish_id is null because the search input currently identifies a parish by its textual name. Published coverage remains in metadata.wpisy and metadata.indeksy; those keys preserve provider wording.

Discovery uses a separate raw-page cache. The default directory is var/discovery-cache; use --output=DIR to select another location and --refresh to bypass the cached provider snapshot.

BASIA stores a successfully parsed catalog at raw/basia/catalog_pl.html together with metadata. Structural parse failures are not promoted to a successful reusable cache entry.

Rate limiting

Geneteka and Metryki-Wołyń default to at least 2000 ms between network requests. BASIA uses a more conservative default of 5000 ms:

Reducing these delays is discouraged. The package is intended for controlled personal research. Before larger acquisitions, verify that the intended use is consistent with the rules and policies of the upstream service.

Resume and restart

The CLI checkpoints complete acquisition work units:

• Geneteka — result page,
• Metryki-Wołyń — year,
• BASIA bounded search — complete bounded query.

Re-running the same acquisition with the same --output resumes from local state.

Geneteka cache/checkpoints are partitioned by a deterministic fingerprint of the full query configuration: region, rid, record type, form parameters, and page size. BASIA bounded search uses an analogous fingerprint of its canonical search configuration and only caches a response after search completeness has been confirmed.

BASIA catalog discovery is a separate provider snapshot. It uses raw cache plus --refresh and does not create a person-record acquisition checkpoint.

Raw responses are retained under raw/. If an acquisition response was retrieved but processing stopped before checkpoint completion, a later run can reuse the local raw response when that provider workflow considers it reusable.

records.jsonl deduplicates by provider_record_id, preventing ordinary resume from creating duplicate semantic rows.

Full restart

For record acquisition, --restart:

• removes records.jsonl,
• removes checkpoints,
• bypasses existing cache during the run,
• replaces corresponding raw responses with newly retrieved responses.

For discovery, prefer the explicit --refresh mechanism.

Output layout

Geneteka example:

Metryki-Wołyń raw responses:

BASIA raw responses:

Result viewer (TUI)

Existing acquisition output can be inspected with the separate read-only/offline viewer:

The viewer supports mytree.index-acquisition-manifest.v1 plus mytree.external-index-record.v1. It reads manifest.json and records.jsonl; it does not contact providers, mutate checkpoints, or parse preserved raw HTML/JSON to reconstruct records.

Main keys:

The initial terminal adapter uses ANSI + stty and therefore targets Unix-like interactive terminals. See docs/RESULT_VIEWER.md for validation rules, details-view behavior, search semantics, and current limitations.

ExternalIndexRecord

Every acquisition JSONL line uses the stable mytree.external-index-record.v1 contract:

fields provides a convenient structured projection, while raw retains provider values. The package does not resolve uncertain values such as 20?, surname variants, or free-text note semantics. Provider/indexer descriptive values use ValueRepresentation::indexerRendering(...) so downstream code does not mistake them for guaranteed wording from the historical document.

See docs/VALUE_REPRESENTATION.md.

Provenance

Acquisition records retain, as applicable:

• request URL or stable provider permalink,
• retrieval time,
• raw-response path,
• raw-response SHA-256,
• provider-specific query configuration/fingerprint,
• page/year/result/row index,
• parser version where relevant.

IndexCatalogUnit also retains the catalog URL, retrieval time, raw-response path/hash, parser version, and catalog-item index. catalog_unit_key is a deterministic provider-local discovery key, not a MyTree SourceId.

This allows a later MyTree importer to preserve the origin of data without collapsing the boundaries between discovery, external index acquisition, and historical source identity.

Tests

Tests use PHPUnit and are designed to run offline.

Direct PHPUnit invocation:

Normal tests must not depend on third-party service availability. BASIA uses static fixtures for bounded search and separate catalog fixtures covering multiple units under one locality, discontinuous ranges, explicit/unknown categories, and structurally incomplete responses.

Laravel / MyTree integration

See docs/BASIA_ACQUISITION.md.