Download the PHP package hirale/openmage-async-index without Composer

Hirale Async Index

Asynchronous core index event and full reindex orchestration for OpenMage and Maho.

The module keeps index_event and index_process_event as the durable source of truth, then dispatches a message to wake a background worker. Handlers are at least once, so the DB-backed pending event state is always used to decide what work remains.

Queue backend

There is no shared queue package any more. The module picks a backend at runtime, in this order:

Maho_Queue wins whenever it is present and enabled, even on a store that also has hirale/queue installed.

2.0.0 is a breaking change.

• hirale/queue moved from require to suggest. OpenMage installs that upgrade from 1.x must require it explicitly, or async indexing silently turns itself off.
• Four settings were removed because nothing read them: Full Reindex Max Runtime Seconds, Max Attempts, Retry Delay Seconds and Coalesce TTL Seconds. Retry behaviour has always come from the queue backend (system/queue/* on Maho, the Hirale Queue section on OpenMage). Values already saved under those paths are ignored; nothing needs to be migrated.

Install

Maho (with core Maho_Queue):

composer dump-autoload is required: it compiles the #[\Maho\Config\MessageHandler] attributes into vendor/composer/maho_attributes.php. Without it the messages have no registered handler, and the queue refuses to decode them.

OpenMage (20.17+, PHP 8.3+) — one-time tweaks first; details in the hirale/queue README:

For local development, add a path repository next to the application:

Queues and worker pools

On Maho, drain messages ride an index_drain queue and full reindex batches an indexer-style full_reindex queue. Neither uses core's shared default queue: no module may reroute that one, so a host could never move drain onto a pool of its own. config.xml leaves index_drain unrouted, which means the catch-all (slow) pool consumes it — the same place it went before.

The catch-all pool is on-demand: it exits after 60 seconds idle and the watchdog cron restarts it once a minute, so a drain can wait up to about a minute on a quiet store. If that matters, route the queue from your own config.xml or app/etc/local.xml:

fast is resident, but it is core's tier for work a human is waiting on, and a drain can run for up to max_runtime_seconds. Declaring a resident pool of your own and routing index_drain to it keeps checkout mail out of the way:

A custom name in Hirale > Async Index > Full reindex queue is used verbatim on both platforms. On OpenMage it maps onto a hirale/queue transport, which has to exist in that module's own configuration; leaving it empty uses the routing in hirale/queue's config.xml.

Deduplication (Maho only)

hirale/queue v3 has no dispatch-time dedup, so on OpenMage every dispatch inserts a job.

• Drain messages carry a shared key, so a drain that is already pending or processing suppresses further ones.
• Full reindex batches carry a key per run, so the once-a-minute reconciler and the batch chain cannot pile up dozens of messages for the same run — while runs still never suppress each other.
• A message a handler dispatches to continue its own chain keeps the key but skips the check, otherwise its own in-flight row would swallow the rest.
• The reconciler's drain carries no key at all. It is the recovery path for a worker killed mid-drain, whose row stays processing for the five minutes the queue takes to call the claim abandoned; an enforced key would suppress the very dispatch meant to get indexing moving again. Its schedule is the rate limit. This makes the reconciler cron the only crash-recovery mechanism there is — disabling it disables recovery.

Cache invalidation

Core purges a saved record's cache tags before the index is rebuilt: both happen in afterCommitCallback(), and cleanModelCache() plus catalog_product_save_commit_after run first, with processEntityAction() after them. Synchronously that is harmless — the index is rebuilt in the same request. Asynchronously it is not: a page rendered between the purge and the drain caches the stale index, and nothing purges it again. Neither Mage_Index nor the catalog indexers invalidate anything of their own.

So the module announces what it touched, once per drain and once per full-reindex batch:

On OpenMage, bind it in config.xml under <global><events><hirale_asyncindex_reindex_after> instead. The payload is the same on both:

Ids are deliberately generous. Core swallows an indexer's exception and marks the event failed rather than reporting it, so an id means "this record was handed to its indexer", not "this record is certainly fresh". Invalidating one record too many is cheap; missing one is the bug this exists to fix.

An id list longer than Invalidation Entity Limit (500) is reported as the entity types wholesale instead — past that point, carrying and acting on the list costs more than invalidating the type.

Never save a model from this observer. The event is dispatched after the index lock is released and outside the drain context, so a save would not deadlock — it would queue another drain, which announces again, which saves again. Invalidate, log, enqueue your own work; do not write.

An observer that throws is caught and logged. Letting it escape would fail the message, and the whole batch would be reindexed on the retry.

If you have no cache of your own, turn on Clean Cache After Reindex. It invalidates catalog_product / catalog_category tags through Mage::app()->cleanCache(), which also fires core's application_clean_cache. It is off by default because the event is the better extension point.

Those two are the only tags the fallback knows, because they are the only ones core models carry. Other entities — cataloginventory_stock_item, catalog_product_attribute, core_store — still appear in the event payload in full; invalidating whatever you derive from them is the host's job.

Operations

These three commands are Maho only. They are registered through lib/MahoCLI/Commands/, which OpenMage has no equivalent of. On OpenMage, read the same state from the database — failed events are index_process_event.status = 'error' joined to index_process for the indexer code, and full-reindex runs live in hirale_asyncindex_full_run — or from the admin, where the index grid shows which indexers need attention.

Long full-reindex batches

A full reindex of a product-backed indexer is split into batches of Full Reindex Batch Size. The rest — catalog_url, catalog_category_flat, catalog_category_product — rebuild in one piece: core exposes no checkpoint inside reindexAll(), so one batch is one message, however long it takes.

Each backend eventually decides a worker has died holding that message:

Re-running is harmless. A batch whose run already finished is skipped, and while one is still going the index lock makes a second wait rather than run alongside it. The cost is wasted work, not a corrupted index. (On several servers, set <global><lock><backend>db</backend></lock></global> — the default file lock is machine-local and cannot serialise across hosts.)

Every batch logs its duration to var/log/asyncindex.log, and one that runs past claim timeout minus 60 seconds logs a forced warning naming the indexer, the elapsed time and the catalog size. For reference, catalog_url took 75s over 3,220 products and 61,893 rewrites on one store, so a catalog several times that size will cross Maho's 300s.

On Maho the fix is ext-pcntl: Maho_Queue's worker refreshes its claim every few seconds through a pcntl_alarm, and without the extension it cannot — the usual php-fpm images do not ship it, and queue:work says so at startup. With it, a long batch stops being reported at all. Note that the refresh is skipped while a handler holds a transaction, so it shortens the exposure rather than removing it.

Failed events are not retried

Core catches an indexer's exception, marks the event failed and returns normally — it never reports the failure to its caller. The drain then only ever selects events marked new, so a failed one is never looked at again: not retried, not counted as pending, and invisible in the admin. The index silently drifts.

hirale:asyncindex:events is that missing view, and a drain that leaves failures behind writes a warning to var/log/asyncindex.log — forced, so it appears even on a store with dev/log/active off, since there is no other channel for it. To clear them, reindex the affected indexer.

There is deliberately no automatic retry. An event fails because its indexer threw, and the input that made it throw is stored in the event row — retrying on a schedule re-runs the same failure forever, burning a worker each time. A retry worth having needs an attempt counter and a give-up state on a table this module does not own. Until then, failing loudly beats failing in a loop.

--prune-done exists for one migration: full reindex runs before 2.0.0 marked their event rows done instead of deleting them, and nothing ever removed a done row. New runs delete as they go, and this clears what the old ones left. It deletes in batches of --prune-batch (5000), each its own transaction, so a store carrying a large backlog is not cleared under one long-held lock.

Runtime

Enable Hirale > Async Index only after the queue backend is configured and working. When async index is disabled, core indexing behaves as it did before.

When enabled, the module switches manual index processes to real_time so the platform records index events. The mode each process had at the moment it was taken over is stored, and handed back when async index is disabled.

An indexer you switch back to manual while async index is running is taken over again on the next reconciler tick — real_time is what makes events land in the first place. The takeover is logged to var/log/asyncindex.log, so a change that does not stick is distinguishable from a bug. Turn async index off first if you want an indexer to stay manual.

Turning Restore Modes on Disable off leaves the indexers on real_time and keeps the stored modes, so a later re-enable still knows what to hand back.

The queue job only wakes the runner. The runner scans pending index events and require_reindex processes from the database, processes them in batches, and publishes continuation jobs while work remains.

Full reindex runs are also queued. Product-backed indexers are split into product ID batches. Indexers that cannot be safely split by product still run through the same async full reindex coordinator as a global batch unit, so admin and CLI calls do not perform the heavy full reindex inside the request process while async index is enabled.