Download the PHP package gurento/kafka-consumer without Composer

On this page you can find all versions of the php package gurento/kafka-consumer. It is possible to download/install these versions without Composer. Possible dependencies are resolved automatically.

FAQ

After the download, you have to make one include require_once('vendor/autoload.php');. After that you have to import the classes with use statements.

Example:
If you use only one package a project is not needed. But if you use more then one package, without a project it is not possible to import the classes with use statements.

In general, it is recommended to use always a project to download your libraries. In an application normally there is more than one library needed.
Some PHP packages are not free to download and because of that hosted in private repositories. In this case some credentials are needed to access such packages. Please use the auth.json textarea to insert credentials, if a package is coming from a private repository. You can look here for more information.

  • Some hosting areas are not accessible by a terminal or SSH. Then it is not possible to use Composer.
  • To use Composer is sometimes complicated. Especially for beginners.
  • Composer needs much resources. Sometimes they are not available on a simple webspace.
  • If you are using private repositories you don't need to share your credentials. You can set up everything on our site and then you provide a simple download link to your team member.
  • Simplify your Composer build process. Use our own command line tool to download the vendor folder as binary. This makes your build process faster and you don't need to expose your credentials for private repositories.
Please rate this library. Is it a good library?

Example code of gurento/kafka-consumer

Informations about the package kafka-consumer

Kafka Consumer

Latest Version on Packagist Total Downloads

A Laravel package for consuming Kafka messages and writing them into Eloquent models using declarative, database-driven topic mappings — no per-topic consumer classes required.

Built for operations teams that need:

Ships with a default engine based on mateusjunges/laravel-kafka, so it works out of the box.

Admin UI available: install gurento/kafka-consumer-filament for consumer topic CRUD, live health monitoring, and one-click replay — and/or gurento/kafka-producer-filament for producer topic CRUD and a Send Message composer. Both are independent Filament panels sharing this package as their backend; install either or both.

Features

Consumer

Producer

Requirements

Dependency Version
PHP 8.2+
Laravel 11 / 12 / 13
ext-rdkafka configured for your PHP runtime
Kafka broker reachable from the app

Installation

This creates four tables:

Environment

Add your Kafka connection settings to .env:

Quick Start

  1. Create a topic configuration (via seeder, tinker, or the Filament UI).

  2. Start the consumer:

  3. Inspect results in kafka_topics.messages_consumed and kafka_consume_logs.

Topic Configuration

Each kafka_topics row defines how a Kafka topic maps into your model.

Core columns:

Column Purpose
topic Kafka topic name
model_class Fully-qualified model class (e.g. App\Models\Office)
upsert_key Unique model column used by updateOrCreate
field_map JSON array of {from, to} mapping pairs
exclude_keys Optional payload keys to strip before mapping
relations Optional BelongsToMany sync definitions
is_active Whether the topic is consumed

Retry/health overrides (fall back to config when NULL): max_reconsume_attempts, retry_backoff_seconds, health_stale_after_seconds.

Operational columns (auto-managed): messages_consumed, messages_failed, messages_reconsumed, last_consumed_at, consumer_last_heartbeat_at, consumer_last_error, consumer_lag_seconds.

Field Mapping Example

Given this Kafka payload:

Use field_map:

And exclude_keys:

The package upserts by upsert_key (for example id).

Command Usage

Normal Consumption

Option Description
--topics=* Consume only selected topics
--limit= Process only N messages, then stop
--group= Consumer group id (defaults to kafka-consumer.default_group config)
--from-beginning Read from earliest offsets (replay mode)
--stop-on-empty Stop after the last available message instead of polling forever

Examples:

Re-consume Failed Logs

Instead of polling Kafka, reprocess failed entries from kafka_consume_logs:

Useful after fixing a wrong field_map, a wrong model_class, or a temporary DB/infrastructure failure.

How Replay Works

--from-beginning sets offset reset to earliest and uses a unique consumer group (unless --group is provided) to avoid corrupting your normal consumer offsets.

Recommended pattern:

  1. Run the normal consumer with a stable group for real-time operations.
  2. Use --from-beginning only for backfills/replays.

Failure & Retry Behavior (Consumer)

On processing errors, the package logs status=failed, the error message, and retry metadata (attempt_count, next_retry_at, retryable). Retries back off linearly per attempt using the topic's retry_backoff_seconds and stop after max_reconsume_attempts. Re-consume mode marks replayed entries reconsumed_success on success.

Producing Messages

The producer side is a mirror of the consumer: declarative topic config, per-message logs, and automatic retry — just in the opposite direction.

Producer Topic Configuration

Each kafka_producer_topics row defines how a topic accepts outgoing messages.

Column Purpose
topic Kafka topic name to publish to
model_class Optional Eloquent model class — used only by gurento/kafka-producer-filament's UI to auto-derive payload_schema/relations; has no effect on send()
key_field Payload field used as the Kafka message key when no explicit key is given
default_headers JSON array of {name, value} pairs merged into every message
payload_schema Optional JSON array of {name, type, required} field definitions (drives the Filament send composer)
payload_template Optional free-form raw JSON text — when set, takes priority over payload_schema/relations as the Filament send composer's default payload
relations Optional JSON array of {relationship, type, related_model} — informational only, surfaced in the Filament UI
is_active Whether the topic accepts sends

Retry overrides (fall back to config('kafka-consumer.producer.*') when NULL): max_send_attempts, send_backoff_seconds.

Operational columns (auto-managed): messages_produced, messages_send_failed, messages_resent, last_produced_at, producer_last_error.

Sending a Message

Programmatically:

Via command:

Option Description
--topic= Producer topic name to send to
--payload= Message payload as a JSON string
--key= Optional message key (overrides the topic's key_field resolution)
--retry-failed Retry failed produce logs instead of sending a new message
--retry-limit= Max failed logs per topic to retry (default 50)

Retry all topics' failed sends:

Failure & Retry Behavior (Producer)

On send errors, the package logs status=failed, the error message, and retry metadata (attempt_count, next_retry_at, retryable) — backing off linearly per attempt using send_backoff_seconds and stopping after max_send_attempts. Retried entries are marked resent_success on success.

At-least-once semantics: if the broker acknowledgment is lost after actually receiving the message, a retry can produce a duplicate. Design consumers to be idempotent (e.g. upsert by a stable key) when replaying producer retries.

Configuration

Values in config/kafka-consumer.php act as fallbacks when a topic row leaves the matching column NULL:

Key Default Purpose
default_group app-consumer Consumer group id when --group is not passed
max_reconsume_attempts 3 Consumer retry attempts before a failure becomes permanent
retry_backoff_seconds 60 Consumer base backoff between retries
health_stale_after_seconds 300 Heartbeat age before a topic is considered stalled
producer.max_send_attempts 3 Producer retry attempts before a send failure becomes permanent
producer.send_backoff_seconds 60 Producer base backoff between retries
producer.health_stale_after_seconds 300 Reserved for future producer health checks

Events

The services dispatch events on every processing outcome — hook them for alerting or metrics:

Event When Payload
Gurento\KafkaConsumer\Events\KafkaMessageConsumed Message (or re-consume) succeeds $topic, $log, $isReconsume
Gurento\KafkaConsumer\Events\KafkaMessageFailed Processing fails $topic, $log, $error, $willRetry
Gurento\KafkaConsumer\Events\KafkaMessageProduced Message (or resend) sends successfully $topic, $log, $isResend
Gurento\KafkaConsumer\Events\KafkaMessageProduceFailed Send fails $topic, $log, $error, $willRetry

Custom Engine (Optional)

By default the package binds:

To use a different transport, override either binding in your host app:

Your engines must implement:

Programmatic Hooks

If you already consume or produce Kafka messages elsewhere, call the services directly:

Production Recommendations

Troubleshooting

No active topics configured — ensure kafka_topics has rows with is_active = true.

Command not behaving as expected — run php artisan optimize:clear && composer dump-autoload.

Kafka connection/auth issues — verify the host app's config/kafka.php and environment values (KAFKA_BROKERS, auth settings).

Replay not processing failed logs — check that rows have status = failed, retryable = true, and next_retry_at <= now() (or NULL). Applies to both kafka_consume_logs and kafka_produce_logs.

Message not sent, no error shown — confirm the producer topic row has is_active = true; inactive topics are silently skipped (logged as a warning) rather than erroring.

Changelog

See CHANGELOG.md for release history.

License

The MIT License (MIT). See LICENSE for details.

All versions of kafka-consumer with dependencies

PHP Build Version
Package Version
Requires php Version ^8.2
illuminate/support Version ^11.0|^12.0|^13.0
illuminate/database Version ^11.0|^12.0|^13.0
illuminate/console Version ^11.0|^12.0|^13.0
mateusjunges/laravel-kafka Version ^2.9
Composer command for our command line client (download client) This client runs in each environment. You don't need a specific PHP version etc. The first 20 API calls are free. Standard composer command

The package gurento/kafka-consumer contains the following files

Loading the files please wait ...