Download the PHP package olexin-pro/data-processing-pipeline without Composer

🧭 Data Processing Pipeline for Laravel

A robust, strictly-typed, and extensible data processing pipeline system for Laravel applications. Process data through a chain of isolated steps with built-in conflict resolution, priority handling, and optional execution history tracking.

✨ Features

• 🔒 Strictly Typed - Full PHP 8.1+ type safety with enums and interfaces
• 🔄 Immutable Payload - Original data never changes during processing
• 🎯 Conflict Resolution - Built-in strategies: MERGE, OVERWRITE, SKIP, CUSTOM
• 📊 Priority System - Control data precedence in merge operations
• 📝 Execution History - Optional database tracking of pipeline runs
• 🚀 Queue-Safe - Fully serializable for Laravel queues
• 🧩 Extensible - Easy to add custom steps and conflict resolvers
• 🧪 Well Tested - Comprehensive test coverage

📦 Installation

Publish Migrations

This creates two tables:

• pipeline_runs - Stores pipeline execution records
• pipeline_steps - Stores individual step execution details

🚀 Quick Start

1. Create a Pipeline Step

2. Run the Pipeline

3. Queue Execution

📚 Core Concepts

Pipeline Context

The context is an immutable container that holds:

• Payload (readonly) - Original input data
• Results - Accumulated step outputs
• Meta - Arbitrary metadata (errors, timestamps, etc.)

Pipeline Results

Every step returns a PipelineResultInterface:

Conflict Policies

When multiple steps write to the same key:

MERGE (default)

Recursively merges arrays, respecting priorities:

OVERWRITE

Replaces previous value completely:

SKIP

Keeps the first result, ignores the rest:

CUSTOM

Define a custom resolver:

🔧 Advanced Usage

Priority System

Higher priority overrides lower ones in merges:

Execution History

Enable automatic logging:

Error Handling

Pipeline continues on failure:

Dynamic Steps

Add steps at runtime:

Conditional Steps

Steps can self-skip:

🎯 Real-World Example

🛠 Step Generator

Create pipeline steps with a single command. By default, classes are placed under App\Pipeline\Steps and saved to app/Pipeline/Steps.

`

Nested directories

Options

• --policy= — conflict policy (MERGE|OVERWRITE|SKIP|CUSTOM). Default: MERGE.
• --priority= — result priority (int). Default: 10.
• --key= — result key (defaults to the snake_case of the class name without the Step suffix, e.g., EmailFormatterStep → email_formatter).
• --force — overwrite the file if it already exists.

Examples:

Customizing the stub

🧪 Testing

📖 API Reference

PipelineContext

GenericPipelineResult

🤝 Contributing

Contributions are welcome! Please feel free to submit a Pull Request.

1. Fork the repository
2. Create your feature branch (git checkout -b feature/amazing-feature)
3. Commit your changes (git commit -m 'Add amazing feature')
4. Push to the branch (git push origin feature/amazing-feature)
5. Open a Pull Request

📄 License

This package is open-sourced software licensed under the MIT license.

🙏 Credits

• Aizharyk Olexin

📧 Support

For support, please open an issue on GitHub

Made with ❤️ for the Laravel community