Download the PHP package ducks-project/encoding-repair without Composer

CharsetHelper

Advanced charset encoding converter with Chain of Responsibility pattern, auto-detection, double-encoding repair, and JSON safety.

🆕 What's New in v1.2

Type Interpreter System

New optimized type-specific processing with Strategy + Visitor pattern:

Batch Processing API

New optimized batch processing methods for high-performance array conversion:

Service-Based Architecture

CharsetHelper now uses a service-based architecture following SOLID principles:

• CharsetProcessor: Instanciable service with fluent API
• CharsetProcessorInterface: Service contract for dependency injection
• Multiple instances: Different configurations for different contexts
• 100% backward compatible: Existing code works unchanged

PSR-16 Cache Support

Optional external cache integration for improved performance:

🌟 Why CharsetHelper?

Unlike existing libraries, CharsetHelper provides:

• ✅ Extensible architecture with Chain of Responsibility pattern
• ✅ PSR-16 cache support for Redis, Memcached, APCu (NEW in v1.2)
• ✅ Type-specific interpreters for optimized processing (NEW in v1.2)
• ✅ Custom property mappers for selective object conversion (NEW in v1.2)
• ✅ Multiple fallback strategies (UConverter → iconv → mbstring)
• ✅ Smart auto-detection with multiple detection methods
• ✅ Double-encoding repair for corrupted legacy data
• ✅ Recursive conversion for arrays AND objects (not just arrays!)
• ✅ Safe JSON encoding/decoding with automatic charset handling
• ✅ Modern PHP with strict typing (PHP 7.4+)
• ✅ Minimal dependencies (only PSR-16 interface for optional caching)

📖 Features

• Robust Transcoding: Implements a Chain of Responsibility pattern trying best providers first (Intl/UConverter -> Iconv -> MbString).
• PSR-16 Cache Support: Optional external cache (Redis, Memcached, APCu) for detection results (NEW in v1.2).
• Type-Specific Interpreters: Optimized processing strategies per data type (NEW in v1.2).
• Custom Property Mappers: Selective object property conversion for security and performance (NEW in v1.2).
• Double-Encoding Repair: Automatically detects and fixes strings like été back to été.
• Recursive Processing: Handles string, array, and object recursively.
• Immutable: Objects are cloned before modification to prevent side effects.
• Safe JSON Wrappers: Prevents json_encode from returning false on bad charsets.
• Secure: Whitelisted encodings to prevent injection.
• Extensible: Register your own transcoders, detectors, interpreters, or cache providers without modifying the core.
• Modern Standards: PSR-12 compliant, strictly typed, SOLID architecture.

📋 Requirements

• PHP: 7.4, 8.0, 8.1, 8.2, or 8.3
• Extensions (required):
  • ext-mbstring
  • ext-json
• Extensions (recommended):
  • ext-intl

📦 Installation

Optional Extensions (for better performance)

🚀 Quick Start

🏗️ Usage

1. Basic Conversion

Convert between different character encodings:

Note: We use Windows-1252 instead of strict ISO-8859-1 by default because it includes common characters like €, œ, ™ which are missing in standard ISO.

Supported Encodings:

• UTF-8
• UTF-16
• UTF-32
• ISO-8859-1
• Windows-1252 (CP1252)
• ASCII
• AUTO (automatic detection)

2. Automatic Encoding Detection

Let CharsetHelper detect the source encoding:

3. Batch Processing (New in v1.2)

Optimized for processing large arrays with single encoding detection:

4. Recursive Conversion (Arrays & Objects)

Convert nested data structures:

5. Double-Encoding Repair 🔧

Fix strings that have been encoded multiple times (common with legacy databases):

How it works:

1. Detects valid UTF-8 strings
2. Attempts to reverse-convert (UTF-8 → source encoding)
3. Repeats until no more layers found or max depth reached
4. Returns the cleaned string

6. Safe JSON Operations

Prevent JSON encoding/decoding errors caused by invalid UTF-8:

7. Conversion Options

Fine-tune the conversion behavior:

Options explained:

• normalize: Applies Unicode NFC normalization to UTF-8 output (combines accents)
• translit: Converts unmappable characters to similar ones (é → e)
• ignore: Skips invalid byte sequences instead of failing
• encodings: List of encodings to try during auto-detection

🎯 Advanced Usage

Using CharsetProcessor Service (New in v1.1)

For better testability and flexibility, use the CharsetProcessor service directly:

Multiple Processor Instances

Dependency Injection

Custom Property Mappers (New in v1.2)

Optimize object processing by converting only specific properties:

Custom Type Interpreters (New in v1.2)

Add support for custom data types:

Registering Custom Transcoders

Extend CharsetHelper with your own conversion strategies using the TranscoderInterface:

Registering Custom Detectors

Add custom encoding detection methods using the DetectorInterface:

Chain of Responsibility Pattern

The class uses a Chain of Responsibility pattern for both detection and transcoding.

CharsetHelper uses multiple strategies with automatic fallback:

Transcoder priorities:

1. UConverter (priority: 100, requires ext-intl): Best precision, supports many encodings
2. iconv (priority: 50): Good performance, supports transliteration
3. mbstring (priority: 10): Universal fallback, most permissive

Custom transcoders can be registered with any priority value. Higher values execute first.

Detector priorities:

1. CachedDetector (priority: 200, wraps MbStringDetector): Caches detection results
2. MbStringDetector (priority: 100, requires ext-mbstring): Fast and reliable using mb_detect_encoding
3. FileInfoDetector (priority: 50, requires ext-fileinfo): Fallback using finfo class

Custom detectors can be registered with any priority value. Higher values execute first.

Cache Support (New in v1.2):

CachedDetector supports PSR-16 cache for persistent detection results:

📊 Performance

Benchmarks on 10,000 conversions (PHP 8.2, i7-12700K):

Tips for performance:

• Install ext-intl for best performance (UConverter is fastest)
• Use specific encodings instead of AUTO when possible
• Use batch methods (toCharsetBatch()) for arrays > 100 items with AUTO detection
• Cache detection results for repeated operations

🆚 Comparison with Alternatives

🔍 Use Cases

1. Database Migration (Latin1 → UTF-8)

2. CSV Import with Unknown Encoding

3. API Response Sanitization

4. Web Scraping

5. Legacy System Integration

🧪 Testing

📚 Glossary

• Changelog
• How To
• About Middleware Pattern
• Type Interpreter System
• CharsetHelper
• CharsetProcessor
• CharsetProcessorInterface
• PrioritizedHandlerInterface
• TypeInterpreterInterface
• PropertyMapperInterface
• InterpreterChain
• StringInterpreter
• ArrayInterpreter
• ObjectInterpreter
• TranscoderInterface
• CallableTranscoder
• IconvTranscoder
• MbStringTranscoder
• UConverterTranscoder
• DetectorInterface
• CallableDetector
• MbStringDetector
• FileInfoDetector
• CallableAdapterTrait
• ChainOfResponsibilityTrait
• CachedDetector
• InternalArrayCache
• ArrayCache

🤝 Contributing

Contributions are welcome! Please:

1. Fork the repository
2. Create a feature branch (git checkout -b feature/amazing-feature)
3. Write tests for your changes
4. Ensure tests pass (composer test)
5. Run static analysis (composer analyse)
6. Fix code style (composer cs-fix)
7. Commit your changes (git commit -m 'Add amazing feature')
8. Push to the branch (git push origin feature/amazing-feature)
9. Open a Pull Request

Development Setup

Code Quality Standards

• PSR-12 / PER Coding Style
• PHPStan level 8
• 100% type coverage
• Minimum 90% code coverage

📄 License

This project is licensed under the MIT license see the LICENSE file for details.

🙏 Credits

• Inspired by ForceUTF8 (simplified approach)
• Uses design patterns from Symfony (extensibility)
• Fallback strategies similar to Portable UTF-8

🔗 Links

• Documentation: https://github.com/ducks-project/encoding-repair/wiki
• Issue Tracker: https://github.com/ducks-project/encoding-repair/issues
• Changelog: CHANGELOG.md
• Packagist: https://packagist.org/packages/ducks-project/encoding-repair

💬 Support

• 📧 Email: [email protected]
• 💬 Discussions: https://github.com/ducks-project/encoding-repair/discussions
• 🐛 Issues: https://github.com/ducks-project/encoding-repair/issues

⭐ Star History

If this project helped you, please consider giving it a ⭐ on GitHub!

Made with ❤️ by the Duck Project Team