Download the PHP package helsingborg-stad/s3-local-index without Composer

S3 Local Index

A WordPress plugin that provides local indexing of S3 files with CLI commands and cache flushing capabilities. This plugin improves performance when working with large S3 buckets by creating local indexes that enable fast file existence checks and directory listings without expensive S3 API calls.

This plugin is tested with a bucket of over 1 000 000 files.

Description

The S3 Local Index plugin creates and maintains local JSON index files that mirror the structure of your S3 bucket. This enables WordPress to quickly determine if files exist on S3 without making API calls for every check. The plugin supports both single-site and multisite WordPress installations.

Key Features

• Fast File Operations: Local indexes eliminate the need for S3 API calls during file existence checks
• WordPress Integration: Seamless integration with WordPress's filesystem API through stream wrappers
• CLI Management: WP-CLI commands for index management
• Cache Support: Multiple caching strategies including WordPress object cache and static memory cache
• Multisite Support: Full support for WordPress multisite networks
• Cached Delegation: Sometimes delegation to original wrapper must occur, but fear not. This is cached!

Index maintanaince

The plugin will maintain the index when files are deleted and uploaded to reflect the changes. This functionality is designed using WordPress standard hooks. This plugin requires that you manage files "the right way", not registering files in WordPress will cause not_found errors. Although you may lessen the issue by running the create command often (this reflects everyting in the bucket).

The flowing file changes will be captured:

• When a image file is resized.
• When a file is deleted
• When a file is uploaded

Good to know & limitations

• The indexing cron / cli should only run on one of the sites in a multisite environment.
• All sites in a network MUST share a bucket. Multiple buckets is not yet supported.
• This plugin does not index directories, a lookup delegation will be made on is_dir querys.

Requirements

• WordPress 5.0 or higher
• PHP 8.0 or higher
• S3 Uploads Plugin (required dependency)
• WP-CLI (for command-line operations)
• Composer (for dependency management)

Installation

1. Install via Composer (recommended):

2. Manual Installation:

   • Download the plugin files
   • Upload to /wp-content/plugins/s3-local-index/
   • Run composer install in the plugin directory
3. Activate the Plugin:
   • Through WordPress admin: Plugins → Installed Plugins → Activate "S3 Local Index"
   • Via WP-CLI: wp plugin activate s3-local-index

Configuration

The plugin is enabled by default when the S3 Uploads plugin is active. You can customize behavior using WordPress filters:

Available Filters

Cache Directory Configuration

The plugin automatically prevents collisions between multiple sites on the same server by using unique cache directories. Each site gets its own cache directory based on its document root:

Examples:

• Site 1: /tmp/s3-index-0f256ede/
• Site 2: /tmp/s3-index-b9ba97a4/

You can override this behavior using the S3_Local_Index/Config/GetCacheDirectory filter to specify a custom cache directory location.

CLI Commands

The plugin provides a WP-CLI command for managing S3 indexes:

Create Full Index

Creates a complete index by scanning all objects in the S3 bucket. It is recommended to setup this action as a daily cron, to eshure that the index is as accurate as possible over time (it is however maintained when a file is deleted or added in realtime).

This command:

• Scans the entire S3 bucket
• Groups files by blog ID, year, and month
• Creates JSON index files in the temporary directory
• Clears existing cache before starting
• Provides progress updates during execution

How It Works

Index Structure

The plugin creates JSON index files organized by:

• Blog ID: 1 for single-site, specific ID for multisite
• Year: 4-digit year (e.g., 2023)
• Month: 2-digit month (e.g., 01)

Index files are stored as: {cache-directory}/s3-index-{blogId}-{year}-{month}.json

The cache directory is automatically generated to be unique per site (e.g., /tmp/s3-index-0f256ede/) to prevent collisions when multiple sites run on the same server.

File Path Patterns

The plugin recognizes these S3 path patterns:

Single Site:

Multisite:

Multisite Multi Network:

Caching Strategy

The plugin uses a multi-layer caching approach:

1. Static Cache: In-memory cache for the current request
2. WordPress Object Cache: Persistent cache using WordPress's object cache system
3. Composite Cache: Combines both strategies for optimal performance

Cache keys follow the pattern: index_{blogId}_{year}_{month}

Stream Wrapper Integration

The plugin registers a custom S3 stream wrapper that:

• Intercepts S3 file operations
• Checks local indexes before making S3 API calls
• Provides transparent integration with WordPress filesystem functions. If any error occurs, it will delegate to the standard s3 wrapper.
• It only should cache file_exist check. This is sufficient to make a noticable performance improvement.

Running Tests

If tests are available, run them with:

Code Standards

The plugin follows PSR-4 autoloading and PSR-12 coding standards. All classes use dependency injection and implement appropriate interfaces for testability.

Troubleshooting

Common Issues

Plugin not working:

• Ensure S3 Uploads plugin is installed and active
• Check that S3 credentials are properly configured
• Verify WordPress has write permissions to the cache directory

Index files not created:

• Run wp s3-index create to generate initial indexes
• Check error logs for permission issues
• Ensure sufficient disk space in the cache directory

Performance issues:

• Verify cache is working: check WordPress object cache status
• Consider increasing cache TTL for static content
• Monitor cache directory size and clean up old indexes periodically

Multiple sites on same server:

• Each site automatically gets its own cache directory to prevent collisions
• Cache directories are generated as /tmp/s3-index-{8-char-uuid}/
• You can verify your site's cache directory with: wp eval "echo (new S3_Local_Index\Config\Config(new WpService\WpService()))->getCacheDirectory();"

Debug Logging

The plugin logs important events to the WordPress error log:

Enable WordPress debug logging in wp-config.php:

Contributing

1. Fork the repository
2. Create a feature branch: git checkout -b feature/your-feature
3. Make your changes with proper documentation
4. Add or update tests as needed
5. Submit a pull request

Please ensure all new code includes proper PHPDoc blocks and follows the existing code style.

License

This plugin is licensed under the MIT License. See the composer.json file for details.

Authors

• Sebastian Thulin - [email protected]

Support

For issues and questions:

• Create an issue on the GitHub repository
• Contact the development team at Helsingborg Stad