Download the PHP package goat/the-cache-warmer without Composer

On this page you can find all versions of the php package goat/the-cache-warmer. It is possible to download/install these versions without Composer. Possible dependencies are resolved automatically.

FAQ

After the download, you have to make one include require_once('vendor/autoload.php');. After that you have to import the classes with use statements.

Example:
If you use only one package a project is not needed. But if you use more then one package, without a project it is not possible to import the classes with use statements.

In general, it is recommended to use always a project to download your libraries. In an application normally there is more than one library needed.
Some PHP packages are not free to download and because of that hosted in private repositories. In this case some credentials are needed to access such packages. Please use the auth.json textarea to insert credentials, if a package is coming from a private repository. You can look here for more information.

  • Some hosting areas are not accessible by a terminal or SSH. Then it is not possible to use Composer.
  • To use Composer is sometimes complicated. Especially for beginners.
  • Composer needs much resources. Sometimes they are not available on a simple webspace.
  • If you are using private repositories you don't need to share your credentials. You can set up everything on our site and then you provide a simple download link to your team member.
  • Simplify your Composer build process. Use our own command line tool to download the vendor folder as binary. This makes your build process faster and you don't need to expose your credentials for private repositories.
Please rate this library. Is it a good library?

Informations about the package the-cache-warmer

Python Unit Tests

The Cache Warmer Module

Overview

The Cache Warmer module for Magento 2 is designed to improve website performance by proactively warming up cache entries for frequently visited pages. This helps reduce the response time for user requests, especially during high-traffic periods or when cache entries have expired.

The module works by analyzing nginx access logs and sitemap data to identify popular pages and then using a Python script to fetch these URLs, thereby pre-populating the cache with the requested content.

Key Features

Architecture

Components

  1. WarmupFromNginx Cron Job: Parses nginx access logs and generates CSV files with top visited URLs
  2. WarmupFromSitemap Cron Job: Processes sitemaps to generate CSV files for cache warming
  3. WarmupFromConfig Cron Job: Uses configured CSV files for warmup operations
  4. UrlParser Service: Extracts page URLs from nginx access log, filters out non-page resources
  5. Sitemap Parser: Parses XML sitemap files to extract product/category pages and other important URLs
  6. CacheWarmer Model: Executes the Python warming script with configured parameters
  7. Message Queue System: Allows for asynchronous cache warmup execution via queue consumers
  8. Console Commands: Manual execution capabilities through CLI commands
  9. Admin Configuration: Provides interface to configure paths, timeout settings, and schedules

Installation

There are two ways to install the Cache Warmer module:

Method 1: Manual Installation (Legacy)

  1. Copy module files to app/code/Goat/TheCacheWarmer/
  2. Run the following commands in your Magento root directory:

Method 2: Composer Installation (Recommended)

Install directly from Packagist using Composer:

After installation via Composer, run:

Performance Improvements & Core Web Vitals (CWV) Results

The Cache Warmer module has demonstrated significant improvements in website performance and Core Web Vitals scores, particularly for default Magento themes.

Before vs After Cache Warming Performance Comparison

Metric Before Warmup After Warmup Improvement
Largest Contentful Paint (LCP) 2.5s+ <1.0s 60%+ reduction
First Input Delay (FID) 150ms+ <50ms 67%+ reduction
Cumulative Layout Shift (CLS) 0.25+ <0.1 60%+ reduction
Time to First Byte (TTFB) 1200ms+ <900ms 25%+ reduction

Key Performance Benefits

Visual Performance Comparison

Real-World Impact

With proper cache warming configuration:

Real-World Performance Results

Real website example showing TTFB improvement after implementing Cache Warmer module

New Relic Monitoring Dashboard

The Cache Warmer module includes a comprehensive New Relic dashboard to monitor cache warming activities and performance metrics. This dashboard provides valuable insights into:

Dashboard Features

The New Relic dashboard includes the following key visualizations:

  1. Cache status by POP's - Shows cache performance across different geographic locations
  2. Warmer cache status - Displays warming activity and success rates
  3. MISS vs HIT Excluding bots - Performance comparison excluding automated traffic
  4. Top requested IPs - Identifies high-traffic sources
  5. TTFB based on Transactions duration - Time to First Byte performance metrics
  6. MISS URLS Excluding bots - Detailed view of cache misses that require warming

Using the Dashboard

To use this dashboard:

  1. Import the New Relic JSON configuration file found at src/monitoring-dashboards/new-relic.json
  2. Configure your project ID in the dashboard variables
  3. Monitor cache performance and warming effectiveness in real-time

Example screenshot of the New Relic monitoring dashboard for the Cache Warmer module

Fastly Configuration Requirements

When using this Cache Warmer module with Fastly as your CDN, ensure that the following configuration is added to your Real Time Log Streaming settings in Fastly:

  1. Navigate to Stores > Configuration > System > Full Page Cache > Fastly Configuration > Tools > Real Time Logging
  2. Add these fields to your Real Time Log Streaming configuration:

These additional fields are required for proper New Relic dashboard functionality with Fastly configurations.

Rate Limiting & Threading Configuration

The Cache Warmer module now supports advanced configuration options for optimizing performance and preventing server overload:

Threaded Execution (Parallel Processing)

By default, threading is disabled to ensure stability across different server configurations. When enabled, the module can process multiple URLs in parallel using Python's ThreadPoolExecutor, which significantly improves performance.

To enable threaded execution:

  1. Navigate to Stores > Configuration > Advanced > Cache Warmer
  2. Set "Use Threads" to "Yes"
  3. Configure "Threads Count" with your desired number of concurrent threads (default is 5)

⚠️ Important: Only enable threading if your server has sufficient CPU resources available to handle multiple parallel requests without performance degradation.

Rate Limiting

The module now supports rate limiting to control how many URLs are warmed per minute, preventing overwhelming the server or CDN with too many concurrent requests.

To configure rate limiting:

  1. Navigate to Stores > Configuration > Advanced > Cache Warmer
  2. Set "Rate Limit (URLs per minute)" with your desired limit
  3. Setting this value to 0 disables rate limiting entirely

When rate limiting is enabled, the module will automatically adjust the timing between requests to maintain the specified rate.

Performance Considerations

These advanced options provide fine-grained control over the cache warming process for optimal performance tuning based on your specific infrastructure requirements.

Configuration

Access configuration through Stores > Configuration > Advanced > Cache Warmer

General Settings

Cron Settings

Fastly Configuration Considerations

When using this Cache Warmer module with Fastly as your CDN, it's crucial to configure the appropriate shielding location for optimal performance.

Importance of Selecting the Best Shielding Location

Fastly offers multiple geographic locations where you can set up shield nodes (also known as "shielding locations"). These shielded locations serve cached content from a central point to reduce latency and improve performance.

However, not all Fastly shielding configurations provide shared caching between nodes. Some setups only cache content within their specific location, meaning:

Performance Impact

This non-shared caching behavior can significantly impact your cache warming effectiveness because:

  1. Reduced Cache Hit Rate: Content warmed at one shield location may not be available to users accessing the site from other regions
  2. Inconsistent Performance: Users in different geographic areas might experience varying performance as content gets progressively cached
  3. Wasted Warmup Effort: Time and resources spent warming up cache in locations that don't share with others are less effective

Recommended Best Practice

Select a single, primary shielding location that:

If you're using this module in conjunction with Fastly and are concerned about cache sharing, we recommend:

  1. Choosing one primary shielding location for your entire configuration
  2. Testing performance metrics before and after implementing this approach
  3. Monitoring cache hit ratios across different geographic regions to ensure consistency

This ensures that content warmed by the Cache Warmer module will be available from multiple Fastly edge nodes, maximizing both cache effectiveness and user experience.

Sitemap Configuration for Robots.txt

To enable sitemap.xml submission in robots.txt file for better search engine indexing, follow one of these methods:

Method 1: Using Configuration Settings (Recommended)

  1. Log in to your Magento 2 Admin Panel
  2. Navigate to Stores > Settings > Configuration
  3. Select Catalog and open the XML Sitemap tab
  4. Expand the Search Engine Submission Settings section
  5. Set Enable Submission to Robots.txt to Yes
  6. Click Save Config

Method 2: Customizing robots.txt

If you need to add a specific or custom sitemap URL:

  1. Go to Content > Design > Configuration
  2. Select your theme and click Edit
  3. Under Search Engine Robots, find the Edit custom instruction of robots.txt File field
  4. Add Sitemap: [your-sitemap-url] (e.g., Sitemap: https://example.com/pub/sitemap.xml) to the field
  5. Save the configuration

Usage

Manual Execution

You can manually trigger the cache warming process by running:

Console Commands

The module provides three console commands for manual execution:

This command is useful when a cache warming process gets stuck (e.g., due to an unexpected crash or server interruption) and leaves behind a lock flag that prevents subsequent executions. Running this command will manually remove the stale lock, allowing cache warming processes to run again.

Via Admin Panel

Configure the module through Stores > Configuration > Advanced > Cache Warmer and set your preferred settings.

Through Queue System

The module supports asynchronous execution via message queues. When enabled, cache warmups are published to a queue for processing by consumers.

Technical Details

CSV Generation Process

  1. Parses nginx access log files (default: /var/log/nginx/access.log)
  2. Extracts URLs from GET requests in the log format
  3. Filters out static assets, API endpoints, and other non-page resources
  4. Counts URL visits to identify top pages
  5. Generates CSV file with up to 1000 most visited page URLs

Python Warmer Script

The module uses a Python script (src/warmer.py) that:

Filtering Logic

The module filters out URLs matching these patterns:

Logging

All operations are logged to /var/log/the_cache_warmer.log. The logs include:

Requirements

Note: In most hosting environments, Python 3 is already installed by default. If it's not available, you'll need to install it via your system's package manager (e.g., apt-get install python3 on Debian/Ubuntu or yum install python3 on CentOS/RHEL). The requests library is required and can be installed using pip: pip3 install requests.

Magento Cloud Note: For Magento Cloud instances, the nginx access log file location may differ from the standard path. Instead of /var/log/nginx/access.log, you might need to use /var/log/platform/project_id/access.log where project_id is your specific project identifier. Ensure that the configured log path matches your environment's actual log file location.

Ignored User Agents Feature

The Cache Warmer module includes an "Ignored User Agents" feature that allows you to exclude specific bot traffic, crawlers, or other automated requests from being included in cache warming operations. This helps prevent unnecessary cache warming for non-human visitors and ensures that your caching efforts focus on actual customer traffic.

How It Works

When enabled, the module will:

  1. Parse nginx access logs line by line
  2. Check each request's User-Agent header against configured patterns
  3. Skip processing any requests from user agents that match the ignored patterns
  4. Only include human visitor traffic in the cache warming process

Configuration

To configure ignored user agents:

  1. Navigate to Stores > Configuration > Advanced > Cache Warmer
  2. Expand the "Ignored User Agents" section
  3. Add one or more user agent patterns (one per line)
  4. Save configuration and flush cache

Patterns can be:

Use Cases

This feature is particularly useful for:

Troubleshooting

Common Issues

  1. Access Log Not Found: Ensure the path in cron job matches your actual nginx access log location
  2. Python Path Issues: Verify Python executable path is correct in admin configuration
  3. Permissions: Make sure Magento has write permissions to media directory
  4. Queue Not Processing: Check that queue consumers are running (bin/magento queue:consumers:start)

Debugging

Enable debug logging by setting log level to DEBUG or check /var/log/the_cache_warmer.log for detailed execution information.

All versions of the-cache-warmer with dependencies

PHP Build Version
Package Version
Requires magento/framework Version *
php Version ~8.1||~8.2||~8.3
Composer command for our command line client (download client) This client runs in each environment. You don't need a specific PHP version etc. The first 20 API calls are free. Standard composer command

The package goat/the-cache-warmer contains the following files

Loading the files please wait ...