Download the PHP package shammaa/laravel-smart-scraper without Composer

Laravel Smart Scraper

Advanced intelligent web scraper for Laravel with caching, rate limiting, middleware, monitoring, and much more. Built on Puppeteer with smart features that make web scraping professional, efficient, and reliable.

🚀 Key Features

• ✅ Intelligent Caching - Automatic caching to avoid redundant requests
• ✅ Rate Limiting - Prevent overwhelming target websites
• ✅ User-Agent Rotation - Rotate user agents automatically to avoid detection
• ✅ Middleware System - Extensible middleware for request manipulation
• ✅ Automatic Retry - Exponential backoff retry logic for failed requests
• ✅ Screenshot & PDF - Capture screenshots and generate PDFs
• ✅ Proxy Support - Full proxy support with authentication
• ✅ Monitoring & Logging - Comprehensive monitoring and logging
• ✅ Schema Validation - Validate extracted data against schemas
• ✅ Concurrent Scraping - Scrape multiple URLs concurrently
• ✅ Queue Support - Process scraping jobs in background
• ✅ Error Handling - Robust error handling and recovery
• ✅ Smart Site Detection - Automatically detect site type and use appropriate selectors
• ✅ Multi-Site Support - Handle multiple websites with different HTML structures intelligently

📦 Installation

Install the package via Composer:

Publish the configuration file:

Install Node.js dependencies (required for Puppeteer):

Note: Make sure Node.js is installed and available in your PATH. If using NVM, see the NVM Configuration section below.

🎯 Quick Start

1. Create a Scraper

Generate a scraper class using the Artisan command:

This creates a file at app/Scrapers/ProductScraper.php:

2. Use the Scraper

📚 Basic Usage

Simple Scraping

With Options

With Parameters

You can pass parameters to the handle() method:

Then use it:

🔧 Advanced Features

Caching

The scraper automatically caches results to avoid redundant requests:

Cache TTL can be configured in config/smart-scraper.php:

Rate Limiting

Prevent overwhelming target websites with rate limiting:

Configure rate limits in config/smart-scraper.php:

User-Agent Rotation

User agents are automatically rotated to avoid detection:

Configure user agents in config/smart-scraper.php:

Proxy Support

Full proxy support with authentication:

Retry Logic

Automatic retry with exponential backoff:

Configure default retry settings in config/smart-scraper.php:

Screenshots

Capture screenshots of web pages:

PDF Generation

Generate PDFs from web pages:

Custom Headers

Add custom headers to requests:

Data Validation

Validate extracted data against a schema:

Middleware

Create custom middleware to modify requests:

🎨 Artisan Commands

Create a Scraper

List All Scrapers

Test a Scraper

🧠 Smart Site Detection & Multi-Site Support

The scraper can intelligently detect different websites and automatically use the appropriate selectors for each site. This means you can scrape multiple websites with different HTML structures using the same scraper class!

How It Works

1. Automatic Site Detection - The scraper detects the site type from URL patterns or HTML patterns
2. Smart Selectors - Uses site-specific selectors if available, falls back to generic selectors
3. Fallback System - If a selector fails, it automatically tries the next one

Creating a Smart Scraper

Use the --smart flag when creating a scraper:

This creates a scraper with smart selectors:

Smart Selector Methods

extract() - Extract text content

Tries multiple selectors until one works:

extractAttribute() - Extract attribute value

Tries multiple selectors to extract an attribute:

extractMultiple() - Extract array of values

Extracts multiple elements:

Site Profiles

Define site profiles in config/smart-scraper.php:

How Site Detection Works

1. URL Pattern Matching - First, tries to match URL patterns
2. HTML Pattern Matching - If URL doesn't match, analyzes HTML structure
3. Selector Priority - Uses site-specific selectors first, then falls back to generic ones

Example: Scraping Multiple Sites

The same scraper automatically adapts to each site's structure!

Manual Site Type Detection

You can also manually set or check the site type:

🔍 Monitoring & Logging

Monitoring is enabled by default. All scraping activities are logged:

Check logs in storage/logs/laravel.log:

Configure monitoring in config/smart-scraper.php:

⚙️ Configuration

All configuration options are available in config/smart-scraper.php:

🐛 Troubleshooting

NVM Configuration

If you're using Node.js via NVM and running scrapers via scheduled tasks, Node might not be available. To fix this:

1. Edit your ~/.bash_profile:

2. Add at the top:

3. Reload:

Note: It's not recommended to use NVM in production environments.

Common Issues

Issue: Puppeteer execution failed

Solution: Make sure Node.js and Puppeteer dependencies are installed:

Issue: Rate limit exceeded

Solution: Adjust rate limit settings or disable rate limiting:

Issue: Data validation failed

Solution: Check your validation schema and ensure data matches expected types.

📝 Examples

Example 1: E-commerce Product Scraper

Example 2: News Article Scraper

Example 3: Multiple URLs Concurrently

🤝 Contributing

Contributions are welcome! Please feel free to submit a Pull Request.

📄 License

This package is open-sourced software licensed under the MIT license.

🙏 Credits

Built with ❤️ by Shadi Shammaa

Made with Laravel Smart Scraper - Professional web scraping made easy! 🚀