Download the PHP package jez500/web-scraper-for-laravel without Composer
On this page you can find all versions of the php package jez500/web-scraper-for-laravel. It is possible to download/install these versions without Composer. Possible dependencies are resolved automatically.
Informations about the package web-scraper-for-laravel
Web Scraper for Laravel
A powerful and flexible Laravel package for scraping external web pages and extracting structured data. Whether you need to fetch static HTML content or scrape JavaScript-rendered pages, this package provides a clean, fluent interface for all your web scraping needs.
Key Features
- HTTP Scraping: Support for standard HTTP requests using Laravel's HTTP client
- API Scraping: Support for scraping JavaScript-rendered pages (using jez500/seleniumbase-scrapper)
- Extract Scraping: Support for article extraction using jez500/extract (Mercury Parser)
- Multiple Extraction Methods: Extract data using CSS selectors, XPath expressions, regular expressions, or JSON dot notation
- Typed Schemas: Support for type-safe scrape schemas via DTOs and
fromDto(...) - User Agent Rotation: Built-in support for rotating user agents to avoid being blocked
- Response Caching: Cache responses to avoid repeated requests
- Comprehensive Error Handling: Robust error handling for network issues and parsing errors
Installation
For JavaScript-rendered page scraping, you'll also need to install:
For article extraction, you'll need to run the Extract service:
Quick Start
Basic Scraping
Extract Structured Data
Usage Examples
Basic HTTP Scraping
CSS Selector Extraction
XPath Extraction
Regular Expression Extraction
JSON Data Extraction
JavaScript-Rendered Page Scraping
Article Extraction
Extract clean article content from web pages using Mercury Parser:
Custom Drivers
Register a custom driver in a service provider, then resolve it by name:
http() and api() remain available for backwards compatibility, and they now
resolve through the same driver layer internally.
Typed Scrape Schemas
fromDto(...) also accepts a single FieldExtractionDto when you only need one
extraction rule:
Advanced Features
Custom User Agents
Response Caching
Error Handling
Testing
When testing scraping logic, use Laravel's HTTP client mocking to avoid making actual network requests:
Best Practices
- Always cache requests when data doesn't change frequently to reduce server load
- Use appropriate delays between requests to respect rate limits and avoid being blocked
- Check robots.txt before scraping to ensure you're allowed to access the content
- Handle errors gracefully and implement retry logic for transient failures
- Use typed schemas for complex data extraction to ensure consistency and type safety
- Test scraping logic with mocked responses to ensure reliability
- Respect website terms of service and implement proper rate limiting
- Consider using API endpoints when available instead of scraping HTML
- Validate extracted data to ensure it meets your expected format
- Log scraping activities for debugging and monitoring purposes
Contributing
Contributions are welcome! Please follow these guidelines:
- Fork the repository
- Create a feature branch (
git checkout -b feature/amazing-feature) - Make your changes
-
Run coding standards and tests locally:
- Commit your changes with clear, descriptive messages
- Push to your branch (
git push origin feature/amazing-feature) - Open a Pull Request
When submitting PRs, please ensure:
- Code follows PSR standards
- Tests are included for new features
- Documentation is updated as needed
- All existing tests pass
Support
For bug reports, feature requests, or questions, please open an issue on GitHub.
License
The MIT License (MIT). Please see LICENSE.md for more information.