Libraries tagged by 64Robots

Crawler for any type of site using robots.txt and sitemap.xml as the source of URL. Useful for cache regenerating.

Go to Download

PHPCrawl is a webcrawler/webspider-library written in PHP. It supports filters, limiters, cookie-handling, robots.txt-handling, multiprocessing and much more.

Go to Download

Define your HTML head attributes in a manifest shared between Laravel and an SPA project so that robots crawling the page have access to up too path-specific metadata without requiring server side rendering on the SPA side

Go to Download

PHPCrawl is a webcrawler/webspider-library written in PHP. It supports filters, limiters, cookie-handling, robots.txt-handling, multiprocessing and much more.

Go to Download

stanford_metatag_nobots for blocking robots

Go to Download

This module blocks (well-behaved) search engine robots from crawling, indexing, or archiving your site by setting a "X-Robots-Tag: noindex,nofollow,noarchive" HTTP header.

Go to Download

PHP package to handle data from liquid handling robots

Go to Download