Libraries tagged by CrawlLib

Translations Manager extension to use Google translation crawling to auto translate your __(), trans() fn

Go to Download

Framework-agnostic IndexNow client and queue for submitting URL changes to participating search engines (Bing, Yandex, Seznam, Naver, Yep).

Go to Download

Clusteer is a Puppeteer wrapper written for PHP, with the super-power of parallelizing pages across multiple browser instances.

Go to Download

A SilverStripe module to optimise the Meta, crawling, indexing, and sharing of your website content (forked from Cyber-Duck/Silverstripe-SEO)

Go to Download

Allows crawling of sitemaps and node-trees

Go to Download

Server-rendered SEO + crawlability for Laravel + Inertia apps — per-route meta/Open Graph/Twitter/JSON-LD on the first byte, dynamic sitemap.xml / robots.txt / llms.txt, and per-page Markdown. The PHP baseline that pairs with @particle-academy/fancy-inertia's .

Go to Download

A Magento 2 module that improves store performance by automatically warming cache through intelligent page crawling and async refresh mechanisms.

Go to Download

TYPO3 extension for generating llms.txt links based on the llmstxt.org specification and an on-site AI search assistant that helps visitors find information via ke_search and indexed_search.

Go to Download

Headless manager for the well-known files every modern web app owes bots AND agents — robots.txt, .well-known/security.txt, llms.txt, humans.txt, sitemap.xml, and an agents/AI manifest. Define them once in code/config and serve them consistently. Zero runtime deps; ships a default-open robots evaluator so scrapers can honor robots honestly, and makes it hard to leak a protected path. Framework-agnostic core + Laravel adapter.

Go to Download

Crawl urls from a webpage and provide a DomCrawler with Scraper Library

Go to Download

Dexi API Client for PHP 5.3+

Go to Download

Extension for the crwlr/crawler package containing steps utilizing a headless browser.

Go to Download

Go to Download

Display fully customizable, responsive, and search engine crawlable Twitter feeds on your TYPO3 website. The TYPO3 Twitter Extension easily integrates to display Tweets that perfectly match your site's appearance and offers many customization options. Easily embed Twitter Timelines, Feeds, Follow buttons, and Like buttons on posts, pages, and widget areas. Documentation & Free Support: https://t3planet.com/typo3-twitter-extension

Go to Download

This module blocks (well-behaved) search engine robots from crawling, indexing, or archiving your site by setting a "X-Robots-Tag: noindex,nofollow,noarchive" HTTP header.

Go to Download