Libraries tagged by crawl

stil/curl-robot

33 Favers
22785 Downloads

Parallel URL crawling extension to curl-easy.

silverstripe-labs/googleanalytics

32 Favers
10112 Downloads

The Google Analytics module consists of 2 components that can be employed independently: The Google Logger injects the google analytics javascript snippet into your source code and logs relevant events (as of now only crawler visits) The Analyzer adds the Google Analytics UI to your CMS.

Go to Download

s25/dom-trawler

0 Favers
5794 Downloads

Yet another DOM crawler, inspired by symfony/dom-crawler

Go to Download

ramphor/rake

3 Favers
222 Downloads

The spider/crawler framework written in PHP

Go to Download

openbuildings/spiderling

62 Favers
134216 Downloads

Crawl the web with kohana or phantomjs.

Go to Download

nitsan/ns-twitter

3 Favers
24213 Downloads

Display fully customizable, responsive, and search engine crawlable Twitter feeds on your TYPO3 website. The TYPO3 Twitter Extension easily integrates to display Tweets that perfectly match your site's appearance and offers many customization options. Easily embed Twitter Timelines, Feeds, Follow buttons, and Like buttons on posts, pages, and widget areas. Documentation & Free Support: https://t3planet.com/typo3-twitter-extension

Go to Download

neclimdul/coveo-push-api

0 Favers
619 Downloads

The Push API allows you to *push* items and security identities, as opposed to letting standard Coveo Cloud V2 crawlers *pull* this data from a content repository. This is especially useful when you need to index content from a cloud or on-premises system for which no dedicated source type exists in the Coveo Cloud V2 platform.

Go to Download

mjp91/sitemap-generator

1 Favers
16802 Downloads

Generates an XML site map for use by web crawlers

Go to Download

mauricerenck/darkvisitors

16 Favers
262 Downloads

Kirby robots.txt plugin for blocking AI Crawlers and Bots

Go to Download

kiwa/link-obfuscator

0 Favers
5402 Downloads

Strong link obfuscation with PHP and JavaScript. Object-oriented, using the Rot47 algorithm. Hides https, email, phone and sms links from being crawled by spambots.

Go to Download