Download the PHP package dissnik/robots-txt without Composer
On this page you can find all versions of the php package dissnik/robots-txt. It is possible to download/install these versions without Composer. Possible dependencies are resolved automatically.
FAQ
Example:
In general, it is recommended to use always a project to download your libraries. In an application normally there is more than one library needed.
- Some hosting areas are not accessible by a terminal or SSH. Then it is not possible to use Composer.
- To use Composer is sometimes complicated. Especially for beginners.
- Composer needs much resources. Sometimes they are not available on a simple webspace.
- If you are using private repositories you don't need to share your credentials. You can set up everything on our site and then you provide a simple download link to your team member.
- Simplify your Composer build process. Use our own command line tool to download the vendor folder as binary. This makes your build process faster and you don't need to expose your credentials for private repositories.
Informations about the package robots-txt
Robots.txt manager for Laravel
A powerful, flexible robots.txt management package for Laravel applications with environment-aware rules, caching, and fluent API.
Features
- Fluent API - Easy-to-use chainable methods with context-aware syntax
- Environment-based rules - Different rules for local, staging, production
- Smart caching - Configurable HTTP caching for performance
- Conflict resolution - Automatic Allow/Disallow priority handling
- Laravel integration - Service provider, facades, and middleware
- Extensible - Custom rules and programmatic control
Installation
Configuration
Publish the configuration file:
Configuration File (config/robots-txt.php)
Conflict Check
[!WARNING] File takes priority! If a
public/robots.txtfile exists on your server, it will OVERRIDE the package's generated content.
This command will:
- Detect if a robots.txt file exists in public directory
- Show file details and potential conflicts
- Help you resolve conflicts
Quick Start
Basic Usage (Configuration Only)
For most use cases, you only need to configure the package. The robots.txt file will be automatically generated and served at /robots.txt.
-
Publish the configuration (if you want to customize):
- Edit the configuration file (config/robots-txt.php) with your rules.
- Access your robots.txt at https://your-domain.com/robots.txt
That's it! The package handles route registration, content generation, and caching automatically.
Programmatic Usage
If you need dynamic rules or programmatic control, you can use the fluent API. Important: When using programmatic rules, you should disable the package's automatic route registration to avoid conflicts.
Step 1: Disable Automatic Route via Environment Variable Add to your .env file:
Or modify directly in config/robots-txt.php:
Step 2: Define Your Custom Route
Create your own route in routes/web.php:
The package includes a caching middleware that's automatically applied to the robots.txt route:
Fluent API Examples
[!NOTE] The examples below show advanced programmatic usage. For basic setup, you only need configuration.
Environment-Specific Rules
Conditional Rules
Helper Methods
Advanced Usage
[!TIP] Most users only need configuration. The following sections are for advanced programmatic control.
Programmatic Rule Management
Rule Conflicts Resolution
The package automatically resolves conflicts where both Allow and Disallow rules exist for the same path (Allow has priority).
Cache Management
Directive Removal
Examples
Complete Production Setup
E-commerce Site Example
Dynamic Rules Based on Conditions
API Reference
Main Methods
| Method | Description | Returns |
|---|---|---|
forUserAgent(string $userAgent, callable $callback) |
Set user agent for subsequent rules | self |
forEnvironment(string\|array $environments, callable $callback) |
Define environment-specific rules | self |
directive(string $directive, mixed $value) |
Add global custom directive | self |
sitemap(string $url) |
Add sitemap directive | self |
host(string $host) |
Add host directive | self |
cleanParam(string $param, ?string $path = null) |
cleanParam(string $param, ?string $path = null) | self |
blockAll() |
Disallow all crawling for all user agents | self |
allowAll() |
Allow all crawling for all user agents | self |
clear() |
Clear all programmatic rules | self |
reset() |
Clear rules and reload from configuration | self |
generate() |
Generate robots.txt content | string |
clearCache() |
Clear cached robots.txt content | bool |
Information Methods
| Method | Description | Returns |
|---|---|---|
getRules() |
Get all defined rules as array | array |
getSitemaps() |
Get all sitemap URLs | array |
getDirectives() |
Get all global directives | array |
getUserAgentDirectives(string $userAgent) |
Get directives for specific user agent | array |
getEnvironmentRules() |
Get registered environment callbacks | array |
checkConflicts() |
Check for rule conflicts (allow/disallow) | array |
getUserAgents() |
Get all defined user agents | array |
hasUserAgent(string $userAgent) |
Check if user agent is defined | bool |
Modification Methods
| Method | Description | Returns |
|---|---|---|
removeDirective(string $directive, mixed $value = null) |
Remove global directive | self |
removeUserAgentDirective(string $userAgent, string $directive, mixed $value = null) |
Remove user agent directive | self |
Context Methods (available inside callbacks)
| Method | Description |
|---|---|
allow(string $path) |
Add allow rule |
disallow(string $path) |
Add disallow rule |
crawlDelay(float $delay) |
Set crawl delay |
cleanParam(string $param, ?string $path = null) |
Add clean-param |
directive(string $directive, mixed $value) |
Add custom directive |
blockAll() |
Disallow all paths |
allowAll() |
Allow all paths |
removeDirective(string $directive, mixed $value = null) |
Remove directive |
EnvironmentContext ($robots in forEnvironment() callbacks):
| Method | Description |
|---|---|
forUserAgent(string $userAgent, callable $callback) |
Define user agent rules |
sitemap(string $url) |
Add global sitemap |
host(string $host) |
Add global host |
cleanParam(string $param, ?string $path = null) |
Add global clean-param |
directive(string $directive, mixed $value) |
Add global custom directive |
blockAll() |
Block all crawlers |
allowAll() |
Allow all crawlers |
Conditional Methods (via Conditionable trait)
| Method | Description |
|---|---|
when(bool $condition, callable $callback) |
Execute if condition is true |
unless(bool $condition, callable $callback) |
Execute if condition is false |
Troubleshooting
Common Issues
-
Rules not applying?
- Make sure you're calling methods in the correct context
- User-agent specific methods (allow(), disallow(), crawlDelay()) must be inside forUserAgent() callbacks
- Check your current environment: dd(app()->environment())
- If your rules are not showing up in
/robots.txt:- Check if a
public/robots.txtfile exists (it overrides package rules) - Run the conflict check:
php artisan robots-txt:check - The package-generated robots.txt will NOT work if a file exists in
public/robots.txt
- Check if a
-
Caching issues?
- Run RobotsTxt::clearCache() to clear cached content
- Check config: config('robots-txt.cache.enabled')
- Disable middleware caching in route if needed
-
Route not working?
- Check if route is enabled: config('robots-txt.route.enabled') or env('ROBOTS_TXT_ROUTE_ENABLED')
- Run php artisan route:list to see if route is registered
- Make sure no physical public/robots.txt file exists
- Configuration not loading?
- Make sure you published config:
php artisan vendor:publish --tag=robots-txt-config - Check config structure matches expected format
- Verify environment is set correctly
- Make sure you published config:
Debug Mode
License
The MIT License (MIT). Please see License File for more information.
