Download the PHP package rindow/rindow-rl-agents without Composer
On this page you can find all versions of the php package rindow/rindow-rl-agents. It is possible to download/install these versions without Composer. Possible dependencies are resolved automatically.
Download rindow/rindow-rl-agents
More information about rindow/rindow-rl-agents
Files in rindow/rindow-rl-agents
Package rindow-rl-agents
Short Description Reinforcement Learning Agents
License BSD-3-Clause
Informations about the package rindow-rl-agents
Rindow RL Agents
Reinforcement-learning agents and training utilities for Rindow Neural Networks.
The package provides discrete and continuous-control algorithms, replay and rollout buffers, training runners, action-mask support, checkpoint persistence, and CPU/GPU-compatible tensor operations.
Features
- Advantage Actor-Critic (A2C)
- Deep Q-Network and Double DQN (DQN/DDQN)
- Proximal Policy Optimization (PPO)
- Deep Deterministic Policy Gradient (DDPG)
- Soft Actor-Critic with generalized State-Dependent Exploration (SAC+gSDE)
- REINFORCE
- Linear Q-learning with tile coding
- True Online Sarsa(λ) with tile coding
- Discrete action masks
- Generalized Advantage Estimation (GAE)
- Replay and on-policy rollout buffers
- Best-model checkpoints and deterministic evaluation
- Host and OpenCL accelerator backends
Installation
Install the package with Composer:
Composer installs Rindow RL Agents and its declared runtime dependencies. Your application only needs the generated Composer autoloader:
The exact PHP extensions and native libraries required for acceleration depend on the selected Rindow Math Matrix backend. A CPU backend is sufficient to use the package.
Quick start: CartPole with DDQN
The following example assumes that a CartPole environment implementing
Interop\Polite\AI\RL\Environment is available. The environment and evaluation
environment are separate so evaluation does not disturb the training trajectory.
Reinforcement learning is stochastic. Treat the values above as a complete API example, not as a convergence guarantee for every backend or random seed.
Choosing an algorithm
| Algorithm | Action space | Data model | Typical use |
|---|---|---|---|
| A2C | Discrete or continuous | On-policy rollout | Compact synchronous actor-critic |
| DQN/DDQN | Discrete | Replay buffer | Discrete value-based control |
| PPO | Discrete or continuous | On-policy rollout | Clipped policy optimization |
| DDPG | Continuous | Replay buffer | Deterministic continuous control |
| SAC+gSDE | Continuous | Replay buffer | Entropy-regularized continuous control |
| REINFORCE | Discrete | Complete episode | Basic Monte Carlo policy gradient |
| Q-learning | Discrete | Per transition | Linear tile-coded value learning |
| True Online Sarsa(λ) | Discrete | Per transition | On-policy learning with Dutch traces |
CPU and GPU execution
Rindow Neural Networks selects the numerical backend. For example, an OpenCL GPU backend can be selected before starting PHP:
Use the agent backend's NDArray objects for observations, replay storage, and
action masks. The bundled device wrappers transfer environment observations to
the configured device and actions back to the host environment.
Environment stepping and runner control flow remain on the PHP host. Batched network operations, losses, reductions, masking, replay gathers, and optimizer updates execute on the selected backend.
CPU and GPU runs need not follow identical stochastic trajectories. Compare multiple seeds and task-level statistics rather than expecting bitwise-identical training logs.
Action masks
DQN, A2C, PPO, Q-learning, and True Online Sarsa(λ) support discrete action masks. Configure dictionary fields when creating the agent:
Each mask must contain one value per action and enable at least one action. Disabled actions are excluded during both exploration and greedy selection.
Reward transformation
Several runners accept a rewardFunction callback. It changes the reward stored
for training without changing the raw reward reported by evaluation. This is
useful for controlled reward-shaping experiments, but arbitrary shaping can
change the optimal policy.
Time-limit truncation and true MDP termination are handled separately. The bundled runners normally bootstrap at truncation and use zero bootstrap value at true termination.
Checkpoints
Agents expose a common persistence pattern:
Neural checkpoints include the online models and any target networks needed to resume training. SAC also stores its learned entropy coefficient. Existing model buffers are updated in place so checkpoints and target synchronization remain compatible with compiled Rindow Neural Networks function pipelines.
Samples
The samples/ directory contains complete programs for:
- CartPole
- Maze
- MountainCar
- Continuous MountainCar
- Pendulum
It includes vector and image-observation examples where applicable. Run samples from the project root:
Some samples accept RL_* environment variables for seeds, training length,
evaluation intervals, model paths, and demo control. See the constants and
environment lookups at the top of each sample.
Documentation
The full reference manual starts at docs/index.md.
- Getting started
- Core concepts
- Configuration reference
- Agent reference
- Buffers
- Training runners
- CPU and GPU backends
- API index
Testing
Install development dependencies and run PHPUnit:
On Windows:
The unit suite checks analytical update targets, losses, GAE, action masks,
buffer layouts, action bounds, target-network synchronization, Polyak updates,
and checkpoint behavior. Learning convergence is intentionally kept outside the
unit-test contract; the scripts under tests/ are experiment and smoke-test
drivers.
Package
Composer package name: rindow/rindow-rl-agents
Primary namespace: Rindow\RL\Agents
All versions of rindow-rl-agents with dependencies
rindow/rindow-neuralnetworks Version ^2.2
rindow/rindow-math-matrix-matlibffi Version ^1.1
interop-phpobjects/polite-ai Version ^1.1.1