Download the PHP package nonsapiens/bigquery-model-sync without Composer
On this page you can find all versions of the php package nonsapiens/bigquery-model-sync. It is possible to download/install these versions without Composer. Possible dependencies are resolved automatically.
Download nonsapiens/bigquery-model-sync
More information about nonsapiens/bigquery-model-sync
Files in nonsapiens/bigquery-model-sync
Package bigquery-model-sync
Short Description A Laravel library for coordinating batch record-level syncs from the local database, to BigQuery
License MIT
FAQ
Example:
In general, it is recommended to use always a project to download your libraries. In an application normally there is more than one library needed.
- Some hosting areas are not accessible by a terminal or SSH. Then it is not possible to use Composer.
- To use Composer is sometimes complicated. Especially for beginners.
- Composer needs much resources. Sometimes they are not available on a simple webspace.
- If you are using private repositories you don't need to share your credentials. You can set up everything on our site and then you provide a simple download link to your team member.
- Simplify your Composer build process. Use our own command line tool to download the vendor folder as binary. This makes your build process faster and you don't need to expose your credentials for private repositories.
Informations about the package bigquery-model-sync
BigQuery Model Sync
A Laravel package to synchronize Eloquent models with Google BigQuery tables.
Installation
-
Install the package via composer:
-
Publish the configuration file and migrations:
- Run the migrations:
Configuration
The configuration is located at config/bigquery.php.
Environment Variables
GOOGLE_CLOUD_PROJECT: Your Google Cloud Project ID.BIGQUERY_DATASET: The BigQuery dataset where tables will be created.BIGQUERY_LOCATION: The location for your BigQuery dataset (default:africa-south1).BIGQUERY_AUTOSYNC: Boolean to enable/disable automatic scheduled syncing (default:true).BIGQUERY_LOGGING: Boolean to enable/disable sync logging (default:true).BIGQUERY_CREDENTIALS: Path to your Google Cloud service account JSON file.
Localhost vs WIF (Workload Identity Federation)
Localhost:
Set BIGQUERY_CREDENTIALS in your .env file to the absolute path of your service account JSON key.
Deployed (WIF):
When deployed on Google Cloud (GKE, Cloud Run, etc.) using Workload Identity Federation, you should NOT provide a service account key. The BigQueryClient will automatically use the environment's default credentials. Leave BIGQUERY_CREDENTIALS empty or unset in your production environment.
Usage
1. Prepare your Model
Add the SyncsToBigQuery trait to the models you want to synchronize.
2. Configure Model Settings (Optional)
You can customize the sync behavior by defining properties or methods in your model:
| Property / Method | Default | Description |
|---|---|---|
$fieldsToSync |
[] (all fields) |
Array of columns to sync. |
$syncStrategy |
BATCH |
BATCH, REPLACE, or ON_INSERT. |
$bigQueryTableName |
Model table name | The target BigQuery table name. |
$syncSchedule |
null |
Cron expression for when the model should sync. |
$batchSize |
10000 |
Number of records per batch. |
$batchField |
sync_batch_uuid |
Database field used to track sync batches. |
3. Create BigQuery Table
Generate the SQL required to create the table in BigQuery:
Copy the output and run it in the BigQuery console.
4. Prepare Database (if using BATCH or ON_INSERT)
If you are using BATCH (default) or ON_INSERT strategies, you need a tracking column in your local database. Generate a migration:
Syncing Strategies
- BATCH: (Default) Synchronizes records in batches. It uses a UUID column to mark records that have been sent. Recommended for most use cases.
- REPLACE: Truncates the BigQuery table and re-uploads all data. Good for small tables or when records are frequently updated.
- ON_INSERT: Dispatches a job to sync an individual record immediately after it is created in the local database.
Geodata Mapping
If your model contains latitude and longitude information, the package can automatically map these into a single GEOGRAPHY field in BigQuery.
Configuration
Add the following properties or methods to your model:
| Property / Method | Default | Description |
|---|---|---|
$hasGeodata |
false |
Set to true to enable geodata mapping. |
$geodataFields |
['latitude', 'longitude'] |
The source fields in your local database. |
$mappedGeographyField |
geolocation |
The name of the GEOGRAPHY column in BigQuery. |
How it Works
- Table Generation: When running
php artisan make:bigquery-table, the package will exclude the individual latitude and longitude fields from the generated SQL and instead include a singleGEOGRAPHYfield (e.g.,geolocation). - Data Transformation: During synchronization, the package combines the latitude and longitude values into a WKT (Well-Known Text)
POINTstring:POINT(longitude latitude). - BigQuery Storage: BigQuery stores this as a native
GEOGRAPHYtype, allowing you to use BigQuery GIS functions for spatial analysis.
Note: The interactive
bigquery:set-modelcommand will automatically detect iflatitudeandlongitudefields are present and offer to configure this mapping for you.
Requirements for Schedule and Queue
Scheduling
The package includes a scheduled task that runs every minute to check if any models are due for synchronization based on their $syncSchedule.
To enable this, ensure the Laravel scheduler is running in your environment:
You can disable automatic scheduling by setting BIGQUERY_AUTOSYNC=false in your .env.
Queuing
The ON_INSERT strategy and the bigquery:sync --queue command rely on Laravel's queue system. Ensure you have a queue worker running:
Commands
bigquery:set-model
An interactive command to configure an Eloquent model for BigQuery synchronization. It guides you through selecting fields, configuring geodata mapping, and choosing a sync strategy.
- Interactive Steps:
- Select a model from your application.
- Choose which fields should be synchronized.
- Configure
GEOGRAPHYfield mapping iflatitudeandlongitudeare present. - Choose between immediate (
ON_INSERT) or scheduled (BATCH) synchronization. - Generate the necessary database migration for the tracking column.
make:bigquery-table
Generates the CREATE TABLE SQL required for BigQuery based on your Eloquent model's schema and configuration.
--class: (Required) The fully qualified class name of the model.
make:bigquery-model-migration
Creates a Laravel migration to add the required tracking column (default: sync_batch_uuid) to your model's database table.
--class: (Required) The fully qualified class name of the model.
bigquery:sync
Synchronizes a specific model with BigQuery. This command is typically called by the scheduler but can be run manually.
--class: (Required) The fully qualified class name of the model.--force: Run the sync even if it's not currently due according to the model's$syncSchedule.--queue: Dispatch the sync job to the queue instead of running it immediately in the foreground.--schedule: Override the cron expression defined in the model.
bigquery:sync-all
Discovers all models using the SyncsToBigQuery trait and runs their synchronization if they are due.
--force: Run synchronization for all discovered models, regardless of their schedule.
bigquery:truncate
Truncates the BigQuery tables associated with your models. Use with caution.
--all: Truncate tables for all models using theSyncsToBigQuerytrait.--class: A comma-separated list of model classes to truncate.- Note: This command requires manual confirmation and a randomly generated verification code to prevent accidental data loss.
