Download the PHP package memeddev/mysql-to-google-bigquery without Composer
On this page you can find all versions of the php package memeddev/mysql-to-google-bigquery. It is possible to download/install these versions without Composer. Possible dependencies are resolved automatically.
FAQ
Example:
In general, it is recommended to use always a project to download your libraries. In an application normally there is more than one library needed.
- Some hosting areas are not accessible by a terminal or SSH. Then it is not possible to use Composer.
- To use Composer is sometimes complicated. Especially for beginners.
- Composer needs much resources. Sometimes they are not available on a simple webspace.
- If you are using private repositories you don't need to share your credentials. You can set up everything on our site and then you provide a simple download link to your team member.
- Simplify your Composer build process. Use our own command line tool to download the vendor folder as binary. This makes your build process faster and you don't need to expose your credentials for private repositories.
Informations about the package mysql-to-google-bigquery
MySQL to Google BigQuery Sync Tool
Table of Contents
- How it works
- Requirements
- Usage
- Credits
- License
How it works
Steps when no order column has been supplied:
- Count MySQL table rows
- Count BigQuery table rows
- MySQL rows > BigQuery rows?
- Get the rows diff, split in batches of XXXXX rows/batch
Steps when order column has been supplied:
- Get max value for order column from MySQL table
- Get max value for order column from BigQuery table
- Max value MySQL > Max value BigQuery?
- Delete all rows with order column value = max value BigQuery to make sure no duplicate records are being created in BigQuery
- Get max value for order column from BigQuery table
- Get the rows diff based on new max value BigQuery, split in batches of XXXXX rows/batch
Final three steps:
- Dump MySQL rows to a JSON
- Send JSON to BigQuery
- Repeat until all batches are sent
Tip: Create a cron job for keep syncing the tables using an interval like 15 minutes (respect the Load Jobs quota policy)
Requirements
The following PHP versions are supported:
- PHP 7
- HHVM
- PDO Extension with MySQL driver
Usage
Download the library using composer:
Now, define some environment variables or create a .env file on the root of the project, replacing the values:
PS: To create the Google Service Account JSON Key File, access https://console.cloud.google.com/apis/credentials/serviceaccountkey
Run:
If you want to auto create the table on BigQuery:
If you want to delete (and create) the table on BigQuery for a full dump:
Credits
:heart: Memed SA (memed.com.br)
License
MIT license, see LICENSE
All versions of mysql-to-google-bigquery with dependencies
vlucas/phpdotenv Version ^2.4
php-di/php-di Version ^5.4
google/cloud Version ^0.11.1
doctrine/dbal Version ^2.5
facile-it/doctrine-mysql-come-back Version ^1.6