Download the PHP package keboola/table-backend-utils without Composer

Table backend utils

Common stuff for table backends (Snowflake|Synapse|Redshift) shared between apps.

Interfaces

Database

Keboola\TableBackendUtils\Database\DatabaseReflectionInterface

Schema

Keboola\TableBackendUtils\Schema\SchemaReflectionInterface

Function to retrieve information's about schema:

Table

Keboola\TableBackendUtils\Table\TableReflectionInterface

Function to retrieve information's about table:

Keboola\TableBackendUtils\Table\TableQueryBuilderInterface

Queries to work with table:

Keboola\TableBackendUtils\Table\TableStatsInterface

Table statistics

Column

Keboola\TableBackendUtils\Column\ColumnInterface

Table column definition:

View

Keboola\TableBackendUtils\View\ViewReflectionInterface

Function to retrieve information's about view:

Auth

Keboola\TableBackendUtils\Auth\UserReflectionInterface

Keboola\TableBackendUtils\Auth\GrantQueryBuilderInterface

Development

Preparation

Set up envs:

Set TEST_PREFIX= ENV

SYNAPSE

Create synapse server on Azure portal or using CLI.

set up env variables:

Teradata

Prepare Teradata servers on AWS/Azure and set following properties. See

create new database for tests:

set up env variables:

AWS

In order to download TD odbc driver, create a user who can access S3 bucket with the driver package.

set up env variables:

Exasol

Run Exasol on your local machine in docker (for this case .env is preconfigured)

Run Exasol server somewhere else and set up env variables:

issues:

• on slower machines Exasol can take a lot of resources. Run docker compose -f docker compose.yml -f docker compose.limits.yml up exasol to limit memory and cpu

• If you are getting error exadt::ERROR: the private interface with address '<ip>/16' either does not exist or is down. https://github.com/exasol/docker-db/issues/45 Exasol saves you (container) current ip address and docker daemon is changing default bridge range. There are two ways to fix this:
  1. each reboot set you current ip range to exasol docket-compose run --rm exasol exaconf modify-node -n 11 -p '<ip>/16' this is have to be done each time when ip addresses are not align.
  2. set docker default bridge to some fixed range: edit or create /etc/docker/daemon.json and set {"bip":"172.0.0.1/24"} (or different range that suites you)

Snowflake

Prepare credentials for Snowflake access

set up env variables:

Bigquery

To prepare the backend you can use Terraform template. You must have the resourcemanager.folders.create permission for the organization.

After terraform apply ends go to the service project in folder created by terraform.

1. go to the newly created service project, the project id are listed at the end of the terraform call. (service_project_id)
2. click on IAM & Admin
3. on left panel choose Service Accounts
4. click on email of service account(there is only one)
5. on to the top choose Keys and Add Key => Create new key
6. select Key type JSON
7. click on the Create button and the file will automatically download
8. convert key to stringawk -v RS= '{$1=$1}1' <key_file>.json >> .env
9. set content on last line of .env as variable BQ_KEY_FILE

setup envs:

Tests

Run tests with following command.

Unit and functional test can be run sepparetly

Code quality check

Full CI workflow

This command will run all checks load fixtures and run tests

Teradata connection

You can use following method to get connection to Teradata for your queries

If you want to use connection via PHPStorm DataGrip or other DB client, remove port when you setting up connection in client. Otherwise test connection will fail.

License

MIT licensed, see LICENSE file.