README

Simple PHP wrapper library for Keboola Management REST API

Installation

Library is available as composer package. To start using composer in your project follow these steps:

Install composer

curl -s http://getcomposer.org/installer | php
mv ./composer.phar ~/bin/composer # or /usr/local/bin/composer

Create composer.json file in your project root folder:

{
    "require": {
        "php" : ">=8.1",
        "keboola/kbc-manage-api-php-client": "~0.0"
    }
}

Install package:

composer install

Add autoloader in your bootstrap script:

require 'vendor/autoload.php';

Read more in Composer documentation

Usage examples

require 'vendor/autoload.php';

use Keboola\ManageApi\Client;

$client = new Client([
    'token' => getenv('MY_MANAGE_TOKEN'),
    'url' => 'https://connnection.keboola.com',
]);

$project = $client->getProject(234);

Authentication options

The client supports exactly one authentication option per instance.

Use the legacy Manage API token header:

$client = new Client([
    'token' => getenv('MY_MANAGE_TOKEN'),
    'url' => 'https://connection.keboola.com',
]);

Use a manually provided JWT for internal-service authentication:

$client = new Client([
    'jwtToken' => getenv('MY_INTERNAL_SERVICE_JWT'),
    'url' => 'https://connection.keboola.com',
]);

Use a projected Kubernetes ServiceAccount token mounted in a pod:

$client = new Client([
    'kubernetesTokenPath' => '/var/run/secrets/keboola/connection-token',
    'url' => 'https://connection.keboola.com',
]);

Both JWT variants send X-Kubernetes-Authorization: Bearer <jwt>. The Kubernetes token file is read for every request so kubelet token rotation is picked up without restarting the PHP process.

Tests

The main purpose of these test is "black box" test driven development of Keboola Connection. These test guards the API implementation. You can run these tests only against non-production environments.

Tests requires valid Keboola Management API tokens and an endpoint URL of the API test environment.

Note: For automated tests, the tests are run again three times by default if they fail. For local development this would be quite annoying, so you can disable this by creating new file phpunit-retry.xml from phpunit-retry.xml.dist

Note: The test environment should be running a cronjob for token-expirator otherwise the testTemporaryAccess test will fail.

Create file .env with environment variables`:

# REQUIRED - must be filled before running any test
KBC_MANAGE_API_URL=https://connection.keboola.com  # URL where Keboola Connection is running
KBC_MANAGE_API_TOKEN=your_token # manage api token assigned to user **with** **superadmin** privileges. Can be created in Account Settings under the title Personal Access Tokens. User must have Multi-Factor Authentication disabled.
KBC_SUPER_API_TOKEN=your_token # can be created in manage-apps on the Tokens tab
KBC_MANAGE_API_SUPER_TOKEN_WITH_PROJECTS_READ_SCOPE=super_token_with_projects_read_scope # can be created in manage-apps on the Tokens tab. Token must have "projects:read" scope
KBC_MANAGE_API_SUPER_TOKEN_WITHOUT_SCOPES=super_token_without_scopes
KBC_MANAGE_API_SUPER_TOKEN_WITH_DELETED_PROJECTS_READ_SCOPE=super_token_with_deleted_projects_read_scope # can be created in manage-apps on the Tokens tab. Token must have "deleted-projects:read" scope
KBC_MANAGE_API_SUPER_TOKEN_WITH_UI_MANAGE_SCOPE=super_token_with_ui_manage_scope # can be created in manage-apps on the Tokens tab. Token must have "connection:ui-manage" scope
KBC_MANAGE_API_SUPER_TOKEN_WITH_ORGANIZATIONS_READ_SCOPE=super_token_with_organizations_read # can be created in manage-apps on the Tokens tab. Token must have "organizations:read" scope
KBC_MANAGE_API_SUPER_TOKEN_WITH_STORAGE_TOKENS_SCOPE=super_token_with_storage_tokens_scope # can be created in manage-apps on the Tokens tab. Token must have "manage:storage-tokens" scope
KBC_TEST_MAINTAINER_ID=id # `id` of maintainer. Please create a new maintainer dedicated to test suite. All maintainer's organizations and projects all purged before tests!
KBC_TEST_ADMIN_EMAIL=email_of_another_admin_having_mfa_disabled # email address of another user without any organizations
KBC_TEST_ADMIN_TOKEN=token_of_another_admin_having_mfa_disabled # is also a Personal Access Token of user **without** **superadmin** privileges , but for a different user than that which has `KBC_MANAGE_API_TOKEN`. User must have Multi-Factor Authentication disabled.
KBC_TEST_ADMIN_WITH_MFA_EMAIL=email_of_another_admin_having_mfa_enabled # email address of another user without any organizations and having Multi-Factor Authentication enabled
KBC_TEST_ADMIN_WITH_MFA_TOKEN=token_of_another_admin_having_mfa_enabled # is also a Personal Access Token of user **without** **superadmin** privileges , but for a different user than that which has `KBC_MANAGE_API_TOKEN` or `KBC_TEST_ADMIN_TOKEN`
KBC_TEST_UNVERIFIED_ADMIN_TOKEN=token_of_unverified_admin_having_mfa_disabled # is a Personal Access Token of user with **isActivated=false** and **without** **superadmin** privileges, but for a different user than that which has `KBC_MANAGE_API_TOKEN`.

# OPTIONAL - required only for running testCreateStorageBackend, you have to have new snowflake backend and fill credentials into following environment variables
KBC_TEST_SNOWFLAKE_BACKEND_NAME=
KBC_TEST_SNOWFLAKE_BACKEND_PASSWORD=
KBC_TEST_SNOWFLAKE_HOST=
KBC_TEST_SNOWFLAKE_WAREHOUSE=
KBC_TEST_SNOWFLAKE_BACKEND_REGION=

OPTIONAL - required only for running StorageBackendTest::testCreateStorageBackendWithCert, you have to have new snowflake backend and fill credentials into following environment variables

Prepare credentials for Snowflake access Create RSA key pair for Snowflake user, you can use the following command to generate it:

openssl genrsa 2048 | openssl pkcs8 -topk8 -inform PEM -out rsa_key.p8 -nocrypt
openssl rsa -in rsa_key.p8 -pubout -out rsa_key.pub

Then you can use the public key in the Snowflake user creation script below.

To get the public key in one line (without header and footer) you can use:

PUBLIC_KEY=$(sed '1d;$d' rsa_key.pub | tr -d '\n')
# add this to CREATE USER statement
echo "RSA_PUBLIC_KEY='${PUBLIC_KEY}'"

CREATE ROLE "CI_MANAGE_TEST_USER";
CREATE DATABASE "CI_MANAGE_TEST_USER";
GRANT CREATE DATABASE ON ACCOUNT TO ROLE "CI_MANAGE_TEST_USER";
GRANT CREATE ROLE ON ACCOUNT TO ROLE "CI_MANAGE_TEST_USER" WITH GRANT OPTION;
GRANT CREATE USER ON ACCOUNT TO ROLE "CI_MANAGE_TEST_USER" WITH GRANT OPTION;

CREATE WAREHOUSE "CI_MANAGE_TEST_USER" WITH WAREHOUSE_SIZE = 'XSMALL' WAREHOUSE_TYPE = 'STANDARD' AUTO_SUSPEND = 60 AUTO_RESUME = TRUE;
GRANT USAGE ON WAREHOUSE "CI_MANAGE_TEST_USER" TO ROLE "CI_MANAGE_TEST_USER" WITH GRANT OPTION;


CREATE USER "CI_MANAGE_TEST_USER"
  DEFAULT_ROLE = "CI_MANAGE_TEST_USER"
  RSA_PUBLIC_KEY = '<your_public_key>'
;

GRANT ROLE ACCOUNTADMIN TO USER CI_MANAGE_TEST_USER;
GRANT ROLE "CI_MANAGE_TEST_USER" TO USER "CI_MANAGE_TEST_USER";
GRANT ROLE "CI_MANAGE_TEST_USER" TO ROLE SYSADMIN;

set up env variables:

For local tests and CI we need to edit the private key to one line and trim -----BEGIN PRIVATE KEY----- -----END PRIVATE KEY----- We can do this with cat rsa_key.p8 | sed '1d;$d' | tr -d '\n'

# set private key in your local .env file
PRIVATE_KEY=$(sed '1d;$d' rsa_key.p8 | tr -d '\n'); if [ -f .env ]; then awk -v v="KBC_TEST_MAIN_SNOWFLAKE_BACKEND_PRIVATE_KEY=\"$PRIVATE_KEY\"" 'BEGIN{r=0} /^KBC_TEST_MAIN_SNOWFLAKE_BACKEND_PRIVATE_KEY=/{print v; r=1; next} {print} END{if(!r) print v}' .env > .env.tmp && mv .env.tmp .env; else printf '%s\n' "KBC_TEST_MAIN_SNOWFLAKE_BACKEND_PRIVATE_KEY=\"$PRIVATE_KEY\"" > .env; fi

# user must be on the same host as KBC_TEST_SNOWFLAKE_HOST
KBC_TEST_MAIN_SNOWFLAKE_BACKEND_NAME=CI_MANAGE_TEST_USER
KBC_TEST_MAIN_SNOWFLAKE_BACKEND_PRIVATE_KEY= # note: it has to be full private key in PEM format, including the header and footer
KBC_TEST_MAIN_SNOWFLAKE_BACKEND_DATABASE=CI_MANAGE_TEST_USER
KBC_TEST_MAIN_SNOWFLAKE_BACKEND_WAREHOUSE=CI_MANAGE_TEST_USER
KBC_TEST_SNOWFLAKE_BACKEND_STACK_PREFIX= same as value `KEBOOLA_STORAGE_API__CLIENT_DB_PREFIX` in connection

If any of the environment variables are not set, the tests will fail with an error message explaining which variable is missing.

Run tests

docker-compose run --rm dev composer tests

File Storage tests

Setup cloud resources for File Storage tests

Prerequisites:

• configured and logged in az, aws and gcp CLI tools
  • az login
  • aws sso login --profile=<your_profile>
  • gcloud auth application-default login
• installed terraform (https://www.terraform.io) and jq (https://stedolan.github.io/jq) to setup local env

# create terraform.tfvars file from terraform.tfvars.dist
cp ./provisioning/terraform.tfvars.dist ./provisioning/terraform.tfvars

# set terraform variables
name_prefix = "<your_nick>" # your name/nickname to make your resource unique & recognizable, allowed characters are [a-zA-Z0-9-]
gcp_storage_location = "<your_region>" # region of GCP resources
gcp_project_id = "<your_project_id>" # GCP project id
gcp_project_region = "<your_region>" # region of GCP project
azure_storage_location = "<your_region>" # region of Azure resources 
azure_tenant_id = "<your_tenant_id>" # Azure tenant id
azure_subscription_id = "<your_subscription_id>" # Azure subscription id
aws_profile = "<your_profile>"  # your aws profile name
aws_region = "<your_region>" # region of AWS resources
aws_account = "<your_account_id>" # your aws account id


# Initialize terraform
terraform -chdir=./provisioning init
# Create resources
terraform -chdir=./provisioning apply

# For destroying resources run 
terraform -chdir=./provisioning apply -destroy

# Setup terraform variables to .env file (will be prepended to .env file)
# For Azure
./provisioning/update-env.sh azure
# For Aws
./provisioning/update-env.sh aws
# For GCP
./provisioning/update-env.sh gcp

Required variables for File Storage tests

These variables are used for testing file storage. You have to copy these values from Azure and AWS portal.

• TEST_ABS_ACCOUNT_KEY - First secret key for Azure Storage account
• TEST_ABS_ACCOUNT_NAME - Name of Azure Storage account
• TEST_ABS_CONTAINER_NAME - Name of container created inside Azure Storage Account
• TEST_ABS_REGION - Name of region where Azure Storage Account is located. (Note: AWS region list is used)
• TEST_ABS_ROTATE_ACCOUNT_KEY - Second secret key for Azure Storage account
• TEST_S3_ROTATE_KEY - Second AWS key
• TEST_S3_ROTATE_SECRET - Second AWS secret
• TEST_S3_FILES_BUCKET - Name of file bucket on S3
• TEST_S3_KEY - First AWS key
• TEST_S3_REGION - Region where your S3 is located
• TEST_S3_SECRET - First AWS secret
• TEST_GCS_KEYFILE_JSON - First GCS key file contents as json string
• TEST_GCS_KEYFILE_ROTATE_JSON - Second GCS key file contents as json string used for testing rotation
• TEST_GCS_FILES_BUCKET - Name of file bucket on GCS
• TEST_GCS_REGION - Region whare GCS is located

Variable prefixed with ROTATE are used for rotating credentials and they MUST be working credentials.

Run File Storage tests

docker-compose run --rm dev composer tests-file-storage

Build OpenAPI document

The OpenAPI document for the Manage API is generated from PHP attributes (#[OA\…]) on Symfony controllers in connection/src/Controller/Manage/** and request DTOs in connection/src/Manage/**/Request/*Request.php.

To dump the current OpenAPI document, run from the monorepo repository root (not from this package directory -- the script lives in the root composer.json and needs the root docker-compose.yml's cli service):

$ docker compose run --rm cli composer openapi:manage:dump

The output is written to openapi/manage.json.

License

MIT licensed, see LICENSE file.