Configuration
Plugin maintains the configuration in the conf/base/vertexai.yaml
file. Sample configuration can be generated using kedro vertexai init
:
# Configuration used to run the pipeline
project_id: my-gcp-mlops-project
region: us-central1
run_config:
# Name of the image to run as the pipeline steps
image: us.gcr.io/my-gcp-mlops-project/example_model:${commit_id}
# Pull policy to be used for the steps. Use Always if you push the images
# on the same tag, or Never if you use only local images
image_pull_policy: IfNotPresent
# Location of Vertex AI GCS root
root: my-gcp-mlops-project-staging-bucket
# Name of the kubeflow experiment to be created
experiment_name: example-model
# Name of the run for run-once, templated with the run-once parameters
run_name: example-model-${run_id}
# Name of the scheduled run, templated with the schedule parameters
scheduled_run_name: example-model
# Optional pipeline description
#description: "Very Important Pipeline"
# Flag indicating if the run-once should wait for the pipeline to finish
wait_for_completion: False
# How long to keep underlying Argo workflow (together with pods and data
# volume after pipeline finishes) [in seconds]. Default: 1 week
ttl: 604800
# What Kedro pipeline should be run as the last step regardless of the
# pipeline status. Used to send notifications or raise the alerts
# on_exit_pipeline: notify_via_slack
# Optional section allowing adjustment of the resources
# reservations and limits for the nodes
resources:
# For nodes that require more RAM you can increase the "memory"
data_import_step:
memory: 2Gi
# Training nodes can utilize more than one CPU if the algoritm
# supports it
model_training:
cpu: 8
memory: 1Gi
# GPU-capable nodes can request 1 GPU slot
tensorflow_step:
nvidia.com/gpu: 1
# Default settings for the nodes
__default__:
cpu: 200m
memory: 64Mi
Dynamic configuration support
kedro-vertexai
contains hook that enables TemplatedConfigLoader.
It allows passing environment variables to configuration files. It reads all environment variables following KEDRO_CONFIG_<NAME>
pattern, which you
can later inject in configuration file using ${name}
syntax.
There are two special variables KEDRO_CONFIG_COMMIT_ID
, KEDRO_CONFIG_BRANCH_NAME
with support specifying default when variable is not set,
e.g. ${commit_id|dirty}