Using environment variables in pipelines

This page describes how to pass environment variables to Kubeflow pipeline components.

Before you start

Set up your environment:

• Install Kubeflow
• Install the Kubeflow Pipelines SDK

Using environment variables

In this example, you pass an environment variable to a lightweight Python component, which writes the variable’s value to the log.

Learn more about lightweight Python components

To build a component, define a stand-alone Python function and then call kfp.components.func_to_container_op(func) to convert the function to a component that can be used in a pipeline. The following function gets an environment variable and writes it to the log.

def logg_env_function():
  import os
  import logging
  logging.basicConfig(level=logging.INFO)
  env_variable = os.getenv('example_env')
  logging.info('The environment variable is: {}'.format(env_variable))

Transform the function into a component using kfp.components.func_to_container_op(func).

image_name = 'tensorflow/tensorflow:1.11.0-py3'
logg_env_function_op = comp.func_to_container_op(logg_env_function,
                                                 base_image=image_name)

Add this component to a pipeline. Use add_env_variable to pass an environment variable into the component. This code is the same no matter if your using python lightweight components or a container operation.

import kfp.dsl as dsl
from kubernetes.client.models import V1EnvVar

@dsl.pipeline(
  name='Env example',
  description='A pipline showing how to use environment variables'
)
def environment_pipline():
  env_var = V1EnvVar(name='example_env', value='env_variable')
  #Returns a dsl.ContainerOp class instance. 
  container_op = logg_env_function_op().add_env_variable(env_var) 

To pass more environment variables into a component, add more instances of add_env_variable(). Use the following command to run this pipeline using the Kubeflow Pipelines SDK.

#Specify pipeline argument values
arguments = {}

#Submit a pipeline run
kfp.Client().create_run_from_pipeline_func(environment_pipline,
                                           arguments=arguments)

Feedback

Was this page helpful?

Glad to hear it! Please tell us how we can improve.

Sorry to hear that. Please tell us how we can improve.