Upgrading a Kubeflow Deployment

Upgrading your deployment to a later version of Kubeflow

Upgrading from Kubeflow v0.7.0 or later

Upgrading your Kubeflow deployment is supported if your deployment is v0.7.0 or later.


  • Download the latest kfctl binary from the Kubeflow releases page.

  • Ensure that your Kubeflow namespace is annotated with the control-plane: kubeflow label. You can verify this by doing:

    kubectl get namespace $NAMESPACE -o yaml

    You should see something like:

        control-plane: kubeflow

    If the label is not present, you can patch the namespace by doing

    kubectl label namespace $NAMESPACE control-plane=kubeflow
  • You must have a local Kubeflow application directory matching your current deployment. We’ll call this ${KF_DIR}.

    • This should be present if you deployed Kubeflow using the kfctl tool.
    • If you deployed Kubeflow using some other method (e.g. the Deployment UI), you will need to first create a local deployment directory:
      1. Follow the instructions for environment preparations.
        • Make sure that the ${PROJECT}, ${ZONE}, and ${KF_NAME} variables match exactly with your deployment.
        • For ${CONFIG_URI}, use the appropriate YAML configuration file.
      2. Create your local deployment files:
        mkdir -p ${KF_DIR}
        cd ${KF_DIR}
        kfctl build -V -f ${CONFIG_URI}

Upgrade instructions

Prepare your upgrade specification:

  1. Create an upgrade specification in the parent directory of your ${KF_DIR} directory. Base your upgrade specification on one of the specifications supplied in the kubeflow/manifests repository. For example, to upgrade a v0.7.0 deployment to v0.7.1 on GCP, use the kfctl_upgrade_gcp_iap_0.7.1.yaml specification. Your directory structure should look like this:

  2. For convenience, set the environment variable ${UPGRADE_SPEC} to the name of your upgrade specification:

    export UPGRADE_SPEC=<your-upgrade-specification.yaml>
  3. Modify the contents of the upgrade specification to suit your deployment:

    apiVersion: kfupgrade.apps.kubeflow.org/v1alpha1
    kind: KfUpgrade
      name: kf-upgrade-v0.7.1
        # Replace with the name of your Kubeflow app
        name: kubeflow-app
        version: v0.7.0
        # Replace with the name of your kubeflow app
        name: kubeflow-app
        version: v0.7.1
      # Replace this with the path to the KfDef that you are upgrading to
      baseConfigPath: https://raw.githubusercontent.com/kubeflow/manifests/v0.7-branch/kfdef/kfctl_gcp_iap.0.7.1.yaml

If you don’t need to change any kustomize parameter values in your configuration, run the apply command now to upgrade your deployment:

kfctl apply -f ${UPGRADE_SPEC} -V

Alternatively you can follow these steps to change the configuration before applying the upgrade:

  1. Run a build command:

    kfctl build -f ${UPGRADE_SPEC} -V

    The above command creates a new Kubeflow application under the same parent directory as your ${KF_DIR}. The new Kubeflow application has a name in the form of a 7-character long hash value. The directory structure should look like this, where ${UPGRADE_DIR} indicates the new Kubeflow application:


    You can examine and change the kustomize parameter values in the ${UPGRADE_DIR} directory.

  2. Run the apply command to upgrade the deployment:

    kfctl apply -f ${UPGRADE_SPEC} -V

Upgrades from earlier versions of Kubeflow

For earlier versions, Kubeflow makes no promises of backwards compatibility or upgradeability. Nonetheless, here are some instructions for updating your deployment:

  1. Check your Kubeflow configuration directory (${KF_DIR}) into source control as a backup.

  2. Delete your existing Kubeflow cluster:

    kfctl delete -V 
  3. Download the kfctl v1.0.2 release from the Kubeflow releases page.

  4. Unpack the tar ball:

    tar -xvf kfctl_<release tag>_<platform>.tar.gz
  5. Update your kustomize manifests:

    export CONFIG_FILE=<the path to your Kubeflow config file>
    kfctl build -V -f ${CONFIG_FILE}

    The ${CONFIG_FILE} environment variable must contain the path to the Kubeflow configuration file in your ${KF_DIR} directory. For example, ${KF_DIR}/kfctl_k8s_istio.v1.0.2.yaml or ${KF_DIR}/kfctl_istio_dex.yaml

  6. Re-apply any customizations that you need.

  7. Update the deployment:

    kfctl apply -V -f ${CONFIG_FILE}

Upgrading Kubeflow Pipelines

See the Kubeflow Pipelines upgrade guide.