Integrating KitOps with KServe

ModelKits let you treat your model as a first‑class, OCI‑versioned artifact with rich metadata and strong supply‑chain guarantees—while KServe handles the runtime plumbing for scalable, pluggable inference. The result is a secure, reproducible, and operationally streamlined path from model build to model serving.

KitOps can be used as a ClusterStorageContainer in KServe to serve ModelKits directly in InferenceServices. This guide shows how to install, configure, and build the KitOps container for KServe.

Prerequisites

• A Kubernetes cluster with KServe installed.
• kubectl configured to access your cluster.
• A built KitOps KServe image (for example, ghcr.io/kitops-ml/kitops-kserve:latest).

Installing the ClusterStorageContainer

Create a ClusterStorageContainer resource that uses the KitOps CLI to unpack ModelKits when storageUri begins with kit://:

apiVersion: serving.kserve.io/v1alpha1
kind: ClusterStorageContainer
metadata:
  name: kitops
spec:
  container:
    name: storage-initializer
    image: ghcr.io/kitops-ml/kitops-kserve:latest
    imagePullPolicy: Always
    env:
      - name: KIT_UNPACK_FLAGS
        value: "" # Additional flags for `kit unpack`
    resources:
      requests:
        memory: 100Mi
        cpu: 100m
      limits:
        memory: 1Gi
  supportedUriFormats:
    - prefix: kit://

After applying this manifest, any InferenceService that references a ModelKit URI with the kit:// prefix will use KitOps to fetch and unpack the model.

Using ModelKits in an InferenceService

Reference your ModelKit in the storageUri field of an InferenceService spec:

apiVersion: serving.kserve.io/v1beta1
kind: InferenceService
metadata:
  name: iris-model
spec:
  predictor:
    model:
      modelFormat:
        name: sklearn
      storageUri: kit://<your-modelkit-reference>

KServe will invoke the KitOps container to unpack the ModelKit layers and make them available to your model server.

Configuration Options

Unpack Flags

Pass extra flags to the kit unpack command via the KIT_UNPACK_FLAGS environment variable. For example:

env:
  - name: KIT_UNPACK_FLAGS
    value: "-v --plain-http"

Registry Credentials

By default, the KitOps container uses KIT_USER and KIT_PASSWORD to log in. You can inject these from a Kubernetes Secret:

env:
  - name: KIT_USER
    valueFrom:
      secretKeyRef:
        name: kit-secret
        key: KIT_USER
        optional: true
  - name: KIT_PASSWORD
    valueFrom:
      secretKeyRef:
        name: kit-secret
        key: KIT_PASSWORD
        optional: true

AWS ECR (IRSA)

If you use AWS ECR and leverage IRSA (IAM Roles for Service Accounts), set AWS_ROLE_ARN on the service account. KitOps will detect it and perform ECR login automatically, unless KIT_USER and KIT_PASSWORD are provided (which take precedence).