優化 Stable Diffusion 在 GKE 上的啟動體驗

●觸發 Cluster Autoscaler 擴容 + Node 啟動并調度 Pod: 225s

●啟動 Pull Image:4s

●拉取鏡像: 5m 23s

●啟動 Pod:1s

●能夠提供 sd-webui 的服務 (大約): > 2m

●優化 Dockerfile，選擇正確的 base image，精簡 runtime 的依賴安裝，減小鏡像大小。

●借助基礎環境與 runtime 依賴分離方式，通過磁盤復制方式加速運行環境的創建。

●通過 GKE Image Streaming 優化鏡像加載時間，利用 Cluster Autoscaler 提升彈性擴縮容速度。

# Base image

FROM nvidia/cuda:11.8.0-runtime-ubuntu22.04

RUN set -ex &&

apt update &&

apt install -y wget git python3 python3-venv python3-pip libglib2.0-0 pkg-config libcairo2-dev &&

rm -rf /var/lib/apt/lists/*

# Pytorch

RUN python3 -m pip install torch==1.13.1+cu117 torchvision==0.14.1+cu117 --extra-index-url https://download.pytorch.org/whl/cu117

…

# Stable Diffusion

RUN git clone https://github.com/AUTOMATIC1111/stable-diffusion-webui.git

RUN git clone https://github.com/Stability-AI/stablediffusion.git /stable-diffusion-webui/repositories/stable-diffusion-stability-ai

RUN git -C /stable-diffusion-webui/repositories/stable-diffusion-stability-ai checkout cf1d67a6fd5ea1aa600c4df58e5b47da45f6bdbf

…

# Stable Diffusion extensions

RUN set -ex && cd stable-diffusion-webui

&& git clone https://gitcode.net/ranting8323/sd-webui-additional-networks.git extensions/sd-webui-additional-networks

&& git clone https://gitcode.net/ranting8323/sd-webui-cutoff extensions/sd-webui-cutoff

&& git clone https://ghproxy.com/https://github.com/toshiaki1729/stable-diffusion-webui-dataset-tag-editor.git extensions/stable-diffusion-webui-dataset-tag-editor

FROM nvidia/cuda:11.8.0-runtime-ubuntu22.04

RUN set -ex &&

apt update &&

apt install -y wget git python3 python3-venv python3-pip libglib2.0-0 &&

rm -rf /var/lib/apt/lists/*

gcloud compute disks create sd-lib-disk-$NOW --type=pd-balanced --size=30GB --zone=$ZONE --image=$IMAGE_NAME

gcloud compute instances attach-disk ${MY_NODE_NAME} --disk=projects/$PROJECT_ID/zones/$ZONE/disks/sd-lib-disk-$NOW --zone=$ZONE

●觸發 Cluster Autoscaler 擴容:38s

●Node 啟動并調度 Pod:89s

●掛載 PVC:4s

●啟動 Pull Image:10s

●拉取鏡像:1s

●啟動 Pod:1s

●能夠提供 sd-webui 的服務 (大約): 65s

●Snapshot 每 10 分鐘只能恢復一次，一小時以內恢復 6 次 Disk 的限制。

●而 Image 可以支持每 30 秒恢復一次，最多 1,000 個 Disk。

apiVersion: snapshot.storage.k8s.io/v1

kind: VolumeSnapshotClass

metadata:

name: image-class

driver: pd.csi.storage.gke.io

deletionPolicy: Delete

parameters:

DiskImageType: images

apiVersion: snapshot.storage.k8s.io/v1

kind: VolumeSnapshotContent

metadata:

name: test-snapshotcontent-from-image

spec:

deletionPolicy: Retain

driver: pd.csi.storage.gke.io

volumeSnapshotClassName: image-class

source:

snapshotHandle:projects/flius-vpc-2/global/images/sd-image

volumeSnapshotRef:

name: test-snapshot

namespace: default

apiVersion: snapshot.storage.k8s.io/v1

kind: VolumeSnapshot

metadata:

name: test-snapshot

namespace: default

spec:

source:

volumeSnapshotContentName:test-snapshotcontent-from-image

apiVersion: apps/v1

kind: StatefulSet

metadata:

name: stable-diffusion-statefulset-image

labels:

app: stable-diffusion

spec:

podManagementPolicy: "Parallel"

replicas: 1

selector:

matchLabels:

app: stable-diffusion

template:

metadata:

labels:

app: stable-diffusion

spec:

containers:

- name: stable-diffusion-webui

image: us-central1-docker.pkg.dev/flius-vpc-2/stable-diffusion-repo/sd-webui-final:0.1

command: ["/bin/bash"]

args: ["-c", "source /runtime-lib/bin/activate; cp/user-watch.py /runtime-lib/stable-diffusion-webui/user-watch.py;cp/start.sh /runtime-lib/stable-diffusion-webui/start.sh; cd /runtime-lib/stable-diffusion-webui; python3 launch.py --listen --xformers --enable-insecure-extension-access--no-gradio-queue" ]

volumeMounts:

- mountPath: "/runtime-lib"

name: runtime-lib

resources:

limits:

nvidia.com/gpu: 1

ports:

- containerPort: 7860

volumeClaimTemplates:

- metadata:

name: runtime-lib

spec:

dataSource:

name: test-snapshot

kind: VolumeSnapshot

apiGroup: snapshot.storage.k8s.io

accessModes: [ "ReadWriteOnce" ]

storageClassName: "standard-rwo"

resources:

requests:

storage: 30Gi

kubectl scale statefulset stable-diffusion-statefulset-image --replicas=15