Kubernetes Driver: switch to 'StatefulSet' from 'Deployment'

[ArielLahiany]

I've switched the Kubernetes driver object to 'StatefulSet' from 'Deployment' In order to support long-living persistent data directories between restart, and to meet the needs of the following issue: #2056
Please let me know if there are any other changes that are required, or if a different PR format is required.
Thank you.

[AkihiroSuda]

• new driver: kubernetes #167 (comment)

I avoided using StatefulSet because creating StatefulSet pods was quite slow at that time due to non-parallelizability.
Has the situation changed?

[AkihiroSuda]

podManagementPolicy should be set to Parallel

[AkihiroSuda]

Also please make sure to measure the startup time of the pods with a good number of replicas (around 5-10?), and there is no significant delay compared to Deployment

[ArielLahiany]

@AkihiroSuda
I've tested that logic on a freshly-installed vanilla K3s and everything seems to work as required.
I believe that the PR is failing duo to way that the GitHub Action is deploying the cluster.
How can we processed and move that PR along?

[AkihiroSuda]

ERROR: expected 1 replicas to be ready, got 0

Needs kubectl get pod -o yaml and kubectl logs for further analysis

[ArielLahiany]

Good afternoon, @AkihiroSuda.
I've run some tests on a freshly installed K3s cluster:

1. For deployment with the buildx-v0.20.0.linux-amd64 version:

function buildx_create_deployment() {
    buildx-v0.20.0.linux-amd64 create \
        --bootstrap \
        --buildkitd-config="/home/debian/github/buildx/bin/build/buildkitd.toml" \
        --driver-opt=image="docker.io/moby/buildkit:v0.20.0" \
        --driver-opt=limits.ephemeral-storage="10Gi" \
        --driver-opt=loadbalance="sticky" \
        --driver-opt=namespace="buildkitd" \
        --driver-opt=replicas=10 \
        --driver-opt=requests.ephemeral-storage="2Gi" \
        --driver="kubernetes" \
        --name="buildkitd"
}

This resulted in

#1 [internal] booting buildkit
#1 waiting for 10 pods to be ready, timeout: 2 minutes
#1 waiting for 10 pods to be ready, timeout: 2 minutes 7.9s done
#1 DONE 8.2s
buildkitd

real    0m8.660s
user    0m0.103s
sys     0m0.126s

2. For StatefulSet with the native local-path provisioner that comes with K3s by default:

function buildx_create_statefulset() {
    $BUILDX_BINARY_PATH create \
        --bootstrap \
        --buildkitd-config="/home/debian/github/buildx/bin/build/buildkitd.toml" \
        --driver-opt=image="docker.io/moby/buildkit:v0.20.0" \
        --driver-opt=limits.persistent-storage="10Gi" \
        --driver-opt=loadbalance="sticky" \
        --driver-opt=namespace="buildkitd" \
        --driver-opt=replicas=10 \
        --driver-opt=requests.persistent-storage="2Gi" \
        --driver="kubernetes" \
        --name="buildkitd"
}

This resulted in

#1 [internal] booting buildkit
#1 waiting for 10 pods to be ready, timeout: 2 minutes
#1 waiting for 10 pods to be ready, timeout: 2 minutes 50.7s done
#1 DONE 51.0s
buildkitd

real    0m51.463s
user    0m0.518s
sys     0m0.286s

As you can see, provisioning of 10 StatefulSet pods took longer time but still in the 2 minutes timeout that was defined by the team.
I do believe that the benefits are greater than the cons of that solution as there are multiple cases in which a user would like to get greater resilience of the BuiltKiD cluster with long-living pods.
Please let me know if there anything else that is required on my end.

[AkihiroSuda]

8.2s vs 51.0s is an extreme difference, even though it is within the threshold of your team.
The default mode should remain Deployment then.

[tonistiigi]

^