Descheduler 使用指南


通过 Descheduler 来实现更高级的调度策略

Descheduler 使用指南

通过 Descheduler 来实现更高级的调度策略

  • Kubernetes 知识图谱
  • 云原生导航
  • 谷歌搜索
  • Raft 协议动画演示
  • 搬瓦工

    kube-scheduler 是 Kubernetes 中负责调度的组件,它本身的调度功能已经很强大了。但由于 Kubernetes 集群非常活跃,它的状态会随时间而改变,由于各种原因,你可能需要将已经运行的 Pod 移动到其他节点:

    一旦 Pod 启动之后 kube-scheduler 便不会再尝试重新调度它。根据环境的不同,你可能会有很多需要手动调整 Pod 的分布,例如:如果集群中新加入了一个节点,那么已经运行的 Pod 并不会被分摊到这台节点上,这台节点可能只运行了少量的几个 Pod,这并不理想,对吧?

    1. Descheduler 如何工作?


    Descheduler 会检查 Pod 的状态,并根据自定义的策略将不满足要求的 Pod 从该节点上驱逐出去。Descheduler 并不是 kube-scheduler 的替代品,而是要依赖于它。该项目目前放在 Kubernetes 的孵化项目中,还没准备投入生产,但经过我实验发现它的运行效果很好,而且非常稳定。那么该如何安装呢?

    2. 部署方法


    你可以通过 JobCronJob 来运行 descheduler。我已经创建了一个镜像 komljen/descheduler:v0.5.0-4-ga7ceb671(包含在下面的 yaml 文件中),但由于这个项目的更新速度很快,你可以通过以下的命令创建你自己的镜像:

    $ git clone https://github.com/kubernetes-incubator/descheduler
    $ cd descheduler && make image
    

    然后打好标签 push 到自己的镜像仓库中。

    通过我创建的 chart 模板,你可以用 Helm 来部署 descheduler,该模板支持 RBAC 并且已经在 Kubernetes v1.9 上测试通过。

    添加我的 helm 私有仓库,然后部署 descheduler:

    $ helm repo add akomljen-charts \
      https://raw.githubusercontent.com/komljen/helm-charts/master/charts/
      
    $ helm install --name ds \
      --namespace kube-system \
      akomljen-charts/descheduler
    

    你也可以不使用 helm,通过手动部署。首先创建 serviceaccount 和 clusterrolebinding:

    # Create a cluster role
    $ cat << EOF| kubectl create -n kube-system -f -
    apiVersion: rbac.authorization.k8s.io/v1beta1
    kind: ClusterRole
    metadata:
      name: descheduler
    rules:
    - apiGroups: [""]
      resources: ["nodes"]
      verbs: ["get", "watch", "list"]
    - apiGroups: [""]
      resources: ["pods"]
      verbs: ["get", "watch", "list", "delete"]
    - apiGroups: [""]
      resources: ["pods/eviction"]
      verbs: ["create"]
    EOF
    
    # Create a service account
    $ kubectl create sa descheduler -n kube-system
    
    # Bind the cluster role to the service account
    $ kubectl create clusterrolebinding descheduler \
        -n kube-system \
        --clusterrole=descheduler \
        --serviceaccount=kube-system:descheduler
    

    然后通过 configmap 创建 descheduler 策略。目前只支持四种策略:

    默认这四种策略全部开启,你可以根据需要关闭它们。下面在 kube-suystem 命名空间中创建一个 configmap:

    $ cat << EOF| kubectl create -n kube-system -f -
    apiVersion: v1
    kind: ConfigMap
    metadata:
      name: descheduler
    data:
      policy.yaml: |-  
        apiVersion: descheduler/v1alpha1
        kind: DeschedulerPolicy
        strategies:
          RemoveDuplicates:
             enabled: false
          LowNodeUtilization:
             enabled: true
             params:
               nodeResourceUtilizationThresholds:
                 thresholds:
                   cpu: 20
                   memory: 20
                   pods: 20
                 targetThresholds:
                   cpu: 50
                   memory: 50
                   pods: 50
          RemovePodsViolatingInterPodAntiAffinity:
            enabled: true
          RemovePodsViolatingNodeAffinity:
            enabled: true
            params:
              nodeAffinityType:
              - requiredDuringSchedulingIgnoredDuringExecution
    EOF
    

    kube-system 命名空间中创建一个 CronJob:

    $ cat << EOF| kubectl create -n kube-system -f -
    apiVersion: batch/v1beta1
    kind: CronJob
    metadata:
      name: descheduler
    spec:
      schedule: "*/30 * * * *"
      jobTemplate:
        metadata:
          name: descheduler
          annotations:
            scheduler.alpha.kubernetes.io/critical-pod: "true"
        spec:
          template:
            spec:
              serviceAccountName: descheduler
              containers:
              - name: descheduler
                image: komljen/descheduler:v0.5.0-4-ga7ceb671
                volumeMounts:
                - mountPath: /policy-dir
                  name: policy-volume
                command:
                - /bin/descheduler
                - --v=4
                - --max-pods-to-evict-per-node=10
                - --policy-config-file=/policy-dir/policy.yaml
              restartPolicy: "OnFailure"
              volumes:
              - name: policy-volume
                configMap:
                  name: descheduler
    EOF
    
    $ kubectl get cronjobs -n kube-system
    
    NAME             SCHEDULE       SUSPEND   ACTIVE    LAST SCHEDULE   AGE
    descheduler      */30 * * * *   False     0         2m              32m
    

    该 CroJob 每 30 分钟运行一次,当 CronJob 开始工作后,可以通过以下命令查看已经成功结束的 Pod:

    $ kubectl get pods -n kube-system -a | grep Completed
    
    descheduler-1525520700-297pq          0/1       Completed   0          1h
    descheduler-1525521000-tz2ch          0/1       Completed   0          32m
    descheduler-1525521300-mrw4t          0/1       Completed   0          2m
    

    也可以查看这些 Pod 的日志,然后根据需要调整 descheduler 策略:

    $ kubectl logs descheduler-1525521300-mrw4t -n kube-system
    
    I0505 11:55:07.554195       1 reflector.go:202] Starting reflector *v1.Node (1h0m0s) from github.com/kubernetes-incubator/descheduler/pkg/descheduler/node/node.go:84
    I0505 11:55:07.554255       1 reflector.go:240] Listing and watching *v1.Node from github.com/kubernetes-incubator/descheduler/pkg/descheduler/node/node.go:84
    I0505 11:55:07.767903       1 lownodeutilization.go:147] Node "ip-10-4-63-172.eu-west-1.compute.internal" is appropriately utilized with usage: api.ResourceThresholds{"cpu":41.5, "memory":1.3635487207675927, "pods":8.181818181818182}
    I0505 11:55:07.767942       1 lownodeutilization.go:149] allPods:9, nonRemovablePods:9, bePods:0, bPods:0, gPods:0
    I0505 11:55:07.768141       1 lownodeutilization.go:144] Node "ip-10-4-36-223.eu-west-1.compute.internal" is over utilized with usage: api.ResourceThresholds{"cpu":48.75, "memory":61.05259502942694, "pods":30}
    I0505 11:55:07.768156       1 lownodeutilization.go:149] allPods:33, nonRemovablePods:12, bePods:1, bPods:19, gPods:1
    I0505 11:55:07.768376       1 lownodeutilization.go:144] Node "ip-10-4-41-14.eu-west-1.compute.internal" is over utilized with usage: api.ResourceThresholds{"cpu":39.125, "memory":98.19259268881142, "pods":33.63636363636363}
    I0505 11:55:07.768390       1 lownodeutilization.go:149] allPods:37, nonRemovablePods:8, bePods:0, bPods:29, gPods:0
    I0505 11:55:07.768538       1 lownodeutilization.go:147] Node "ip-10-4-34-29.eu-west-1.compute.internal" is appropriately utilized with usage: api.ResourceThresholds{"memory":43.19826999287199, "pods":30.90909090909091, "cpu":35.25}
    I0505 11:55:07.768552       1 lownodeutilization.go:149] allPods:34, nonRemovablePods:11, bePods:8, bPods:15, gPods:0
    I0505 11:55:07.768556       1 lownodeutilization.go:65] Criteria for a node under utilization: CPU: 20, Mem: 20, Pods: 20
    I0505 11:55:07.768571       1 lownodeutilization.go:69] No node is underutilized, nothing to do here, you might tune your thersholds further
    I0505 11:55:07.768576       1 pod_antiaffinity.go:45] Processing node: "ip-10-4-63-172.eu-west-1.compute.internal"
    I0505 11:55:07.779313       1 pod_antiaffinity.go:45] Processing node: "ip-10-4-36-223.eu-west-1.compute.internal"
    I0505 11:55:07.796766       1 pod_antiaffinity.go:45] Processing node: "ip-10-4-41-14.eu-west-1.compute.internal"
    I0505 11:55:07.813303       1 pod_antiaffinity.go:45] Processing node: "ip-10-4-34-29.eu-west-1.compute.internal"
    I0505 11:55:07.829109       1 node_affinity.go:40] Executing for nodeAffinityType: requiredDuringSchedulingIgnoredDuringExecution
    I0505 11:55:07.829133       1 node_affinity.go:45] Processing node: "ip-10-4-63-172.eu-west-1.compute.internal"
    I0505 11:55:07.840416       1 node_affinity.go:45] Processing node: "ip-10-4-36-223.eu-west-1.compute.internal"
    I0505 11:55:07.856735       1 node_affinity.go:45] Processing node: "ip-10-4-41-14.eu-west-1.compute.internal"
    I0505 11:55:07.945566       1 request.go:480] Throttling request took 88.738917ms, request: GET:https://100.64.0.1:443/api/v1/pods?fieldSelector=spec.nodeName%3Dip-10-4-41-14.eu-west-1.compute.internal%2Cstatus.phase%21%3DFailed%2Cstatus.phase%21%3DSucceeded
    I0505 11:55:07.972702       1 node_affinity.go:45] Processing node: "ip-10-4-34-29.eu-west-1.compute.internal"
    I0505 11:55:08.145559       1 request.go:480] Throttling request took 172.751657ms, request: GET:https://100.64.0.1:443/api/v1/pods?fieldSelector=spec.nodeName%3Dip-10-4-34-29.eu-west-1.compute.internal%2Cstatus.phase%21%3DFailed%2Cstatus.phase%21%3DSucceeded
    I0505 11:55:08.160964       1 node_affinity.go:72] Evicted 0 pods
    

    哇哦,现在你的集群中已经运行了一个 descheduler!

    3. 总结


    Kubernetes 的默认调度器已经做的很好,但由于集群处于不断变化的状态中,某些 Pod 可能运行在错误的节点上,或者你想要均衡集群资源的分配,这时候就需要 descheduler 来帮助你将某些节点上的 Pod 驱逐到正确的节点上去。我很期待正式版的发布!

    4. 原文链接



    -------他日江湖相逢 再当杯酒言欢-------

    「真诚赞赏,手留余香」

    米开朗基杨

    真诚赞赏,手留余香

    使用微信扫描二维码完成支付


    相关推荐