跳转至

实例规格和偏好设置

实例类型(Instance Type)和配置偏好(Perference)

实例类型

kubevirt 提供了2种CRDs来定义实例类型，一个是集群级别的 VirtualMachineClusterInstancetype， 另一个是名称空间namespace级别VirtualMachineInstancetype。这两个CRDs定义了 一组标准的资源配置，包括： - cpu：必须属性；定义虚拟机CPU相关属性，包括vcpu数量(guest)、cpu模式(model,host-passthrough)、cpu独占核心(dedicatedCPUPlacement)、控制是否将QEMU虚拟化模拟器的线程与其他虚拟机实例隔离开来(isolateEmulatorThread)、NUMA（Non-Uniform Memory Access）为虚拟机分配特定的 NUMA 节点等。 - memory：必须属性；定义虚拟机内存相关属性，为虚拟机分配的内存大小(guest)、如果使用大页内存(hugepages)可指定大页的大小(pageSize)、定义内存超配比(overcommitPercent)，默认为0不进行内存超配，设置内存超配会增加系统内存压力和潜在的 OOM（Out Of Memory）风险。 - gpus：可选属性；定义了虚拟机使用vGPU或GPU直通相关属性。 - hostDevices：可选属性；定义了一组要直接透传给虚拟机使用的主机设备，例如透传 NVME SSD - ioThreadsPolicy：可选属性；定义虚拟机使用的I/O线程策略，目前支持shared和auto。 - launchSecurity：可选属性；定义虚拟机实例启动时安全性相关的设置，比如支持AMD安全加密虚拟化(Secure Encrypted Virtualization, SEV)相关的设置等。 - nodeSelector：可选属性；指定虚拟机将被调度到的节点标签，例如kubernetes.io/hostname等。 - schedulerName：可选属性；指定义调度器

注意 在实例类型中定义的任何资源都不能在VirtualMachine中被重写。例如，用户在定义VirtualMachine的yaml时，如果引用了实例类型，就不能再在该yaml中定义cpu或内存等资源属性，否则会与实例类型发生冲突，请求将在创建过程中会被拒绝。

# kubectl explain virtualmachineclusterinstancetype --recursive
GROUP:      instancetype.kubevirt.io
KIND:       VirtualMachineClusterInstancetype
VERSION:    v1beta1

DESCRIPTION:
    VirtualMachineClusterInstancetype is a cluster scoped version of
    VirtualMachineInstancetype resource.

FIELDS:
  apiVersion    <string>
  kind  <string>
  metadata      <ObjectMeta>
    annotations <map[string]string>
    creationTimestamp   <string>
    deletionGracePeriodSeconds  <integer>
    deletionTimestamp   <string>
    finalizers  <[]string>
    generateName        <string>
    generation  <integer>
    labels      <map[string]string>
    managedFields       <[]ManagedFieldsEntry>
      apiVersion        <string>
      fieldsType        <string>
      fieldsV1  <FieldsV1>
      manager   <string>
      operation <string>
      subresource       <string>
      time      <string>
    name        <string>
    namespace   <string>
    ownerReferences     <[]OwnerReference>
      apiVersion        <string> -required-
      blockOwnerDeletion        <boolean>
      controller        <boolean>
      kind      <string> -required-
      name      <string> -required-
      uid       <string> -required-
    resourceVersion     <string>
    selfLink    <string>
    uid <string>
  spec  <Object> -required-
    annotations <map[string]string>
    cpu <Object> -required-
      dedicatedCPUPlacement     <boolean>
      guest     <integer> -required-
      isolateEmulatorThread     <boolean>
      model     <string>
      numa      <Object>
        guestMappingPassthrough <Object>
      realtime  <Object>
        mask    <string>
    gpus        <[]Object>
      deviceName        <string> -required-
      name      <string> -required-
      tag       <string>
      virtualGPUOptions <Object>
        display <Object>
          enabled       <boolean>
          ramFB <Object>
            enabled     <boolean>
    hostDevices <[]Object>
      deviceName        <string> -required-
      name      <string> -required-
      tag       <string>
    ioThreadsPolicy     <string>
    launchSecurity      <Object>
      sev       <Object>
        attestation     <Object>
        dhCert  <string>
        policy  <Object>
          encryptedState        <boolean>
        session <string>
    memory      <Object> -required-
      guest     <Object> -required-
      hugepages <Object>
        pageSize        <string>
      overcommitPercent <integer>
    nodeSelector        <map[string]string>
    schedulerName       <string>

instancetype 目前已是v1beta1，目前kubevirt最新版本是1.1.1。切记，在VirtualMachineInstancetype中定义的内容都不能再在VirtualMachine再定义 VirtualMachineInstancetype 是名称空间级的，即kind是 VirtualMachineInstancetype 创建的规格，只限定在指定名称间空下使用。 

---
# 名称空间级
## kubectl label node k8s04 cpu=true
apiVersion: instancetype.kubevirt.io/v1beta1
kind: VirtualMachineInstancetype
metadata:
  name: p1-2
  namespace: default
spec:
  cpu:
    guest: 1
    model: host-passthrough
    dedicatedCPUPlacement: false
    isolateEmulatorThread: false
  memory:
    guest: 2Gi
  nodeSelector:
    cpu: true

# kubectl get VirtualMachineInstancetype[vminstancetype|vminstancetypes|vmf|vmfs]
# kubectl get vmf
NAME   AGE
p1-2   93s

集群级别 

---
# 名称空间级
apiVersion: instancetype.kubevirt.io/v1beta1
kind: VirtualMachineClusterInstancetype
metadata:
  name: p1-1
  namespace: default
spec:
  cpu:
    guest: 1
    model: host-passthrough
    dedicatedCPUPlacement: true
    isolateEmulatorThread: true
  memory:
    guest: 1Gi
  nodeSelector:
    cpu: true
注意 实验环境建议将dedicatedCPUPlacement和isolateEmulatorThread设置为fase，cpu资源不够会导致调度不成功。

# kubectl get vmclusterinstancetype,vmclusterinstancetypes,vmcf,vmcfs
# kubectl get vmcf
NAME   AGE
p1-2   93s


apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: pvc-ubuntu01-bootdisk
  namespace: default
spec:
  storageClassName: "ceph-hdd-block"
  volumeMode: Block
  accessModes:
    - ReadWriteMany
  dataSource:
    kind: PersistentVolumeClaim
    name: img-ubuntu2204
  resources:
    requests:
      storage: 10Gi
---
apiVersion: kubevirt.io/v1
kind: VirtualMachine
metadata:
  name: vm-ubuntu01
spec:
  instancetype:
    kind: VirtualMachineInstancetype
    name: p1-2
  running: true
  template:
    metadata:
      labels:
        kubevirt.io/domain: vm-ubuntu01
    spec:
      architecture: amd64
      domain:
        devices:
          disks:
          - disk:
              bus: virtio
            name: root-disk
          - disk:
              bus: virtio
            name: cloudinitdisk
          rng: {}
          interfaces:
            - name: default
              masquerade: {}
      networks:
        - name: default
          pod: {}
      terminationGracePeriodSeconds: 0
      volumes:
      - name: root-disk
        persistentVolumeClaim:
          claimName: pvc-ubuntu01-bootdisk
      - name: cloudinitdisk
        cloudInitConfigDrive:
          userData: |-
            #cloud-config
            hostname: vm-ubuntu01
            ssh_pwauth: True
            disable_root: False
            timezone: Asia/Shanghai
            password: 123456
            chpasswd: {"list":"root:12345678",expire: False}
            bootcmd:
              - setenforce 0
              - sed -i 's/^SELINUX=enforcing$/SELINUX=disabled/' /etc/selinux/config
            runcmd:
              - sed -i "/PermitRootLogin/s/^.*$/PermitRootLogin yes/g" /etc/ssh/sshd_config
              - systemctl restart sshd.service
              #- echo root:12345678 |chpasswd
注意 使用了instancetype后，就不能再在VirtualMachine中定义spec.domain.cpu和spec.domain.memory了，也不能用spec.resources。

偏好设置

KubeVirt 还提供了两个偏好设置的自定义资源定义（CRDs），包括集群范围内的VirtualMachineClusterPreference 和名称空间级别的VirtualMachinePreference。 这两个CRD封装了VirtualMachine 中剩余（即实例类型不包括的）属性的首选值，通过共享VirtualMachinePreferenceSpec实现。与实例类型不同，偏好设置仅表示首选值，并且可以被用户提供的 VirtualMachine 中的值覆盖。

---
apiVersion: instancetype.kubevirt.io/v1beta1
kind: VirtualMachinePreference
metadata:
  name: pref01
spec:
  devices:
    # 磁盘总线首选类型
    preferredDiskBus: virtio
    # 网络接口首选类型
    preferredInterfaceModel: virtio
下面的示例，我们定义一个 VirtualMachine，使用前面定义好的偏好设置pref01。然后，在spec.domain.device中，我们使用disks.disk.bus来覆盖pref01中的设置。

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: pvc-ubuntu02-bootdisk
  namespace: default
spec:
  storageClassName: "ceph-hdd-block"
  volumeMode: Block
  accessModes:
    - ReadWriteMany
  dataSource:
    kind: PersistentVolumeClaim
    name: img-ubuntu2204
  resources:
    requests:
      storage: 10Gi
---
apiVersion: kubevirt.io/v1
kind: VirtualMachine
metadata:
  name: vm-ubuntu02
spec:
  instancetype:
    kind: VirtualMachineInstancetype
    name: p1-1
  preference:
    kind: VirtualMachinePreference
    name: pref01
  running: true
  template:
    metadata:
      labels:
        kubevirt.io/domain: vm-ubuntu02
    spec:
      architecture: amd64
      domain:
        devices:
          disks:
          - disk:
              bus: sata
            name: root-disk
          - disk:
              bus: virtio
            name: cloudinitdisk
          rng: {}
          interfaces:
            - name: default
              masquerade: {}
      networks:
        - name: default
          pod: {}
      terminationGracePeriodSeconds: 0
      volumes:
      - name: root-disk
        persistentVolumeClaim:
          claimName: pvc-ubuntu02-bootdisk
      - name: cloudinitdisk
        cloudInitConfigDrive:
          userData: |-
            #cloud-config
            hostname: vm-ubuntu02
            ssh_pwauth: True
            disable_root: False
            timezone: Asia/Shanghai
            password: 123456
            chpasswd: {"list":"root:12345678",expire: False}
            bootcmd:
              - setenforce 0
              - sed -i 's/^SELINUX=enforcing$/SELINUX=disabled/' /etc/selinux/config
            runcmd:
              - sed -i "/PermitRootLogin/s/^.*$/PermitRootLogin yes/g" /etc/ssh/sshd_config
              - systemctl restart sshd.service
              #- echo root:12345678 |chpasswd

# kubectl get vms/vm-ubuntu02 -o json | jq .spec.template.spec.domain.devices.disks
[
  {
    "disk": {
      "bus": "sata"
    },
    "name": "root-disk"
  },
  {
    "disk": {
      "bus": "virtio"
    },
    "name": "cloudinitdisk"
  }
]

在 VirtualMachineInstance 中，剩余其它的磁盘仍然将使用偏好设置中"preferredDiskBus"的值virtio。 

# kubectl get vmis/vm-ubuntu02 -o json | jq .spec.domain.devices.disks
[
  {
    "disk": {
      "bus": "sata"
    },
    "name": "root-disk"
  },
  {
    "disk": {
      "bus": "virtio"
    },
    "name": "cloudinitdisk"
  }
]

注意 如果我们在创建虚拟机时使用了某个实例类型(此处实例类型以 t1 代替)，之后面我们修改了该实例类型cpu的数量，那么在修改实例类型之前创建的虚拟机cpu不会发生变化。 如果想让虚拟机cpu数量发生变化，需要先停止虚拟机，然后清空虚拟机spec.instancetype.revisionName字段的值。

示例：创建实例类型t1 

---
# 名称空间级
apiVersion: instancetype.kubevirt.io/v1beta1
kind: VirtualMachineInstancetype
metadata:
  name: t1
  namespace: default
spec:
  cpu:
    guest: 1
    model: host-passthrough
    dedicatedCPUPlacement: false
    isolateEmulatorThread: false
  memory:
    guest: 1Gi
  nodeSelector:
    cpu: true
vm-ubuntu03的yaml 
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: pvc-ubuntu03-bootdisk
  namespace: default
spec:
  storageClassName: "ceph-hdd-block"
  volumeMode: Block
  accessModes:
    - ReadWriteMany
  dataSource:
    kind: PersistentVolumeClaim
    name: img-ubuntu2204
  resources:
    requests:
      storage: 10Gi
---
apiVersion: kubevirt.io/v1
kind: VirtualMachine
metadata:
  name: vm-ubuntu03
spec:
  instancetype:
    kind: VirtualMachineInstancetype
    name: t1
  running: true
  template:
    metadata:
      labels:
        kubevirt.io/domain: vm-ubuntu03
    spec:
      architecture: amd64
      domain:
        devices:
          disks:
          - disk:
              bus: virtio
            name: root-disk
          - disk:
              bus: virtio
            name: cloudinitdisk
          rng: {}
          interfaces:
            - name: default
              masquerade: {}
      networks:
        - name: default
          pod: {}
      terminationGracePeriodSeconds: 0
      volumes:
      - name: root-disk
        persistentVolumeClaim:
          claimName: pvc-ubuntu03-bootdisk
      - name: cloudinitdisk
        cloudInitConfigDrive:
          userData: |-
            #cloud-config
            hostname: vm-ubuntu03
            ssh_pwauth: True
            disable_root: False
            timezone: Asia/Shanghai
            password: 123456
            chpasswd: {"list":"root:12345678",expire: False}
            bootcmd:
              - setenforce 0
              - sed -i 's/^SELINUX=enforcing$/SELINUX=disabled/' /etc/selinux/config
            runcmd:
              - sed -i "/PermitRootLogin/s/^.*$/PermitRootLogin yes/g" /etc/ssh/sshd_config
              - systemctl restart sshd.service
              #- echo root:12345678 |chpasswd


# kubectl get vmi/vm-ubuntu03 -o json |jq .spec.domain.cpu
{
  "cores": 1,
  "model": "host-passthrough",
  "sockets": 1,
  "threads": 1
}

# kubectl get vm/vm-ubuntu03 -o json |jq .spec.instancetype   
{
  "kind": "VirtualMachineInstancetype",
  "name": "t1",
  "revisionName": "vm-ubuntu03-t1-5a799df9-f92c-46b6-93dc-ff6f4ea08c96-1"
}


# kubectl get controllerrevisions/vm-ubuntu03-t1-5a799df9-f92c-46b6-93dc-ff6f4ea08c96-1 -o json |jq .
{
  "apiVersion": "apps/v1",
  "data": {
    "apiVersion": "instancetype.kubevirt.io/v1beta1",
    "kind": "VirtualMachineInstancetype",
    "metadata": {
      "creationTimestamp": "2024-02-15T05:42:34Z",
      "generation": 1,
      "name": "t1",
      "namespace": "default",
      "resourceVersion": "546900",
      "uid": "5a799df9-f92c-46b6-93dc-ff6f4ea08c96"
    },
    "spec": {
      "cpu": {
        "dedicatedCPUPlacement": false,
        "guest": 1,
        "isolateEmulatorThread": false,
        "model": "host-passthrough"
      },
      "memory": {
        "guest": "1Gi"
      },
      "nodeSelector": {
        "cpu": "true"
      }
    }
  },
  "kind": "ControllerRevision",
  "metadata": {
    "creationTimestamp": "2024-02-15T05:42:40Z",
    "labels": {
      "instancetype.kubevirt.io/object-generation": "1",
      "instancetype.kubevirt.io/object-kind": "VirtualMachineInstancetype",
      "instancetype.kubevirt.io/object-name": "t1",
      "instancetype.kubevirt.io/object-uid": "5a799df9-f92c-46b6-93dc-ff6f4ea08c96",
      "instancetype.kubevirt.io/object-version": "v1beta1"
    },
    "name": "vm-ubuntu03-t1-5a799df9-f92c-46b6-93dc-ff6f4ea08c96-1",
    "namespace": "default",
    "ownerReferences": [
      {
        "apiVersion": "kubevirt.io/v1",
        "blockOwnerDeletion": true,
        "controller": true,
        "kind": "VirtualMachine",
        "name": "vm-ubuntu03",
        "uid": "2071c3ec-ef7e-4c64-abba-9ef4c9b41ba8"
      }
    ],
    "resourceVersion": "546948",
    "uid": "1b8704d6-c1d4-47bd-8c4e-0187f2b27f22"
  },
  "revision": 0
}

接下来，我们调整实例类型中cpu数量，从1改成2 

# kubectl patch VirtualMachineInstancetype/t1 --type merge -p '{"spec":{"cpu":{"guest":2}}}'
为了使已创建的虚拟机cpu数量发生变化，需要先停止虚拟机，然后清空虚拟机spec.instancetype.revisionName字段的值。

# virtctl stop vm-ubuntu03
# kubectl patch vm/vm-ubuntu03 --type merge -p '{"spec":{"instancetype":{"revisionName":""}}}'

# kubectl get vm/vm-ubuntu03 -o json | jq .spec.instancetype

这个时候，我们会发现，vm-ubuntu03引用了新的"ControllerRevision"，然后，我们再次启动vm-ubuntu03，并查看"VirtualMachineInstance"正在使用的新 vCPU 数量： 

# virtctl start vm-ubuntu03


# kubectl get vmi/vm-ubuntu03 -o json |jq .spec.domain.cpu
{
  "cores": 1,
  "model": "host-passthrough",
  "sockets": 2,
  "threads": 1
}

Current CPU Topology并没有刷新，是否为bug?已向社区提 issue 

  Current CPU Topology:
    Cores:    1
    Sockets:  1
    Threads:  1


# kubectl explain vmp --recursive
GROUP:      instancetype.kubevirt.io
KIND:       VirtualMachinePreference
VERSION:    v1beta1

DESCRIPTION:
    VirtualMachinePreference resource contains optional preferences related to
    the VirtualMachine.

FIELDS:
  apiVersion    <string>
  kind  <string>
  metadata      <ObjectMeta>
    annotations <map[string]string>
    creationTimestamp   <string>
    deletionGracePeriodSeconds  <integer>
    deletionTimestamp   <string>
    finalizers  <[]string>
    generateName        <string>
    generation  <integer>
    labels      <map[string]string>
    managedFields       <[]ManagedFieldsEntry>
      apiVersion        <string>
      fieldsType        <string>
      fieldsV1  <FieldsV1>
      manager   <string>
      operation <string>
      subresource       <string>
      time      <string>
    name        <string>
    namespace   <string>
    ownerReferences     <[]OwnerReference>
      apiVersion        <string> -required-
      blockOwnerDeletion        <boolean>
      controller        <boolean>
      kind      <string> -required-
      name      <string> -required-
      uid       <string> -required-
    resourceVersion     <string>
    selfLink    <string>
    uid <string>
  spec  <Object> -required-
    annotations <map[string]string>
    clock       <Object>
      preferredClockOffset      <Object>
        timezone        <string>
        utc     <Object>
          offsetSeconds <integer>
      preferredTimer    <Object>
        hpet    <Object>
          present       <boolean>
          tickPolicy    <string>
        hyperv  <Object>
          present       <boolean>
        kvm     <Object>
          present       <boolean>
        pit     <Object>
          present       <boolean>
          tickPolicy    <string>
        rtc     <Object>
          present       <boolean>
          tickPolicy    <string>
          track <string>
    cpu <Object>
      preferredCPUFeatures      <[]Object>
        name    <string> -required-
        policy  <string>
      preferredCPUTopology      <string>
    devices     <Object>
      preferredAutoattachGraphicsDevice <boolean>
      preferredAutoattachInputDevice    <boolean>
      preferredAutoattachMemBalloon     <boolean>
      preferredAutoattachPodInterface   <boolean>
      preferredAutoattachSerialConsole  <boolean>
      preferredBlockMultiQueue  <boolean>
      preferredCdromBus <string>
      preferredDisableHotplug   <boolean>
      preferredDiskBlockSize    <Object>
        custom  <Object>
          logical       <integer> -required-
          physical      <integer> -required-
        matchVolume     <Object>
          enabled       <boolean>
      preferredDiskBus  <string>
      preferredDiskCache        <string>
      preferredDiskDedicatedIoThread    <boolean>
      preferredDiskIO   <string>
      preferredInputBus <string>
      preferredInputType        <string>
      preferredInterfaceMasquerade      <Object>
      preferredInterfaceModel   <string>
      preferredLunBus   <string>
      preferredNetworkInterfaceMultiQueue       <boolean>
      preferredRng      <Object>
      preferredSoundModel       <string>
      preferredTPM      <Object>
        persistent      <boolean>
      preferredUseVirtioTransitional    <boolean>
      preferredVirtualGPUOptions        <Object>
        display <Object>
          enabled       <boolean>
          ramFB <Object>
            enabled     <boolean>
    features    <Object>
      preferredAcpi     <Object>
        enabled <boolean>
      preferredApic     <Object>
        enabled <boolean>
        endOfInterrupt  <boolean>
      preferredHyperv   <Object>
        evmcs   <Object>
          enabled       <boolean>
        frequencies     <Object>
          enabled       <boolean>
        ipi     <Object>
          enabled       <boolean>
        reenlightenment <Object>
          enabled       <boolean>
        relaxed <Object>
          enabled       <boolean>
        reset   <Object>
          enabled       <boolean>
        runtime <Object>
          enabled       <boolean>
        spinlocks       <Object>
          enabled       <boolean>
          spinlocks     <integer>
        synic   <Object>
          enabled       <boolean>
        synictimer      <Object>
          direct        <Object>
            enabled     <boolean>
          enabled       <boolean>
        tlbflush        <Object>
          enabled       <boolean>
        vapic   <Object>
          enabled       <boolean>
        vendorid        <Object>
          enabled       <boolean>
          vendorid      <string>
        vpindex <Object>
          enabled       <boolean>
      preferredKvm      <Object>
        hidden  <boolean>
      preferredPvspinlock       <Object>
        enabled <boolean>
      preferredSmm      <Object>
        enabled <boolean>
    firmware    <Object>
      preferredUseBios  <boolean>
      preferredUseBiosSerial    <boolean>
      preferredUseEfi   <boolean>
      preferredUseSecureBoot    <boolean>
    machine     <Object>
      preferredMachineType      <string>
    preferSpreadSocketToCoreRatio       <integer>
    preferredSubdomain  <string>
    preferredTerminationGracePeriodSeconds      <integer>
    requirements        <Object>
      cpu       <Object>
        guest   <integer> -required-
      memory    <Object>
        guest   <Object> -required-
    volumes     <Object>
      preferredStorageClassName <string>

实例类型和偏好设置可参考：common-instancetypes ，包括了很多的实例类型和偏好设置，简化了我们在对虚拟机资源配置过程，算得上最佳实践了。 也可以直接在CR中开启feature-gatesCommonInstancetypesDeploymentGate，kubevirt-operator会自动添加一些常用的集群级别的实例类型和偏好设置。