CoreOS Container Linux on ESXi with OVFTool

According to the Internet, there are a lot of different methods for installing CoreOS Container Linux on ESXi. Unfortunately, most of them requires some manual steps in ESXi's management interface: clicking around in menus, mounting ISO files, browsing for floppies with cloud-config stuff, and so on. This is probably fine if you just want to try it out, but if you want to provision many nodes this becomes pretty tedious.

OVFTool is a really neat tool that can automate all of this, including injection of instance configuration! Download and install it and then get the latest stable version of CoreOS:

$ wget \
    https://stable.release.core-os.net/amd64-usr/current/coreos_production_vmware_ova_image.vmdk.bz2 \
    https://stable.release.core-os.net/amd64-usr/current/coreos_production_vmware_ova.ovf

$ bunzip2 coreos_production_vmware_ova_image.vmdk.bz2

To configure the instance, you should definitely use Ignition. This is the successor of Cloud-Config and it's better in every way, except that is has no support for comments or multi-line strings. Fortunately, there is another tool called ct which can transform a more human friendly yaml-based language to Ignition.

To keep things simple, these examples will use vanilla Ignition. The following configuration will set the hostname, ssh keys and a static address for the instance:

$ cat > coreos-0.json << EOF
{
  "ignition": {
    "version": "2.0.0"
  },
  "storage": {
    "files": [
      {
        "filesystem": "root",
        "path": "/etc/hostname",
        "mode": 420,
        "contents": {
          "source": "data:,core-0"
        }
      }
    ]
  },
  "passwd": {
    "users": [
      {
        "name": "core",
        "sshAuthorizedKeys": [
          "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDGdByTgSVHq......."
        ]
      }
    ]
  },
  "networkd": {
    "units": [
      {
        "name": "00-ens192.network",
        "contents": "[Match]\nName=ens192\n\n[Network]\nDNS=1.2.3.4\nAddress=10.0.0.101/24\nGateway=10.0.0.1"
      }
    ]
  }
}
EOF

There are some issues with the original OVF file and ESXi 6.5. The machine will boot just fine, but you can get warnings that it's running another OS than specified. You can also bump the virtual hardware version to 6.5 (vmx-13). Let's fix that:

$ sed \
    -e 's/other26xLinux64Guest/other3xLinux64Guest/' \
    -e 's/<vssd:VirtualSystemType>.*<\/vssd:VirtualSystemType>/<vssd:VirtualSystemType>vmx-13<\/vssd:VirtualSystemType>/' \
    < coreos_production_vmware_ova.ovf > coreos_production_vmware_ova-esxi65.ovf

Now we're ready to create a new CoreOS instance with OVFTool:

$ ovftool \
    -ds=datastore1 \
    --name=coreos-0 \
    --net:"VM Network=VM Network" \
    --X:guest:coreos.config.data="$(gzip < coreos-0.json | base64)" \
    --X:guest:coreos.config.data.encoding=gzip+base64 \
    --powerOn --skipManifestCheck --noSSLVerify \
    coreos_production_vmware_ova.ovf \
    vi://username:password@your-esxi-host

Your new virtual machine will soon be available on the specified address and you should hopefully be able to login using ssh -l core the-ip-address. If you need to make changes to the Ignition config, you can use the parameter --overwrite to OVFTool to automatically destroy the old instance before creating a new.

Other resources

• OVF Tool user's guide
• Running CoreOS Container Linux on VMware
• Getting Started with Ignition
• Ignition Configuration Specification
• Container Linux Config Transpiler

Complete example

#!/bin/bash
set -eou pipefail

vm_name="core0"
vm_net="VM Network"
ignition_config="core-0.json"

esxi_host="username:password@your-esxi-host"
datastore="datastore1"
coreos_ovf="coreos_production_vmware_ova.ovf"
esxi65_ovf="coreos_production_vmware_ova-esxi65.ovf"


wget \
    https://stable.release.core-os.net/amd64-usr/current/coreos_production_vmware_ova_image.vmdk.bz2 \
    https://stable.release.core-os.net/amd64-usr/current/coreos_production_vmware_ova.ovf

bunzip2 coreos_production_vmware_ova_image.vmdk.bz2

# Patch ovf file to improve compatibility with ESXi 6.5
sed \
  -e 's/other26xLinux64Guest/other3xLinux64Guest/' \
  -e 's/<vssd:VirtualSystemType>.*<\/vssd:VirtualSystemType>/<vssd:VirtualSystemType>vmx-13<\/vssd:VirtualSystemType>/' \
  < "$coreos_ovf" > "$esxi65_ovf"

ignition_config_data=$(gzip < "$ignition_config" | base64)

ovftool \
  ${ovf_params} \
  -ds="$datastore" \
  --name="$vm_name" \
  --net:"VM Network=$vm_net" \
  --X:guest:"coreos.config.data=$ignition_config_data" \
  --X:guest:"coreos.config.data.encoding=gzip+base64" \
  --powerOn --skipManifestCheck --X:noPrompting --noSSLVerify \
  $esxi65_ovf \
  vi://$esxi_host