April 13, 2018

Beyond the Basics: Adding New VM Types to Pivotal Cloud Foundry on Microsoft Azure

Pivotal Cloud Foundry is a beast. A full blown deployment with high availability deployment configurations is >50 VMs. Let’s explore how we can optimise our deployment footprint a bit.

Azure makes a lot of VM types available in a given region:

az vm list-sizes -l westus2
[
 {
 "additionalProperties": {},
 "maxDataDiskCount": 2,
 "memoryInMb": 2048,
 "name": "Standard_B1ms",
 "numberOfCores": 1,
 "osDiskSizeInMb": 1047552,
 "resourceDiskSizeInMb": 4096
 },
 {
 "additionalProperties": {},
 "maxDataDiskCount": 2,
 "memoryInMb": 1024,
 "name": "Standard_B1s",
 "numberOfCores": 1,
 "osDiskSizeInMb": 1047552,
 "resourceDiskSizeInMb": 2048
 },
 {
 "additionalProperties": {},
 "maxDataDiskCount": 4,
 "memoryInMb": 8192,
 "name": "Standard_B2ms",
 "numberOfCores": 2,
 "osDiskSizeInMb": 1047552,
 "resourceDiskSizeInMb": 16384
 },
 {
 "additionalProperties": {},
 "maxDataDiskCount": 4,
 "memoryInMb": 4096,
 "name": "Standard_B2s",
 "numberOfCores": 2,
 "osDiskSizeInMb": 1047552,
 "resourceDiskSizeInMb": 8192
 },
{…}
 ]

I’ve omitted 98% of the results because the West US 2 region has 168 different VM configurations, but the important aspect is there is a lot of choice. This is very beneficial when it comes to looking at deployment footprint configurations, whether it’s due to cost or other considerations.

Now that we understand what our choices are, nay, how many choices we have, we need to figure out how to get those VM configurations into Pivotal Operations Manager (OpsMan) so Pivotal Application Services (PAS) can utilise them. First we need to understand what OpsMan needs.

Pivotal Engineers have authored a tool called om; you can find find om on Github. The great part about om is that you don’t need to spend a whole lot of time learning how to get authentication tokens from OpsMan using curl, you can just dive right in and be successful. So let’s explore what our options are with om:

λ  om
ॐ
om helps you interact with an Ops Manager
Usage: om [options] <command> [<args>]
  --client-id, -c            string  Client ID for the Ops Manager VM (not required for unauthenticated commands, $OM_CLIENT_ID)
  --client-secret, -s        string  Client Secret for the Ops Manager VM (not required for unauthenticated commands, $OM_CLIENT_SECRET)
  --format, -f               string  Format to print as (options: table,json) (default: table)
  --help, -h                 bool    prints this usage information (default: false)
  --password, -p             string  admin password for the Ops Manager VM (not required for unauthenticated commands, $OM_PASSWORD)
  --request-timeout, -r      int     timeout in seconds for HTTP requests to Ops Manager (default: 1800)
  --skip-ssl-validation, -k  bool    skip ssl certificate validation during http requests (default: false)
  --target, -t               string  location of the Ops Manager VM
  --trace, -tr               bool    prints HTTP requests and response payloads
  --username, -u             string  admin username for the Ops Manager VM (not required for unauthenticated commands, $OM_USERNAME)
  --version, -v              bool    prints the om release version (default: false)
Commands:
  activate-certificate-authority  activates a certificate authority on the Ops Manager
  apply-changes                   triggers an install on the Ops Manager targeted
  available-products              list available products
  certificate-authorities         lists certificates managed by Ops Manager
  certificate-authority           prints requested certificate authority
  config-template                 **EXPERIMENTAL** generates a config template for the product
  configure-authentication        configures Ops Manager with an internal userstore and admin user account
  configure-bosh                  configures Ops Manager deployed bosh director
  configure-director              configures the director
  configure-product               configures a staged product
  create-certificate-authority    creates a certificate authority on the Ops Manager
  create-vm-extension             creates a VM extension
  credential-references           list credential references for a deployed product
  credentials                     fetch credentials for a deployed product
  curl                            issues an authenticated API request
  delete-certificate-authority    deletes a certificate authority on the Ops Manager
  delete-installation             deletes all the products on the Ops Manager targeted
  delete-product                  deletes a product from the Ops Manager
  delete-unused-products          deletes unused products on the Ops Manager targeted
  deployed-manifest               prints the deployed manifest for a product
  deployed-products               lists deployed products
  errands                         list errands for a product
  export-installation             exports the installation of the target Ops Manager
  generate-certificate            generates a new certificate signed by Ops Manager's root CA
  generate-certificate-authority  generates a certificate authority on the Opsman
  help                            prints this usage information
  import-installation             imports a given installation to the Ops Manager targeted
  installation-log                output installation logs
  installations                   list recent installation events
  pending-changes                 lists pending changes
  regenerate-certificates         regenerates a certificate authority on the Opsman
  revert-staged-changes           reverts staged changes on the Ops Manager targeted
  set-errand-state                sets state for a product's errand
  stage-product                   stages a given product in the Ops Manager targeted
  staged-manifest                 prints the staged manifest for a product
  staged-products                 lists staged products
  unstage-product                 unstages a given product from the Ops Manager targeted
  upload-product                  uploads a given product to the Ops Manager targeted
  upload-stemcell                 uploads a given stemcell to the Ops Manager targeted
  version                         prints the om release version

Looking through the commands, there is generally a command for each major feature of OpsMan. However, as we look further into it, there is no command that will let us configure VM types. There is a command, om curl, though, that will let us replicate certain aspects of curl commands, without having to use curl natively. Let’s see what the JSON structure is so we know what our target payload needs to look like:

om \
    --skip-ssl-validation \
    --target $opsman \
    --username $opsmanUser \
    --password $opsmanPassword \
    curl -x GET --path /api/v0/vm_types
{
  "vm_types": [
    {
      "name": "Standard_B1s",
      "ram": 1024,
      "cpu": 1,
      "ephemeral_disk": 16384,
      "created_at": "2018-04-03T02:26:30.199Z",
      "updated_at": "2018-04-03T02:26:30.199Z",
      "builtin": false
    },
    {
      "name": "Standard_B1ms",
      "ram": 2048,
      "cpu": 1,
      "ephemeral_disk": 16384,
      "created_at": "2018-04-03T02:26:30.201Z",
      "updated_at": "2018-04-03T02:26:30.201Z",
      "builtin": false
    },
    {...}
  ]
}

So we know we need to have a structure like this:

{
  "vm_types": [ 
    {
      "name": "",
      "ram": "",
      "cpu": "",
      "ephemeral_disk": "" 
    }
  ]
}

How do we easily go from Azure’s JSON to OpsMan’s JSON? With jq. jq is a tool written by Stephen Dolan that allows users to extract, transform, and load against JSON on the command line. If you haven’t used jq, I would highly recommend learning it, it’s a great tool to have in the toolbox.

Let’s see how we can transform the Azure JSON into OpsMan JSON:

az vm list-sizes -l westus2 | jq '{"vm_types": [.[] | {"name": .name, "ram": .memoryInMb, "ephemeral_disk": .resourceDiskSizeInMb, "cpu": .numberOfCores}]}'

Let’s explain this a bit. First we grab the VM configurations with az vm list-sizes -l <region>, which will give us a JSON payload on stdout for us to work with. Then we | it to jq so we can start transforming the JSON. '{"vm_types": [...]}' will encompass the Azure JSON into the target structure we need for OpsMan. '{"vm_types": [ .[] | {...}]}' will traverse the Azure JSON array of VM configurations and then | it as one JSON object per Azure JSON object. Now comes the nifty part.

{
  "name": .name,
  "ram": .memoryInMb,
  "ephemeral_disk": .resourceDiskSizeInMb,
  "cpu": .numberOfCores
}

Because jq is a powerful processor, we can actually reference values from piped objects. So if we look at one instance of the Azure JSON:

{
 "additionalProperties": {},
 "maxDataDiskCount": 2,
 "memoryInMb": 2048,
 "name": "Standard_B1ms",
 "numberOfCores": 1,
 "osDiskSizeInMb": 1047552,
 "resourceDiskSizeInMb": 4096
 }

We can actually reference individual fields and pull their values. So in building our target JSON payload, we are really just referencing the Azure configurations so we can directly pass them to OpsMan.

Putting it all together looks like this:

λ  om \
   --skip-ssl-validation \
   --username $opsmanUser \
   --password $opsmanPassword \
   --target $opsmanHttpAddress \
   curl --path /api/v0/vm_types \
   --request PUT --data $(
   az vm list-sizes -l westus2 | \
   jq -c '
{
  "vm_types": [.[] | 
    {
     "name": .name,
     "ram": .memoryInMb,
     "ephemeral_disk": .resourceDiskSizeInMb,
     "cpu": .numberOfCores
     }
   ]
}')

And voila! You now have many choices on how to want to deploy Pivotal Cloud Foundry on Azure with a modifiable deployment footprint. If you want to scale with large VM types, you can; if you want to scale with smaller, burstable VM types, you can do that as well.

Mike’s Note: It is recommended to set the VM types before deploying PAS, otherwise there is the potential to run into an “Out of Resources” error from Azure. This is due to how Azure’s Availability Set manages varying capacity across the internal deployment architecture. If you want to migrate to a new VM type that doesn’t come with OpsMan, it is recommended to scale that PAS component to 1, make the change, and then reset the PAS component to the desired scale. It is a bit time consuming, but it is the safest way to change the VM types.

It is not recommended to use burstable instance types with Diego components due to the nature of the workload.