January 23, 2019

Public to Private: Stateful Firewalling with Pivotal Cloud Foundry on Azure

What we should actually be doing is thinking about what are our key controls that will mitigate the risks. How do we have those funneled and controlled through the team that we have, how do we work through that in a well formatted, formulated process and pay attention to those controls we have chosen? Not a continual, add more, add more, add more.

Dr. Chris Pierson, Chief Executive Officer at Binary Sun Cyber Risk Advisors, SecureWorld Charlotte

In the great plane of attack surfaces, you can see a lot of risks at your Point of Presence (PoP):

  • Protocol connections with malicious protocols
  • Injections, command or SQL!
  • Man-in-the-Middle (MITM) attacks
  • And more!

It’s a very dangerous world on the Internet, and it’s always important to not only protect your customers, but also to protect yourself. Let’s talk about ingress and egress controls with Azure. Azure’s firewalling solution is fantastic, as it’s stateful. To recap from Wikipedia:

[A] stateful firewall is a network firewall that tracks the operating state and characteristics of network connections traversing it. The firewall is configured to distinguish legitimate packets for different types of connections. Only packets matching a known active connection are allowed to pass the firewall.

The Azure Firewall has a lot of great features we can leverage to secure our ingress and egress traffic:

  • High availability
  • Network- and application-level connectivity policies
  • Communicate with Internet resources using SNAT and DNAT
  • Central logging and analytics
  • Stateful!

Now, let’s talk about how to firewall your Pivotal Cloud Foundry PoP.

The easiest way to deploy Pivotal Cloud Foundry is with an Azure Load Balancer which has a Public IP. It’s the easiest and fastest go to production path and has a very small surface area. As Pivotal Cloud Foundry hairpins it’s DNS traffic through the Load Balancer, this means you are accepting that the platform’s internal traffic is exposed over the public internet. Not only does that expose all of Pivotal Cloud Foundry to attack, but it exposes you and your customers through any app logs you have.

Let’s see which requirements this doesn’t comply with:

  • App log traffic can’t be exposed to the internet.
  • cf push should be secured behind a firewall.
  • Internal platform components need to be secured from the world.
  • Egress traffic needs to be secured.

So how do you get from a Public IP with a Load Balancer to a firewalled Foundation? It takes 8 steps and a bit of downtime.

Mike’s Note: Only do this with Production Foundations if you have deployed Pivotal Cloud Foundry in an active-active or active-standby high availability pattern. Following these migration steps results in downtime. Expect about two hours of downtime as you need DNS records to propagate, less if your A records expire quickly or if you reuse the Public IP.

Before we begin, here is a list of new Azure components we’ll be using:

  • Azure Firewall
  • User-Defined Routes with Custom Routing

Step 1: Public IPs

We have two options ahead of us:

1. Create a new Public IP.
2. Reuse the existing Public IP.

It really doesn’t matter which you choose, both end in the same result. If you elect to keep the existing Public IP, you can skip to Step 2.

To create our new Public IP:

az network public-ip create \
    -g <rg> \
    -n <fw-pip-name> \
    --allocation-method Static \
    --sku Standard

It’s important to note the Azure Firewall requires a Standard Sku Public IP, Basic Sku will not work.

Step 2: Migrate the Azure Load Balancer

Here is a crucial decision to make:

  • Keep the existing Azure Load Balancer, OR
  • Migrate the existing one.

Both will result in downtime, however a migration of the existing Load Balancer that Pivotal Application Service leverages won’t require any configuration changes in Pivotal Cloud Foundry. In this guide, I opt for a migration as it’s simpler and you won’t need to go through and apply configuration changes to Pivotal Cloud Foundry.

Let’s migrate:

az network lb update \
    -n <lb> \
    -g <rg> \
    --set frontendIpConfigurations[0].privateIPAddress="<private-ip-on-pas-subnet>" \
    --set frontendIpConfigurations[0].privateIPAllocationMethod="Static" \
    --set frontendIpConfigurations[0].subnet.id="/subscriptions/<sub>/resourceGroups/<rg>/providers/Microsoft.Network/virtualNetworks/<pcf-vnet>/subnets/<pas-subnet>" \
    --set frontendIpConfigurations[0].publicIpAddress=null

Oof, that’s rough. Time to explain. We’re leveraging a wrapper around `az resource update` for the Load Balancer so we can update specific properties without requiring a redeployment. While I did formally try adding a new Frontend IP Configuration for private IPs, the Azure Load Balancer only supports multiple Frontend IP Configurations of the same type, all public or all private. Let’s inspect the properties.

# This is where we set the private IP address of the Load Balancer. My test Azure Load Balancers had only one Frontend IP Configuration, so it was index 0 for me. It's likey to be the same for you. It doesn't inherently matter which subnet you leverage, I put mine in the PAS subnet for simplicity's sake. If you do leverage any subnet that you've deployed tiles into, make sure to place the Load Balancer in the reserved IP range so BOSH doesn't try to use that IP.
frontendIpConfigurations[0].privateIPAddress="<private-ip-on-pas-subnet>"

# We set the private IP allocation method to Static so we can manually specify the private IP. This makes it much easier for DNS as the IP will never change over the course of the Load Balancer lifecycle.
frontendIpConfigurations[0].privateIPAllocationMethod="Static"

# We need to make sure the Load Balancer ends up in the same subnet as the private IP range we gave it.
frontendIpConfigurations[0].subnet.id="/subscriptions/<sub>/resourceGroups/<rg>/providers/Microsoft.Network/virtualNetworks/<pcf-vnet>/subnets/<pas-subnet>"

# Wipe out the Public IP configuration.
frontendIpConfigurations[0].publicIpAddress=null

Step 3: Migrate PCF Internal DNS Records

Now we need to update the DNS records that PCF uses to leverage the Private IP of the Azure Load Balancer we just migrated. The steps required to do this completely depends on your DNS provider, so you’re on your own. Once you’ve updated the records, make sure when you try to manually resolve the records you get the new Private IP. Don’t move on until all the records have been cleared. By default, most DNS providers set the A records to 3600 seconds, so you’ll have to wait an hour. If you don’t wait, things could end up in a really bad state. Just wait, go get coffee, catch up on emails, it’s worth waiting.

Step 4: Verify PCF Operates Correctly

This step isn’t required, but it’s a midway safety check to make sure PCF is still operating as intended.

If you are BOSH savvy:

1. Log into your jumpbox.
2. SSH into OpsMan.
3. Log into the BOSH CLI.
4. Run bosh deployments to get the name of the PAS deployment.
5. Run bosh -d cf-<uuid> run-errand smoke_tests

If you like OpsMan:

1. Log into OpsMan
2. Hit “Review Pending Changes” (assuming there are no pending changes)
3. Deselect “Select All Products”, then select just Pivotal Application Service.
4. Hit “Apply Changes”

What this step verifies is that PCF is still doing what you think it’s doing. It should run the smoke tests and it should complete successfully. If you get to this point and it’s broken, you have a problem. Please call Pivotal Support if you get stuck.

Step 5: Create a Subnet & the Azure Firewall

The Azure Firewall supports one Public and one Private IP, but in order to operate as intended, it needs it’s own very specific subnet so it can scale to meet your traffic demands. It requires a /26 network to be named AzureFirewallSubnet. So let’s create it.

# Create our subnet. Remember, a /26
az network vnet subnet create -g <rg> --vnet-name <pcf-vnet> -n AzureFirewallSubnet --address-prefix <prefix>

# First, add the firewall extension.
az extension add -n azure-firewall

# Create the firewall
az network firewall create -g <rg> -n <fw-name>

# Create it's Public and Private IP configs.
az network firewall ip-config create \
    -g <rg> \
    -f <fw> \
    -n <fw-ip-config> \
    --vnet-name <pcf-vnet> \
    --public-ip-address <fw-pip> \
    --private-ip-address <private-ip-in-fw-subnet-range>

Time to dig deeper:

# Name of the Firewall.
-f <fw>
# Name of the Firewall IP Configuration
-n <fw-ip-config>
# Name of the Virtual Network you are putting the Firewall on.
--vnet-name <pcf-vnet>
# This is the Public IP from Step 1.
--public-ip-address <fw-pip>
# A private IP in the AzureFirewallSubnet. I generally went with x.x.x.5 as that's the first available.
--private-ip-address <private-ip-in-fw-subnet-range>

Step 6: Route Egress Traffic through the Azure Firewall

While Pivotal Cloud Foundry doesn’t phone home, it does integrate with 1st/3rd Party solutions which do. A good example of this would be the OMS agent for Azure’s Log Analytics, it needs to talk over the internet to the OMS endpoints.

Let’s configure our custom route so all egress traffic flows through the Azure Firewall.

# Create the baseline route table.
az network route-table create -g <rg> -n <egress-table-name>

# Create the route in the route table.
az network route-table route create \
    -g <rg> \
    -n <route-name> \
    --next-hop-type VirtualAppliance \
    --route-table-name <egress-table-name> \
    --next-hop-ip-address <fw-private-ip> \
    --address-prefix 0.0.0.0/0

Deeper:

# Name of the route for egress traffic.
-n <route-name>

# TypeOf next hop. This must be VirtualAppliance so the Azure network fabric can properly map it to the Azure Firewall.
--next-hop-type VirtualAppliance

# Name of the route table created above.
--route-table-name <egress-table-name>

# The Private IP address of the Azure Firewall.
--next-hop-ip-address <fw-private-ip>

# All traffic!
--address-prefix 0.0.0.0

Now, while we are creating the route, the Azure Firewall will deny all by default, so traffic doesn’t actually go anywhere. At this point, feel free to add any egress rules you need.

Step 7: Configure Destination NAT to PCF

We need to make sure external internet traffic makes it to our newly minted Internal Load Balancer (ILB), otherwise traffic stops at the firewall.

az resource update \
    --ids /subscriptions/<sub>/resourceGroups/<rg>/providers/Microsoft.Network/azureFirewalls/<fw> \
    --add properties.natRuleCollections '{"name":"allow-pcf","properties":{"action":{"type":"Dnat"},"priority":100,"rules":[{"destinationAddresses":["<fw-pip>"],"destinationPorts":["80"],"name":"allow-http-pas","protocols":["TCP"],"sourceAddresses":["*"],"translatedAddress":"<ilb-private-ip>","translatedPort":"80"},{"destinationAddresses":["<fw-pip>"],"destinationPorts":["443"],"name":"allow-https-pas","protocols":["TCP"],"sourceAddresses":["*"],"translatedAddress":"<ilb-private-ip>","translatedPort":"443"}]}}'

DEEPER:

# Updates an arbitrary resource.
az resource update

# Specify it's our firewall.
--ids /subscriptions/<sub>/resourceGroups/<rg>/providers/Microsoft.Network/azureFirewalls/<fw>

# Add our configs! It's a JSON payload so it's scriptable, currently this is the only way to programmatically update the Azure Firewall as the azure-firewall extension doesn't support this feature yet.
--add properties.natRuleCollections '{"name":"allow-pcf","properties":{"action":{"type":"Dnat"},"priority":100,"rules":[{"destinationAddresses":["<fw-pip>"],"destinationPorts":["80"],"name":"allow-http-pas","protocols":["TCP"],"sourceAddresses":["*"],"translatedAddress":"<ilb-private-ip>","translatedPort":"80"},{"destinationAddresses":["<fw-pip>"],"destinationPorts":["443"],"name":"allow-https-pas","protocols":["TCP"],"sourceAddresses":["*"],"translatedAddress":"<ilb-private-ip>","translatedPort":"443"}]}}'
# Human Readable YAML™
# Same rule as above, just a bit more readable.
---
name: allow-pcf
properties:
  # our DNAT configs.
  action:
    type: Dnat
  # this can be whatever you want, if it's only fronting PCF, leave it at 100.
  priority: 100
  rules:
  # Here is where we convert fw-pip:80 to ilb:80
  - destinationAddresses:
    - "<fw-pip>"
    destinationPorts:
    - '80'
    name: allow-http-pas
    protocols:
    # HTTP is TCP
    - TCP
    sourceAddresses:
    - "*"
    # our ILB
    translatedAddress: "<ilb-private-ip>"
    translatedPort: '80'
  # Here is where we convert fw-pip:443 to ilb:443
  - destinationAddresses:
    - "<fw-pip>"
    destinationPorts:
    - '443'
    name: allow-https-pas
    protocols:
    # HTTP is TCP
    - TCP
    sourceAddresses:
    - "*"
    # our ILB
    translatedAddress: "<ilb-private-ip>"
    translatedPort: '443'

STEP 8: FINISH HIM

Validate. Validate. Validate. Validate. Validate. Validate. Validate. Validate. Validate. Validate. Validate. Validate. Validate. Validate. Validate.

At this point, your external-facing apps should be available, assuming you used the same Public IP as before. Otherwise make sure to update any publicly facing DNS records with the Public IP you created in Step 1.

Errata?

Poke me on Twitter.