Azure Private Endpoints and DNS Name resolving

Introduction

Private Endpoints in Azure are way to make PaaS services privately available from within your environment and are part of Azure Private Link. For more details on how these components relate to each other, multiple resource are available online, for instance: https://global.hitachi-solutions.com/blog/azure-private-link/

What people sometimes do not seem to understand is that the use of private endpoints involves DNS changes that can have some unexpected side effects, especially if you communicate with customers or partners that also make use of Private Endpoints.

Let’s use some examples to highlight what changes.

Resolving PaaS Services in Azure

Say we create an Azure Storage Account, for the moment without Private Endpoint, with the name agopepdemo. A storage account contains multiple services. In this example we will only look at blob.
The storage account endpoint for blob, in this case, will be agopepdemo.blob.core.windows.net. To check how this is resolved in DNS, we can use nslookup in the commandline:

What this tells us, is:

• agopepdemo.blob.core.windows.net is a CNAME record which points to blob.ams09prdstr07a.store.core.windows.net (The Azure stamp / farm that the storage account is hosted by)
• blob.ams09prdstr07a.store.core.windows.net is and A record and resolves to 20.60.223.100

So far it is relatively straight forward. Let’s add a private endpoint to this account (for the blob service), but leave public access enabled. If we now look with nslookup, we see this:

This shows us the process is now a little bit different:

• agopepdemo.blob.core.windows.net is still a CNAME, however now it points to agopepdemo.privatelink.blob.core.windows.net
• agopepdemo.privatelink.blob.core.windows.net is also a CNAME. Because I am resolving this externally, this CNAME points to blob.ams09prdstr07a.store.core.windows.net
• blob.ams09prdstr07a.store.core.windows.net is still an A record and points to 20.60.223.100 As part of the setup of a Private Endpoint, you would normally also create a record in an Azure Private DNS zone linked to your VNET(s) or potentially in your own internal DNS server. This setup falls outside of the scope of this example.

If we try the same thing from an internal resource, we see something else. In this case I am using the serial console of a VM in the same VNET:

• agopepdemo.blob.core.windows.net is still points to agopepdemo.privatelink.blob.core.windows.net
• agopepdemo.privatelink.blob.core.windows.net is now an A record and it points to the internal IP of the private endpoint But what happens if we create another storage account, with the same setup: public access enabled, but in another VNET and using another Azure Private DNS zone?

Externally, it shows (of course) a similar result:

The storage cluster and IP are different, but the process is identical. However, if we attempt to do this from the VM in the original VNET, we see that we cannot resolve (and thus access) the storage account. Even though public access is enabled:

In order to see why it is failing, we can enable debug mode in nslookup and try it again:

What do we see here?

• agopepdemo2.blob.core.windows.net is still a CNAME to agopepdemo2.privatelink.blob.core.windows.net
• The (local) domain privatelink.blob.core.windows.net does not contain a record for agopepdemo2 and DNS returns a failure

If a DNS zone has a copy of a zone, privatelink.blob.core.windows.net in this case, it will consider itself authoritative for the zone. This means that when you query that DNS server for a record, it will return the record if it has it and if it does not, it will return a failure and any further attempts to resolve the record stop. More information about recursive and authoritative DNS servers is available in many places, but for example: https://umbrella.cisco.com/blog/what-is-the-difference-between-authoritative-and-recursive-dns-nameservers

Workarounds

So, can we work around this? Yes, with some effort: We could create a CNAME record for the external storage account in our private DNS pointing to the stamp. For instance in order to resolve the last example we can add the following to our privatelink.blob.core.windows.net DNS zone:

If we now try to resolve the agopepdemo2 blob endpoint again:

An other solution could be create a subdomain for the local private endpoint address rather than a record. This would mean creating an Azure Private DNS zone agopepdemo2.privatelink.blob.core.windows.net. In this zone we would need to create an @ record pointing to the private IP of the Private Endpoint:

The advantage of this method is that we do not need to add records for external accounts (again, with private endpoint), but the obvious downside is that we now need to create a separate DNS zone for each and every Private Endpoint locally. If there are a lot of Private Endpoint this easily gets cumbersome.

Complications

To complicate matters, some Azure Resources ALWAYS have a CNAME to a privatelink record. For example the portal for Azure Data Factory (adf.azure.com):

This means that if you have a Private Endpoint enabled for your local Azure Data Factory and created a privatelink.adf.azure.com zone to resolve the private IP address, you can no longer access any other ADF instance, even if they are not using Private Endpoints. This is probably an example where it could be worth it to create a subdomain rather than a record (especially if you are a MSP).