AKS

The following arguments are supported:

• name - (Required) The name of the Managed Kubernetes Cluster to create. Changing this forces a new resource to be created.
• location - (Required) The location where the Managed Kubernetes Cluster should be created. Changing this forces a new resource to be created.
• resource_group_name - (Required) Specifies the Resource Group where the Managed Kubernetes Cluster should exist. Changing this forces a new resource to be created.
• default_node_pool - (Required) A default_node_pool block as defined below.
• dns_prefix - (Optional) DNS prefix specified when creating the managed cluster. Possible values must begin and end with a letter or number, contain only letters, numbers, and hyphens and be between 1 and 54 characters in length. Changing this forces a new resource to be created.
• dns_prefix_private_cluster - (Optional) Specifies the DNS prefix to use with private clusters. Changing this forces a new resource to be created.

Note:

You must define either a dns_prefix or a dns_prefix_private_cluster field.

In addition, one of either identity or service_principal blocks must be specified.

• aci_connector_linux - (Optional) A aci_connector_linux block as defined below. For more details, please visit Create and configure an AKS cluster to use virtual nodes.
• automatic_channel_upgrade - (Optional) The upgrade channel for this Kubernetes Cluster. Possible values are patch, rapid, node-image and stable. Omitting this field sets this value to none.

Note:

Cluster Auto-Upgrade will update the Kubernetes Cluster (and its Node Pools) to the latest GA version of Kubernetes automatically - please see the Azure documentation for more information.

Note:

Cluster Auto-Upgrade only updates to GA versions of Kubernetes and will not update to Preview versions.

• api_server_access_profile - (Optional) An api_server_access_profile block as defined below.
• auto_scaler_profile - (Optional) A auto_scaler_profile block as defined below.
• azure_active_directory_role_based_access_control - (Optional) A azure_active_directory_role_based_access_control block as defined below.

Note:

This requires that the Preview Feature Microsoft.ContainerService/AKS-PrometheusAddonPreview is enabled, see the documentation for more information.

• azure_policy_enabled - (Optional) Should the Azure Policy Add-On be enabled? For more details please visit Understand Azure Policy for Azure Kubernetes Service
• confidential_computing - (Optional) A confidential_computing block as defined below. For more details please the documentation
• disk_encryption_set_id - (Optional) The ID of the Disk Encryption Set which should be used for the Nodes and Volumes. More information can be found in the documentation. Changing this forces a new resource to be created.
• edge_zone - (Optional) Specifies the Edge Zone within the Azure Region where this Managed Kubernetes Cluster should exist. Changing this forces a new resource to be created.
• http_application_routing_enabled - (Optional) Should HTTP Application Routing be enabled?

Note:

At this time HTTP Application Routing is not supported in Azure China or Azure US Government.

• http_proxy_config - (Optional) A http_proxy_config block as defined below.
• identity - (Optional) An identity block as defined below. One of either identity or service_principal must be specified.

Note:

A migration scenario from service_principal to identity is supported. When upgrading service_principal to identity, your cluster's control plane and addon pods will switch to use managed identity, but the kubelets will keep using your configured service_principal until you upgrade your Node Pool.

• image_cleaner_enabled - (Optional) Specifies whether Image Cleaner is enabled.
• image_cleaner_interval_hours - (Optional) Specifies the interval in hours when images should be cleaned up. Defaults to 48.

Note:

This requires that the Preview Feature Microsoft.ContainerService/EnableImageCleanerPreview is enabled and the Resource Provider is re-registered, see the documentation for more information.

• ingress_application_gateway - (Optional) A ingress_application_gateway block as defined below.
• key_management_service - (Optional) A key_management_service block as defined below. For more details, please visit Key Management Service (KMS) etcd encryption to an AKS cluster.
• key_vault_secrets_provider - (Optional) A key_vault_secrets_provider block as defined below. For more details, please visit Azure Keyvault Secrets Provider for AKS.
• kubelet_identity - (Optional) A kubelet_identity block as defined below.
• kubernetes_version - (Optional) Version of Kubernetes specified when creating the AKS managed cluster. If not specified, the latest recommended version will be used at provisioning time (but won't auto-upgrade). AKS does not require an exact patch version to be specified, minor version aliases such as 1.22 are also supported. - The minor version's latest GA patch is automatically chosen in that case. More details can be found in the documentation.

Note:

Upgrading your cluster may take up to 10 minutes per node.

• linux_profile - (Optional) A linux_profile block as defined below.
• local_account_disabled - (Optional) If true local accounts will be disabled. See the documentation for more information.

Note:

If local_account_disabled is set to true, it is required to enable Kubernetes RBAC and AKS-managed Azure AD integration. See the documentation for more information.

• maintenance_window - (Optional) A maintenance_window block as defined below.
• microsoft_defender - (Optional) A microsoft_defender block as defined below.
• monitor_metrics - (Optional) Specifies a Prometheus add-on profile for the Kubernetes Cluster. A monitor_metrics block as defined below.
• network_profile - (Optional) A network_profile block as defined below. Changing this forces a new resource to be created.

Note:

If network_profile is not defined, kubenet profile will be used by default.

• node_resource_group - (Optional) The name of the Resource Group where the Kubernetes Nodes should exist. Changing this forces a new resource to be created.

Note:

Azure requires that a new, non-existent Resource Group is used, as otherwise, the provisioning of the Kubernetes Service will fail.

• oidc_issuer_enabled - (Optional) Enable or Disable the OIDC issuer URL
• oms_agent - (Optional) A oms_agent block as defined below.
• open_service_mesh_enabled - (Optional) Is Open Service Mesh enabled? For more details, please visit Open Service Mesh for AKS.
• private_cluster_enabled - (Optional) Should this Kubernetes Cluster have its API server only exposed on internal IP addresses? This provides a Private IP Address for the Kubernetes API on the Virtual Network where the Kubernetes Cluster is located. Defaults to false. Changing this forces a new resource to be created.
• private_dns_zone_id - (Optional) Either the ID of Private DNS Zone which should be delegated to this Cluster, System to have AKS manage this or None. In case of None you will need to bring your own DNS server and set up resolving, otherwise, the cluster will have issues after provisioning. Changing this forces a new resource to be created.
• private_cluster_public_fqdn_enabled - (Optional) Specifies whether a Public FQDN for this Private Cluster should be added. Defaults to false.

Note:

If you use BYO DNS Zone, the AKS cluster should either use a User Assigned Identity or a service principal (which is deprecated) with the Private DNS Zone Contributor role and access to this Private DNS Zone. If UserAssigned identity is used - to prevent improper resource order destruction - the cluster should depend on the role assignment, like in this example:

resource "azurerm_resource_group" "example" {
 name     = "example"
 location = "West Europe"
}

resource "azurerm_private_dns_zone" "example" {
 name                = "privatelink.eastus2.azmk8s.io"
 resource_group_name = azurerm_resource_group.example.name
}

resource "azurerm_user_assigned_identity" "example" {
 name                = "aks-example-identity"
 resource_group_name = azurerm_resource_group.example.name
 location            = azurerm_resource_group.example.location
}

resource "azurerm_role_assignment" "example" {
 scope                = azurerm_private_dns_zone.example.id
 role_definition_name = "Private DNS Zone Contributor"
 principal_id         = azurerm_user_assigned_identity.example.principal_id
}

resource "azurerm_kubernetes_cluster" "example" {
 name                    = "aksexamplewithprivatednszone1"
 location                = azurerm_resource_group.example.location
 resource_group_name     = azurerm_resource_group.example.name
 dns_prefix              = "aksexamplednsprefix1"
 private_cluster_enabled = true
 private_dns_zone_id     = azurerm_private_dns_zone.example.id

 # rest of configuration omitted for brevity

 depends_on = [
   azurerm_role_assignment.example,
 ]
}

• service_mesh_profile - (Optional) A service_mesh_profile block as defined below.

Note:

This requires that the Preview Feature Microsoft.ContainerService/AzureServiceMeshPreview is enabled and the Resource Provider is re-registered, see the documentation for more information.

• workload_autoscaler_profile - (Optional) A workload_autoscaler_profile block defined below.
• workload_identity_enabled - (Optional) Specifies whether Azure AD Workload Identity should be enabled for the Cluster. Defaults to false.

Note:

To enable Azure AD Workload Identity oidc_issuer_enabled must be set to true.

Note:

This requires that the Preview Feature Microsoft.ContainerService/EnableWorkloadIdentityPreview is enabled and the Resource Provider is re-registered, see the documentation for more information.

• public_network_access_enabled - (Optional) Whether public network access is allowed for this Kubernetes Cluster. Defaults to true. Changing this forces a new resource to be created.

Note:

When public_network_access_enabled is set to true, 0.0.0.0/32 must be added to authorized_ip_ranges in the api_server_access_profile block.

• role_based_access_control_enabled - (Optional) Whether Role Based Access Control for the Kubernetes Cluster should be enabled. Defaults to true. Changing this forces a new resource to be created.
• run_command_enabled - (Optional) Whether to enable run command for the cluster or not. Defaults to true.
• service_principal - (Optional) A service_principal block as documented below. One of either identity or service_principal must be specified.

Note:

A migration scenario from service_principal to identity is supported. When upgrading service_principal to identity, your cluster's control plane and addon pods will switch to use managed identity, but the kubelets will keep using your configured service_principal until you upgrade your Node Pool.

• sku_tier - (Optional) The SKU Tier that should be used for this Kubernetes Cluster. Possible values are Free, and Standard (which includes the Uptime SLA). Defaults to Free.

Note:

Whilst the AKS API previously supported the Paid SKU - the AKS API introduced a breaking change in API Version 2023-02-01 (used in v3.51.0 and later) where the value Paid must now be set to Standard.

• storage_profile - (Optional) A storage_profile block as defined below.
• tags - (Optional) A mapping of tags to assign to the resource.
• web_app_routing - (Optional) A web_app_routing block as defined below.
• windows_profile - (Optional) A windows_profile block as defined below.

An aci_connector_linux block supports the following:

• subnet_name - (Required) The subnet name for the virtual nodes to run.

Note:

At this time ACI Connectors are not supported in Azure China.

Note:

AKS will add a delegation to the subnet named here. To prevent further runs from failing you should make sure that the subnet you create for virtual nodes has a delegation, like so.

resource "azurerm_subnet" "virtual" {

 #...

 delegation {
   name = "aciDelegation"
   service_delegation {
     name    = "Microsoft.ContainerInstance/containerGroups"
     actions = ["Microsoft.Network/virtualNetworks/subnets/action"]
   }
 }
}

An api_server_access_profile block supports the following:

• authorized_ip_ranges - (Optional) Set of authorized IP ranges to allow access to API server, e.g. ["198.51.100.0/24"].
• subnet_id - (Optional) The ID of the Subnet where the API server endpoint is delegated to.
• vnet_integration_enabled - (Optional) Should API Server VNet Integration be enabled? For more details please visit Use API Server VNet Integration.

Note:

This requires that the Preview Feature Microsoft.ContainerService/EnableAPIServerVnetIntegrationPreview is enabled and the Resource Provider is re-registered, see the documentation for more information.

An auto_scaler_profile block supports the following:

• balance_similar_node_groups - (Optional) Detect similar node groups and balance the number of nodes between them. Defaults to false.
• expander - (Optional) Expander to use. Possible values are least-waste, priority, most-pods and random. Defaults to random.
• max_graceful_termination_sec - (Optional) Maximum number of seconds the cluster autoscaler waits for pod termination when trying to scale down a node. Defaults to 600.
• max_node_provisioning_time - (Optional) Maximum time the autoscaler waits for a node to be provisioned. Defaults to 15m.
• max_unready_nodes - (Optional) Maximum Number of allowed unready nodes. Defaults to 3.
• max_unready_percentage - (Optional) Maximum percentage of unready nodes the cluster autoscaler will stop if the percentage is exceeded. Defaults to 45.
• new_pod_scale_up_delay - (Optional) For scenarios like burst/batch scale where you don't want CA to act before the kubernetes scheduler could schedule all the pods, you can tell CA to ignore unscheduled pods before they're a certain age. Defaults to 10s.
• scale_down_delay_after_add - (Optional) How long after the scale up of AKS nodes the scale down evaluation resumes. Defaults to 10m.
• scale_down_delay_after_delete - (Optional) How long after node deletion that scale down evaluation resumes. Defaults to the value used for scan_interval.
• scale_down_delay_after_failure - (Optional) How long after scale down failure that scale down evaluation resumes. Defaults to 3m.
• scan_interval - (Optional) How often the AKS Cluster should be re-evaluated for scale up/down. Defaults to 10s.
• scale_down_unneeded - (Optional) How long a node should be unneeded before it is eligible for scale down. Defaults to 10m.
• scale_down_unready - (Optional) How long an unready node should be unneeded before it is eligible for scale down. Defaults to 20m.
• scale_down_utilization_threshold - (Optional) Node utilization level, defined as sum of requested resources divided by capacity, below which a node can be considered for scale down. Defaults to 0.5.
• empty_bulk_delete_max - (Optional) Maximum number of empty nodes that can be deleted at the same time. Defaults to 10.
• skip_nodes_with_local_storage - (Optional) If true cluster autoscaler will never delete nodes with pods with local storage, for example, EmptyDir or HostPath. Defaults to true.
• skip_nodes_with_system_pods - (Optional) If true cluster autoscaler will never delete nodes with pods from kube-system (except for DaemonSet or mirror pods). Defaults to true.

An azure_active_directory_role_based_access_control block supports the following:

• managed - (Optional) Is the Azure Active Directory integration Managed, meaning that Azure will create/manage the Service Principal used for integration.
• tenant_id - (Optional) The Tenant ID used for Azure Active Directory Application. If this isn't specified the Tenant ID of the current Subscription is used.

When managed is set to true the following properties can be specified:

• admin_group_object_ids - (Optional) A list of Object IDs of Azure Active Directory Groups which should have Admin Role on the Cluster.
• azure_rbac_enabled - (Optional) Is Role Based Access Control based on Azure AD enabled?

When managed is set to false the following properties can be specified:

• client_app_id - (Optional) The Client ID of an Azure Active Directory Application.
• server_app_id - (Optional) The Server ID of an Azure Active Directory Application.
• server_app_secret - (Optional) The Server Secret of an Azure Active Directory Application.

A confidential_computing block supports the following:

• sgx_quote_helper_enabled - (Required) Should the SGX quote helper be enabled?

An monitor_metrics block supports the following:

• annotations_allowed - (Optional) Specifies a comma-separated list of Kubernetes annotation keys that will be used in the resource's labels metric.
• labels_allowed - (Optional) Specifies a Comma-separated list of additional Kubernetes label keys that will be used in the resource's labels metric.

A default_node_pool block supports the following:

• name - (Required) The name which should be used for the default Kubernetes Node Pool. Changing this forces a new resource to be created.
• vm_size - (Required) The size of the Virtual Machine, such as Standard_DS2_v2.

Note:

Resizing the default_node_pool Virtual Machine is done by cycling the system node pool of the cluster. temporary_name_for_rotation must be specified when attempting a resize.

• capacity_reservation_group_id - (Optional) Specifies the ID of the Capacity Reservation Group within which this AKS Cluster should be created. Changing this forces a new resource to be created.
• custom_ca_trust_enabled - (Optional) Specifies whether to trust a Custom CA.

Note:

This requires that the Preview Feature Microsoft.ContainerService/CustomCATrustPreview is enabled and the Resource Provider is re-registered, see the documentation for more information.

• enable_auto_scaling - (Optional) Should the Kubernetes Auto Scaler be enabled for this Node Pool?

Note:

This requires that the type is set to VirtualMachineScaleSets.

Note:

If you're using AutoScaling, you may wish to use Terraform's ignore_changes functionality to ignore changes to the node_count field.

• enable_host_encryption - (Optional) Should the nodes in the Default Node Pool have host encryption enabled? Changing this forces a new resource to be created.

Note:

This requires that the Preview Feature Microsoft.ContainerService/EnableEncryptionAtHostPreview is enabled and the Resource Provider is re-registered.

• enable_node_public_ip - (Optional) Should nodes in this Node Pool have a Public IP Address? Changing this forces a new resource to be created.
• host_group_id - (Optional) Specifies the ID of the Host Group within which this AKS Cluster should be created. Changing this forces a new resource to be created.
• kubelet_config - (Optional) A kubelet_config block as defined below. Changing this forces a new resource to be created.
• linux_os_config - (Optional) A linux_os_config block as defined below. Changing this forces a new resource to be created.
• fips_enabled - (Optional) Should the nodes in this Node Pool have Federal Information Processing Standard enabled? Changing this forces a new resource to be created.
• kubelet_disk_type - (Optional) The type of disk used by kubelet. Possible values are OS and Temporary.
• max_pods - (Optional) The maximum number of pods that can run on each agent. Changing this forces a new resource to be created.
• message_of_the_day - (Optional) A base64-encoded string which will be written to /etc/motd after decoding. This allows customization of the message of the day for Linux nodes. It cannot be specified for Windows nodes and must be a static string (i.e. will be printed raw and not executed as a script). Changing this forces a new resource to be created.
• node_network_profile - (Optional) A node_network_profile block as documented below.
• node_public_ip_prefix_id - (Optional) Resource ID for the Public IP Addresses Prefix for the nodes in this Node Pool. enable_node_public_ip should be true. Changing this forces a new resource to be created.
• node_labels - (Optional) A map of Kubernetes labels which should be applied to nodes in the Default Node Pool.
• node_taints - (Optional) A list of the taints added to new nodes during node pool create and scale. Changing this forces a new resource to be created.
• only_critical_addons_enabled - (Optional) Enabling this option will taint default node pool with CriticalAddonsOnly=true:NoSchedule taint. Changing this forces a new resource to be created.
• orchestrator_version - (Optional) Version of Kubernetes used for the Agents. If not specified, the default node pool will be created with the version specified by kubernetes_version. If both are unspecified, the latest recommended version will be used at provisioning time (but won't auto-upgrade). AKS does not require an exact patch version to be specified, minor version aliases such as 1.22 are also supported. - The minor version's latest GA patch is automatically chosen in that case. More details can be found in the documentation.

Note:

This version must be supported by the Kubernetes Cluster - as such the version of Kubernetes used on the Cluster/Control Plane may need to be upgraded first.

• os_disk_size_gb - (Optional) The size of the OS Disk which should be used for each agent in the Node Pool. Changing this forces a new resource to be created.
• os_disk_type - (Optional) The type of disk which should be used for the Operating System. Possible values are Ephemeral and Managed. Defaults to Managed. Changing this forces a new resource to be created.
• os_sku - (Optional) Specifies the OS SKU used by the agent pool. Possible values include: Ubuntu, CBLMariner, Mariner, Windows2019, Windows2022. If not specified, the default is Ubuntu if OSType=Linux or Windows2019 if OSType=Windows. And the default Windows OSSKU will be changed to Windows2022 after Windows2019 is deprecated. Changing this forces a new resource to be created.
• pod_subnet_id - (Optional) The ID of the Subnet where the pods in the default Node Pool should exist. Changing this forces a new resource to be created.
• proximity_placement_group_id - (Optional) The ID of the Proximity Placement Group. Changing this forces a new resource to be created.
• scale_down_mode - (Optional) Specifies the autoscaling behaviour of the Kubernetes Cluster. Allowed values are Delete and Deallocate. Defaults to Delete.
• temporary_name_for_rotation - (Optional) Specifies the name of the temporary node pool used to cycle the default node pool for VM resizing.
• type - (Optional) The type of Node Pool which should be created. Possible values are AvailabilitySet and VirtualMachineScaleSets. Defaults to VirtualMachineScaleSets. Changing this forces a new resource to be created.

Note:

When creating a cluster that supports multiple node pools, the cluster must use VirtualMachineScaleSets. For more information on the limitations of clusters using multiple node pools see the documentation.

• tags - (Optional) A mapping of tags to assign to the Node Pool.

Note

At this time there's a bug in the AKS API where Tags for a Node Pool are not stored in the correct case - you may wish to use Terraform's ignore_changes functionality to ignore changes to the casing until this is fixed in the AKS API.

• ultra_ssd_enabled - (Optional) Used to specify whether the UltraSSD is enabled in the Default Node Pool. Defaults to false. See the documentation for more information. Changing this forces a new resource to be created.
• upgrade_settings - (Optional) A upgrade_settings block as documented below.
• vnet_subnet_id - (Optional) The ID of a Subnet where the Kubernetes Node Pool should exist. Changing this forces a new resource to be created.

Note:

A Route Table must be configured on this Subnet.

• workload_runtime - (Optional) Specifies the workload runtime used by the node pool. Possible values are OCIContainer and KataMshvVmIsolation.

Note:

Pod Sandboxing / KataVM Isolation node pools are in Public Preview - more information and details on how to opt into the preview can be found in this article

• zones - (Optional) Specifies a list of Availability Zones in which this Kubernetes Cluster should be located. Changing this forces a new Kubernetes Cluster to be created.

Note:

This requires that the type is set to VirtualMachineScaleSets and that load_balancer_sku is set to standard.

If enable_auto_scaling is set to true, then the following fields can also be configured:

• max_count - (Optional) The maximum number of nodes which should exist in this Node Pool. If specified this must be between 1 and 1000.
• min_count - (Optional) The minimum number of nodes which should exist in this Node Pool. If specified this must be between 1 and 1000.
• node_count - (Optional) The initial number of nodes which should exist in this Node Pool. If specified this must be between 1 and 1000 and between min_count and max_count.

Note:

If specified you may wish to use Terraform's ignore_changes functionality to ignore changes to this field.

If enable_auto_scaling is set to false, then the following fields can also be configured:

• node_count - (Optional) The number of nodes which should exist in this Node Pool. If specified this must be between 1 and 1000.

Note:

If enable_auto_scaling is set to false both min_count and max_count fields need to be set to null or omitted from the configuration.

An identity block supports the following:

• type - (Required) Specifies the type of Managed Service Identity that should be configured on this Kubernetes Cluster. Possible values are SystemAssigned or UserAssigned.
• identity_ids - (Optional) Specifies a list of User Assigned Managed Identity IDs to be assigned to this Kubernetes Cluster.

Note:

This is required when type is set to UserAssigned.

A key_management_service block supports the following:

• key_vault_key_id - (Required) Identifier of Azure Key Vault key. See key identifier format for more details. When Azure Key Vault key management service is enabled, this field is required and must be a valid key identifier. When enabled is false, leave the field empty.
• key_vault_network_access - (Optional) Network access of the key vault Network access of key vault. The possible values are Public and Private. Public means the key vault allows public access from all networks. Private means the key vault disables public access and enables private link. The default value is Public.

A key_vault_secrets_provider block supports the following:

• secret_rotation_enabled - (Optional) Should the secret store CSI driver on the AKS cluster be enabled?
• secret_rotation_interval - (Optional) The interval to poll for secret rotation. This attribute is only set when secret_rotation is true and defaults to 2m.

Note:

To enablekey_vault_secrets_provider either secret_rotation_enabled or secret_rotation_interval must be specified.

A kubelet_config block supports the following:

• allowed_unsafe_sysctls - (Optional) Specifies the allow list of unsafe sysctls command or patterns (ending in *). Changing this forces a new resource to be created.
• container_log_max_line - (Optional) Specifies the maximum number of container log files that can be present for a container. must be at least 2. Changing this forces a new resource to be created.
• container_log_max_size_mb - (Optional) Specifies the maximum size (e.g. 10MB) of container log file before it is rotated. Changing this forces a new resource to be created.
• cpu_cfs_quota_enabled - (Optional) Is CPU CFS quota enforcement for containers enabled? Changing this forces a new resource to be created.
• cpu_cfs_quota_period - (Optional) Specifies the CPU CFS quota period value. Changing this forces a new resource to be created.
• cpu_manager_policy - (Optional) Specifies the CPU Manager policy to use. Possible values are none and static, Changing this forces a new resource to be created.
• image_gc_high_threshold - (Optional) Specifies the percent of disk usage above which image garbage collection is always run. Must be between 0 and 100. Changing this forces a new resource to be created.
• image_gc_low_threshold - (Optional) Specifies the percent of disk usage lower than which image garbage collection is never run. Must be between 0 and 100. Changing this forces a new resource to be created.
• pod_max_pid - (Optional) Specifies the maximum number of processes per pod. Changing this forces a new resource to be created.
• topology_manager_policy - (Optional) Specifies the Topology Manager policy to use. Possible values are none, best-effort, restricted or single-numa-node. Changing this forces a new resource to be created.

The kubelet_identity block supports the following:

• client_id - (Optional) The Client ID of the user-defined Managed Identity to be assigned to the Kubelets. If not specified a Managed Identity is created automatically. Changing this forces a new resource to be created.
• object_id - (Optional) The Object ID of the user-defined Managed Identity assigned to the Kubelets.If not specified a Managed Identity is created automatically. Changing this forces a new resource to be created.
• user_assigned_identity_id - (Optional) The ID of the User Assigned Identity assigned to the Kubelets. If not specified a Managed Identity is created automatically. Changing this forces a new resource to be created.

Note:

When kubelet_identity is enabled - The type field in the identity block must be set to UserAssigned and identity_ids must be set.

A linux_os_config block supports the following:

• swap_file_size_mb - (Optional) Specifies the size of the swap file on each node in MB. Changing this forces a new resource to be created.
• sysctl_config - (Optional) A sysctl_config block as defined below. Changing this forces a new resource to be created.
• transparent_huge_page_defrag - (Optional) specifies the defrag configuration for Transparent Huge Page. Possible values are always, defer, defer+madvise, madvise and never. Changing this forces a new resource to be created.
• transparent_huge_page_enabled - (Optional) Specifies the Transparent Huge Page enabled configuration. Possible values are always, madvise and never. Changing this forces a new resource to be created.

A node_network_profile block supports the following:

• node_public_ip_tags - (Optional) Specifies a mapping of tags to the instance-level public IPs. Changing this forces a new resource to be created.

Note:

This requires that the Preview Feature Microsoft.ContainerService/NodePublicIPTagsPreview is enabled and the Resource Provider is re-registered, see the documentation for more information.

A linux_profile block supports the following:

• admin_username - (Required) The Admin Username for the Cluster. Changing this forces a new resource to be created.
• ssh_key - (Required) An ssh_key block. Only one is currently allowed. Changing this will update the key on all node pools. More information can be found in the documentation.

A maintenance_window block supports the following:

• allowed - (Optional) One or more allowed blocks as defined below.
• not_allowed - (Optional) One or more not_allowed block as defined below.

An allowed block exports the following:

• day - (Required) A day in a week. Possible values are Sunday, Monday, Tuesday, Wednesday, Thursday, Friday and Saturday.
• hours - (Required) An array of hour slots in a day. For example, specifying 1 will allow maintenance from 1:00am to 2:00am. Specifying 1, 2 will allow maintenance from 1:00am to 3:00m. Possible values are between 0 and 23.

A not_allowed block exports the following:

• end - (Required) The end of a time span, formatted as an RFC3339 string.
• start - (Required) The start of a time span, formatted as an RFC3339 string.

A microsoft_defender block supports the following:

• log_analytics_workspace_id - (Required) Specifies the ID of the Log Analytics Workspace where the audit logs collected by Microsoft Defender should be sent to.

A network_profile block supports the following:

• network_plugin - (Required) Network plugin to use for networking. Currently supported values are azure, kubenet and none. Changing this forces a new resource to be created.

Note:

When network_plugin is set to azure - the vnet_subnet_id field in the default_node_pool block must be set and pod_cidr must not be set.

• network_mode - (Optional) Network mode to be used with Azure CNI. Possible values are bridge and transparent. Changing this forces a new resource to be created.

Note:

network_mode can only be set to bridge for existing Kubernetes Clusters and cannot be used to provision new Clusters - this will be removed by Azure in the future.

Note:

This property can only be set when network_plugin is set to azure.

• network_policy - (Optional) Sets up network policy to be used with Azure CNI. Network policy allows us to control the traffic flow between pods. Currently supported values are calico and azure. Changing this forces a new resource to be created.

Note:

When network_policy is set to azure, the network_plugin field can only be set to azure.

• dns_service_ip - (Optional) IP address within the Kubernetes service address range that will be used by cluster service discovery (kube-dns). Changing this forces a new resource to be created.
• docker_bridge_cidr - (Optional) IP address (in CIDR notation) used as the Docker bridge IP address on nodes. Changing this forces a new resource to be created.

Note:

docker_bridge_cidr has been deprecated as the API no longer supports it and will be removed in version 4.0 of the provider.

• ebpf_data_plane - (Optional) Specifies the eBPF data plane used for building the Kubernetes network. Possible value is cilium. Changing this forces a new resource to be created.

Note:

When ebpf_data_plane is set to cilium, the network_plugin field can only be set to azure.

Note:

When ebpf_data_plane is set to cilium, one of either network_plugin_mode = "Overlay" or pod_subnet_id must be specified.

Note:

This requires that the Preview Feature Microsoft.ContainerService/CiliumDataplanePreview is enabled and the Resource Provider is re-registered, see the documentation for more information.

• network_plugin_mode - (Optional) Specifies the network plugin mode used for building the Kubernetes network. Possible value is Overlay. Changing this forces a new resource to be created.

Note:

When network_plugin_mode is set to Overlay, the network_plugin field can only be set to azure.

Note:

This requires that the Preview Feature Microsoft.ContainerService/AzureOverlayPreview is enabled and the Resource Provider is re-registered, see the documentation for more information.

• outbound_type - (Optional) The outbound (egress) routing method which should be used for this Kubernetes Cluster. Possible values are loadBalancer, userDefinedRouting, managedNATGateway and userAssignedNATGateway. Defaults to loadBalancer. Changing this forces a new resource to be created.
• pod_cidr - (Optional) The CIDR to use for pod IP addresses. This field can only be set when network_plugin is set to kubenet. Changing this forces a new resource to be created.
• pod_cidrs - (Optional) A list of CIDRs to use for pod IP addresses. For single-stack networking a single IPv4 CIDR is expected. For dual-stack networking an IPv4 and IPv6 CIDR are expected. Changing this forces a new resource to be created.
• service_cidr - (Optional) The Network Range used by the Kubernetes service. Changing this forces a new resource to be created.
• service_cidrs - (Optional) A list of CIDRs to use for Kubernetes services. For single-stack networking a single IPv4 CIDR is expected. For dual-stack networking an IPv4 and IPv6 CIDR are expected. Changing this forces a new resource to be created.

Note:

This range should not be used by any network element on or connected to this VNet. Service address CIDR must be smaller than /12. docker_bridge_cidr, dns_service_ip and service_cidr should all be empty or all should be set.

Examples of how to use AKS with Advanced Networking can be found in the ./examples/kubernetes/ directory in the GitHub repository.

• ip_versions - (Optional) Specifies a list of IP versions the Kubernetes Cluster will use to assign IP addresses to its nodes and pods. Possible values are IPv4 and/or IPv6. IPv4 must always be specified. Changing this forces a new resource to be created.

Note:

To configure dual-stack networking ip_versions should be set to ["IPv4", "IPv6"].

Note:

Dual-stack networking requires that the Preview Feature Microsoft.ContainerService/AKS-EnableDualStack is enabled and the Resource Provider is re-registered, see the documentation for more information.

• load_balancer_sku - (Optional) Specifies the SKU of the Load Balancer used for this Kubernetes Cluster. Possible values are basic and standard. Defaults to standard. Changing this forces a new resource to be created.
• load_balancer_profile - (Optional) A load_balancer_profile block as defined below. This can only be specified when load_balancer_sku is set to standard. Changing this forces a new resource to be created.
• nat_gateway_profile - (Optional) A nat_gateway_profile block as defined below. This can only be specified when load_balancer_sku is set to standard and outbound_type is set to managedNATGateway or userAssignedNATGateway. Changing this forces a new resource to be created.

A load_balancer_profile block supports the following:

Note:

The fields managed_outbound_ip_count, outbound_ip_address_ids and outbound_ip_prefix_ids are mutually exclusive. Note that when specifying outbound_ip_address_ids (azurerm_public_ip) the SKU must be standard.

• idle_timeout_in_minutes - (Optional) Desired outbound flow idle timeout in minutes for the cluster load balancer. Must be between 4 and 120 inclusive. Defaults to 30.
• managed_outbound_ip_count - (Optional) Count of desired managed outbound IPs for the cluster load balancer. Must be between 1 and 100 inclusive.
• managed_outbound_ipv6_count - (Optional) The desired number of IPv6 outbound IPs created and managed by Azure for the cluster load balancer. Must be in the range of 1 to 100 (inclusive). The default value is 0 for single-stack and 1 for dual-stack.

Note:

managed_outbound_ipv6_count requires dual-stack networking. To enable dual-stack networking the Preview Feature Microsoft.ContainerService/AKS-EnableDualStack needs to be enabled and the Resource Provider re-registered, see the documentation for more information.

• outbound_ip_address_ids - (Optional) The ID of the Public IP Addresses which should be used for outbound communication for the cluster load balancer.

Note:

Set outbound_ip_address_ids to an empty slice [] in order to unlink it from the cluster. Unlinking a outbound_ip_address_ids will revert the load balancing for the cluster back to a managed one.

• outbound_ip_prefix_ids - (Optional) The ID of the outbound Public IP Address Prefixes which should be used for the cluster load balancer.

Note:

Set outbound_ip_prefix_ids to an empty slice [] in order to unlink it from the cluster. Unlinking a outbound_ip_prefix_ids will revert the load balancing for the cluster back to a managed one.

• outbound_ports_allocated - (Optional) Number of desired SNAT port for each VM in the clusters load balancer. Must be between 0 and 64000 inclusive. Defaults to 0.

A nat_gateway_profile block supports the following:

• idle_timeout_in_minutes - (Optional) Desired outbound flow idle timeout in minutes for the cluster load balancer. Must be between 4 and 120 inclusive. Defaults to 4.
• managed_outbound_ip_count - (Optional) Count of desired managed outbound IPs for the cluster load balancer. Must be between 1 and 100 inclusive.

An oms_agent block supports the following:

• log_analytics_workspace_id - (Required) The ID of the Log Analytics Workspace which the OMS Agent should send data to.
• msi_auth_for_monitoring_enabled - Is managed identity authentication for monitoring enabled?

An ingress_application_gateway block supports the following:

• gateway_id - (Optional) The ID of the Application Gateway to integrate with the ingress controller of this Kubernetes Cluster. See this page for further details.
• gateway_name - (Optional) The name of the Application Gateway to be used or created in the Nodepool Resource Group, which in turn will be integrated with the ingress controller of this Kubernetes Cluster. See this page for further details.
• subnet_cidr - (Optional) The subnet CIDR to be used to create an Application Gateway, which in turn will be integrated with the ingress controller of this Kubernetes Cluster. See this page for further details.
• subnet_id - (Optional) The ID of the subnet on which to create an Application Gateway, which in turn will be integrated with the ingress controller of this Kubernetes Cluster. See this page for further details.

Note:

If specifying ingress_application_gateway in conjunction with only_critical_addons_enabled, the AGIC pod will fail to start. A separate azurerm_kubernetes_cluster_node_pool is required to run the AGIC pod successfully. This is because AGIC is classed as a "non-critical addon".

A service_mesh_profile block supports the following:

• mode - (Required) The mode of the service mesh. Possible value is Istio.

A service_principal block supports the following:

• client_id - (Required) The Client ID for the Service Principal.
• client_secret - (Required) The Client Secret for the Service Principal.

A ssh_key block supports the following:

• key_data - (Required) The Public SSH Key used to access the cluster.

A storage_profile block supports the following:

• blob_driver_enabled - (Optional) Is the Blob CSI driver enabled? Defaults to false.
• disk_driver_enabled - (Optional) Is the Disk CSI driver enabled? Defaults to true.
• disk_driver_version - (Optional) Disk CSI Driver version to be used. Possible values are v1 and v2. Defaults to v1.

Note:

Azure Disk CSI driver v2 is currently in Public Preview on an opt-in basis. To use it, the feature EnableAzureDiskCSIDriverV2 for namespace Microsoft.ContainerService must be requested.

• file_driver_enabled - (Optional) Is the File CSI driver enabled? Defaults to true.
• snapshot_controller_enabled - (Optional) Is the Snapshot Controller enabled? Defaults to true.

A sysctl_config block supports the following:

Note

For more information, please refer to Linux Kernel Doc.

• fs_aio_max_nr - (Optional) The sysctl setting fs.aio-max-nr. Must be between 65536 and 6553500. Changing this forces a new resource to be created.
• fs_file_max - (Optional) The sysctl setting fs.file-max. Must be between 8192 and 12000500. Changing this forces a new resource to be created.
• fs_inotify_max_user_watches - (Optional) The sysctl setting fs.inotify.max_user_watches. Must be between 781250 and 2097152. Changing this forces a new resource to be created.
• fs_nr_open - (Optional) The sysctl setting fs.nr_open. Must be between 8192 and 20000500. Changing this forces a new resource to be created.
• kernel_threads_max - (Optional) The sysctl setting kernel.threads-max. Must be between 20 and 513785. Changing this forces a new resource to be created.
• net_core_netdev_max_backlog - (Optional) The sysctl setting net.core.netdev_max_backlog. Must be between 1000 and 3240000. Changing this forces a new resource to be created.
• net_core_optmem_max - (Optional) The sysctl setting net.core.optmem_max. Must be between 20480 and 4194304. Changing this forces a new resource to be created.
• net_core_rmem_default - (Optional) The sysctl setting net.core.rmem_default. Must be between 212992 and 134217728. Changing this forces a new resource to be created.
• net_core_rmem_max - (Optional) The sysctl setting net.core.rmem_max. Must be between 212992 and 134217728. Changing this forces a new resource to be created.
• net_core_somaxconn - (Optional) The sysctl setting net.core.somaxconn. Must be between 4096 and 3240000. Changing this forces a new resource to be created.
• net_core_wmem_default - (Optional) The sysctl setting net.core.wmem_default. Must be between 212992 and 134217728. Changing this forces a new resource to be created.
• net_core_wmem_max - (Optional) The sysctl setting net.core.wmem_max. Must be between 212992 and 134217728. Changing this forces a new resource to be created.
• net_ipv4_ip_local_port_range_max - (Optional) The sysctl setting net.ipv4.ip_local_port_range max value. Must be between 1024 and 60999. Changing this forces a new resource to be created.
• net_ipv4_ip_local_port_range_min - (Optional) The sysctl setting net.ipv4.ip_local_port_range min value. Must be between 1024 and 60999. Changing this forces a new resource to be created.
• net_ipv4_neigh_default_gc_thresh1 - (Optional) The sysctl setting net.ipv4.neigh.default.gc_thresh1. Must be between 128 and 80000. Changing this forces a new resource to be created.
• net_ipv4_neigh_default_gc_thresh2 - (Optional) The sysctl setting net.ipv4.neigh.default.gc_thresh2. Must be between 512 and 90000. Changing this forces a new resource to be created.
• net_ipv4_neigh_default_gc_thresh3 - (Optional) The sysctl setting net.ipv4.neigh.default.gc_thresh3. Must be between 1024 and 100000. Changing this forces a new resource to be created.
• net_ipv4_tcp_fin_timeout - (Optional) The sysctl setting net.ipv4.tcp_fin_timeout. Must be between 5 and 120. Changing this forces a new resource to be created.
• net_ipv4_tcp_keepalive_intvl - (Optional) The sysctl setting net.ipv4.tcp_keepalive_intvl. Must be between 10 and 75. Changing this forces a new resource to be created.
• net_ipv4_tcp_keepalive_probes - (Optional) The sysctl setting net.ipv4.tcp_keepalive_probes. Must be between 1 and 15. Changing this forces a new resource to be created.
• net_ipv4_tcp_keepalive_time - (Optional) The sysctl setting net.ipv4.tcp_keepalive_time. Must be between 30 and 432000. Changing this forces a new resource to be created.
• net_ipv4_tcp_max_syn_backlog - (Optional) The sysctl setting net.ipv4.tcp_max_syn_backlog. Must be between 128 and 3240000. Changing this forces a new resource to be created.
• net_ipv4_tcp_max_tw_buckets - (Optional) The sysctl setting net.ipv4.tcp_max_tw_buckets. Must be between 8000 and 1440000. Changing this forces a new resource to be created.
• net_ipv4_tcp_tw_reuse - (Optional) The sysctl setting net.ipv4.tcp_tw_reuse. Changing this forces a new resource to be created.
• net_netfilter_nf_conntrack_buckets - (Optional) The sysctl setting net.netfilter.nf_conntrack_buckets. Must be between 65536 and 147456. Changing this forces a new resource to be created.
• net_netfilter_nf_conntrack_max - (Optional) The sysctl setting net.netfilter.nf_conntrack_max. Must be between 131072 and 1048576. Changing this forces a new resource to be created.
• vm_max_map_count - (Optional) The sysctl setting vm.max_map_count. Must be between 65530 and 262144. Changing this forces a new resource to be created.
• vm_swappiness - (Optional) The sysctl setting vm.swappiness. Must be between 0 and 100. Changing this forces a new resource to be created.
• vm_vfs_cache_pressure - (Optional) The sysctl setting vm.vfs_cache_pressure. Must be between 0 and 100. Changing this forces a new resource to be created.

A web_app_routing block supports the following:

• dns_zone_id - (Required) Specifies the ID of the DNS Zone in which DNS entries are created for applications deployed to the cluster when Web App Routing is enabled. For Bring-Your-Own DNS zones this property should be set to an empty string "".

A windows_profile block supports the following:

• admin_username - (Required) The Admin Username for Windows VMs. Changing this forces a new resource to be created.
• admin_password - (Optional) The Admin Password for Windows VMs. Length must be between 14 and 123 characters.
• license - (Optional) Specifies the type of on-premise license which should be used for Node Pool Windows Virtual Machine. At this time the only possible value is Windows_Server.
• gmsa - (Optional) A gmsa block as defined below.

A gmsa block supports the following:

• dns_server - (Required) Specifies the DNS server for Windows gMSA. Set this to an empty string if you have configured the DNS server in the VNet which was used to create the managed cluster.
• root_domain - (Required) Specifies the root domain name for Windows gMSA. Set this to an empty string if you have configured the DNS server in the VNet which was used to create the managed cluster.

Note:

The properties dns_server and root_domain must both either be set or unset, i.e. empty.

A workload_autoscaler_profile block supports the following:

• keda_enabled - (Optional) Specifies whether KEDA Autoscaler can be used for workloads.

Note:

This requires that the Preview Feature Microsoft.ContainerService/AKS-KedaPreview is enabled and the Resource Provider is re-registered, see the documentation for more information.

• vertical_pod_autoscaler_enabled - (Optional) Specifies whether Vertical Pod Autoscaler should be enabled.

Note:

This requires that the Preview Feature Microsoft.ContainerService/AKS-VPAPreview is enabled and the Resource Provider is re-registered, see the documentation for more information.

A http_proxy_config block supports the following:

• http_proxy - (Optional) The proxy address to be used when communicating over HTTP. Changing this forces a new resource to be created.
• https_proxy - (Optional) The proxy address to be used when communicating over HTTPS. Changing this forces a new resource to be created.
• no_proxy - (Optional) The list of domains that will not use the proxy for communication. Changing this forces a new resource to be created.

Note:

If you specify the default_node_pool.0.vnet_subnet_id, be sure to include the Subnet CIDR in the no_proxy list.

Note:

You may wish to use Terraform's ignore_changes functionality to ignore the changes to this field.

• trusted_ca - (Optional) The base64 encoded alternative CA certificate content in PEM format.

A upgrade_settings block supports the following:

• max_surge - (Required) The maximum number or percentage of nodes which will be added to the Node Pool size during an upgrade.

Note:

If a percentage is provided, the number of surge nodes is calculated from the node_count value on the current cluster. Node surge can allow a cluster to have more nodes than max_count during an upgrade. Ensure that your cluster has enough IP space during an upgrade.

‍

‍