initial commit

.gitignore

@@ -0,0 +1,5 @@
+.terraform*
+*.tfstate*
+crash.log
+kubeconfig.yaml
+terraform.tfvars

LICENSE

@@ -0,0 +1,5 @@
+Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.

README.md

@@ -0,0 +1,240 @@
+[![Contributors][contributors-shield]][contributors-url]
+[![Forks][forks-shield]][forks-url]
+[![Stargazers][stars-shield]][stars-url]
+[![Issues][issues-shield]][issues-url]
+[![MIT License][license-shield]][license-url]
+[![LinkedIn][linkedin-shield]][linkedin-url]
+
+<!-- PROJECT LOGO -->
+<br />
+<p align="center">
+  <a href="https://github.com/mysticaltech/kube-hetzner">
+    <img src=".images/kube-hetzner-logo.png" alt="Logo" width="112" height="112">
+  </a>
+
+  <h2 align="center">Kube-Hetzner</h2>
+
+  <p align="center">
+    A fully automated, optimized and auto-upgradable, HA-able, k3s cluster on <a href="https://hetzner.com" target="_blank">Hetzner Cloud</a> 🤑
+  </p>
+  <hr />
+  <br />
+</p>
+
+## About The Project
+
+![Product Name Screen Shot][product-screenshot]
+
+[Hetzner Cloud](https://hetzner.com) is a good cloud provider that offers very affordable prices for cloud instances. The goal of this project was to create an optimal Kubernetes installation with it. We wanted functionality that was as close as possible to GKE's auto-pilot.
+
+Here's what is working at the moment:
+
+- Lightweight and resource-efficient Kubernetes with [k3s](https://github.com/k3s-io/k3s), and Fedora nodes to take advantage of the latest Linux kernels.
+- Optimal [Cilium](https://github.com/cilium/cilium) CNI with full BPF support and Kube-proxy replacement. It uses the Hetzner private subnet underneath to communicate between the nodes, so no encryption is not needed.
+- Automatic OS upgrades, supported by [kured](https://github.com/weaveworks/kured) instances that initiate a reboot of the node only when necessary and after having drained it properly.
+- Automatic HA by setting the required number of servers and agents nodes.
+- Automatic k3s upgrade by using Rancher's [system-upgrade-controller](https://github.com/rancher/system-upgrade-controller) and tracking the latest 1.x stable branch.
+- Optional [Nginx ingress controller](https://kubernetes.github.io/ingress-nginx/) that will automatically use Hetzner's private network to allocate a Hetzner load balancer.
+
+It uses Terraform to deploy as it's easy to use, and Hetzner provides a great [Hetzner Terraform Provider](https://registry.terraform.io/providers/hetznercloud/hcloud/latest/docs).
+
+<!-- GETTING STARTED -->
+
+## Getting started
+
+Follow those simple steps and your world cheapest Kube cluster will be up and running in no time.
+
+### Prerequisites
+
+First and foremost, you need to have a Hetzner Cloud account. You can sign up for free [here](https://hetzner.com/cloud/).
+
+Then you'll need you have both the [terraform](https://learn.hashicorp.com/tutorials/terraform/install-cli) and [helm](https://helm.sh/docs/intro/install/), and [kubectl](https://kubernetes.io/docs/tasks/tools/) cli installed. The easiest way is to use the [gofish](https://gofi.sh/#install) package manager to install them.
+
+```sh
+gofish install terraform
+gofish install helm
+gofish install kubectl
+```
+
+### Creating terraform.tfvars
+
+1. Create a project in your Hetzner Cloud Console, and go to **Security > API Tokens** of that project to grab the API key.
+2. Generate an ssh key pair for your cluster, unless you already have one that you'd like to use.
+3. Rename terraform.tfvars.example to terraform.tfvars and replace the values from steps 1 and 2.
+
+### (Optional) Customize other variables
+
+The number of control plane nodes and worker nodes, and the Hetzner datacenter location, can be customized by adding the variables to your newly created [terraform.tfvars](terraform.tfvars) file.
+
+See the default values in the [variables.tf](variables.tf) file, they correspond to (you can copy-paste and customize):
+
+```tfvars
+servers_num      = 2
+agents_num       = 2
+server_location  = "fsn1"
+```
+
+### Installation
+
+```sh
+terraform init
+terraform apply -auto-approve
+```
+
+It will take a few minutes to complete, and then you should see a green output with the IP addresses of the nodes. Then you can immediately kubectl into it without any further action.
+
+<!-- USAGE EXAMPLES -->
+
+## Usage
+
+When the cluster is on, you can do whatever you wish with it! 🎉
+
+### Useful commands
+
+- List your nodes IPs, with either of those:
+
+```sh
+terraform outputs
+hcloud server list
+```
+
+- See the Hetzner network config:
+
+```sh
+hcloud network describe k3s-net
+```
+
+- Log into one of your nodes (replace the location of your private key if needed):
+
+```sh
+ssh root@xxx.xxx.xxx.xxx -i ~/.ssh/id_ed25519 -o StrictHostKeyChecking=no
+```
+
+### Cilium commands
+
+- Check the status of cilium with the following commands (get the cilium pod name first and replace it in the command):
+
+```sh
+kubectl -n kube-system exec --stdin --tty cilium-xxxx -- cilium status
+kubectl -n kube-system exec --stdin --tty cilium-xxxx -- cilium status --verbose
+```
+
+- Monitor cluster traffic with:
+
+```sh
+kubectl -n kube-system exec --stdin --tty cilium-xxxx -- cilium monitor
+```
+
+- See the list of kube services with:
+
+```sh
+kubectl -n kube-system exec --stdin --tty cilium-xxxx -- cilium service list
+```
+
+_For more cilium commands, please refer to their corresponding [Documentation](https://docs.cilium.io/en/latest/cheatsheet)._
+
+### Automatic upgrade
+
+The nodes and k3s versions are configured to self-upgrade unless you turn that feature off.
+
+- To turn OS upgrade off, log in to each node and issue:
+
+```sh
+systemctl disable --now dnf-automatic.timer
+```
+
+- To turn off k3s upgrade, you can just use kubectl to set the k3s_upgrade label to false for each node:
+
+```sh
+kubectl label node node-name k3s_upgrade=false
+```
+
+### Individual components upgrade
+
+To upgrade individual components, you can use the following commands:
+
+- Hetzner CCM
+
+```sh
+kubectl apply -f manifests/hcloud-ccm-net.yaml
+```
+
+- Hetzner CSI
+
+```sh
+kubectl apply -f https://raw.githubusercontent.com/hetznercloud/csi-driver/master/deploy/kubernetes/hcloud-csi.yml
+```
+
+- Rancher's system upgrade controller
+
+```sh
+kubectl apply -f https://raw.githubusercontent.com/rancher/system-upgrade-controller/master/manifests/system-upgrade-controller.yaml
+```
+
+- Kured (used to reboot the nodes after upgrading and draining them)
+
+```sh
+latest=$(curl -s https://api.github.com/repos/weaveworks/kured/releases | jq -r .[0].tag_name)
+kubectl apply -f https://github.com/weaveworks/kured/releases/download/$latest/kured-$latest-dockerhub.yaml
+```
+
+- Last but not least, upgrading Cilium itself
+
+```sh
+helm upgrade --values=manifests/helm/cilium/values.yaml cilium cilium/cilium -n kube-system
+```
+
+<!-- ROADMAP -->
+
+## Roadmap
+
+See the [open issues](https://github.com/mysticaltech/kube-hetzner/issues) for a list of proposed features (and known issues).
+
+<!-- CONTRIBUTING -->
+
+## Contributing
+
+Any contributions you make are **greatly appreciated**.
+
+1. Fork the Project
+2. Create your Branch (`git checkout -b AmazingFeature`)
+3. Commit your Changes (`git commit -m 'Add some AmazingFeature'`)
+4. Push to the Branch (`git push origin AmazingFeature`)
+5. Open a Pull Request
+
+<!-- LICENSE -->
+
+## License
+
+The following code is distributed as-is and under the MIT License. See [LICENSE](LICENSE) for more information.
+
+<!-- CONTACT -->
+
+## Contact
+
+Karim Naufal - [@myticaltech](https://twitter.com/mysticaltech) - karim.naufal@me.com
+
+Project Link: [https://github.com/mysticaltech/kube-hetzner](https://github.com/mysticaltech/kube-hetzner)
+
+<!-- ACKNOWLEDGEMENTS -->
+
+## Acknowledgements
+
+[k-andy](https://github.com/StarpTech/k-andy) was the starting point for this project. It wouldn't have been possible without it.
+
+<!-- MARKDOWN LINKS & IMAGES -->
+<!-- https://www.markdownguide.org/basic-syntax/#reference-style-links -->
+
+[contributors-shield]: https://img.shields.io/github/contributors/mysticaltech/kube-hetzner.svg?style=for-the-badge
+[contributors-url]: https://github.com/mysticaltech/kube-hetzner/graphs/contributors
+[forks-shield]: https://img.shields.io/github/forks/mysticaltech/kube-hetzner.svg?style=for-the-badge
+[forks-url]: https://github.com/mysticaltech/kube-hetzner/network/members
+[stars-shield]: https://img.shields.io/github/stars/mysticaltech/kube-hetzner.svg?style=for-the-badge
+[stars-url]: https://github.com/mysticaltech/kube-hetzner/stargazers
+[issues-shield]: https://img.shields.io/github/issues/mysticaltech/kube-hetzner.svg?style=for-the-badge
+[issues-url]: https://github.com/mysticaltech/kube-hetzner/issues
+[license-shield]: https://img.shields.io/github/license/mysticaltech/kube-hetzner.svg?style=for-the-badge
+[license-url]: https://github.com/mysticaltech/kube-hetzner/blob/master/LICENSE.txt
+[linkedin-shield]: https://img.shields.io/badge/-LinkedIn-black.svg?style=for-the-badge&logo=linkedin&colorB=555
+[linkedin-url]: https://www.linkedin.com/in/karimnaufal/
+[product-screenshot]: .images/kubectl-pods-screenshot.png

agents.tf

@@ -0,0 +1,51 @@
+resource "hcloud_server" "agents" {
+  count = var.agents_num
+  name  = "k3s-agent-${count.index}"
+
+  image       = data.hcloud_image.linux.name
+  server_type = local.agent_server_type
+  location    = local.agent_locations[count.index][1]
+
+  ssh_keys = [hcloud_ssh_key.default.id]
+
+  labels = {
+    provisioner = "terraform",
+    engine      = "k3s",
+    node_type   = "agent"
+    k3s_upgrade = "true"
+  }
+
+  user_data = data.template_cloudinit_config.init_cfg.rendered
+
+  provisioner "remote-exec" {
+    inline = var.initial_commands
+
+    connection {
+      user        = "root"
+      private_key = file(var.private_key)
+      host        = self.ipv4_address
+    }
+  }
+
+  provisioner "remote-exec" {
+    inline = [
+      "curl -sfL https://get.k3s.io | K3S_TOKEN=${random_password.k3s_cluster_secret.result} sh -s - agent --server https://${local.first_control_plane_ip}:6443 --kubelet-arg='cloud-provider=external' --no-flannel"
+    ]
+
+    connection {
+      user        = "root"
+      private_key = file(var.private_key)
+      host        = self.ipv4_address
+    }
+  }
+
+  network {
+    network_id = hcloud_network.k3s.id
+    ip         = cidrhost(hcloud_network.k3s.ip_range, 2 + var.servers_num + count.index)
+  }
+
+  depends_on = [
+    hcloud_server.first_control_plane,
+    hcloud_network_subnet.k3s
+  ]
+}

init.cfg

@@ -0,0 +1,41 @@
+  #cloud-config
+  write_files:
+    - path: /etc/sysctl.d/k8s.conf
+      content: |
+        net.bridge.bridge-nf-call-ip6tables = 1
+        net.bridge.bridge-nf-call-iptables = 1
+    - path: /etc/sysctl.d/99-override_cilium_rp_filter.conf
+      content: |
+        net.ipv4.conf.lxc*.rp_filter = 0
+    - path: /etc/fail2ban/jail.local
+      content: |
+        [sshd]
+        enabled = true
+        banaction = iptables-multiport
+    - path: /etc/yum.repos.d/rpm-rancher-io.repo
+      content: |
+        [rancher] 
+        name=Rancher 
+        baseurl=https://rpm.rancher.io/k3s/stable/common/centos/8/noarch
+        enabled=1 
+        gpgcheck=1 
+        gpgkey=https://rpm.rancher.io/public.key
+    - path: /etc/dnf/automatic.conf
+      content: |
+        [commands]
+        upgrade_type = default
+        random_sleep = 10000
+        network_online_timeout = 60
+        download_updates = yes
+        apply_updates = yes
+        [emitters]
+        emit_via = stdio
+        [base]
+        debuglevel = 1
+  runcmd:
+    - sed -i -e '/^PasswordAuthentication/s/^.*$/PasswordAuthentication no/' /etc/ssh/sshd_config
+    - sed -i -e '/^X11Forwarding/s/^.*$/X11Forwarding no/' /etc/ssh/sshd_config
+    - sed -i -e '/^#MaxAuthTries/s/^.*$/MaxAuthTries 2/' /etc/ssh/sshd_config
+    - sed -i -e '/^#AllowTcpForwarding/s/^.*$/AllowTcpForwarding no/' /etc/ssh/sshd_config
+    - sed -i -e '/^#AllowAgentForwarding/s/^.*$/AllowAgentForwarding no/' /etc/ssh/sshd_config
+    - sed -i -e '/^#AuthorizedKeysFile/s/^.*$/AuthorizedKeysFile .ssh\/authorized_keys/' /etc/ssh/sshd_config

k3s.tf

@@ -0,0 +1,8 @@
+locals {
+  control_plane_server_type = "cx11"
+  agent_server_type         = "cx21"
+  first_control_plane_ip    = cidrhost(hcloud_network.k3s.ip_range, 2)
+  locations                 = [var.server_location, "fsn1", "fsn1"]
+  agent_locations           = setproduct(range(var.agents_num), local.locations)
+  server_locations          = setproduct(range(var.servers_num), local.locations)
+}

main.tf

@@ -0,0 +1,51 @@
+resource "random_password" "k3s_cluster_secret" {
+  length  = 48
+  special = false
+}
+
+resource "hcloud_ssh_key" "default" {
+  name       = "K3S terraform module - Provisioning SSH key"
+  public_key = file(var.public_key)
+}
+
+resource "hcloud_network" "k3s" {
+  name     = "k3s-net"
+  ip_range = "10.0.0.0/8"
+}
+
+resource "hcloud_network_subnet" "k3s" {
+  network_id   = hcloud_network.k3s.id
+  type         = "cloud"
+  network_zone = "eu-central"
+  ip_range     = "10.0.0.0/16"
+}
+
+data "hcloud_image" "linux" {
+  name = "fedora-34"
+}
+
+data "template_file" "init_cfg" {
+  template = file("init.cfg")
+}
+
+# Render a multi-part cloud-init config making use of the part
+# above, and other source files
+data "template_cloudinit_config" "init_cfg" {
+  gzip          = true
+  base64_encode = true
+
+  # Main cloud-config configuration file.
+  part {
+    filename     = "init.cfg"
+    content_type = "text/cloud-config"
+    content      = data.template_file.init_cfg.rendered
+  }
+}
+
+data "template_file" "ccm_manifest" {
+  template = file("${path.module}/manifests/hcloud-ccm-net.yaml")
+}
+
+data "template_file" "upgrade_plan" {
+  template = file("${path.module}/manifests/upgrade/plan.yaml")
+}

manifests/hcloud-ccm-net.yaml

@@ -0,0 +1,87 @@
+apiVersion: v1
+kind: ServiceAccount
+metadata:
+  name: cloud-controller-manager
+  namespace: kube-system
+---
+kind: ClusterRoleBinding
+apiVersion: rbac.authorization.k8s.io/v1
+metadata:
+  name: system:cloud-controller-manager
+roleRef:
+  apiGroup: rbac.authorization.k8s.io
+  kind: ClusterRole
+  name: cluster-admin
+subjects:
+  - kind: ServiceAccount
+    name: cloud-controller-manager
+    namespace: kube-system
+---
+apiVersion: apps/v1
+kind: Deployment
+metadata:
+  name: hcloud-cloud-controller-manager
+  namespace: kube-system
+spec:
+  replicas: 1
+  revisionHistoryLimit: 2
+  selector:
+    matchLabels:
+      app: hcloud-cloud-controller-manager
+  template:
+    metadata:
+      labels:
+        app: hcloud-cloud-controller-manager
+      annotations:
+        scheduler.alpha.kubernetes.io/critical-pod: ''
+    spec:
+      serviceAccountName: cloud-controller-manager
+      dnsPolicy: Default
+      tolerations:
+        # this taint is set by all kubelets running `--cloud-provider=external`
+        # so we should tolerate it to schedule the cloud controller manager
+        - key: "node.cloudprovider.kubernetes.io/uninitialized"
+          value: "true"
+          effect: "NoSchedule"
+        - key: "CriticalAddonsOnly"
+          operator: "Exists"
+        # cloud controller manages should be able to run on masters
+        - key: "node-role.kubernetes.io/master"
+          effect: NoSchedule
+        - key: "node-role.kubernetes.io/control-plane"
+          effect: NoSchedule
+        - key: "node.kubernetes.io/not-ready"
+          effect: "NoSchedule"
+      hostNetwork: true
+      containers:
+        - image: hetznercloud/hcloud-cloud-controller-manager:latest
+          name: hcloud-cloud-controller-manager
+          command:
+            - "/bin/hcloud-cloud-controller-manager"
+            - "--cloud-provider=hcloud"
+            - "--leader-elect=false"
+            - "--allow-untagged-cloud"
+            - "--allocate-node-cidrs=true"
+            - "--cluster-cidr=10.42.0.0/16"
+          resources:
+            requests:
+              cpu: 100m
+              memory: 50Mi
+            limits:
+              cpu: 500m
+              memory: 500Mi
+          env:
+            - name: NODE_NAME
+              valueFrom:
+                fieldRef:
+                  fieldPath: spec.nodeName
+            - name: HCLOUD_TOKEN
+              valueFrom:
+                secretKeyRef:
+                  name: hcloud
+                  key: token
+            - name: HCLOUD_NETWORK
+              valueFrom:
+                secretKeyRef:
+                  name: hcloud
+                  key: network

manifests/helm/cilium/values.yaml

@@ -0,0 +1,126 @@
+upgradeCompatibility: '1.10'
+
+debug:
+  # -- Enable debug logging
+  enabled: false
+  # verbose:
+
+# gke:
+  # enabled: true
+
+ipam:
+  # -- Configure IP Address Management mode.
+  # ref: https://docs.cilium.io/en/stable/concepts/networking/ipam/
+  mode: kubernetes
+
+# -- Configure the encapsulation configuration for communication between nodes.
+# Possible values:
+#   - disabled (breaks csi-controller)
+#   - vxlan (default)
+#   - geneve
+tunnel: geneve
+
+# -- Specify the IPv4 CIDR for native routing (ie to avoid IP masquerade for).
+# This value corresponds to the configured cluster-cidr.
+nativeRoutingCIDR: 10.0.0.0/8
+
+# When enabled, causes legacy routing
+# endpointRoutes:
+  # -- Enable use of per endpoint routes instead of routing via
+  # the cilium_host interface.
+  # enabled: false
+
+# -- Enable installation of PodCIDR routes between worker
+# nodes if worker nodes share a common L2 network segment.
+autoDirectNodeRoutes: false
+
+bpf:
+  # -- Allow cluster external access to ClusterIP services.
+  lbExternalClusterIP: false
+
+  # -- Enable native IP masquerade support in eBPF
+  masquerade: true
+
+endpointHealthChecking:
+  # -- Enable connectivity health checking between virtual endpoints.
+  enabled: true
+
+# -- Configure ClusterIP service handling in the host namespace (the node).
+hostServices:
+  # -- Enable host reachable services.
+  enabled: true
+
+  # -- Supported list of protocols to apply ClusterIP translation to.
+  protocols: tcp,udp
+
+externalIPs:
+  # -- Enable ExternalIPs service support.
+  enabled: true
+
+hostPort:
+  # -- Enable hostPort service support.
+  enabled: true
+
+# -- Configure N-S k8s service loadbalancing
+nodePort:
+  # -- Enable the Cilium NodePort service implementation.
+  enabled: true
+
+# -- Enable connectivity health checking.
+healthChecking: true
+
+ipv4:
+  # -- Enable IPv4 support.
+  enabled: true
+
+ipv6:
+  # -- Enable IPv6 support.
+  enabled: false
+
+# -- Configure Kubernetes specific configuration
+k8s:
+  # -- requireIPv4PodCIDR enables waiting for Kubernetes to provide the PodCIDR
+  # range via the Kubernetes node resource
+  requireIPv4PodCIDR: true
+
+# -- Configure the kube-proxy replacement in Cilium BPF datapath
+# Valid options are "disabled", "probe", "partial", "strict".
+# ref: https://docs.cilium.io/en/stable/gettingstarted/kubeproxy-free/
+kubeProxyReplacement: strict
+
+# -- Enables masquerading of IPv4 traffic leaving the node from endpoints.
+enableIPv4Masquerade: true
+
+monitor:
+  # -- Enable the cilium-monitor sidecar.
+  enabled: false
+
+# -- Configure service load balancing
+loadBalancer:
+  # -- standalone enables the standalone L4LB which does not connect to
+  # kube-apiserver.
+  # standalone: false
+
+  # -- algorithm is the name of the load balancing algorithm for backend
+  # selection e.g. random or maglev
+  # algorithm: random
+
+  # -- mode is the operation mode of load balancing for remote backends
+  # e.g. snat, dsr, hybrid
+  mode: snat
+
+  # -- acceleration is the option to accelerate service handling via XDP
+  # e.g. native, disabled
+  # Gives "Error: virtio_net: Too few free TX rings available."
+  # acceleration: native
+
+# Breaks csi
+# devices: eth1
+
+# -- The agent can be put into one of the three policy enforcement modes:
+# default, always and never.
+# ref: https://docs.cilium.io/en/stable/policy/intro/#policy-enforcement-modes
+policyEnforcementMode: never
+
+# -- Enables the enforcement of host policies in the eBPF datapath.
+hostFirewall: false

manifests/helm/nginx/values.yaml

@@ -0,0 +1,8 @@
+controller:
+  kind: DaemonSet
+  service: 
+    annotations:
+      load-balancer.hetzner.cloud/location: "fsn1"
+      load-balancer.hetzner.cloud/use-private-ip: "true"
+      load-balancer.hetzner.cloud/type: "lb11"
+    type: LoadBalancer

manifests/upgrade/plan.yaml

@@ -0,0 +1,50 @@
+# Doc: https://rancher.com/docs/k3s/latest/en/upgrades/automated/
+# agent plan
+apiVersion: upgrade.cattle.io/v1
+kind: Plan
+metadata:
+  name: k3s-agent
+  namespace: system-upgrade
+  labels:
+    k3s_upgrade: agent
+spec:
+  concurrency: 1
+  channel: https://update.k3s.io/v1-release/channels/stable
+  nodeSelector:
+    matchExpressions:
+      - {key: k3s_upgrade, operator: Exists}
+      - {key: k3s_upgrade, operator: NotIn, values: ["disabled", "false"]}
+      - {key: node-role.kubernetes.io/master, operator: NotIn, values: ["true"]}
+  serviceAccountName: system-upgrade
+  prepare:
+    image: rancher/k3s-upgrade
+    args: ["prepare", "k3s-server"]
+  drain:
+    force: true
+    skipWaitForDeleteTimeout: 60
+  upgrade:
+    image: rancher/k3s-upgrade
+---
+# server plan
+apiVersion: upgrade.cattle.io/v1
+kind: Plan
+metadata:
+  name: k3s-server
+  namespace: system-upgrade
+  labels:
+    k3s_upgrade: server
+spec:
+  concurrency: 1
+  channel: https://update.k3s.io/v1-release/channels/stable
+  nodeSelector:
+    matchExpressions:
+      - {key: k3s_upgrade, operator: Exists}
+      - {key: k3s_upgrade, operator: NotIn, values: ["disabled", "false"]}
+      - {key: node-role.kubernetes.io/master, operator: In, values: ["true"]}
+  tolerations:
+    - {key: node-role.kubernetes.io/master, effect: NoSchedule, operator: Exists}
+    - {key: CriticalAddonsOnly, effect: NoExecute, operator: Exists}
+  serviceAccountName: system-upgrade
+  cordon: true
+  upgrade:
+    image: rancher/k3s-upgrade

master.tf

@@ -0,0 +1,71 @@
+resource "hcloud_server" "first_control_plane" {
+  name = "k3s-control-plane-0"
+
+  image       = data.hcloud_image.linux.name
+  server_type = local.control_plane_server_type
+  location    = local.server_locations[0][1]
+
+  ssh_keys = [hcloud_ssh_key.default.id]
+
+  labels = {
+    provisioner = "terraform",
+    engine      = "k3s",
+    node_type   = "control-plane"
+    k3s_upgrade = "true"
+  }
+
+  user_data = data.template_cloudinit_config.init_cfg.rendered
+
+  provisioner "remote-exec" {
+    inline = var.initial_commands
+
+    connection {
+      user        = "root"
+      private_key = file(var.private_key)
+      host        = self.ipv4_address
+    }
+  }
+
+  provisioner "remote-exec" {
+    inline = [
+      "curl -sfL https://get.k3s.io | K3S_TOKEN=${random_password.k3s_cluster_secret.result} sh -s - server --cluster-init ${var.k3s_extra_args}",
+      "until systemctl is-active --quiet k3s.service; do sleep 1; done",
+      "until kubectl get node ${self.name}; do sleep 1; done",
+      "kubectl -n kube-system create secret generic hcloud --from-literal=token=${var.hcloud_token} --from-literal=network=${hcloud_network.k3s.name}",
+      "kubectl apply -f -<<EOF\n${data.template_file.ccm_manifest.rendered}\nEOF",
+      "kubectl -n kube-system create secret generic hcloud-csi --from-literal=token=${var.hcloud_token}",
+      "kubectl apply -f https://raw.githubusercontent.com/hetznercloud/csi-driver/master/deploy/kubernetes/hcloud-csi.yml",
+      "kubectl apply -f https://raw.githubusercontent.com/rancher/system-upgrade-controller/master/manifests/system-upgrade-controller.yaml",
+      "kubectl apply -f -<<EOF\n${data.template_file.upgrade_plan.rendered}\nEOF",
+      "latest=$(curl -s https://api.github.com/repos/weaveworks/kured/releases | jq -r .[0].tag_name)",
+      "kubectl apply -f https://github.com/weaveworks/kured/releases/download/$latest/kured-$latest-dockerhub.yaml"
+    ]
+
+    connection {
+      user        = "root"
+      private_key = file(var.private_key)
+      host        = self.ipv4_address
+    }
+  }
+
+  provisioner "local-exec" {
+    command = "scp -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -i ${var.private_key} root@${self.ipv4_address}:/etc/rancher/k3s/k3s.yaml ./kubeconfig.yaml"
+  }
+
+  provisioner "local-exec" {
+    command = "sed -i -e 's/127.0.0.1/${self.ipv4_address}/g' ./kubeconfig.yaml"
+  }
+
+  provisioner "local-exec" {
+    command = "helm install --values=manifests/helm/cilium/values.yaml cilium cilium/cilium -n kube-system"
+  }
+
+  network {
+    network_id = hcloud_network.k3s.id
+    ip         = local.first_control_plane_ip
+  }
+
+  depends_on = [
+    hcloud_network_subnet.k3s
+  ]
+}

output.tf

@@ -0,0 +1,9 @@
+output "controlplanes_public_ip" {
+  value       = concat([hcloud_server.first_control_plane.ipv4_address], hcloud_server.control_planes.*.ipv4_address)
+  description = "The public IP addresses of the controlplane server."
+}
+
+output "agents_public_ip" {
+  value       = hcloud_server.agents.*.ipv4_address
+  description = "The public IP addresses of the agent server."
+}

servers.tf

@@ -0,0 +1,51 @@
+resource "hcloud_server" "control_planes" {
+  count = var.servers_num - 1
+  name  = "k3s-control-plane-${count.index + 1}"
+
+  image       = data.hcloud_image.linux.name
+  server_type = local.control_plane_server_type
+  location    = local.server_locations[count.index + 1][1]
+
+  ssh_keys = [hcloud_ssh_key.default.id]
+
+  labels = {
+    provisioner = "terraform",
+    engine      = "k3s",
+    node_type   = "control-plane"
+    k3s_upgrade = "true"
+  }
+
+  user_data = data.template_cloudinit_config.init_cfg.rendered
+
+  provisioner "remote-exec" {
+    inline = var.initial_commands
+
+    connection {
+      user        = "root"
+      private_key = file(var.private_key)
+      host        = self.ipv4_address
+    }
+  }
+
+  provisioner "remote-exec" {
+    inline = [
+      "curl -sfL https://get.k3s.io | K3S_TOKEN=${random_password.k3s_cluster_secret.result} sh -s - server --server https://${local.first_control_plane_ip}:6443 ${var.k3s_extra_args}"
+    ]
+
+    connection {
+      user        = "root"
+      private_key = file(var.private_key)
+      host        = self.ipv4_address
+    }
+  }
+
+  network {
+    network_id = hcloud_network.k3s.id
+    ip         = cidrhost(hcloud_network.k3s.ip_range, 3 + count.index)
+  }
+
+  depends_on = [
+    hcloud_server.first_control_plane,
+    hcloud_network_subnet.k3s
+  ]
+}

terraform.tfvars.example

@@ -0,0 +1,3 @@
+hcloud_token = "mL2oj8Xaxc0AC0uo5LM5NnBlqW8UFjPclc9ZoDIhcKc7ejKgNsjDT6EuObcf5Ia7I"
+public_key   = "/home/myuser/.ssh/id_ed25519.pub"
+private_key  = "/home/myuser/.ssh/id_ed25519"

variables.tf

@@ -0,0 +1,51 @@
+variable "hcloud_token" {
+  description = "Hetzner API tokey"
+  type        = string
+}
+
+provider "hcloud" {
+  token = var.hcloud_token
+}
+
+variable "public_key" {
+  description = "SSH public Key."
+  type        = string
+}
+
+variable "private_key" {
+  description = "SSH private Key."
+  type        = string
+}
+
+variable "servers_num" {
+  description = "Number of control plane nodes."
+  default     = 2
+}
+
+variable "agents_num" {
+  description = "Number of agent nodes."
+  default     = 2
+}
+
+variable "server_location" {
+  description = "Default server location"
+  default     = "fsn1"
+}
+
+variable "k3s_extra_args" {
+  description = "Important flags to make our setup work"
+  default     = "--disable-cloud-controller --disable-network-policy --no-deploy=traefik --no-deploy=servicelb --disable local-storage --disable traefik --disable servicelb --kubelet-arg='cloud-provider=external' --no-flannel"
+}
+
+variable "initial_commands" {
+  description = "Initial commands to run on each machines."
+  default = [
+    "dnf upgrade -y",
+    "dnf install -y container-selinux selinux-policy-base fail2ban k3s-selinux dnf-automatic jq",
+    "systemctl enable --now fail2ban",
+    "systemctl enable --now dnf-automatic.timer",
+    "systemctl disable firewalld",
+    "grubby --args='systemd.unified_cgroup_hierarchy=0' --update-kernel=ALL",
+    "sleep 10; shutdown -r +0"
+  ]
+}

versions.tf

@@ -0,0 +1,7 @@
+terraform {
+  required_providers {
+    hcloud = {
+      source  = "hetznercloud/hcloud"
+    }
+  }
+}