Kubernetes environment with Vagrant and Ansible

Setting up local kubernetes development environment.

Vagrant is a tool for building and managing virtual machines, just like Docker is use to build and manage containers. We going to use it because it allow us to build/reproduce ready to use single disposable, consistent development environment. More info at https://www.vagrantup.com/intro/

Ansible is automation tool that gonna help us to preconfigure new virtual machine created by vagrant.

Pre installations:

  • Ansible (2.8.5)
  • Vagrant (2.2.5)
  • VirtualBox (5.2.32) – its used as vagrant virtualization provider

Now we going to create simple Vargrantfile witch will describe our virtual machine configuration.

Vagrant.configure("2") do |config|

# use ubuntu/xenial64 as base operating system
   config.vm.box = "ubuntu/xenial64"

# set vm hostname to kube-dev
   config.vm.hostname = "kube-dev"

# communicate guest with host on ip 
   config.vm.network "private_network", ip: ""

# synchronize host /home/dev directory to /home/dev dir on guest
# with virtualbox 
   config.vm.synced_folder "/home/dev", "/home/dev", type: "virtualbox"

# use virtualbox as a virtualization provider and
# configure vm with 3 cpu, 6GB RAM and some other custom options
   config.vm.provider "virtualbox" do |vb|
     vb.gui = true
     vb.name = "kube-dev"
     vb.cpus = 3
     vb.memory = "6144"
     vb.customize ["modifyvm", :id, "--clipboard", "bidirectional"]

# in this part we are saying to vagrant to use ansible as 
# configuration provisioner with some extra options.
# playbook - location of ansible playbook
# extra_vars - pass some variables to playbook
   config.vm.provision "ansible" do |ansible|
        ansible.playbook = "./ansible/playbook.yml"    
        ansible.extra_vars = {
            node_ip: "",


Ansible playbook its a file in yaml fromat describing ordered tasks to run in the new vm. Task are based on modules that are translated to shell commands by ansible engine. List of all playbook modules can be found here: https://docs.ansible.com/ansible/latest/modules/list_of_all_modules.html

- hosts: all
  become: true 
    - name: Install packages that allow apt to be used over HTTPS
        name: "{{ packages }}"
        state: present
        update_cache: yes
          - apt-transport-https
          - ca-certificates
          - curl
          - gnupg-agent
          - software-properties-common
          - bash-completion
    - name: Add an apt signing key for Docker
        url: https://download.docker.com/linux/ubuntu/gpg
        state: present
    - name: Add apt repository for stable version
        repo: deb [arch=amd64] https://download.docker.com/linux/ubuntu xenial stable
        state: present
        update_cache: yes
    - name: Install docker and its dependecies
        name: "{{ packages }}"
        state: present
        update_cache: yes
          - docker-ce=5:18.09.9~3-0~ubuntu-xenial
          - docker-ce-cli=5:18.09.9~3-0~ubuntu-xenial
          - containerd.io
          - python3-docker
        - docker status
    - name: Add vagrant user to docker group
        name: vagrant
        group: docker
        - docker status   
    - name: DockerRegistery Container
        name: registry
        hostname: registry
        image: registry:2
        state: started
          - 5000:5000
          - local-registry:/var/lib/registry
    - name: Remove swapfile from /etc/fstab
        name: "{{ item }}"
        fstype: swap
        state: absent
        - swap
        - none
    - name: Disable swap
      command: swapoff -a
      when: ansible_swaptotal_mb > 0
    - name: Add an apt signing key for Kubernetes
        url: https://packages.cloud.google.com/apt/doc/apt-key.gpg
        state: present
    - name: Adding apt repository for Kubernetes
        repo: deb https://apt.kubernetes.io/ kubernetes-xenial main
        state: present
        filename: kubernetes.list
    - name: Install Kubernetes binaries
        name: "{{ packages }}"
        state: present
        update_cache: yes
          - kubelet=1.15.4-00
          - kubeadm=1.15.4-00
          - kubectl=1.15.4-00
    - name: Configure node ip
        create: true
        path: /etc/default/kubelet
        line: KUBELET_EXTRA_ARGS=--node-ip={{ node_ip }}
    - name: Restart kubelet
        name: kubelet
        daemon_reload: yes
        state: restarted
    - name: Initialize kubernetes
      command: kubeadm init --ignore-preflight-errors=all --apiserver-advertise-address={{ node_ip }} --apiserver-cert-extra-sans={{ node_ip }}
    - name: Setup kubeconfig for vagrant user
      command: "{{ item }}"
        - mkdir -p /home/vagrant/.kube
        - cp -i /etc/kubernetes/admin.conf /home/vagrant/.kube/config
        - chown vagrant:vagrant /home/vagrant/.kube/config
    - name: Adding bash completion for kube commands
      become_user: vagrant
      shell: "{{ item }}"
        - echo "source <(kubectl completion bash)" >> ~/.bashrc
        - echo "source <(kubeadm completion bash)" >> ~/.bashrc
    - name: Taining master node
      become_user: vagrant
      command: kubectl taint nodes --all node-role.kubernetes.io/master-
    - name: Labeling node
      become_user: vagrant
      command: "{{ item }}"
        - kubectl label node kube-dev env=development
        - kubectl label node kube-dev role=ingress-controller
    - name: Instaling weavenet network addon
      become_user: vagrant
      shell: kubectl apply -f "https://cloud.weave.works/k8s/net?k8s-version=$(kubectl version | base64 | tr -d '\n')"
    - name: Instaling ingress controller
      become_user: vagrant
      command: kubectl apply -f https://raw.githubusercontent.com/jcmoraisjr/haproxy-ingress/master/docs/haproxy-ingress.yaml
    - name: docker status
      service: name=docker state=started

I will not go in details through this playbook because its standard ansible stuff. It just install configure and run all the required packages to have working kubernetes installation for a developer. But there are some things that I wanted to talk abut:

  • python3-docker package need to be installed because ansible runs in guest os and we need docker module which is not installed by default
  • sometimes I run shell module because command module don’t have access to environment variables and stream operators required for example to install weavnet addon
  • if u declare become true on top level You need to remember to switch users when needed
  • we need local docker repository to be able push local images to kubernetes on vm

Post install:

  • install on host same version of docker engine
  • install on host same version of kubectl
  • copy from guest os /home/vagrant/.kube/config to host ~/.kube/config so u can manage kubernetes without login to vm
  • add on host this entry kube-dev to /etc/hosts

When You need to use in ur kubernetes environment local image just tag it kube-dev:5000/image_name and push it

Of course You can add much more stuff, like bring up container with database engine, generate private key to sign jwt, login to private docker registry etc.

When You need to connect from inside kubernetes cluster to local instance of application because for example You are working on some bug, just swap Your service with endpoint like this:


apiVersion: v1
kind: Service
    name: my_service_name
        - name: svc
          protocol: TCP
          port: 8080
          targetPort: 8080
apiVersion: v1
kind: Endpoints
    name: my_service_name
    - addresses:
        - ip:
        - name: svc
          port: 8080

Then run app from IDE on localhost 8080. To switch back use ur regular service configuration.

To bring all of that to life just run vagrant up 🙂

Dodaj komentarz