Kubernetes environment with Vagrant and Ansible

Setting up local kubernetes development environment.

Vagrant is a tool for building and managing virtual machines, just like Docker is use to build and manage containers. We going to use it because it allow us to build/reproduce ready to use single disposable, consistent development environment. More info at https://www.vagrantup.com/intro/

Ansible is automation tool that gonna help us to preconfigure new virtual machine created by vagrant.

Pre installations:

  • Ansible (2.8.5)
  • Vagrant (2.2.5)
  • VirtualBox (5.2.32) – its used as vagrant virtualization provider

Now we going to create simple Vargrantfile witch will describe our virtual machine configuration.

Vagrant.configure("2") do |config|

# use ubuntu/xenial64 as base operating system
   config.vm.box = "ubuntu/xenial64"

# set vm hostname to kube-dev
   config.vm.hostname = "kube-dev"

# communicate guest with host on ip 10.10.100.10 
   config.vm.network "private_network", ip: "10.10.100.10"

# synchronize host /home/dev directory to /home/dev dir on guest
# with virtualbox 
   config.vm.synced_folder "/home/dev", "/home/dev", type: "virtualbox"

# use virtualbox as a virtualization provider and
# configure vm with 3 cpu, 6GB RAM and some other custom options
   config.vm.provider "virtualbox" do |vb|
     vb.gui = true
     vb.name = "kube-dev"
     vb.cpus = 3
     vb.memory = "6144"
     vb.customize ["modifyvm", :id, "--clipboard", "bidirectional"]
   end

# in this part we are saying to vagrant to use ansible as 
# configuration provisioner with some extra options.
# playbook - location of ansible playbook
# extra_vars - pass some variables to playbook
   config.vm.provision "ansible" do |ansible|
        ansible.playbook = "./ansible/playbook.yml"    
        ansible.extra_vars = {
            node_ip: "10.10.100.10",
        }       
   end

end

Ansible playbook its a file in yaml fromat describing ordered tasks to run in the new vm. Task are based on modules that are translated to shell commands by ansible engine. List of all playbook modules can be found here: https://docs.ansible.com/ansible/latest/modules/list_of_all_modules.html

- hosts: all
  become: true 
  tasks:   
    - name: Install packages that allow apt to be used over HTTPS
      apt:
        name: "{{ packages }}"
        state: present
        update_cache: yes
      vars:
        packages:
          - apt-transport-https
          - ca-certificates
          - curl
          - gnupg-agent
          - software-properties-common
          - bash-completion
    - name: Add an apt signing key for Docker
      apt_key:
        url: https://download.docker.com/linux/ubuntu/gpg
        state: present
    - name: Add apt repository for stable version
      apt_repository:
        repo: deb [arch=amd64] https://download.docker.com/linux/ubuntu xenial stable
        state: present
        update_cache: yes
    - name: Install docker and its dependecies
      apt:
        name: "{{ packages }}"
        state: present
        update_cache: yes
      vars:
        packages:
          - docker-ce=5:18.09.9~3-0~ubuntu-xenial
          - docker-ce-cli=5:18.09.9~3-0~ubuntu-xenial
          - containerd.io
          - python3-docker
      notify:
        - docker status
    - name: Add vagrant user to docker group
      user:
        name: vagrant
        group: docker
      notify:
        - docker status   
    - name: DockerRegistery Container
      docker_container:
        name: registry
        hostname: registry
        image: registry:2
        state: started
        published_ports:
          - 5000:5000
        volumes:
          - local-registry:/var/lib/registry
    - name: Remove swapfile from /etc/fstab
      mount:
        name: "{{ item }}"
        fstype: swap
        state: absent
      with_items:
        - swap
        - none
    - name: Disable swap
      command: swapoff -a
      when: ansible_swaptotal_mb > 0
    - name: Add an apt signing key for Kubernetes
      apt_key:
        url: https://packages.cloud.google.com/apt/doc/apt-key.gpg
        state: present
    - name: Adding apt repository for Kubernetes
      apt_repository:
        repo: deb https://apt.kubernetes.io/ kubernetes-xenial main
        state: present
        filename: kubernetes.list
    - name: Install Kubernetes binaries
      apt:
        name: "{{ packages }}"
        state: present
        update_cache: yes
      vars:
        packages:
          - kubelet=1.15.4-00
          - kubeadm=1.15.4-00
          - kubectl=1.15.4-00
    - name: Configure node ip
      lineinfile:
        create: true
        path: /etc/default/kubelet
        line: KUBELET_EXTRA_ARGS=--node-ip={{ node_ip }}
    - name: Restart kubelet
      service:
        name: kubelet
        daemon_reload: yes
        state: restarted
    - name: Initialize kubernetes
      command: kubeadm init --ignore-preflight-errors=all --apiserver-advertise-address={{ node_ip }} --apiserver-cert-extra-sans={{ node_ip }}
    - name: Setup kubeconfig for vagrant user
      command: "{{ item }}"
      with_items:
        - mkdir -p /home/vagrant/.kube
        - cp -i /etc/kubernetes/admin.conf /home/vagrant/.kube/config
        - chown vagrant:vagrant /home/vagrant/.kube/config
    - name: Adding bash completion for kube commands
      become_user: vagrant
      shell: "{{ item }}"
      with_items:
        - echo "source <(kubectl completion bash)" >> ~/.bashrc
        - echo "source <(kubeadm completion bash)" >> ~/.bashrc
    - name: Taining master node
      become_user: vagrant
      command: kubectl taint nodes --all node-role.kubernetes.io/master-
    - name: Labeling node
      become_user: vagrant
      command: "{{ item }}"
      with_items:
        - kubectl label node kube-dev env=development
        - kubectl label node kube-dev role=ingress-controller
    - name: Instaling weavenet network addon
      become_user: vagrant
      shell: kubectl apply -f "https://cloud.weave.works/k8s/net?k8s-version=$(kubectl version | base64 | tr -d '\n')"
    - name: Instaling ingress controller
      become_user: vagrant
      command: kubectl apply -f https://raw.githubusercontent.com/jcmoraisjr/haproxy-ingress/master/docs/haproxy-ingress.yaml
  handlers:
    - name: docker status
      service: name=docker state=started

I will not go in details through this playbook because its standard ansible stuff. It just install configure and run all the required packages to have working kubernetes installation for a developer. But there are some things that I wanted to talk abut:

  • python3-docker package need to be installed because ansible runs in guest os and we need docker module which is not installed by default
  • sometimes I run shell module because command module don’t have access to environment variables and stream operators required for example to install weavnet addon
  • if u declare become true on top level You need to remember to switch users when needed
  • we need local docker repository to be able push local images to kubernetes on vm

Post install:

  • install on host same version of docker engine
  • install on host same version of kubectl
  • copy from guest os /home/vagrant/.kube/config to host ~/.kube/config so u can manage kubernetes without login to vm
  • add on host this entry 10.10.100.10 kube-dev to /etc/hosts

When You need to use in ur kubernetes environment local image just tag it kube-dev:5000/image_name and push it

Of course You can add much more stuff, like bring up container with database engine, generate private key to sign jwt, login to private docker registry etc.

When You need to connect from inside kubernetes cluster to local instance of application because for example You are working on some bug, just swap Your service with endpoint like this:

---

apiVersion: v1
kind: Service
metadata:
    name: my_service_name
spec:
    ports:
        - name: svc
          protocol: TCP
          port: 8080
          targetPort: 8080
---
apiVersion: v1
kind: Endpoints
metadata:
    name: my_service_name
subsets:
    - addresses:
        - ip: 10.10.100.1
      ports:
        - name: svc
          port: 8080

Then run app from IDE on localhost 8080. To switch back use ur regular service configuration.

To bring all of that to life just run vagrant up 🙂

Dodaj komentarz