Welcome to My Homelab!

Dotfiles Management with Dotbot and Chezmoi!

Thu, 26 Jun 2025 22:32:59 +0000

Dotfiles Management with Dotbot and Chezmoi!

Summary

Hey everyone! Long time no see!

As a lazy guy myself, always trying to find ways to automate tasks and minimize, as much as possible, manual actions.

One of my struggles, is keeping my different macOS machines synced. And by synced I mean from aliases, macOS system settings and dotfiles to gpg/ssh keys, homebrew packages and development tools.

So you can probably understand the pain of trying to keep the same environment, your commands, your tooling, your git or zsh config, while moving from your personal machine to your work related one.

Dotfiles

One of the most important aspects of working across multiple machines, at least for me, is ensuring that my dotfiles are always accessible. In my case, there are 2 different dotfile categories, the normal ones like aliases, vimrc, zshrc etc, which are the same in each machine and the templated ones, as I like to call them.

By templated I mean dotfiles that should be different for each machine. For example, I don’t want my .netrc to have work related tokens. Also I don’t need a .terraformrc in my personal machine.

Haven’t found any dotfile manager that handles both cases, so I decided to combine two different solutions to be able to keep my systems synced.

Dotbot

Dotbot is an awesome tool that I’m using for the non-templated dotfiles. It’s very easy and straight forward to setup and use.

In my setup, I have a repo where dotbot is initialized and my dotfiles are present. The folder structure looks like this:

├── centos/
│   ├── dotfiles/
├── common/
│   ├── dotfiles/
│   └── vim/
├── dotbot/
│   ├── bin/
│   ├── dotbot/
│   ├── lib/
│   ├── tests/
│   ├── tools/
│   ├── CHANGELOG.md
│   ├── CONTRIBUTING.md
│   ├── LICENSE.md
│   ├── pyproject.toml
│   ├── README.md
│   ├── setup.py
│   └── tox.ini
├── mac/
│   ├── docker/
│   ├── dotfiles/
│   ├── gnupg/
│   ├── launchagents/
│   ├── m2/
│   └── vim_runtime/
├── install*
├── install.config.yaml
└── README.md

As you can see, I have separated files in different folders,

centos folder, linux specific dotfiles in case I want to bootstrap a linux machine
common folder, dotfiles that are relevant for both linux and mac
dobot folder, is created from dotbot installation (check https://github.com/anishathalye/dotbot?tab=readme-ov-file#getting-started)
mac folder, mac specific dotfiles and other config files
install , script that will install the dotfiles
install.config.yaml , config used by the installer to create the relevant symlinks

The install.config.yaml looks like this:

# cleans deadlinks , useful if we want to delete where we can rename the main folders and run dotbot to clean deadlinks
- clean: ['~']

- defaults:
    link:
      relink: true
      force: true

- link:
    ~/:
      glob: true
      path: common/dotfiles/.*
    ~/.vim/templates:
      glob: true
      path: common/vim/*
      create: true

- defaults:
    link:
      if: '[ `uname` = Darwin ]'
      relink: true
      force: true

- link:
    ~/:
      glob: true
      path: mac/dotfiles/.*
    ~/.docker:
      glob: true
      path: mac/docker/*
      create: true
    ~/.gnupg:
      glob: true
      path: mac/gnupg/*
    ~/.vim_runtime:
      glob: true
      path: mac/vim_runtime/*
    ~/Library/LaunchAgents:
      glob: true
      path: mac/launchagents/*
      create: true

- shell:
  - [launchctl unload ~/Library/LaunchAgents/]
  - [launchctl load ~/Library/LaunchAgents/]

- defaults:
    link:
      if: '[ `uname` = Linux ]'
      relink: true
      force: true

- link:
    ~/:
      glob: true
      path: centos/dotfiles/.*
      create: true

- shell:
  - [git submodule update --init --recursive, Installing submodules]

As you can easily see from the config, if executed it will create symbolic links at specified locations that point to files in my dotfiles repository.

There is a separation between operating systems, Linux and Darwin in my case, and some shell commands that are specific to my needs, but you can adapt it accordingly.

So after executing ./install -c install.config.yaml, you should something like that in your home folder :

lrwxr-xr-x     - testuser 26 Jun  2025 .zshr -> /Users/path/to/your/repo/mac/dotfiles/.zshr
lrwxr-xr-x     - testuser 26 Jun  2025 .zshrc -> /Users/path/to/your/repo/mac/dotfiles/.zshrc
lrwxr-xr-x     - testuser 26 Jun  2025 .gitignore_global -> /Users/path/to/your/repo/common/dotfiles/.gitignore_global
lrwxr-xr-x     - testuser 26 Jun  2025 .fzf.zsh -> /Users/path/to/your/repo/mac/dotfiles/.fzf.zsh

This is a rough explanation of how I’m handling non-templated files via Dotbot.

[!TIP] As my repo keeps sensitive data, like docker auth, gitlab tokens etc, I use transcrypt to encrypt such data.

Chezmoi

Chezmoi is another awesome highly customizable tool that manages my dotfiles across multiple different machines, in a secure way.

Chezmoi documentation is very comprehensive, so I won’t spend time explaining the initial setup but you can start here. It supports encryption and also templating which is my favorite part! Have a look on the official official User Guide.

For my setup, I have yet another repo 😆 which is initialized via chezmoi and folder structure looks like this:

├── private_dot_ssh/
│   ├── encrypted_private_executable_config.tmpl.age
│   ├── encrypted_private_executable_id_rsa.tmpl.age
│   └── private_executable_id_rsa.pub.tmpl
├── dot_gitconfig.tmpl
├── dot_vimrc.tmpl
├── encrypted_dot_netrc.tmpl.age
├── encrypted_dot_terraformrc.tmpl.age
├── Makefile
└── README.md

Now lets see how my .terraformrc looks like: ❯ chezmoi edit ~/.terraformrc

credentials "gitlab.com" {
   token = "<REDUCTED>"
 }

With chezmoi’s templating power, a ~/.terraformrc file will only be created if the user of the machine that I’m running chezmoi apply is customuser . Otherwise, no such file will be created.

The above is just an example and you can use the automatically-populated available variables found here,based on your setup.

Automation

Dotbot and Chezmoi managers can be of course configured and used independently of each other as well as standalone.

Since I wanted to automate the process, I created MacOS-Ansible-Bootstraprepository which uses both dotfiles managers to configure the files needed in every machine.

In addition, the repository makes use of Bash and Ansible to configure macOS system settings, import gpg keys, install homebrew packages and plugins with the help of Bitwarden, my favorite Password Manager!

You can check the repo, adapt it accordingly and star it if you like it! 😉

Final Words 💡

Let’s be honest, dotfile managers are awesome but all the heavy work is done from git.

Having my most useful files encrypted and tracked in a repository, significantly saves time both productivity and troubleshooting wise.

These awesome tools then act as wrappers adding advanced features that handle file symlinking automatically, saving me the time and effort of writing custom scripts.

Thanks for reading, until the next post and as Dr Wallace Breen says in Half-Life 2.. Be wise. Be safe. Be aware!

Track your personal tasks with the help of Docker and Kanboard!

Mon, 02 Oct 2023 22:32:59 +0000

Track your personal tasks by using Kanboard and Nginx-Proxy docker containers.

Summary

Hey everyone!

If you are a person that cannot operate unless you write down to a paper or add a reminder or use one of the thousand different tools around the web to track your task, then you landed in the correct place!

I’ve been struggling for years to keep my notes (and this is another project on it’s own) and personal tasks in one place.

Some time ago, I stumbled upon an open source project called Kanboard and I have been using this awesome tool for some months now in order to track different tasks , engineering or daily ones.

So, let’s dive into the agile world!

Background

Kanboard is an Kanban open source project management software! You can see how it looks in Picture 1 below.

Long story short, Kanban is an agile project management tool designed to help us visualize our tasks making it easier to establish an order, create a flow and most importantly understand the amount of work that we are planning to do!

Picture 1

If you are unfamiliar with terms like agile, scrum or kanban don’t be discouraged, you can start here.

Concept

As you may have seen already in one of my latest guides, on creating a MariaDB database host via docker found here , I wanted to have a single database that can host different projects. And one of them was Kanboard, so db wise, we are almost ready! 🚀

While configuring Kanboard though and because other projects are pending in line, it seemed to me that having to access different services from different published ports would slow me down and add overhead trying to check or remember which port is connected with which service.

So I thought that it would be better in the long run to have an nginx proxy to handle the requests and redirect them to appropriate services based on different ports.
In that way I could also use my local SSL Certificate Authority and enable SSL termination instead of plain http.

You can check this great post if you have doubts on why it’s a good idea to use a reverse proxy infront of your containers!

Configuration

Following my networking setup decision described in my previous article, I decided to assign the two new images in my custom network called customnetwork.

nginx-proxy

Now before running the command that will create our nginx-proxy container, need to prepare a bit the folders and files that nginx will need in order to redirect traffic to requested service based on the port.

Below folder paths/names can be different and will be created on our host machine where docker is gonna be running:

mkdir -p /docker_mounts/nginx-proxy/{html,conf.d,ssl,logs}

For now I will mainly focus to folders that are necessary in this setup and especially to the conf.d and ssl. The other two ones are optional and good to have but it depends on your need and setup.

/docker_mounts/nginx-proxy/conf.d : this folder will hold the custom configuration that is needed for the traffic redirection.
/docker_mounts/nginx-proxy/ssl : this folder will hold our custom ssl private authority files (have a look in this article to better understand or create your own local ssl private authority)

nginx-proxy custom.conf

Let’s copy our wildcard.homelab.home.* private authority files to the newly ssl created folder:

cp /tmp/wildcard.homelab.home.* /docker_mounts/nginx-proxy/ssl

For our custom config, copy/paste the below block to /docker_mounts/nginx-proxy/conf.d/custom.conf

server {
        listen 80;
        return 301 https://$host$request_uri;
}

server {
        listen 443 ssl;
        server_name nginx-proxy.homelab.home;
        root /usr/share/nginx/html;

        ssl_certificate /etc/nginx/ssl/wildcard.homelab.home.crt;
        ssl_certificate_key /etc/nginx/ssl/wildcard.homelab.home.key;

        access_log      /var/log/nginx/nginx-proxy.homelab.home_ssl_access.log;
        error_log       /var/log/nginx/nginx-proxy.homelab.home_ssl_error.log;

        ssl_session_cache  builtin:1000  shared:SSL:50m;
        ssl_protocols  TLSv1 TLSv1.1 TLSv1.2;
        ssl_ciphers HIGH:!aNULL:!eNULL:!EXPORT:!CAMELLIA:!DES:!MD5:!PSK:!RC4;
        ssl_prefer_server_ciphers on;
}

server {
        listen 443 ssl;
        server_name kanboard.homelab.home;

        ssl_certificate /etc/nginx/ssl/wildcard.homelab.home.crt;
        ssl_certificate_key /etc/nginx/ssl/wildcard.homelab.home.key;

        access_log      /var/log/nginx/kanboard.homelab.home_ssl_access.log;
        error_log       /var/log/nginx/kanboard.homelab.home_ssl_error.log;

        ssl_session_cache  builtin:1000  shared:SSL:50m;
        ssl_protocols  TLSv1 TLSv1.1 TLSv1.2;
        ssl_ciphers HIGH:!aNULL:!eNULL:!EXPORT:!CAMELLIA:!DES:!MD5:!PSK:!RC4;
        ssl_prefer_server_ciphers on;

        location / {
      	proxy_set_header        Host $host;
      	proxy_set_header        X-Real-IP $remote_addr;
      	proxy_set_header        X-Forwarded-For $proxy_add_x_forwarded_for;
      	proxy_set_header        X-Forwarded-Proto $scheme;
      	proxy_pass              http://172.17.0.1:8081;
        }
}

Some major points to understand in the file above is that

in the first line, we instruct our nginx that needs to redirect any traffic coming to port 80 , to port 443.
in the ssl related lines, I use my custom wildcard.homelab.home.* private authority files which I have already copied to the folder that is going to be mounted between host and container
nginx-proxy.homelab.home server section, is only relevant to make sure that our config proxy is working as expected before moving on to kanboard configuration. Add below lines in /docker_mounts/nginx-proxy/html/index.html

<h1>Hey there!</h1>
<p>Here is my test HTML file</p>

the actual line that tells our nginx where to send the traffic is mainly this one: proxy_pass http://172.17.0.1:8081;
This will instruct nginx, to send traffic that comes to kanboard.homelab.home host, to the the docker default bridge network IP and port 8081 (which for now doesn’t have any service behind it but will have when we get to the Kanboard section! 😉

Now, you may ask yourself why not use localhost instead of 172.17.0.1 (hint: run docker network inspect bridge to find the bridge IP) and this is due to the way networking is implemented in Docker Desktop for Mac (yeap, I know.. I know.. ), and the lack of a docker0 interface on the host. In a linux environment, I would guess that this won’t be a problem and localhost will do the trick!

Last but not least, append these lines to your /etc/hosts file in order to be able to resolv the hosts locally.

127.0.0.1	nginx-proxy.homelab.home
127.0.0.1	kanboard.homelab.home

We finally have everything ready for our nginx-container!

nginx-proxy docker run

Putting things together, the below command will start an nginx-proxy container that will listen to port 80 and 443.
It will also mount the already created host folders to the container, providing ngxinx with the relevant files needed for our configuration.

docker run --name nginx-proxy \
--hostname nginx-proxy.homelan.home \
--dns-search=homelab.home. \
-p 80:80 \
-p 443:443 \
-v /var/run/docker.sock:/tmp/docker.sock:ro \
-v /docker_mounts/nginx-proxy/html:/usr/share/nginx/html \
-v /docker_mounts/nginx-proxy/nginx.conf:/etc/nginx/nginx.conf \
-v /docker_mounts/nginx-proxy/conf.d:/etc/nginx/conf.d \
-v /docker_mounts/nginx-proxy/ssl:/etc/nginx/ssl \
-v /docker_mounts/nginx-proxy/logs:/var/log/nginx \
-itd --restart always --network customnetwork \
jwilder/nginx-proxy

If everything worked as expected, a new container should be running and the our test host nginx-proxy.homelab.home should be accessible and secure.

If you want to do any changes to the custom.conf, for example adding a new server section for a new service, then edit the file locally and then run the below commands that will check the config for errors and reload nginx service.

docker exec nginx-proxy nginx -t
docker exec nginx-proxy nginx -s reload

Troubleshooting

In case the above configuration notes didn’t work for you, then some troubleshooting steps to pin point the issue would be:

If nginx-proxy container exited with error , you can check the logs for the reason behind it : docker logs <container_name>.
If the container started correctly, then check the logs found in the mounted logs folder /docker_mounts/nginx-proxy/logs and the nginx-proxy.homelab.home_ssl_error.log for errors.
Make sure that host can be resolved and the entries in your hosts file are correct based on your local setup.
If none of the above fixed the problem, then a number of issues can be the root cause.
Go back and make sure that you followed the steps and have replaced correctly your private auth, folder paths and filenames.

kanboard

Now that we have our nginx-proxy container up and running, we need a service running in port 8081 and this is where kanboard comes into play!

Prerequisites

Kanboard needs a database, so we will use our already configured MariaDB host in order to create the database and user that is going to be used in the docker run command.

In order to connect to the MariaDB docker host, you can use either phpmyadmin GUI or CLI commands both of which choices can be found in my previous article.

After you have successfully connected, run below commands that will create a new database, a new user with a password and will give user full access to the specific database. :

CREATE DATABASE kanboard;
CREATE USER '<KANBOARD_USER>'@'localhost' IDENTIFIED BY '<KANBOARD_PASSWD>';
CREATE USER '<KANBOARD_USER>'@'%' IDENTIFIED BY '<KANBOARD_PASSWD>';
GRANT ALL ON kanboard.* TO '<KANBOARD_USER>'@'localhost' WITH GRANT OPTION;
GRANT ALL ON kanboard.* TO '<KANBOARD_USER>'@'%' WITH GRANT OPTION;
FLUSH PRIVILEGES;

Hint: Make sure that you can login with the newly created user and credentials before starting kanboard container, this will save you a lot of troubleshooting time!

It is also a good idea to have our data accessible directly in a host mounted folder so go ahead and create some basic ones (kanboard_ssl isn’t needed in this case as we are now offloading SSL with the help of our ngxinx-proxy container).

mkdir -p /docker_mounts/kanboard/{kanboard_data,kanboard_plugins,kanboard_ssl}

kanboard docker run

Now that we have confirmed that the new database is accessible, let’s run the kanboard docker container.

As you can see below:

we expose port 8081 that will be used by the nginx-proxy to understand where to send traffic when kanboard.homelab.home is requested
we mount the needed folders
we use an environmental varialble in order for kanboard application to be able to connect to the database host, in our case mariadb. Replace variables below with the username and password configured here!

docker run -d \
--name kanboard \
--hostname kanboard.homelab.home \
-p 8081:80 \
--volume /docker_mounts/kanboard/kanboard_data:/var/www/app/data \
--volume /docker_mounts/kanboard/kanboard_plugins:/var/www/app/plugins \
--volume /docker_mounts/kanboard/kanboard_ssl:/etc/nginx/ssl \
-e DATABASE_URL=mysql://<KANBOARD_USER>:<KANBOARD_PASSWD>@mariadb/kanboard \
--restart always --network customnetwork  -t \
kanboard/kanboard

We are ready to run the docker command that will create our kanboard container.

If no errors occur, then our we should have 3 containers successfully running!

e4904b212567   kanboard/kanboard         "/usr/local/bin/entr…"   8 minutes ago   Up 7 minutes   443/tcp, 0.0.0.0:8081->80/tcp                                                                        kanboard
401b545ab0a8   jwilder/nginx-proxy       "/app/docker-entrypo…"   4 minutes ago   Up 3 minutes   0.0.0.0:80->80/tcp, 0.0.0.0:443->443/tcp                                                             nginx-proxy
ceac6732a896   mariadb:latest            "docker-entrypoint.s…"   4 weeks ago   Up 8 hours   0.0.0.0:61115->3306/tcp, 0.0.0.0:61116->4444/tcp, 0.0.0.0:61117->4567/tcp, 0.0.0.0:61118->4568/tcp   mariadb

Open a browser and head to https://kanboard.homelab.home , if all the above configuration is correct then we should be able to see our Kanboard login page!

Hint: The default login and password is admin/admin.

There are many guides and videos on how to get started, so Happy Kanboarding! 🥳

Final Words 💡

In this article we saw not only how to configure and start a kanboard container to create projects and track your tasks but also combined it with an nginx reverse proxy that can be used to server more services and projects which are pending in line!

With the use of nginx, we can now choose a port, add a new server block in the custom.conf, as described here, add a hostname or IP to your hosts file and boom, we can access our service with an FQDN which is much easier to remember especially if you don’t use a service daily.

Yes, there is some overhead updating custom.conf and hosts file, maybe at some point I will have to also create a dns container, but on the other hand as services are increasing, running localhost:<port> isn’t so charming!

Thanks for reading, until the next post and as Dr Wallace Breen says in Half-Life 2..
Be wise. Be safe. Be aware!

Create MariaDB and phpMyAdmin containers locally via Docker

Sat, 02 Sep 2023 09:32:59 +0000

Create MariaDB and phpMyAdmin containers locally via Docker.

Summary

Long time no see, right! 😅

Some time ago, and when I say some time ago I mean for more than 2 years now, I decided to stop creating VM’s in my local oVirt Manager for each service that I wanted to experiment on and started using docker containers. Between us, I tried vagrant for a bit but this wasn’t my cup of tee so I decided to use Docker.

I assume that whatever reason brought you to this article, you already know your way around docker containers but nevertheless I will try to explain as much as possible what this article is about and provide some major key points which are useful not only for the specific project but for docker handling in general.

As with all my posts, the main reason for this one is to keep some personal notes on what I did locally in order to make everything work based on my own need of having a central local db that will host multiple databases from different projects. Of course, as different people are reaching out to answer their questions or ask for support , hopefully this will be also helpful for other people out there.

In case you don’t have enough knowledge around docker and everything sounds like gibberish, please head to the docker starting page which is a great starting point.

Concept

As already described, the initial idea was that I needed a main database server but didn’t want to have a VM running all the time. There are multiple articles and maybe I can also create one at some point on how I configured my own Galera Cluster in Docker but for now we will only focus to a single server.

One database container to rule them all! 😈

Configuration

Networking prerequisite

After some network related reading, some googling and multiple container deletions 😅, I decided to create a specific docker network so as 1) to be able to isolate specific docker containers and 2) because I had in mind that probably I will need an nginx-proxy and having their own network would save me from some networking trouble, but this is going to be part of my next article.

In order to create a new docker network, run:

docker network create customnetwork

If we want to check whether the new network is successfully created or which other local networks are available in our host, run:

docker network ls

4d59c36a3et5   bridge              bridge    local
47ee586abggv   host                host      local
f0a26628c86c   customnetwork       bridge    local

MariaDB Container

Now let’s create the configuration for our custom MariaDB container:

docker run -d \
        --name mariadb \
        --hostname mariadb.homelab.prv \
        --publish "3306" \
        --publish "4444" \
        --publish "4567" \
        --publish "4568" \
        --dns-search=homelab.prv. \
        --env MARIADB_ROOT_PASSWORD="somepassword" \
        --env MARIADB_USER="example-ruser" \
        --env MARIADB_PASSWORD="example-password" \
        --volume /docker_mounts/mariadb/datadir:/var/lib/mysql \
        --volume /docker_mounts/mariadb/conf.d:/etc/mysql/mariadb.conf.d \
        --network customnetwork \
        --restart always \
library/mariadb:latest

Before running the above, let’s see what we are doing in each option:

--name : we override defult container’s name with a custom one,
--hostname : we override the default hostname with our own one,
--publish : we expose specific ports to the host so as to be reachable outside our container,
--dns-search : we use DNS search domain to search non-fully-qualified hostnames,
--env : next comes the environment variables for root password and an custom user,
--volume : we mount mariadb datadir and conf dir to a specific path that we keep our mounted files,
--network : we connect our container to our custom created network,
the last parameter is the image used for our container which is the mariadb latest one.

When we run the above command, docker will check if the image mariadb:latest is present on our host else it will pull the related layers and then create the container with the given option.

If no errors occured, we should see something like

$ docker ps
ceac6320a732   mariadb:latest        "docker-entrypoint.s…"   8 seconds ago   Up 7 seconds   0.0.0.0:56158->3306/tcp, 0.0.0.0:56159->4444/tcp, 0.0.0.0:56156->4567/tcp, 0.0.0.0:56157->4568/tcp   mariadb

We can also follow (-f) and check the container logs with:

docker logs mariadb -f

Now that the container is up, we should be able to login to our newly created MariaDB docker container with multiple ways, either by creating a new maridb container that runs the mariadb command line client against our original mariadb container, allowing us to execute SQL statements against our database instance:

docker run -it --network customnetwork --rm mariadb mariadb -hmariadb -p

or by opening a shell to the existing container which will also allow us to run mariadb commands

docker exec -it mariadb /bin/bash

root@mariadb:/# mariadb -p
Enter password:

Your MariaDB connection id is 6
Server version: 11.0.2-MariaDB-1:11.0.2+maria~ubu2204 mariadb.org binary distribution

Copyright (c) 2000, 2018, Oracle, MariaDB Corporation Ab and others.

Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

MariaDB [(none)]> show databases;

+--------------------+
| Database           |
+--------------------+
| information_schema |
| mysql              |
| performance_schema |
| sys                |
+--------------------+
4 rows in set (0.024 sec)

Great, our new host is successfully created and accessible!

phpMyAdmin Container

Even though cli clients are awesome, software or web based clients are also a great option and oftenly even preferred by technical and non-technical people when it comes to more advanced database tasks.
For that reason and because I wanted to make sure that my docker containers can communicate with each other , I decided to create a phpMyAdmin container and connect it to my existing mariadb host.

phpmyadmin isn’t so compicated to run as you can see from the relevant command below and one of the reasons is because it doesn’t need a specific mount point as it doesn’t actually hold any data or config that you may need to change out of the box.

docker run --restart always --name phpmyadmin --hostname phpmyadmin.homelab.prv --dns-search=homelab.prv. -d -p 8081:80 -e PMA_HOST=mariadb --network customnetwork phpmyadmin

Before running the above command, let’s see again what we are doing in each option apart from the already known ones that we already discussed in MariaDB section:

-p 8081:80 : equivalent to --publish , we expose container’s port 80 to our host 8081 port so as to be able to access the GUI, after the container is up and running. We will see how to do that in the next section.
-e PMA_HOST=mariadb : equivalent to --env , we use PMA_HOST variable to define the address or the host name of our MariaDB server.
If you want to use phpMyAdmin to access multiple databases , then you will need also PMA_ARBITRARY=1 variable.

Let’s check docker status after running the latest command.

$ docker ps
ceac6320a732   mariadb:latest        "docker-entrypoint.s…"   2 hours ago   Up 2 hours   0.0.0.0:56158->3306/tcp, 0.0.0.0:56159->4444/tcp, 0.0.0.0:56156->4567/tcp, 0.0.0.0:56157->4568/tcp   mariadb
24304fe258f2   phpmyadmin                "/docker-entrypoint.…"   5 seconds ago   Up 3 seconds   0.0.0.0:8081->80 tcp   phpmyadmin

Both containers are up and running! 🥳

Access our DB via GUI

Now it’s time to open phpMYAdmin GUI and try to connect to our MariaDB.

Open your favourite browser and type http://localhost:8081 as shown in the image below and you should be able to at least see the phpMyAdmin welcome message!

Final and most important part of the whole setup is to be able to login to our MariaDB host via GUI, as we did via terminal already in a previous step, and be able to see and created new databases and any other related config via GUI.

Use either root user and the MARIADB_ROOT_PASSWORD or credentials of the MARIADB_USER configured here.
If no errors occur, then you should see something like:

Awesome! 🥳

Final Words 💡

You may have spotted already in the last screenshot, that among other databases I have one called kanboard and one called phpipam.

This is because of the next step and the actual reason behind this idea as I want to keep my tasks tracked in a Kanban Board and this will be achieved with this amazing free open source Kanban project management software.

Let me point out that every configuration described in this article, can be packed into a file which can be applied via docker compose but this is not the time to go through it and this is where the other database phpipam comes into play! So stay tuned!

Thanks for reading, until the next post and as Dr Wallace Breen says in Half-Life 2..
Be wise. Be safe. Be aware!

Amateur Stop Motion Short Film - Klouvi

Sun, 20 Nov 2022 20:32:59 +0000

This is how I proposed to my girlfriend!

Welcome to the first post of 2022!! Yes yes I know it’s November.. but it’s never too late, right? 😂

I’ve been trying to write something for months now but kept delaying it for some unknown, to me, reason! So I decided to write about a project that I did back in 2015 which was a stop motion short film that I would then use to propose to my girlfriend!

But before moving on, let me give some background before explaining how I combined my engineering and artistic skills!

Background 📝

Not sure if this was genuinely my idea or if my girlfriend spend time whispering “you are now reaaaaady to propose to meeeeeee” in my ear during sleep but in any case I decided that it was time for me to do the next step after being together for 3 years.

I’ve always been a fan of stop motion films/animations. From Jan Svankmajer and Quay Brothers to Tim Burton and Wes Anderson, from a young age I was fascinated by the idea of giving life to figures. So it was clear to me that creating my own stop motion short film would be a very romantic and unique way to propose!

What I needed now was a set, a scenario, the appropriate music track and the actors!

The Set

Our first kiss took place in my favorite bar called Klouvi which is located in the heart of Athens, Greece in a neighborhood called Petralona! So I needed to recreate the bar which seemed duable and all I needed was some photos, some foam boards for the furniture and some miniatures to equip my fake bar!

The Actors

While being an Erasmus student in Czech Republic back in 2005, I visited Prague where in one of my usual walks I found an awesome little toy store that had these 2 cute finger puppets, an old man and a old woman!

One thing that someone needs to know about me, is that I tend to buy stuff just because I think that will be used at some point in the future. And how correct was I after 10 years, I had my actors!

My future wife and myself, as an old couple going back to the place in which we had our first kiss! The scenario was almost ready!

The Scenario

Everything starts at a bar before it’s opening, we see a broom and a mop that are cleaning the floor, then chairs that start taking their place and glasses and cups are parading on the bar!

When everything is ready, our two protagonists enter the bar and sit on the same place that sat years ago. They order a wine, they kiss and something suddenly grabs their attention..

The Music

For the music track I decided to use a song called “Tin Can Parade” from a great Greek band called “Your Hand In Mine” for two reasons. Firstly because they share the same name with my favorite song from a post-rock band called “Explosions In The Sky” and secondly because they are one of my favorite Greek bands and I really pictured me and my wife holding hands for a lot of years when I heard that particular song!

Everything was ready for me to start building the bar and start shooting! 🎥

Building the Sets 🧰

I started small by creating a couple of chairs to make sure that the scale is correct. Didn’t want to have to deal with any complaints from my protagonists, at least not from the beginning!

Not bad with just one try, right? So I decided to move on by creating some more chairs, tables and the bar related furniture.

As it didn’t make sense to create my own equipment , I decided to order some miniature glasses, water and wine bottles from Ebay and luckily also the scale was spot on!

Next I saw that the original Klouvi bar had an espresso machine but couldn’t find one that was at the same size, scale wise, with the rest of the furniture.. so I created my own!

The main bar was almost ready!

Then I started enriching it with the wine bottles and glasses that I had bought. I also added mops, grooms and a bucket with water needed for the first part of the scenario, the cleaning!

Some final touches with a Wine Cooler with ice cubs made from silicon and straws made from colored paperclips (I had to be creative as production was a bit low on cash 😂 ).

At this point I had what seemed to me like a full set which was pretty close to the actual bar!

Everything was ready and shootings got the green light!

Camera rolling 🎬

As many films, mine faced its own difficulties with the main one being the constant moving! I had to take the sets out of the closet each evening to take some pictures and then hide it again. I was leaving alone and my girlfriend would visit and of course she couldn’t see what I was preparing!

This meant that I had to recreate the scene, re arrange the bottles or glasses and sometimes fix broken furniture! The positive thing was that I at least had the time and space to finish this project as we weren’t still leaving together.

The whole procedure took around 9 months and after ~1500 photos, I had the material to start putting together my first stop motion short film!

Montage 🎞️

Next step was to create a video out of the all the photos that I had gathered. This could be done with various ways but I will document the one that worked for me and keep in mind that I had already been working on this project for 9 months and Im was in the brink of giving up!

The tools that I used in order to join all the photos in one video are the ones below!

dnf install pyrenamer
dnf install ffmpeg

First tool that I installed was pyRenamer which is a powerful application with a graphic interface designed for renaming many files at once. Yes I know what you are thinking, why not using awk, sed and other linux commands but keep in mind that at that point, I was already working on this project for more than 8 months and my time window was closing! Second tool was ffmpeg, which is a complete cross-platform solution to record, convert and stream audio and video.

So pyRenamer did exactly what I wanted quickly and easily. The only thing that someone has to do, is choose the folder that keeps all the relevant photos and add the below pattern in the “Renamed file name pattern” field, check the Preview and hit Rename.

frame{num6}.jpg

This was needed for the next tool to be able to join the photos in the correct order!

Then ffmpeg comes into play to perform it’s magic! In case you have questions on the options used below, you can find some very useful examples here

ffmpeg -i frame%06d.jpg -c:v libx264 -r 30 -pix_fmt yuv420p klouvi_stop_motion.mp4
ffmpeg -i klouvi_stop_motion.mp4 -filter:v "setpts=7.0*PTS" klouvi_stop_motion.mp4
ffmpeg -i klouvi_stop_motion.mp4 -i audio.ogg -c copy -map 0:v -map 1:a klouvi_stop_motion.mkv

As you can see above, there are 3 commands that will create a video out of a bunch of photos.

With the first command we are creating the initial klouvi_stop_motion.mp4 by joining the jpg files. With the second we are slowing down the video speed by changing the presentation timestamp (PTS) of each video frame. And finally with the third command, we are adding our audio.ogg to the finalized version of our short film!

Voila! My first and most important stop motion short film was ready!

Final Words 💡

This was a journey that lasted almost a year, had its ups and downs but the end result was very fulfilling as I got the chance to combine my creativity and my engineering skills in order to create a marriage proposal that was quite unique and totally me!

Probably there were better ways to do the photoshoots and the editing than with a single One Plus X mobile (that I was using at the time), a basic camera stand and some ffmpeg commands but it totally worth the struggle!

Thanks for reading, until the next post and as Dr Wallace Breen says in Half-Life 2..
Be wise. Be safe. Be aware!

PS: Let’s see what the future brings.. but for now I guess you are wondering whether my girlfriend said yes or no, so no need to keep you waiting any longer! Enjoy!

Automatically create virtual machines via Ansible and oVirt API - Part Two

Wed, 17 Nov 2021 20:32:59 +0000

Automatically create virtual machines via Ansible and oVirt API - Part Two.

Welcome to the second part of our oVirt API manipulation via Ansible guide!

I guess you’ve already gone through the first part, played around with Ansible on your own and waited anxiously for the second one (Noooot). Or maybe you’ve already found a way to use Ansible and create your own VM’s automatically after cursing for losing time reading the first part which was useless in your case! :D

But waiting time is over!!

Architecture ✏️

To get a reminder of what our goal is and what we are gonna try to achieve in this project, check below picture

and for a more detailed explanation head to the relevant post.

Playbooks 📚

In the first part we managed, as an example, to get a list of our active VM’s and their snapshot description. Now we will head directly to the interesting part without wasting any more time.

I assume that you have basic understanding on how Ansible works by now, and if not head to the first part to find useful informations and links.

But just as a reminder in order to be able to connect to the remote host you need an inventory. This is an essential part of the initial configuration, otherwise you will end up with below output if you try to run your playbook:

PLAY [10.0.0.11] **********************************************************************************************************************************************************************************************
skipping: no hosts matched

After making sure you have a correctly configured Ansible environment, we are ready to proceed.

Create our first VM via Ansible

Open your favourite terminal and paste the command below so as to create the playbook ovirt_create_vm.yml which will be used to connect to oVirt and automatically instruct oVirt to create our next VM for us !

:~$ cat <<EOF > ovirt_create_vm.yml
- hosts: ovirt.homelab.home
  connection: local

  tasks:
  - name: Obtain SSO token
    ovirt_auth:
      url: https://ovirt.homelab.home/ovirt-engine/api
      username: admin@internal
      password: "<add your password here>"
      
  - name: Create VM with cloud-init
    ovirt_vms:
      name: apivm
      template: "<add your template here>"
      cluster: "<add your cluster name here>"
      memory: 1GiB
      auth: "{{ ovirt_auth }}"
      state: running
      cloud_init:
        nic_boot_protocol: static
        nic_ip_address: 10.0.0.11
        nic_netmask: 255.255.255.0
        nic_gateway: 10.0.0.254
        nic_name: eth0
        nic_on_boot: true
        host_name: apivm
        custom_script: |
          write_files:
           - content: |
               Hello, world!
             path: /tmp/greeting.txt
             permissions: '0644'
          runcmd:
            - yum -y update
        user_name: root
        root_password: apivm

  - name: Always revoke the SSO token
    ovirt_auth:
      state: absent
      ovirt_auth: "{{ ovirt_auth }}"
EOF

Easy right? I think it’s much more straight forward than the list VM’s playbook that we used in the part one .We are going to breakdown and explain each task as good as possible.

Playbook breakdown

The first section is the host which tells our playbook which is the target host (or group of hosts in other cases) that the playbook is going to be executed on. In our case the oVirt host of our infrastructure ,ovirt.homelab.home, which provides the REST-API.
Next we have the tasks section and the first play where we are using the ovirt_auth module to authenticate to oVirt engine and create an SSO token which will be used in the next play.
In this play is where the magic happens! By using the ovirt_vms module we are configuring our new VM to have/use a:
- name: The name of our choosing, apivm in our case
- template: A template as described in the previous parts. You can either use your own or use the ones provided by ovirt-image-repository found in Storage -> Storage Domains.
  I use my own one which is already configured in a way that every new VM will have basic configuration to make it manageable from the Ansible VM, ansible.homelab.home (ex. ansible ssh keys , ansible user, cloud-init service etc).
- resources & state: I’m only using 1GB of RAM and change the VM state to running.
- cloud-init: This is the time to setup a static IP instead of letting your DHCP assign a dynamic one and having to login via console each time to find out!
  We can do a lot more with cloud-init, in my example I also create a greetings.txt so for debugging reasons, I’m updating the OS so as to have the latest packages, not a great idea in a production environment, and finally changing the root credentials. And all of that with just one click! Cool right?
- In the final task we use the ovirt_auth module in order to revoke the created SSO token from the first play. This step is optional as the SSO will be revoked automatically after some days based on the default oVirt configuration but I always find it more secure to revoke it after I have finished my task as it is not needed any more.

Let’s see what we have configured!

Execution 🐧

Create VM ⏫

Before running our playbook let’s make sure that IP 10.0.010 isn’t already used by any other VM,

:~$ ping 10.0.0.11  -c 5
PING 10.0.0.11 (10.0.0.11) 56(84) bytes of data.  
From 10.0.0.227 icmp_seq=1 Destination Host Unreachable  
From 10.0.0.227 icmp_seq=2 Destination Host Unreachable  
From 10.0.0.227 icmp_seq=3 Destination Host Unreachable  
From 10.0.0.227 icmp_seq=4 Destination Host Unreachable  
From 10.0.0.227 icmp_seq=5 Destination Host Unreachable  
  
--- 10.0.0.11 ping statistics ---  
5 packets transmitted, 0 received, +14 errors, 100% packet loss, time 17414ms\

It’s probably the first time that I got so much pleasure from 100% failure.. and we are good to go! Open your favourite terminal and run your creation playbook:

:~$ ansible-playbook ovirt_create_vm.yml

and check the output:

PLAY [ovirt.homelab.home] ********************************************************************************************************************************************************************************************  
  
TASK [Gathering Facts] ********************************************************************************************************************************************************************************************  
ok: [ovirt.homelab.home]  
  
TASK [Obtain SSO token] *******************************************************************************************************************************************************************************************  
ok: [ovirt.homelab.home]  
  
TASK [Run VM with cloud init] *************************************************************************************************************************************************************************************

If everything is configured correctly you should be stuck in the “Run VM with cloud init” task which for now looks promising! It means that something is at least happening in the background.
Let’s connect to our oVirt GUI and see what is Ansible doing, login and head to the Events section.

What a sight for sore eyes! oVirt was instructed to create a new vm and as we can see above, and apivm is now being prepared! We can check the progress in Compute -> Virtual Machines.

The VM has already started and we can successfully login via console!

The VM is added to our network and we are able to ping it and connect directly via SSH instead of having to check which IP would have been assigned to our VM via console.

ping 10.0.0.11 -c 5  
PING 10.0.0.11 (10.0.0.11) 56(84) bytes of data.  
64 bytes from 10.0.0.11: icmp_seq=1 ttl=64 time=0.575 ms  
64 bytes from 10.0.0.11: icmp_seq=2 ttl=64 time=0.802 ms  
64 bytes from 10.0.0.11: icmp_seq=3 ttl=64 time=0.869 ms  
64 bytes from 10.0.0.11: icmp_seq=4 ttl=64 time=0.782 ms  
64 bytes from 10.0.0.11: icmp_seq=5 ttl=64 time=0.874 ms  
  
--- 10.0.0.11 ping statistics ---  
5 packets transmitted, 5 received, 0% packet loss, time 4083ms

In the meantime and after having created successfully the requested VM, the playbook continued by revoking the SSO token that we have created in the second task!

TASK [Always revoke the SSO token] ********************************************************************************************************************************************************************************  
ok: [ovirt.homelab.home]  
  
PLAY RECAP ********************************************************************************************************************************************************************************************************  
ovirt.homelab.home            : ok=4    changed=1    unreachable=0    failed=0

Playbook finished it’s job and the recap informs us that that all 4 tasks run successfully and 1 change was applied!!

Strangely everything worked out exactly as planned!! 😂 🥳

Shutdown VM ⏬

The next playbook is the one that will conclude the second part and we will be used to shutdown the VM that we created in the previous section.
Open your favourite terminal and run below command that will create ovirt_stop_vm.yml :

:~$ cat <<EOF > ovirt_restart_vm.yml
- hosts: ovirt.homelab.home
  connection: local

  tasks:
  - name: Obtain SSO token
    ovirt_auth:
      url: https://ovirt.homelab.home/ovirt-engine/api
      username: admin@internal
      password: "<add your password here>"

  - name: Stop vm
    ovirt_vms:
      auth: "{{ ovirt_auth }}"
      name: apivm
      state: stopped

  - name: Always revoke the SSO token
    ovirt_auth:
      state: absent
      ovirt_auth: "{{ ovirt_auth }}"

In addition with the previous ones , i can safely say that the above playbook is quite simple. The only point to keep in mind regarding the state handling above is that the ovirt_vms module doesn’t support restart, at this point of time at least, and the available states are: absent, next_run, present, registered, running, stopped, suspended. So in case you want to restart a VM you need one more task , with state: running!

Let’s run it and put our new VM to sleep until the third part of this awsome series arrives! 😉

:~$ ansible-playbook ovirt_restart_vm.yml

PLAY [ovirt.homelab.home] ***************************************************************************************************************  
  
TASK [Gathering Facts] ***************************************************************************************************************  
ok: [ovirt.homelab.home]  
  
TASK [Obtain SSO token] **************************************************************************************************************  
ok: [ovirt.homelab.home]  
  
TASK [Stop vm] ***********************************************************************************************************************  
changed: [ovirt.homelab.home]  
  
TASK [Always revoke the SSO token] ***************************************************************************************************  
ok: [ovirt.homelab.home]  
  
PLAY RECAP ***************************************************************************************************************************  
ovirt.homelab.home            : ok=4    changed=1    unreachable=0    failed=0

With the successful run of this playbook, the actual task of manipulating the oVirt API in order to create our new VM via Ansible, is now completed! Head to your infrastructure and happy automating!!

Final Words 💡

I hope that I managed to explain, or at least make a bit clearer, what is an API, and how to interact with it with the help of Ansible automation tool.

I think that both a beginner and an intermediate user can find very useful information’s for getting started or dig deeper to get the most out of the oVirt REST API by extending the provided playbooks to create storage pools, new hosts, network profiles ,users and many more.. automatically!

As Ansible is a huge chapter, I will use the momentum to stay a bit longer in the automation area and use the next post to experiment with tasks that can be run directly on the remote server and create playbooks that will act as a “single source of truth” in our infrastructure.

We will see how to disable services (that I don’t like 😆) , copy our own configuration files, use a bash script to wrap everything around it and see how we can pass pass extra variables to our playbooks!

Thanks for reading, until the next post and as Dr Wallace Breen says in Half-Life 2..
Be wise. Be safe. Be aware!

Automatically create virtual machines via Ansible and oVirt API - Part One

Sun, 05 Sep 2021 21:10:59 +0000

Automatically create virtual machines via Ansible and oVirt API - Part One.

At this point my (virtual) infrastructure was ready to welcome more virtual machines that could be successfully backed up and restored to a previous state. If you want to check the backup/restore procedure that I’m following, you can find all related info to the Create and Export Full oVirt VM Backups post.

So nothing could go wrong or get between me and my fancy ideas any more!

But even though i was super excited and wanted to directly create as many virtual machines as I could, I didn’t like the idea of having to follow all the manual steps that I have documented in my previous posts for each and every new vm!

Then it hit me! What if I could automate the procedure and create a new virtual machine with a push of a button? To do that I needed an automation tool that could make use of the oVirt REST-API.

It was time to bring out the big guns for my first official project! Ansible!

Appendix 📓

First things first though and before moving to the interesting part we need to explain the terms that are going to be used in this post.

What’s a(n) API / REST-API?

First let’s start by explaining what an API is. I won’t get into much detail because there are tons of great material to get information from but for the purposes of this post I will document below the explanation that made sense to me when I first encountered this concept.

Think of the API as the Waiter in your favourite restaurant. He is the one standing between you (Application/Client) and your delicious meal. Of course you have the menu to decide but let’s say you want the BBQ Burger 🍔 but with the tomato removed! You need to ask the Waiter (API) if this is possible, he will then have to forward your request to the Kitchen (Server) and return with an answer (and probably with a spit on your burger 😂 ).

Now let’s get a bit more professional shall we? API (Application Programming Interface) is a set of definitions and protocols for building application software that acts as an intermediary allowing two applications to talk to each other by granting all the required permissions.

Wait, we are not done yet!! oVirt Engine is providing a REST-API, a subset of APIs, which is a set of guidelines for building a web service API which uses URIs and HTTP protocol and JSON for data format. REST-APIs are more flexible as they can handle different types of calls and return data in different formats (XML, JSON, YAML etc).

And what about Ansible?

There are multiple automation tools out there and each one has it’s own strengths and weaknesses and let me explain what I mean! For example, I din’t want to mess with ruby at this point (or at any point of time honestly), so this ruled out ~~Chef~~ immediately. I also wanted an agentless push configuration solution which also ruled out ~~Puppet~~. I didn’t have any experience with Salt, so this left me with .. 🥁.. Ansible!

I don’t have a fancy example this time to be honest, but I have an awesome Fun Fact that you probably aren’t aware of!
The term “ansible” was first introduced in 1966 Ursula K. Le Guin’s novel Rocannon’s World!! As I am a huge Ursula Le Guin and Sc-fi fan, this was mind blowing for me back when I first came across to the Ansible software.

Ansible is the Swiss army knife equivalent of IT automation engines, a powerful open-source automation tool that acts as a configuration management, application deployment, orchestrator and cloud provisioning.

Ansible is an agentless tool that runs in a ‘push’ model, temporarily connecting remotely via SSH or Windows Remote Management to make them manageable. This makes it the best option for our project as it is a flexible and highly customizable tool which provides an ad-hoc way to configure and manage various parts of our oVirt infrastructure.

And the best part is that oVirt has everything ready for us as it is already maintaining multiple Ansible roles! Head to the official site so as to get started with Ansible in case you haven’t done so already!

Say to hello to my little Cloud-Init friend (with the voice of Toni Montana) !

Red Hat’s official documentation has a great explanation regarding cloud-init and I think that I shouldn’t interfere thus I quote:

“cloud-init is a software package that automates the initialization of cloud instances during system boot. You can configure cloud-init to perform a variety of tasks. Some sample tasks that cloud-init can perform include:

Configuring a host name

Installing packages on an instance

Running scripts

Suppressing default virtual machine behaviour”

As you can understand there are many different scenarios that cloud-init could come in quite handy.
We will use this software to apply our own custom setup (Static IP/Subnet/Resolv/DNS/Domain etc) for each and every vm that we create.

Prerequisites 📝

In order to be able to understand and test the concept we will need the below setup/configurations available:

Our oVirt infrastructure with a correctly configured SSL certificate (self signed or official one). You can always bypass the SSL configuration but you will need to adapt accordingly the commands that are used during the post as I’m using a secure connection, playbook wise.
a CentOS 7 virtual machine ( hopefully the last machine that you will have to install and configure manually 🙏 ) that has ansible package installed and configured.
This vm will host our yaml configuration files and eventually execute the playbooks that will create the new vm’s to our oVirt host. Check out this great post if you haven’t configured one yet.
a Centos7 template with cloud-init software configured. Again you can find lots of great posts regarding this concept of creating your own cloud-init ready CentOS ISO image or install and update your oVirt template. You can combine for example this cloud-init configuration post with my previous article on how to create your own template!
In case you haven’t come across yet to one of my previous posts (for example on how to backup your oVirt vm’s) , and even if it’s not a prerequisite for this post , it’s good to have Python SDK installed for the oVirt Engine API which is the collection of classes that will allows us to interact and run our oVirt system tasks. For EL distros (such as CentOS, RHEL, etc.):

[ovirt-tested-$VERSION]
baseurl=http://resources.ovirt.org/repos/ovirt/tested/$VERSION/rpm/el$releasever
name=oVirt-$VERSION
enabled=1
gpgcheck=0

:~$ yum install python-ovirt-engine-sdk4 ovirt-engine-sdk-python

or download the rpm directly from here in case you stumble upon package manager issues that are difficult for you to solve and install it manually,

:~$ rpm -iv python-ovirt-engine-sdk4-4.2.7-1.el7.src.rpm

Architecture ✏️

I find it quite useful to have a short sketch which can easily show me what I have already configured and what I want to achieve.

Time for a drawing and more information 😅!

In the above picture we see what we have already configured:

oVirt Host : inside of which I create VM’s, based on a template because i don’t want to configure users, packages, ssh keys etc, by following more or less these steps. So you understand, or you’ll see on your own when creating more virtual machines, that those are a lot of clicks, especially if you intend to keep your services separated and in different machines.
- hostname: hl-ovirt
- DNS: ovirt.homelab.home
- IP: 10.0.0.3
Ansible VM : This is a virtual machine, created in our oVirt infrastructure and which hosts all the ansible related configuration/files.
- hostname: hl-ansible
- DNS: ansible.homelab.home
- IP: 10.0.0.10

and what we want to achieve:

our goal here is to be able to run a playbook from the ansible.homelab.home which will call the REST-API provided by the oVirt host that will instruct the host to create a new machine with the Name & IP of our choosing by using the already configured cloud-init template hosted on our oVirt host.

Confusing 😕? Nahhhh you, you’ll be alright, everything is going to be fine!

Configuration 🔨

First step in projects that APIs are involved, is to configure the authorization and make sure that our account has appropriate permissions. I will start by trying to get a list of the installed VM’s that are hosted in my oVirt host and see what happens.

I will use the admin user , the same one that I’m using to connect to the oVirt GUI, just for testing. In a production environment you should never use the admin account for various security reasons but for my local home-lab there is no need to go down that road!

But before starting interacting with oVirt API, let’s see if we can fetch system related information on our host and make sure that Ansible is correctly configured and has all the appropriate permissions.

Ansible Ad-Hoc 🏃

Let’s see in action some examples of what we can do with Ansible and then we move to the playbooks part.

I guess that you have a working ansible vm at this point and maybe another host that ansible can connect and retrieve informations from. You can always apply the commands below on your local machine but for our case we will stick to the plan!

Let’s try and ping our oVirt host and confirm that we have connectivity:

:~$ ansible ovirt.homelab.home -m ping'
ovirt.homelab.home | SUCCESS => {
    "changed": false, 
    "ping": "pong"
}

Ok, we can connect to our host! Let’s check it’s hostname:

:~$ ansible ovirt.homelab.home -m shell -a 'echo $HOSTNAME'
ovirt.homelab.home | SUCCESS | rc=0 >>
hl-ovirt

ohh nice! Let’s get the uptime of our host:

ansible ovirt.homelab.home -m shell -a uptime 
ovirt.homelab.home | SUCCESS | rc=0 >>
 14:18:13 up 8 days, 21:31,  2 users,  load average: 0.10, 0.09, 0.11

And we can do this for multiple servers! We can check RAM/CPU resources, copy/fetch files, change file/folder permissions, install packages and check if a service is up and running with the use of some ansible magic! Awesome right?

Time to get to the grown up stuff though!

Ansible Playbooks 📚

I won’t get into too much detail on the playbook format and the spaces and dashes that needs to have as I take for a fact that you have some experience on Ansible and if not please head to the Appendix to find info and useful links.

But in order to understand the next code snippets I must point out that playbooks are expressed in YAML format and composed of one or more ‘plays’ that run in order from top to bottom against all machines matched by the host pattern.

Start by adding below yaml code to a .yml file. I’m doing it with the use of EOF (End of file) operator so as to minimize any structure related yaml errors, so just copy/paste below output to your terminal:

:~$ cat <<EOF > ovirt_list_vms.yml
---
- hosts: ovirt.homelab.home
  connection: local

  tasks:
  - name: Obtain SSO token
    ovirt_auth:
      url: https://ovirt.homelab.home/ovirt-engine/api
      username: admin@internal
      password: "<add your password here>"

  - name: List vms
    ovirt_vms_facts:
      fetch_nested: true
      nested_attributes:
        - description
      auth: "{{ ovirt_auth }}"

  - name: set vms
    set_fact:
       vm: "{{ item.name }}: {{ item.snapshots |
map(attribute='description') | join(',') }}"
    with_items: "{{ ovirt_vms }}"
    loop_control:
      label: "{{ item.name }}"
    register: all_vms

  - name: make a list
    set_fact: vms="{{ all_vms.results | map(attribute='ansible_facts.vm') | list }}"

  - name: Print vms
    debug:
      var: vms
EOF

Ok, don’t jump off the window yet, please 😆! If this is the first time that you stumble upon yaml code then keep in mind that the normal feeling after seeing the above is.. 😱 !!

Let’s have a look on each section of the playbook. Keep in mind that I’m not an expert and you must not take the next steps break down as a granted, this is my personal notes and understanding of the playbook that we’ve just created:

The first section is the host which tells our playbook which is the target host (or group of hosts in other cases) that the playbook is going to be executed on. In our case the oVirt host of our infrastructure (ovirt.homelab.home) which provides the REST-API.
Next we have the tasks section and the first play where we are using the ovirt_auth module to authenticate to oVirt engine and create an SSO token which will be used in the next play.
In the second play we are using ovirt_vms_facts module to retrieve the description of all vm’s currently deployed in our home-lab infrastructure by using the SSO token created in the previous task. We could always also use a pattern instead of the a nested_attribute, for example :
```
ovirt_vms_facts:
 pattern: name=centos* and cluster=<our_cluster_name>
```
Then we use the set_fact module which allows us to dynamically add or change facts during execution. In our case we want to output a list of vm names and their snapshot description, filter (|) by description separated by comma for multiple snapshots (if there are any..) while looping though the active virtual machines. Then we register the output to all_vms variable so as to be filtered in the next play. Again this is just an example and you can create your own list columns with the help of a little JSON PATH magic!
We then apply another filter to the previous all_vms variable with the use of map that just allows us to change every item in a list, using the ‘attribute’ keyword and do the transformation based on attributes of the list elements and assign the list to a new variable called vms.
To better understand what would be the output at this exact point, you can remove the “make a list” play and in “print vms” one replace vms variable with the all_vms. In that way you can test different tasks and figure out on your own what would be the outcome of each one if run separetly.
This would be the output in such case (do you see the all_vms.result that we are using in the “make a list” task?) :

ok: [ovirt.homelab.home] => {
    "all_vms": {
        "changed": false, 
        "msg": "All items completed", 
        "results": [
            {
                "_ansible_ignore_errors": null, 
                "_ansible_item_label": "hl-dns", 
                "_ansible_item_result": true, 
                "_ansible_no_log": false, 
                "ansible_facts": {
                    "vm": "hl-dns: Active VM"
                }, 
                "changed": false, 
                "failed": false, 
                "item": {
                    "affinity_labels": [], 
                    "applications": [], 
                    "bios": {
                        "boot_menu": {
                            "enabled": false
                        }
                    }, 
[more output]

Finally we print the finished filtered list that is assigned to the vms variable.

And again, maybe I’m mistaken, maybe I didn’t explain something correctly or adequately so please either use the ansible-doc to search for more detailed module information,

:~$ ansible-doc set_fact

or head to the official Ansible documentation to clear up anything that seems incorrect or not legit! Then please drop me a line so as to update my notes 😂 🙏!

Time to run our playbook and see what happens!

Execution 🖱️

Open a terminal, head to the folder that you have saved our .yml file and run the command below:

$ ansible-playbook ovirt_list_vms.yml

PLAY [ovirt.homelab.home] **********************************************************************************************************************************

TASK [Gathering Facts] **********************************************************************************************************************************
ok: [ovirt.homelab.home]

TASK [Obtain SSO token] *********************************************************************************************************************************
ok: [ovirt.homelab.home]

TASK [List vms] *****************************************************************************************************************************************
ok: [ovirt.homelab.home]

TASK [set vms] ******************************************************************************************************************************************
ok: [ovirt.homelab.home] => (item=hl-ansible)
ok: [ovirt.homelab.home] => (item=hl-dns)

TASK [make a list] **************************************************************************************************************************************
ok: [ovirt.homelab.home]

TASK [Print vms] ****************************************************************************************************************************************
ok: [ovirt.homelab.home] => {
    "vms": [
        "hl-ansible: Active VM, before_ansible_setup", 
        "hl-dns: Active VM"
    ]
}

PLAY RECAP **********************************************************************************************************************************************
ovirt.homelab.home           : ok=6    changed=0    unreachable=0    failed=0   

Success!!

So our playbook did exactly what we told it to do! It connected to our oVirt via the REST-API, obtained an SSO token based on the username/password login credentials and then created a list of our currently deployed virtual machines and their snapshot description (I forgot that I had a snapshot for my hl-ansible vm, time to delete that and save some time and backup space..) !

The Play Recap output part, informs us that no failures and no changes were applied, as we didn’t update anything, and all 6 plays run successfully!🎉🎉

Intermission ☕

It seems to me like a good point to conclude this first part. I have given more than enough information and configuration links that you can refer to in case you want to dig deeper on your own regarding both Ansible and APIs while waiting for part two!

Kudos for managing to get to this point, really, even if you have a headache and you are seriously thinking of jumping to another job field! Before doing so though, grab a well reserved beverage, go through the notes, configure and play with Ansible a bit and I think that you will change your mind!

In the second part of this post, we will try to create our next virtual machine automatically via a playbook and maybe wrap it around a bash script! Stay tuned!! 🎸

Thanks for reading, until the next post and as Dr Wallace Breen says in Half-Life 2..
Be wise. Be safe. Be aware!

Pair Logitech MX Keys with Xubuntu 18.04 via Bluetooth

Wed, 25 Aug 2021 19:32:59 +0000

Pair Logitech MX Keys with Xubuntu 18.04 via Bluetooth

Usually I don’t write posts about hardware configurations but this time I got really frustrated while trying to pair my new, much anticipated, fancy Logitech MX Keys keyboard! Maybe this is helpful to someone experiencing the same issue there is a catch in the pairing procedure and Logitech’s documentation included in the package isn’t clear enough regarding this.

Summary

Let me start by saying that I’ve been using this keyboard for about a week and I’m totally pleased with my choice. It’s the first time that I bought a high-end keyboard, I was always using descent keyboards but never really gave more than 50 euros for any of them and happy with what I was getting with my money. And honestly, why someone wouldn’t be happy with his Microsoft Comfort Curve Keyboard 2000 ?? Thank you for your services MS Comfort, you’ve been a loyal comrade!

Then, Covid-19 hit the world and home office increased exponentially! So my home became my actual office, which meant having a keyboard/mouse for my gaming PC and a keyboard/mouse for my work laptop. Of course this is the least of the problems that this new situation brought along but still it didn’t make sense to have 2 full size keyboards/2mouses on a desk , wasting so much working space in 2021!

I thought about getting a keyboard/mouse hub but from my working experience, I have never came across to a robust solution that would do the actual work. Every single time those hubs where extremely glitchy and I would end up swearing and throwing them away!

So I though of buying a keyboard that would have both Bluetooth & USB Adapter available. In this way I could connect the keyboard to laptop via BT and use the USB adapter to my PC which doesn’t have BT!

And my search led me to this rechargeable (USB Type C), stylish, nearly silent and great built design with smart backlighting and ,based on the reviews, fantastic battery life ( 10 days on full charge and 5 months with backlighting turned off which is awesome as I’m working in a well lighted environment )!

Did I think of it correctly? Where there other solutions that would be better? Sure! Only time will show but honestly I wanted to treat myself with a new keyboard some time now 😆 !!

Enough with the chit chat, let’s cut to the chase!

Pairing!

Now let’s head to the pairing catch that you’ve probably came across and was the reason that landed you here!

Here it was! My new robust keyboard in my hands and ready to start typing! Following the described procedure, I turned on the switch of the keyboard, opened my BT manager on the laptop and MX Keys new device directly popped up in the discovered devices.

But when I chose Pair, multiple pop up pairing messages like “Pairing MX Keys” started flooding the right upper corner of my screen but I couldn’t figure out what was needed from my side.

After googling a bit, I found the below image in Logitech’s official Setup Guide..

.. and it hit me! MX Keys wanted me to type a 6 digits code to my keyboard and hit ENTER! But for some reason my Xubuntu wasn’t giving me any code to use ..

So back to google ✊ ! Eventually my search 🔎 led me to this very helpful guide which explains how to the follow the exact same pairing steps that GUI is using but from the terminal ( always go with the terminal choice people, always! ).

Console Time!

We will use the bluetoothctl command to try and get the 6 digits code in order to pair our controller and the new keyboard. This command tool is provided by bluez_5.48-0ubuntu3_amd64 🐞 package.

Run the command to get at the prompt and tell bluetoothctl to look only for keyboards,
```
:~$ :> bluetoothctl
[bluetooth]# agent KeyboardOnly
Agent is already registered
```

Put your controller in pairable mode

[bluetooth]# pairable on
Changing pairable on succeeded

Instruct the controller to start scanning for a suitable device

[bluetooth]# scan on
Discovery started
[CHG] Controller 38:DE:AD:2A:BE:CA Discovering: yes
[NEW] Device 52:9B:32:5E:BF:3E 47-6B-25-5E-BF-3E
[NEW] Device D4:AD:1D:C5:77:B5 MX Keys

When the correct device is found, use the MAC to pair

[bluetooth]# pair D4:AD:1D:C5:77:B5
Attempting to pair with D4:AD:1D:C5:77:B5
[CHG] Device D4:AD:1D:C5:77:B5 Connected: yes
[agent] Passkey: 508203
[NEW] Primary Service
/org/bluez/hci0/dev_D4_AD_1D_C1_67_B4/service000a
00001801-0000-1000-8000-00805f9b34fb
Generic Attribute Profile
[..]

And there it is, the passkey is revelled! At this point type the 6 digit code to your MX keyboard, hit Enter and boom,

[..]
[CHG] Device D4:AD:1D:C1:67:B4 ServicesResolved: yes
[CHG] Device D4:AD:1D:C1:67:B4 Paired: yes
Pairing successful

The pairing via BT, which was previously failing, is now successful 🎉 !

Conclusion

From my point of view, I think that Logitech’s quick guide included in the package should have a note regarding the 6 digits code needed to be typed in order to complete the pairing and not rely on people having to go through their online guide to find this important configuration step. Of course Xubuntu’s bug of not showing the code on the other side is also something that should be addressed and fixed but maybe this is not a general issue and just something broken on my side!

Either way, now that I’m typing this post in my new awesome keyboard I can say that it totally worth the search and fix 🙌! I can now get rid of all the unneeded keyboards and free up my desk space!

Yes I know that i didn’t speak about an equivalent 1 mouse to save the world kind of thing but I have the feeling that this will be a much easier task. If not.. then expect a new post where I’m complaining about something else I guess 😂!

Even though the are multiple great videos and reviews out there regarding this keyboard, I think that I should create my own after the first 3 months and compare notes of current and future behaviour! We will see 😊 ..

Thanks for reading, until the next post and as Dr Wallace Breen says in Half-Life 2..
Be wise. Be safe. Be aware!

Create and Export Full oVirt VM Backups to NFS Storage Domain

Mon, 13 Jul 2020 20:32:59 +0000

Create and Export Full oVirt VM Backups to NFS Storage Domain.

Now that I had my first virtual machine up and running and before moving on by configuring all the cool stuff that I had in mind, I needed to find a way to create daily virtual machine backups and store them in another device.

You see, the most common nightmare among system engineers is the absence of a backup policy 😱 !There are a lot of different approaches when it comes to backup and it depends on the provided services and its criticality!

You can have a Database backup, a File or Disk backup,a Bare metal or Virtual machine backup etc, then you have to decide if you want a Full or an Incremental one, Daily/Hourly/Weekly, locally or remotely and as you understand the task of creating a concrete backup policy can become a huge headache but in the end of the day is the only thing that will allow you to have some sleep at night!

Prerequisites 📝

As documented in the official oVirtBackup repo, in order to be able to use this tool it is necessary to install oVirt Python-sdk. At the time that i configured this tool the python-ovirt-engine-sdk4 package wasn’t available to CentOS repository so have a look via yum or download it from here and install it via rpm as I did.

CentOS/Fedora

:~$ rpm -i python-ovirt-engine-sdk4-4.2.7-1.el7.src.rpm

(Optional) A package that is also used in this guide and which will be needed in order to clone the public repository of oVritBackup is git and if you haven’t installed it yet, then use the below command to do so.

yum install git

Creating and Attaching NFS Storage Domain

Configure NFS in QNAP ( Optional )

I will be using my QNAP NAS as an NFS server to store my backups remotely. If you have any machine lying around then you can setup your own NFS server without the need of a Nas. You can of course skip this step if you want to store the backups locally but keep in mind that the script that I will be using for backing up my virtaul machines needs an NFS export.

All major descent NAS devices provide NFS functionality, I’m using a QNAP TS-409 Pro which is quite old but does the job without any problems.

First step is to enable NFS service by going to Network Services -> NFS Service, click on Enable and then Apply. Then we have to create a New Folder and set the NFS rights for the network share.

Go to Access Right Management -> Share Folders and click on New Share Folder. Follow the steps as you would in any other similar folder creation and after finishing choose NFS from the Action Column of the newly created folder ( we will give our folder an imaginative and original name, let’s call it nfs_share 😆 )

As you can see from the above screenshot I have given full rights from my /24 internal network but you can limit it for specific IP’s. Click Apply and our NFS share is ready to be attached to our oVirt Storage Domains.

You can bypass this section as this is not mandatory for the backup procedure. I would personally continue with the next steps as it will save you a lot of time and troubleshooting in case your NFS Export fails to be attached to our oVirt infrastructure.

First install the appropriate utils for your Centos7 to be able to act as an nfs client:

:~$ yum install nfs-utils

Create the path that your NFS will be mounted on

:~$ mkdir /mnt/nfs_share

and edit your /etc/fstab like below by changing your IP and share folder name:

## You can also use your favorite editor
:~$ cat <<EOF >>/etc/fstab
10.0.0.4:/nfs_share /mnt/nfs_share/      nfs     defaults        0 0
EOF

Let’s mount our new share and create a file so as to make sure that we have Write Permissions which is of course required by oVirt export domain addition.

:~$ mount -a

:~$ touch /mnt/nfs_share/test
:~$ ls /mnt/nfs_share/test
-rw-r--r--  1 root root    0 May  02 17:43 test
:~$ rm -f /mnt/nfs_share/test

Preparing and Adding NFS Storage Domain to oVirt

Fire up your favorite browser, type your oVirt endpoint or IP and login with your admin account. By now I guess you have seen my previous posts on local Certificate Authority and DNS configuration and you are familiar with your secure oVirt Administration Portal!

From the left panel go to Storage -> Domains. Now if you remember, and if not just have a look here ⬅️, in the storage domain configuration I used in the first part of my oVirt setup I added two domains, a Data and an ISO.

Time to add a new Export Domain!

Click on New Domain and add a Name, a Description and Choose NFS for Storage Type. In the Export Path use either FQDN or IP followed by the nfs folder that we created in the previous step, in my case 10.0.0.4:/nfs_share and click OK.

If everything is ok with the NFS configuration, you will be able to see your NFS in an Active status as well as the Total and Free Disk Space! If not though, check your NFS server configuration ( share permissions, IP or FQDN etc) or follow the previous section which documents the steps to mount NFS to our Centos7 and make sure that everything is configured exactly as they are documented.

Our NFS is now attached and ready to host all of our VM backups!

Creating and Exporting Full VM Backups with oVirtBackup Tool

oVirtBackup is a free tool written in Python which makes an awesome job on creating VM backups. I have been using this for the past months and never had a problem, this is a solid solution that you configure once and then forget about it.
More information can be found in it’s official repository and I will document my configuration in order to get this up and running.

Taken from the official Git page, the workflow that this script is using is:

Create a snapshot
Clone the snapshot into a new VM
Delete the snapshot
Delete previous backups (if set)
Export the VM to the NFS share
Delete the VM

oVirtBackup Setup

First step is to clone the repository! Navigate to your favorite folder or path that you keep your awesome scripts, run the following commands and

:~$ cd /home/user/scripts
:~$ git clone https://github.com/wefixit-AT/oVirtBackup.git

oVirtBackup Config

The only file that you need to update in order to start creating backups is the config.cfg (if this doesn’t exist just copy one from config_example.cfg).

:~$ cd /home/scripts/oVirtBackup
:~$ cp config_example.cfg config.cfg

Let’s have a look at the basic lines found in the config file and that you need to edit in order to have this up and running.

A list of names which VM’s should be backed up
vm_names: [“myfirstvm”,”mysecondvm”] ## Check backup.py help for an option to backup all vm’s
Url to connect to your engine
server=https:///ovirt-engine/api
Username to connect to the engine
username=admin@internal
Password for the above user
password=
Name of the NFS Export Domain (the one that we created in our first section)
export_domain=nfs_share
The name of the cluster where the VM should be cloned
cluster_name=ov_cluster
Storage domain where the VM’s are located. This is important to check space usage during backup
storage_domain=data
</em> Optional :
Redirect log to a specific file
logger_file_path=/var/log/vmbackup.log
How long backups should be keeped, this is in days
backup_keep_count=1
How many backups should be kept, this is the number of backups to keep
backup_keep_count_by_number=1

So open the config.cfg with your favor editor ,add or update the appropriate lines and save your changes.

Our First VM Backup!

Now we are ready to start our first ever VM backup, let’s see the availble options that can be used,

:~$ ./backup -h

and let’s run it in a debug mode to see what happens!

:~$ /backup.py -c config.cfg -d

If everything is ok and no error occurs then you should see logs like the ones below

2020-06-22 04:00:01,745: Start backup for: myfirstvm
2020-06-22 04:00:01,846: Snapshot creation started ...
2020-06-22 04:00:05,176: Snapshot operation(creation) in progress ...
2020-06-22 04:00:10,268: Snapshot operation(creation) in progress ...
2020-06-22 04:00:15,357: Snapshot operation(creation) in progress ...
2020-06-22 04:00:20,437: Snapshot created
2020-06-22 04:00:30,533: Clone into VM (myfirstvm__20200622_040001) started ...
..
..
2020-06-22 04:01:33,503: Cloning in progress (VM myfirstvm__20200622_040001 status is 'image_locked') ...
2020-06-22 04:01:38,536: Cloning in progress (VM myfirstvm__20200622_040001 status is 'image_locked') ...
..
..
2020-06-22 04:02:59,918: Snapshot operation(deletion) in progress ...
2020-06-22 04:03:05,007: Snapshot operation(deletion) in progress ...
..
..
2020-06-22 04:07:21,170: Exporting in progress (VM myfirstvm__20200622_040001 status is 'image_locked') ...
2020-06-22 04:07:26,214: Exporting in progress (VM myfirstvm__20200622_040001 status is 'image_locked') ...
2020-06-22 04:07:31,251: Exporting finished
2020-06-22 04:07:31,278: Delete cloned VM (myfirstvm__20200622_040001) started ...
2020-06-22 04:07:33,485: Cloned VM (myfirstvm__20200622_040001) deleted
2020-06-22 04:07:33,485: Duration: 7:32 minutes
..

The exported backups can be then found by going to Storage -> Domains -> nfs_share -> VM Import.

You can import any of the backups by right clicking and choosing Import.

So now that the command is tested manually, it’s time to add it in a cronjob and create as many backups as you like! I’m running a backup at 4:00 AM every Monday,Wednesday,Saturday and a backup of all my VM’s every 1st Sunday of each Month!

:~$ crontab -l
0 4 * * 1,3,5 /home/scripts/oVirtBackup/backup.py -c /home/scripts/oVirtBackup/running.cfg -d
0 4 1-7 * * [ "$(date +\%a)" = "Sun" ] && /home/scripts/oVirtBackup/backup.py -c /home/scripts/oVirtBackup/running.cfg -d -a

Hint: You can remove -d flag so as to avoid filling up your logs with unnecessary debug informaion. You can also add -a flag which will back all of your VM’s! Check –help as pointed out in previous section for more flag magic. 😉

Final Words 💡

I don’t know how you feel but I am way more relaxed now than I was before configuring this awesome free tool! A huge thanks to wefixit-AT which is the repo owner!

Our VM backups are successfully exported/imported to our NFS and can be used to either to revert to a previous state or even create a new VM to just get a configuration and then delete it!

Having backups to revert to in case of a miscofiguration or HW failure is great. Now we can safely move on with our next projects in line!

In the near future I will also document a simple script that I’m using to backup only the VM’s that are currently up and running, so stay tuned! 🎶🎶

Thanks for reading, until the next post and as Dr Wallace Breen says in Half-Life 2..
Be wise. Be safe. Be aware!

Configure oVirt Engine on CentOS 7 - Part Two

Mon, 06 Apr 2020 20:32:59 +0000

Create your first Virtual Machine and Save it as a Template.

In the First Part of this tutorial, we documented in detail everything needed to setup our oVirt Infrastructure which is ready to host new virtual machines.

I suppose that you had enought time to familiarize yourself with the oVirt GUI by now, so let’s move on by creating our first virtual machine and save it as a template!

After this step we will be finally ready to start our journey of conquering the world with your amazing projects!

Prerequisites 📝

In order to be able to connect to remote virtual machines and troubleshoot in case any of those becomes unavailable for any reason, we need to configure a Console Client.
I’m using the oVirt Remote Viewer which can be either downloaded from the Official Site for both Linux/Windows or installed via a package manager:

CentOS/Fedora

yum install -y virt-viewer

Ubuntu/Mint

sudo apt-get install virt-viewer

Create your First VM 🖥️

From the left panel go to Compute -> Virtual Machines. As you can see our VM section is empty, so let’s not waste our precious time and create one!

After clicking on New button, we are introduced with the General Options configuration Tab.
Add a Name for your first VM, a Description and at the Instance Images section click on Create option to specify the disk space that is going to be used by your VM.

Choose the disk size in GB and leave the other options with the default values. In the bottom left, choose Show Advanced Options and go to System Tab.

Here we can choose the Memory Size and Total CPU that we want to assign to our virtual machine. We can also choose if we to install from a Template, which is not possible for now as we haven’ configured one yet.

Let me point out that there are a lot of different configuration options that can be changed based on your experience and needs, but I will only document the ones that are important for a basic VM creation.

So the last but not least configuration option is handled in the Boot Options Tab.

Enable Attach CD and choose the ISO file that we have already uploaded and is documented in the Prerequisites Section of Part One.

Click OK and voilà! 🥂

Hm.. I made it seem like a big thing by using a French word, right?

Because it actually was!

Like the time you created your first PC you had to go hunting for a CPU, RAM and a Hard Disk, among other important parts 😛 , now you just had those and needed to assign them to your new VM! Time to install our good old Penguin!

Installing CentOS7 to our VM 🔨

Let’s Run our VM and see what happens!

Choose you virtual machine, click on Run and then when Console is available, click on it and choose to open the file with the Remote Viewer, which we have configured in the Prerequisites Section, in order to have a look on what happened when we booted up our VM.

If you followed the steps documented in the previous section then you should be able to start the Centos Installation Process by choosing the Install Centos 7.

Press Enter to start the installation.

Installation Intermission: Due to the fact that there are a lot of great tutorials out there to help you get a new Centos7 up and running, I will skip the actual OS installation.

Instead here’s a gif that keeps you company during Centos7 installation! ⏱️

After the installation is finished you will be able to either connect to your VM via SSH or directly from the Console!

Congratulations, your first VM is now a reality! 🎉

Cleanup and Export your VM as a Template 🚿

Templates can save us a huge amount of time in cases where we need to deploy a large number of similar VM’s, especially in a corporate environemt, but they are nothing more than a pre-installed/configured virtual machine with it’s own OS and resources.

I found templates to be very useful in my homelab setup where I’m using the same distribution and basic resources for all my VM’s. Whenever there is a need for a new virtual machine, I clone one from my Centos7 template, change it accordingly and have it ready in some minutes instead of going through all the steps documented in the previous section.

In order to be able to use the template we have to configure and make it a bit generic.
This means that we will have for example to:

Add a general hostname
Clear machine’s id
Remove HDADD value from network device
Delete any ssh keys and logs
Unconfigure VM

In this way every time we clone a machine, it will have it’s unique information which we can login and change/update appropriately.

Seal your VM 🛡️

Login to your virtual machine
Set a generic hostname

:~$ hostnamectl set-hostname localhost.localdomain

Clear machine’s id

:~$ :> /etc/machine-id

Delete any host and root ssh keys

:~$ rm -f /etc/ssh/ssh_host_*
:~$ rm -rf /root/.ssh/
:~$ rm -f /root/anaconda-ks.cfg
:~$ rm -f /root/.bash_history
:~$ unset HISTFILE

Configure Network Interface

This step depends on your configuration as there are different changes based on the IP aasingment. My only note in this section is to remove the HWADDR= line. If you need more information you can check this great Github page or on the official oVirt Documentation.

Install oVirt agent (Optional)

This is the oVirt management agent running inside the guest (our VM) which supplys run-time data and heart-beat info.
Even though it’s great to have this enabled in every machine, i saw that it needs resources that i didn’t want to spare but if you want to install it here is the way to do so,

:~$ yum -y install epel-release
:~$ yum install ovirt-guest-agent-common
:~$ systemctl enable ovirt-guest-agent && systemctl start ovirt-guest-agent

Set VM to an unconfigured state

This final command will create a file called /.unconfigured and shutwdown the machine immediately. The presence of this file will cause /etc/rc.d/rc.sysinit to act like it is the first time the machine is booting up and thus reconfigure networking/routing, time/date, services and delete all rules from /etc/udev/rules.d/,among other actions! You can check sys-config man page for more info.

:~$ sys-unconfig

Our VM is now stopped and ready to be exported!

Export VM to Template 📤

Now in order to export our virtual machine to template, we have to choose it and click on the 3 dots found the upper right corner. Then choose Make Template.

This will open up the New Template configuration where we can add a Name, Description or even change Disk Alias(optional).

Click OK to start templating!

You can check the progress in the Events Tab.

After this task is finished, we should be able to find our new template in Compute -> Templates menu.

Our template is successfully created and waiting to be used! If you feel safe, just delete the already created VM and recreate, but this time use the template!

Click on New VM , add a Name and click OK!

Final Words 💡

You are finally able to start creating as many VM’s or templates as you like based on your oVirt hardware capacity!

There are so many advantages on having your own homelab to play with and experiment without having to spend a fortune!
I hope those two detailed parts provided a better understanding on which are the most important ones that will help you get started on configuring your own Open Source Virtual Infrastructure and creating your first virtual machine.

In the near future I will also document the cloud-init configuration that I’m using in order to create new virtual machines automatically via Ansible through the oVirt API, so stay tuned! 🎶🎶

Thanks for reading, until the next post and as Dr Wallace Breen says in Half-Life 2..
Be wise. Be safe. Be aware!

Configure oVirt Engine on CentOS 7 - Part One

Wed, 04 Mar 2020 20:32:59 +0000

Create a basic Cluster/Host and Configure a Storage Data and ISO Domain.

In my previous post I gave a general overview of my oVirt setup that followed the official installation guidelines and default suggested values.

So it’s time to provide a more detailed configuration on how to prepare everything, from a new cluster and host to ISO domain and local storage, in order to start creating new virtual machines.

In the second part of this tutorial I will also create a VM and save it as a template which will be used for all of our future machines! (Note to Future Me: Automate VM creation via the oVirt API!)

Let’s move on!

Prerequisites

In case you want to configure ovirt-engine and host on the same machine as I did, you can login to the server that ovirt-engine is installed and in the hosts file add something like this (replace with your own IP and FQDN).

10.0.0.3 	node1.homelab.home

This FQDN will be used at a later point when asked to provide an IP address for the configured Host.

Create the appropriate folders that will host the virtual hard disks, ovf files and ISO files. I am using /opt but you can use any other local or network path based on the configuration you have in mind,

:~$ mkdir /opt/{ovirt_data,ovirt_iso}
:~$ chown vdsm:kvm /opt/{ovirt_data,ovirt_iso}

We need an ISO file in order to setup our first VM after we finish the Cluster/Host configuration. For this step you can download your favorite distribution and either setup a Samba Share for /opt/ovirt_iso and upload iso files from your Windows machine or just scp them from your Unix workstation,

:~$ wget \
http://ftp.ntua.gr/pub/linux/centos/7.7.1908/isos/x86_64/CentOS-7-x86_64-Minimal-1908.iso
:~$ scp CentOS-7-x86_64-Minimal-1708.iso \
root@10.0.0.3:/opt/ovirt_iso/<id>/images/11111111-1111-1111-1111-111111111111/

As a final step I have also added an entry like the one below to my local hosts file,

##
# Host Database
#
# localhost is used to configure the loopback interface
# when the system is booting.  Do not change this entry.
##
127.0.0.1	localhost
10.0.0.3	ovirt.homelab.home

so as to be able to login to my oVirt endpoint without having to remember the IP of the oVirt host.
(NOTE: If you want to get rid of those uggly host entries, I will start doing so after this tutorial, checkout my post on how to create your own local DNS found here!)

oVirt Data Center/Cluster/Host Configuration

If you are the type of person that doesn’t start copying commands from random tutorials and wants to have the background behind any of the configuration documented below, then have a look on the official Data Centers and Clusters documentation before proceeding.

If you are like me though, open your favourite browser and type your oVirt endpoint, in my case https://ovirt.homelab.home/ovirt-engine/. Login with the admin account which was created during the initial engine setup.

As you can see from the screenshot above the connection is secure because as I have already documented in my previous posts, I have configured my own Self-Signed wildcard certificate and replaced the default one provided by oVirt installation with my own! If you want to do the same, just have a look here and here!

After our successful login we are presented with the Dashboard. This is where we will find a general overview of the status of your Host, our Global CPU/Memory/Storage information and Cluster Utilization.

You will have all the time in the world to wonder around so for now let’s cut to the chase!

From the left panel go to Compute -> Data Centers, select the Default and Remove it! Trust me!

Click on New, choose a Name and a Local Storage Type.

After clicking Ok, it’s Cluster’s turn!

Add a Name, a Description and in the Optimization Tab choose Memory Optimization -> For Desktop Load which will allow you to overallocate memory and run more virtual machines.

Finally Enable Memory Balloon and click Ok.

Final step will be to setup a Host.

Type a Name, an IP Address (or an FQDN as pointed out in the Prerequisites Section), the root password for the host that is going to be configured (in my case the same machine that the engine is installed) and in the Advanced Parameters deselect the Automatically configure host firewall.

Once you click Ok, the engine will start installing apropriate packages to the configured host as shown in the Host Status.

You can also see the installation progress in the Events Menu.

Once the host is ready, we have a final chapter in order to complete the basic configuration needed for creating virtual machines and this is the Storage!

oVirt Data/ISO Storage Configuration

Directly from the official oVirt documentation:

oVirt uses a centralized storage system for virtual machine disk images, ISO files, snapshots and provides three types of storage domains:

Data Domain: A data domain holds the virtual hard disks and OVF files of all the virtual machines and templates in a data center. In addition, snapshots of the virtual machines are also stored in the data domain.
ISO Domain: ISO domains store ISO files (or logical CDs) used to install and boot operating systems and applications for the virtual machines. An ISO domain removes the data center’s need for physical media. An ISO domain can be shared across different data centers. ISO domains can only be NFS-based. Only one ISO domain can be added to a data center.
Export Domain: Export domains are temporary storage repositories that are used to copy and move images between data centers and oVirt environments. Export domains can be used to backup virtual machines. An export domain can be moved between data centers, however, it can only be active in one data center at a time. Export domains can only be NFS-based. Only one export domain can be added to a data center.

If you need more information head to the official Storage Chapter. For our case we will skip the Export Domain for now and will only create the Data and ISO domains.

Navigate to Storage and click on New Domain.

Choose Data as a Domain Function, Local on Host as Storage Type, add a Name and finally the Path which is already created in the Prerequisites Section, /opt/ovirt_data.

Do the same for iso domain but choose ISO for a Domain Function and change the path to /opt/ovirt_iso.

Check that both data and iso domains are Active, then click on the newly iso created domain and then choose Images Tab.

If everything is correct then you should be able to see the iso that we have already uploaded in the Prerequisites Section!

In case you don’t want to upload any iso file, you can also check the ovirt-image-repository domain which as the name clearly states is a repository that contains images from all major distributions (CentOS,Fedora,Ubuntu,Debian etc) that are ready to be exported as virual machines.

The first part of the tutorial is finished and oVirt is now configured and ready to host new kick-ass virtual machines!!

Final Words

Rome wasn’t built in a day, right?

Time to take a break and familiarize with your new infrastructure. Get confortable with all the available tabs and configuration options, before moving to Part Two which will show you how to create your first Virtual Machine and save it as a Template for future use!

Thanks for reading, until the next post and as Dr Wallace Breen says in Half-Life 2.. Be wise. Be safe. Be aware!

Welcome to My Homelab!

Dotfiles Management with Dotbot and Chezmoi!

Dotfiles Management with Dotbot and Chezmoi!

Summary

Dotfiles

Dotbot

Chezmoi

Automation

Final Words 💡

Track your personal tasks with the help of Docker and Kanboard!

Track your personal tasks by using Kanboard and Nginx-Proxy docker containers.

Summary

Background

Concept

Configuration

nginx-proxy

nginx-proxy custom.conf

nginx-proxy docker run

Troubleshooting

kanboard

Prerequisites

kanboard docker run

Final Words 💡

Create MariaDB and phpMyAdmin containers locally via Docker

Create MariaDB and phpMyAdmin containers locally via Docker.

Summary

Concept

Configuration

Networking prerequisite

MariaDB Container

phpMyAdmin Container

Access our DB via GUI

Final Words 💡

Amateur Stop Motion Short Film - Klouvi

This is how I proposed to my girlfriend!

Background 📝

The Set

The Actors

The Scenario

The Music

Building the Sets 🧰

Camera rolling 🎬

Montage 🎞️

Final Words 💡

Automatically create virtual machines via Ansible and oVirt API - Part Two

Automatically create virtual machines via Ansible and oVirt API - Part Two.

Architecture ✏️

Playbooks 📚

Create our first VM via Ansible

Playbook breakdown

Execution 🐧

Create VM ⏫

Shutdown VM ⏬

Final Words 💡

Automatically create virtual machines via Ansible and oVirt API - Part One

Automatically create virtual machines via Ansible and oVirt API - Part One.

Appendix 📓

What’s a(n) API / REST-API?

And what about Ansible?

Say to hello to my little Cloud-Init friend (with the voice of Toni Montana) !

Prerequisites 📝

Architecture ✏️

Configuration 🔨

Ansible Ad-Hoc 🏃

Ansible Playbooks 📚

Execution 🖱️

Intermission ☕

Pair Logitech MX Keys with Xubuntu 18.04 via Bluetooth

Pair Logitech MX Keys with Xubuntu 18.04 via Bluetooth

Summary

Pairing!

Console Time!

Conclusion

Create and Export Full oVirt VM Backups to NFS Storage Domain

Create and Export Full oVirt VM Backups to NFS Storage Domain.

Prerequisites 📝

Creating and Attaching NFS Storage Domain

Configure NFS in QNAP ( Optional )

Mounting NFS Share on CentOS7 (Optional)

Preparing and Adding NFS Storage Domain to oVirt