Lab #12: Virtualization automation and networking

As usual, update your clone of teaching/nswi177/2020-summer/upstream/examples repository. We will be using examples from vagrant subdirectory.

Update: if you are on Fedora, we recommend setting export VAGRANT_DEFAULT_PROVIDER=virtualbox into your .bashrc to use VirtualBox provider as the default.

We highly recommend to run this exercise on your machine. If you are planning to use unixadmin, please, follow these steps before continuing.

• Create directory /mnt/big/LOGIN (use your login!).
• Create vagrant.d directory inside /mnt/big/LOGIN.
• Create vbox directory inside /mnt/big/LOGIN.
• Create a symbolic link from your home directory to /mnt/big/LOGIN/vagrant.d named .vagrant.d.
• Run vboxmanage setproperty machinefolder /mnt/big/LOGIN/vbox

Hint.Solution.

If your installation of Linux is virtualized, it is probably easier to run this exercise on your host system directly. While nested virtualization is usually possible, the performance penalty can be inconvenient.

This lab is about management of virtual machines in the small. It will be useful for you later on when you will need to check that your software can be easily installed and run.

That is best tried on a fresh installation. But OS installation can be time consuming (even if you only plan to go with defaults all the time). Hence you may need some kind of automation that will give you easily a fresh installation of a Linux box. That you will later destroy.

Think of this as a bigger virtualenv. Virtualenv gave the illusion of clean Python environment, this lab will be about clean OS (Linux) environment.

We will be using Vagrant on top of VirtualBox.

We will first check that we have Vagrant installed. Try running

vagrant version

Before playing with Vagrant, let’s learn what is it good for. Citing from documentation,

Vagrant is a tool for building and managing virtual machine environments in a single workflow. With an easy-to-use workflow and focus on automation, Vagrant lowers development environment setup time, increases production parity, and makes the “works on my machine” excuse a relic of the past.

In other words: Vagrant allows you to define how the virtual machine should look (similar to requirements.txt we know from virtualenv) and thus allows you to share the configuration with other developers easily.

I.e. instead of stating “I used Fedora with pandoc and make installed”, you provide a script that creates virtual machine with these packages installed automatically.

As stated before: may not make much sense for small programs, but it is a time-saver for bigger ones.

And it even allows you to build a small network of connected machines, making testing of network applications possible even when offline.

Let’s create our first Vagrant box (machine).

Move into simple directory and execute vagrant up (this will take some time).

While Vagrant is upping your machine, open the Vagrantfile in your favourite text editor (note that you may want to turn-on syntax highlighting for Ruby).

We will not describe the syntax in detail, just go over the important parts.

We define a new machine with config.vm.define and we specify that we will be using generic/fedora32 base image. To speed-up the machine creation, you rarely create a completely clean install. Instead, there are prepared boxes that you can use as a starting point. We will not create our own box but only used the prepared ones.

These prepared boxes already contain some pre-installed software and are configured to work smoothly with Vagrant. For Linux boxes it means they have running SSH daemon and contain various guest additions for maximum performance.

Note that you should approach the prepared boxes as you approach any third-party library or software. There could be instances of boxes that contain malicious code (as there could be Python modules with the same kind of issues). But it is much more probable that your machine will be attacked through a web-browser vulnerability than through a Vagrant box (or a Python module).

Back to the Vagrantfile. We specify that if our machine will be using VirtualBox as the backend (Vagrant can work with Docker or even Amazon cloud too), we want only half gigabyte of RAM and one CPU. VirtualBox is the default provider so you do not need to specify anything else.

And that is actually all.

If vagrant up finished without any error, you should see somewhere the text Machine booted and ready! and you just launched your first automated virtual machine.

Our first machine is running. Open your VirtualBox GUI and check what machines are there. You should see something like simple_machine_NNNNNNNN_NNNN there. That is the machine we created via Vagrant.

It is running without GUI (after all, you do not need GUI for automation, right?) and you should see that it really has only 512MB of RAM and some 32GB of hard drive.

You can close your VirtualBox GUI now, we will not need it anymore for this lab.

Let’s now log-in to the machine we just created. Obviously, we will not use any graphical interface but instead SSH that allows automation (see where we are heading in this course?).

vagrant ssh

You should see [vagrant@fedora32 ~]$ prompt. The ssh subcommand of vagrant logs you as user vagrant into the virtual machine.

Let’s look around.

free -h
df -h
sudo fdisk -l

The first command shows amount of free RAM (among other things), the second one displays information about mounted disks (volumes). In both cases, the -h instructs the commands to use human-readable sizes. fdisk prints information about partitions of your hard-drives.

You should see that you have about 300 MB of available (it is not exactly the same as free, but close) memory and your root directory (/) is mounted to /dev/sda3 partition that is 15 GB big.

Hitting Ctrl-D will log you out from the machine (not surprising, right?). Note that the machine is still running.

On your own, upgrade the virtual machine and reboot it. Solution.

On your own, install MidnightCommander. Solution.

Let’s run vagrant status.

It shows that we have one machine running and that it uses VirtualBox.

You can stop the machine with vagrant halt and start it again with vagrant up.

Let’s remove the machine now. In Vagrant, it is called destroy.

vagrant destroy

This asks for confirmation, append -f option if you want to destroy without any prompting.

If you now open the VirtualBox GUI, the machine will be gone.

Often, you do not need the clean installation but a system with some packages installed etc.

Switch to install subfolder and open the Vagrantfile. You can run vagrant up again.

The file is almost the same, except now there are the lines

    machine.vm.provision "shell",
      inline: "sudo dnf --assumeyes --quiet install mc htop"

This so called provisioning is description what needs to be done when the machine is created. Here we will be using shell provisioner: basically we specify which commands we want to run once the machine boots.

In our example, we only install MidnightCommander and htop monitoring utility.

If you now SSH to the machine, you will see that mc is installed.

For 11 lines of code we have fully-installed Linux system with custom tweaks. And we can recreate the machine with simple vagrant up anywhere.

Note that we will be using only Linux boxes for our examples but it is possible to create even Windows boxes (they are rarely freely available because of licensing) and use also PowerShell to script the installation process.

Typically, Windows boxes have Chocolatey installed to make the installation simple. If you are a Windows user and do not know about choco, check it out. It is something that should be part of Windows as it makes installation of software much easier than the usual process.

On your own, create a new virtual machine and create a shell provisioner that installs httpd package and starts the httpd service (it is the famous Apache webserver). You can enable the service with systemctl too.

Note that we will need curl for testing. But that is already part of the base fedora32 image.

Hint.Solution.

Check that the webserver is actually running in your virtual machine.

Inside the machine, run wget http://localhost/ or curl http://localhost.

You should see that the HTML page has title of Test Page for the HTTP Server on Fedora.

If you want to know which network services are running on a machine you can try connecting to all of its ports to check which are opened.

Since that is often useful when checking availability of a machine, there is a tool called nmap.

Login to your virtual machine, install nmap and run

nmap localhost

It should print that ports 22 and 80 are opened (i.e. SSH and HTTP).

If you want to see more information, you can try adding -A switch.

nmap -A localhost

And if you run it under root (i.e. sudo nmap -A localhost) nmap can try to detect the remote operating system too.

By default, nmap scans only first 1000 ports. You can specify a different range with -p option.

nmap -p1-65535 localhost

This instructs nmap to scan all TCP ports (-p1-65535) on localhost.

Which web server is used on our GitLab?

And which one is on our University website?

Solution.

We will extend your solution from previous steps. Our target is a virtual machine with web server that can be accessed from the host machine (for convenience).

Look into apache subfolder. Notice how we have changed the inline script to a more readable form (basically, it is a here-doc syntax in Ruby (like """ in Python)).

Our solution also contains

config.vm.network "forwarded_port", guest: 80, host: 8080

that forwards the HTTP server port to 8080 on your host.

You have also probably noticed that we run two firewall-cmd commands to enable port 80 on the firewall.

Open http://localhost:8080 on your host to check that you can access the webpage from your host too.

Note that port forwarding may trigger a confirmation from your firewall (depending on your system configuration). If that causes the script to hang, terminate it with Ctrl-C, enable the firewall rule and re-provision (vagrant provision).

Create real index.html inside /var/www/html/ with your own content. Check that it is displayed on http://localhost:8080.

Make this index.html part of the provisioning.

Use vagrant provision to run only the provisioning script on an existing machine (it is part of vagrant up when the machine is created).

Solution.

We will deviate from the main topic a bit. We have seen the webpage in a browser or used cURL or wget to fetch it.

We will try to communicate with the webserver manually. Recall that HTTP is a text protocol so it shall not be that difficult. Note that most of the protocols today run inside SSL tunnel so by-hand testing is often a bit more complicated (though, cURL is extremely powerful tool).

Log into your virtual machine with Apache, install package telnet and run

telnet localhost 80

It specifies that you want to connect to localhost on port 80 (i.e. web server port). Note that we are connecting over plain HTTP. For HTTPS we would be using port 443 and we would need some kind of SSL wrapper around our communication.

After the welcome message, type the standard HTTP GET request.

GET / HTTP/1.0

Note that the second line is empty (i.e. hit Enter twice).

You should get back headers and the HTML code of the index page. Note what information you get back in the headers.

The virtual machine that we are creating with Vagrant are by default behind a NAT (the translation is part of the VirtualBox engine).

Therefore, the virtual machine has a private IP address.

Most distributions today use NetworkManager and we will be using it here as well. Note that ArchLinux Wiki page about NetworkManager contains a lot of information too.

NetworkManager has a GUI too (you probably used its applet without knowing about it), but we will focus on the command-line interface here.

nmcli

will display a lot of information about your current connection.

You should see line inet4 10.0.2.15/24 that tells you the IP address of the virtual machine.

Can you ping this machine from your host computer (i.e. ping 10.0.2.15)?

Solution.

A follow-up question: how are you connecting to the machine via SSH if you cannot ping the machine? Solution.

Switch to ansible subdirectory.

The example here is a bit more advanced and only serves as a demonstration that things can be structured a bit more with extra tools.

Instead of using a prepared shell script to do the post-installation tweaks, we can use special tools for that.

One of them is Ansible that is another tool for system management automation. You will need to install ansible first on your host for this example to work.

Look inside playbook.yml. It is a YAML file where we specify what we want to execute on the target machine. The package task can be used for software installation, copy is used for copying files to the remote machine.

If you run vagrant up, you should see quite a lot of colorful output as the machine is provisioned by Ansible. The result is the same as if we would run dnf install and scp, however Ansible allows you to keep it better organized.

Look into ~vagrant and check that mc is really installed.

If Ansible is something you want to learn more about, consider enrolling into NSWI106.

Also note that with Vagrant and Ansible it is extremely easy to test your system configuration. You first test it with Vagrant and once you debug it there, you only apply the (same) Ansible playbooks to the real machine and you can be pretty sure that you will not break things. Well, unpleasant things still happen but this reduces the chances of trivial bugs quite a lot ;-).

Switch to multi subfolder and run vagrant up there again. As usual, open Vagrantfile.

Do not forget to destroy other machines to keep your workspace clean.

The Vagrantfile now contains definition of two machines, one and two. Running vagrant up creates both machines automatically. You can check it with vagrant status.

If you run vagrant ssh, you will learn that you need to provide virtual machine name in a multi-VM environment.

So, let’s run vagrant ssh one and vagrant ssh two in two terminals.

What are the IP addresses of both machines?

Note that we have added vm.hostname to better distinguish the machines.

Solution.

We have already mentioned that Vagrant can emulate whole network environment for you. So far, we have two machines behind two NATs. Not very useful.

Switch to network subfolder where we have added a second network card to each machine. We have also specified IP address for this second card (i.e. Vagrant/VirtualBox runs DHCP for us on that interface).

Again, SSH to both machines and run nmcli.

Both machines now have two interfaces, one has the typical 10.0.2.15 IP address, the second one has address from the 192.168.177.0/24 network.

Run ping 192.168.177.102 from one and ping 192.168.177.101 from two to verify that the machines are directly accessible (use Ctrl-C to stop pinging).

With two interfaces, we can try to shut one down. Stay logged-in to both machines and run nmcli connection in one. Start ping 192.168.177.101 in two (use a different terminal so that ping can continue to run).

nmcli connection shall show three connections (eth0, eth1 and Wired connection). eth1 is the one on private network (verify it by running nmcli).

Let’s bring that interface down.

sudo nmcli connection down UUID-OF-THE-ETH1-CONNECTION

Look into the other terminal. The ping should stop printing any more messages as the packets are lost.

Bring the connection back again.

sudo nmcli connection up UUID-OF-THE-ETH1-CONNECTION

The ping will be alive again. You shall notice a space in the sequence numbers that account for the lost packets when the interface was down.

Note that if you bring down the other interface (eth0), your SSH connection will stop working because that is the interface that is used by Vagrant for SSH connection. The easiest way out is to vagrant reload the machine. If you do not want to do that, you can try logging through the second machine or logging in inside VirtualBox GUI.

We are slowly getting to the point where we can use Vagrant to simulate a simple network and test network applications in it. Note that if you want something more complex, you may need to resort to more powerful tools such as VDE and QEMU.

Switch to nm folder, start the machines and open Vagrantfile (i.e. as usual).

You will notice that the private network interface has auto_config set to false. Vagrant prepares the network but does not initialize the interfaces (i.e. the wire is plugged but no connection was attempted).

We will now set the IP addresses manually via nmcli now. For the fun of it, we will use different ones than the ones in the Vagrantfile.

Why there has to be IP adresses set in the Vagrantfile even with aut_config: false?

Execute the following command in one that adds connection named eth1-static that will be on ethernet (type) interface eth1 and will use IP address 192.168.177.201 with /24 netmask.

sudo nmcli connection add \
    con-name eth1-static \
    ifname eth1 \
    type ethernet \
    ip4 192.168.177.201/24

To bring that connection up, just run

sudo nmcli connection up eth1-static

On your own, configure machine two to use 192.168.177.202/24 and check that you can ping between the machines.

Solution.

Check that the network connection is okay by connecting from one to two with SSH (vagrant user has password vagrant).

On your own, prepare Vagrant configuration with three boxes. Box central will be connected to two different private networks. In each of the (private) networks will be one machine (name them web1 and web2). Both web1 and web2 will have Apache installed (and enabled/running). Write a shell script that will be automatically copied to central that prints status of both web servers. Hint.Solution.

Do you want more exercises related to Linux administration? Enroll to Linux administration (NSWI106) in winter semester!