9) Validating Task (0 Points but Mandatory, No Deadline)

  • Once you feel your infrastructure is ready for thorough check (including the Advanced Task) please contact me and arrange a final zoom call where your grade will be awarded and questions asked. Be sure that:
    • You are done with all tasks.
    • You can receive/send emails from/to other groups.
    • You narrowed down what is the contribution of every team member.
    • You have webcam working so we can see each other and I can validate your identity.
    • This list can be further extended…

8) Advanced Task (40 Points, Deadline 10.01.202114.02.2021)

Task 8 description

  • The last task is going to be unique for every team. So far we did a basic setup of various technologies. Now we will do the opposite:
    • Choose an area you are interested in and propose what will you deploy/improve in your infrastructure. It has to be a non-trivial task. Prepare a proposal and implementation plan and present it via zoom to get an approval. Please contact me for arranging time and date. I prefer to do it no later than Tuesday 22nd.
    • Do the implementation and create a screencast where you will explain the technology and your implementation. It should be similar to the tutorial for an advanced user. Upload your recording to the home directory of the group user with the smallest UID and name it screencast.mkv (you can use different video container than mkv). It should be 10 to 20 minutes long. (No hard constraints.)
    • Possible topics for inspiration (topics can be combined):
      • Finding vulnerabilities in the whole .una network.
      • Deploying Spark for Machine Learning.
      • Automated setup via ansible or similar.
      • Kubernetes + docker demonstration.
      • Deploying ceph.
      • ZFS or btrfs advanced usage.
      • Monitoring deployment (zabbix, icinga, nagios, own solution…)
      • IPv6, DNSSEC deployment.

7) Storage (10 Points, Deadline 18.12.2020)

Task 6 recap

  • To be done, as well as grading of previous tasks. Will be done no later than Friday 11th on Sunday 13th.

    • Grading done on Sunday 13th.
    • Recap will be published on 18th, since some groups are fixing their solutions.

  • In task 6 you added MTA (postfix) and most probably also MDA (dovecot) into your infrastructure. The key think was also to add MX record into your authoritative DNS server, so other people know where to send emails for your domain. Most groups did this task without any hassles.

Task 7 description

  • Create a VM storage.<groupname>.una which will serve as a storage server for your network with roughly following configuration:
    • 10 drives, 1GB each.
    • Deploy MD-RAID on all drives (use RAID6, also experiment with different RAID mode)
    • Encrypt the whole RAID block device with LUKS.
    • On the encrypted block device create LVM.
    • Create 3 logical volumes on top of LVM.
    • Format each logical volume with a filesystem (ext4, xfs…)
    • Fill up the array with data.
    • Simulate drive failure, e.g. shutting down VM, removing one 1-2 drives and booting up.
    • Try to mount the degraded array.
    • Try to add new 2 new drives and restore fully working array.
    • Do not forget to keep the VM running and to deploy teacher’s ssh keys there.
    • Your created volume has to be mounted in /mnt/big.

6) E-mail (10 Points, Deadline 27.11.2020)

Task 5 recap

  • Task 5 was about setting up recursive and authoritative DNS servers. As you already know, their job is completely different. Recursive server should be accessible just for trusted clients in your network because it can be used for DNS mirroring attack. Contrary, authoritative servers have to be accessible world-wide, because they provide information about your domain to other recursive servers. Hence it is recommended to run them on separate machines (different performance/security requirements…).

  • Furthermore you had to setup bird to distribute your routes to your colleagues routers via the backbone router. It was a straightforward application of the template of configuration file I provided.

Task 6 description

  • Right now we have a practical experience with setting up simple networks and, most probably, we found a lot of blank spots in our theoretical knowledge. That is actually very good. You have to first see the blind spot before you can fill it.

  • Task 6 is going to be about creating an e-mail server for our domain. You will need to remind yourself what MTA, MDA and MUA is. Furthermore, it is good idea to refresh the knowledge how protocols related to e-mail works (smtp, imap, pop3, dns).

  • Your mandatory task is to combine two softwares - postfix and dovecot - into complete mail server solution for your domain. Start gathering information on Arch Wiki (for both, postfix and dovecot) and try to find simple and elegant solution. Do not use database or some sort of web application. More specifically:

    • Set MX records appropriately.
    • Start with default postfix settings and try to send and receive e-mail via telnet to it. Watch postfix logs!
    • Install dovecot and use it as MDA.
    • The final state is following: You can send e-mail to login@groupname.una and receive it. (Of course, just withing our network.)
    • At the end, you can use port forwarding and actually use graphical client like Thunderbird for sending and receiving e-mails.
    • I will repeat it once more, try to set it up as simply as possible.

5) Routing and DNS (10 Points, Deadline 13.11.2020)

Task 4 recap

  • Task 4 was just about using already known technologies (vde switch, qemu) in slightly different scenario where you created router to your subnetworks.

Task 5 description

  • First of all install ssh on all your machines. From now on it is mandatory to have installed ssh on all machines and use it as your primary way how to control your VMs. Every VM has to have user teacher with following public key and teacher has to have sudo without password ability.

  • Task 5 is about finishing basic networking setup, i.e. properly setup routing and DNS servers. To be able to access your VMs inside your subnetworks from for example machine c you have to propagate the routing information. I.e. what is the next hop for the IP range of your 10.0.X.0/24 and 10.0.100+X.0/24 networks. You already know that routing protocols are used for this (OSPF, RIP, BGP……). We will use BGP protocol and bird daemon. It means, that your router will have bird installed and will send the routing information about your networks to the main router which will work as a routes concentrator. The main router will send back all the routing information it acquired so far. Because of configuring bird from scratch is too difficult here is configuration template. However read the documentation first and be sure you know what all the options mean. Validation:

    • ip r shows imported routes to other groups networks
    • You can ping machines in your networks from c,d,e,f.
    • You can ping from machines in your network to Internet.
    • You can ping machines of other groups.

  • The second task is to install recursive DNS server accessible just from your network. The recommended one is unbound and you can forward your queries to 10.0.0.1. Validation:

    • You can query your recursive DNS server for any domain on Internet. I.e. drill cuni.cz @<address of your recursive DNS server>

  • The third task is to create an authoritative DNS servers (primary and secondary NS) for your domain. We recommend to use nsd. Your domain will be groupname.una . .una is totally artificial TLD and will work just inside our network. groupname is the unix group you are part of. Authoritative nameserver has to be accessible also from outside of your network. Following records are mandatory:

    • rt.groupname.una. has to have A record 10.0.0.X

    • gw.groupname.una. has to have CNAME record rt.groupname.una.

    • ns1.groupname.una. has to have A record 10.0.X.10

    • ns2.groupname.una. has to have A record 10.0.100+X.10

    • PTR records for all A records

    • Make ns1 primary DNS and ns2 secondary DNS and configure AXFR zone transfer between ns1 and ns2.

    • Validation:

      • You can query your authoritative DNS server for records of your domain. I.e. drill rt.groupname.una @<address of your authoritative DNS server>

4) Connecting Together (10 Points, Deadline 30.10.2020)

Task 2 and task 3 recap

  • In previous tasks you made yourself familiar with the environment and got the feeling how to solve completely unknown and new task on your own with typical sources of information for junior Linux administrator (Arch Wiki, manpages and google). Note that tasks were trivial BUT…! Even with trivial systems related task, sometimes you cannot make it in a moment and you spend tens of hours on it. This is because in real systems world huge amount of technologies interact together and it is important to actually understand them. This is especially true for networking topics. This means, it is completely expected you will be stuck for several hours figuring out what is going on. This is a must! During this struggling part you actually validate your current understanding of systems and you fill the blank spots in you knowledge or fix wrongly understood concepts. Let us recapitulate tasks 2 and 3 and mention most common problems you most probably had:

    • Task 2 was about to install Arch Linux in QEMU running on remote computer. To run qemu you just execute qemu-system-x86_64 with arguments -cpu,-enable-kvm,-m,-drive,-cdrom,-display vnc mentioned in the task description. The Arch Linux installation is done just by following the step-by-step Arch Linux installation guide. Probable problem was to make the VM running even if you logout. You could use -daemonize parameter to qemu or run it under tmux or screen. Sometimes you forgot to kill the existing VM and ran the VM with the same drive. Hence you most probably corrupted your drive filesystem because 2 computers were writing to one drive.

    • Task 3 was about to create a virtual switch and connect multiple machines to it. This is done by running vde_switch with parameters -mod -sock. Manpage of vde_switch is very clear and short and you should have been able to come up with these parameters on your own. To connect a VM into the switch you had to open qemu manpage, search for “vde” keyword and find -nic vde argument. Then you just set VM IP addresses with for example ip command or any other tool like systemd-networkd (personally recommended) described in Arch Wiki. Probable problem was having the same MAC address in every VM you connected to the switch. You already know from your first year that is it not possible for machines connected to the same network segment.

Task 4 description

  • Compute your group X number with following algorithm:

  • On machine c find UIDs of all team members.

  • Let Y be the minimal one.

  • X is the decimal number composed from two last digits of Y.

  • X += 10

  • E.g. UIDs 1009, 1010, 1011 -> X = 19

  • E.g. UIDs 1019, 1020, 1021 -> X = 29

  • Create a VM which will be part of the backbone network and will provide a gateway for your networks created bellow. The backbone network is composed just from group routers and the router which is connected to the Internet. The network configuration of your VM is following:

    • MAC: 52:54:00:36:X:01
    • IP: 10.0.0.X/24
    • GW: 10.0.0.1
    • DNS: 10.0.0.1
    • Connect it to the backbone virtual switch which is located at /tmp/vde-backbone.sock
    • Now you have to be able to ping 10.0.0.1, ping routers of other groups and ping addresses on Internet. Try nmap 10.0.0.1/24 to scan for all the computer on that network.

  • Create 2 networks, 10.0.X.0/24 and 10.0.100+X.0/24 respectively. This means to create 2 switches and connect at least 1 VM to each of them. The network configuration for the VM in the first network is following:

    • MAC: up to you
    • IP: 10.0.X.2/24
    • GW: 10.0.X.1
    • DNS: 10.0.0.1
    • Connect it to the first created switch.

  • Add 2 more networking cards to your gateway machine and connect it also to your just created networks. The network configuration of these 2 interfaces could be following (only interface for the first network is shown, the second one is analogous):

    • MAC: up to you
    • IP: 10.0.X.1/24
    • Connect it to the first created switch.

  • The expected state after this task is that from the gateway you can ping everywhere (Internet, other groups gateways and VMs in your 2 networks). The machines in your networks can ping just the gateway. This is because routing is not working now and we will work on it next week.

  • A few advices at the end:

    • It is good idea to automate everything - try to do things in an elegant way, designed such that it is easy to automate it and write a shell script to do it. It is not just about this course but about all work you do on computers. It is very time consuming and error-prone to do things by hand if they can be done by computer without any mistake. It is worth spending 5x more time on automating the task because it pays you off with 1) automatization; you can now run it several times with different inputs 2) no typos; the result will be always correct 3) different point of view on the problem; when you try to automatize things you need to organize your thoughts first. During organization of your thoughts you will discover the real structure of the problem. This is incredibly important.
    • For networking setup I recommend to use systemd-networkd. It is very easy way how to create persistent networking setup.
    • Sources where to find information are still (in this order) Arch Wiki, manpages, google and lecturer.
    • Learn how to use ssh efficiently because you will use it a lot. Learn how to use ssh keys for password-less login, how to write ssh config such that you can just type ssh c and you connect to machine c. Also learn how to use ssh to connect to machines which are not directly accessible. I.e. your gateway VM is not accessible from the Internet, but machines {c,d,e,f} are. See option -J of ssh or proxyjump option in ssh_config.
    • Some nice utilities good to know about: terminator, parallel, pssh, screen, tmux, ranger, mc, htop, top, iostat, iotop, iftop, df, free, uptime. Feel free to install them on your VMs.
    • As always, contact me in any troubles! We can solve your issues via e-mail or zoom call.

3) Virtual Switch and Real Networking (10 Points, Deadline 23.10.2020)

  • Last time you set up one isolated machine in the remote environment. Let me first add some comments. When you deploy virtual machine you have to solve similar problems as when you deploy real hardware. You need to have some display to see the output, you have to use some input device like keyboard and you need connection to the network. Display and keyboard can be easily done with vnc which is a protocol made exactly for this. You can actually install vnc server on your computer and control it from remote machine via some vnc client. As you probably noticed qemu contains its own vnc server, hence you can control you new VM remotely.

  • Having just single VM can be useful when you do operating system development. It is not good idea to load the kernel to the same computer you use for its writing. Also when you need to emulate some different CPU architecture like ARM, MIPS or even some different microarchitecture like Intel Haswell. Or when you really need Intel 486. To take a look what architectures are supported do ls /bin/qemu-*. To learn what microarchitecture of x86_64 is supported write qemu-system-x86_64 -cpu help.

  • The networking setup from your machine was “it just works” without any options. It is because qemu is user friendly and provides dhcp server and “gateway with NAT capabilities” (actually SLIRP). We will fix this trivial setup by setting up our real network. This means that you have to add networking card into your VM and connect this card to some networking switch. How can we do this in the virtual environment? We just need virtual networking switch. We will use vde_switch.

  • Your task is following:

    • Add networking card to your VM. Read qemu man page how to do it.
    • Create a virtual switch with vde_switch.
    • Connect the VM to your vde_switch. qemu has direct support for vde_switch hence you can connect the VM to the switch via socket with qemu parameters.
    • Create another VM and connect it to the same switch. You can just copy the image you use and start another VM. Don’t start more VMs with the same image (hard drive).
    • Configure the network on both VMs to be able to ping between them.
    • Try (not mandatory) to spawn even more machines and create larger network.
    • Try (not mandatory) to create more switches and create multiple networks.
    • Try (not mandatory) to create machines with more networking cards and connect them to multiple networks.

  • Furthermore please create a file ~/.group_members in your home directory on machine c with alphabetically sorted comma separated list with 4 columns of your group members logins. I.e. login1,login2,login3,login4 in case your team has 4 members and login1,login2,, if you have just 2 members.

  • Your points for the first task will be published next week. If you were able to install Arch Linux on the machine and boot it up, you have 10 points.

  • And as always, if the class is too difficult for you, please contact me. If the class is too easy don’t worry, it will become intense soon.

2) Arch Linux Installation (10 Points, Deadline 13.10.2020)

  • Last time you warmed up with refreshing what virtualization is and how you can use it on your own computer. This is good for local testing but usually you will deploy virtual machines on some remote and very powerful server in the data center with proper cooling, fast Internet access and physical security. We have four powerful servers like this and we call them c, d, e and f. These machines will be used for all following tasks in this course. You will get login credentials for each of them by mailing list.

  • We hope that study groups work well for you and you are able to meet (at least) on weekly basis to discuss all the stuff you need to learn to finish tasks in this course. If you cannot figure something out after long brain-storming or have any other trouble with organizational or technical issues, feel free sending me e-mail. Oh, and one more think; try to be active inside the group, at the end you will need to settle down on what portion of work every member of the team did.

  • First, let me describe the path we would like to follow in this course. In the roughly first half we will do tasks more low level to be able to understand all the details in systems administration. I.e. we will install our virtual machines by hand, we will install Arch Linux which does not have installer and you have to get familiar with all the necessary steps you have to do for system installation. We will setup all the applications by editing the configuration files by hand to get familiar how it really works. Later on we will focus on more high level approach and we will use applications which will do most of the things for us. I.e. we will use orchestration mechanism to deploy multiple virtual machines at once, we will use declarative approach of configuration where we will just describe the state of our servers and network and everything will be done by help of some application. With this approach you will be able to effectively deploy big infrastructure in the infrastructure-as-a-code manner. On the other hand you will precisely know how to fix any low level detail in your system (thanks to the first part).

  • Let us start with the low-level task. The task is to install Arch Linux as a QEMU/KVM guest on one of our servers. It doesn’t matter if it will be server c, d, e or f, because for now it will be just one virtual machine for every group. Everyone should be able to login via ssh to the servers with credentials from the mailing list. QEMU is an application for virtualization like VirtualBox. However it is a bit better on Linux, because it can use KVM (Kernel Virtual Machine) - the hypervizor in Linux kernel instead of external one like VirtualBox does. QEMU can be also used nicely from command line and has very detailed manpage which I recommend to consult during task solving. You will find there all the needed parameters for VM execution (-cpu,-enable-kvm,-m,-drive,-cdrom,-display vnc). Please use just 1 CPU, 4GB RAM and at most 8GB of drive space. The important things for VM creation are:

    • Use qemu-img to create a drive. Please use raw format and NOT qcow2. Performance with raw is way better.

    • Use VNC display output and connect to it with help of ssh local forwarding from your personal computer. See manpage of ssh.

    • Don’t specify any NIC (network interface controller). QEMU will automatically add one in so called SLIRP mode. It will make TCP and UDP Internet connections work for the installation. For more details see QEMU/Networking.

    • Create a shell script for machine execution, it will be easier to run it repeatedly.

  • When you can execute your VM you will have to install Arch Linux. Follow the Arch Linux Installation Guide and be patient. Read carefully and discuss stuff inside the study group. If you don’t understand something, try to learn it. Very good source of information is Arch Linux Wiki, man pages, google, mailing list and you can also send us e-mail if you want alternative explanation. Zoom calls are also possible. But remember, most of the knowledge you should get from the study group learning together from mentioned materials. And realize, this is not just about system installation but also about learning tools you have to use for all the actions performed during the installation and later on during system administration.

  • This task can be perceived as very difficult but after you overcome all the problems, you will learn plenty of new and very useful tools and technologies. It is still just operating system installation, right? Good luck and enjoy the learning path from the beginning. The result of the task is running one installed Arch Linux VM in QEMU per team on any server you choose. I.e. operating system has to be installed to drive, running just archiso is not sufficient and the virtual machine has to be running all the time (even after you logout).

1) Warm Up (0 Points but mandatory, Deadline 6.10.2020)

  • Form a study groups with three members in each group. Feel free to use course mailing list for finding a partner. The last group can have different number of members (two to four). You will arrange regular weekly video-call meetings among group members, hence it is good idea to be in the same timezone.

  • Among the group members form a video-meeting and discuss what a virtualization is and what software you can use for it. Then learn together how to use VirtualBox and how to install Ubuntu Linux as a guest. Finally install Ubuntu Linux guest under VirtualBox hypervizor.