Lab #2 (February 26 - March 1)

Labs: 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11.

In this lab we will start on learning the most effective way to control your Linux machine – via command-line interface.

The lab starts with a bit motivation why we should care about command-line interface at all. Then we do a short recap of what is a filename and a file path and continue to a brief explanation of the Linux file system hierarchy.

After this more theoretical introduction we dive into using the terminal. You will learn how to navigate through directories and how to display file contents in the terminal.

Preflight checklist

Please, make sure that you have completed the following before moving on.

We will need it for the rest of the course.

Why shall we use the command line at all

First of all, it is explicit and precise. There is no danger that a user would use a different style or a different set of taskbars when describing an action to make. Using an exact command leaves no place minimizes misunderstanding.

Next, it is also rather fast. Once we start comparing the possible speed of a mouse when clicking on icons versus the speed of keyboard keystrokes, the keyboard would be a clear winner (assuming in both approaches we would know what we want to do).

And partially connected with the above reasons, it is also easy to save the typed commands into a file and re-run later.

Such (text) files are called scripts in the Linux world. They can simply be a list of commands to execute, but they can also consist of loops and conditions to execute more sophisticated actions. We will devote several labs to these.

And from the machine side, it is also extremely efficient. Especially when we talk about remote access over an unstable connection. The difference between sharing even a small 800×600 screen vs. sending keystrokes is substantial. Managing Linux servers over a flaky 2G connection is possible, managing a server offering only GUI over such a poor connection is out of question.

The downside is that an exploratory approach is a bit more complicated. In a GUI-oriented system, you can try clicking on various icons and explore the system relatively easily. On the command line, you need to learn a lot of commands to use the system effectively. However, you are not forced to remember every single option for every single command, as there is an easily accessible comprehensive documentation (more on this topic in later sections).

Actually, it is exactly the same as with any programming language: you need to know the API before being able to write a program. Except in Linux, the API are not functions in a typical programming language, but rather complete programs.

The fact is that the shell we will be using was born about 50 years ago. But it is still used today. It may mean that we were not able to come with anything better for quite a long time. But, more likely, it may suggest that the pros are worth it.

The beginning might be difficult, but you will not regret it in the long run.

From the practical point of view, using the command line is somewhat similar to using Python in an interactive session. You type commands, you can edit them and once you are happy, you execute the command by hitting <Enter>.

From now on, our interactive with the system would mostly look like this.

Do not be scared, though. Linux often trades eye-candy for efficiency. Try to approach it more like a new programming language: you need to learn (and remember too!) a bit about the constructs and the standard library first before writing big programs. In Linux it is the same. And it is definitely worth it if you mean it with computers seriously. We mean it. Seriously.

Filenames and paths

As a matter of fact, you probably know all of this. Feel free to skim over this part if that is so. We have highlighted the important parts for you.

Basic terms

In our text, we will use the term filename to refer to a plain file name without any directory specification. Filenames are what you see when you open any kind of file browser or in an e-mail with attachments.

Note that on Linux we prefer to use the word directory over the term folder. Folder usually refers to something virtual that is not present on the file system (i.e., as if not physically existing as-is on the hard drive). Therefore, we can talk about folders in your e-mail client or in a cloud storage.

Path means that the filename is prefixed with some kind of directory specification.

Linux does not have any notion of disk drives: everything is found under a so-called root which is a single forward slash /.

Relative and absolute path, working directory

A path can be relative or absolute. When a path is absolute, it refers to a specific file on a given computer. No matter what directory you are currently in. A relative path is always combined with another directory to form an absolute path.

On Linux, each absolute path must start with a slash; if a path does not start with a slash, it is treated as a path relative to the working (current) directory. Intuitively, the working directory refers to the directory that you just opened in the file browser.

Special directories

A path can contain references to parent directories via .. (two dots). For example, relative path ../documents/letter.odt means that the file is located in directory documents that is one level up from the current directory. Assuming we are in directory /home/intro/movies (note that this is an absolute path), the absolute path for the letter.odt would be /home/intro/movies/../documents/letter.odt which can be resolved (shortened) to /home/intro/documents/letter.odt.

Apart from the special directory name of .., there is also a special directory . (dot) that refers to the current directory. Therefore ./bin/run_tests.sh refers to a file run_tests.sh in a bin directory that is a subdirectory of the current one (i.e., it is exactly the same as bin/run_tests.sh). Later, we will see why the dot . directory is needed.

Filename extensions

Linux does not enforce or restrict the use of an extension in the filename (e.g., .zip or .pdf). In fact, a file can exist without it and it can even have multiple ones. A typical example of multi-extension file is file.tar.gz which denotes that the file is a tape archive (.tar) later compressed with gzip.

Hidden files

File names that start with a dot . are by default hidden.

Everything in Linux is a file

We have started our Linux exploration with paths and filenames for a very good reason. Virtually everything in a Linux system is a file.

You already know that there are plain files (e.g., the letter.odt file we mentioned above that represents a word processor document) and directories (for organizing other files).

Be aware that the word file in Linux can refer to both normal files as well as directories, i.e., a directory is a file.

There are also other special types of files that can represent hardware devices, systems state etc. We will talk about these later.

Fire up your terminal, please

Enough of theory. Please, locate the Terminal program and start it. Depending on your environment, it will be either Terminal, Console, or perhaps even Shell (although, technically, shell is the program running inside a terminal emulator).

We recommend you spend some time configuring the look of your terminal, such as having a nice font family and a reasonable font size. You will be spending quite a lot of time with it, so make the experience nice. Below are some possibilities of what you might get :-).

You will see something like [intro@localhost ~] and a blinking cursor after that. This is called a prompt and if you see it, it means you can enter your commands.

The prompt is displayed by your shell which is an interpreter of the commands you enter. The shell is actually a full-fledged programming language, but in this lab we will use it to launch very simple commands only.

Type uptime and start this command by submitting it with <Enter>. Until you hit <Enter>, you can easily edit the command. Shortcuts such as <Ctrl>-<Arrow> for jumping over words work, too.

As we already mentioned, the experience is somewhat similar to an interactive Python session (editing etc.).

Quick copy-paste (and forceful program termination)

Whenever you select a text in the terminal with your mouse, it is automatically copied. This text then can be inserted by simply clicking the middle mouse-button (or the wheel).

Note that these are actually two distinct clipboards – the special one bound to the middle mouse button and the one bound to <Ctrl>-C (<Ctrl>-<Shift>-C) and <Ctrl>-V. In graphical applications, <Ctrl>-C and <Ctrl>-V work as usual.

Closing the terminal

To close the terminal, you can simply close the whole window (e.g., via mouse) but you can also type exit or hit <Ctrl>-D on an empty line. Because we are moving away from needing mouse (in a sense), you should prefer <Ctrl>-D ;-).

Debugging issues

When running programs in a terminal, never paste their output as a screenshot. Instead, select the text (including the command you have run) and paste where needed.

Recall that selected (marked) text is automatically copied: simply clicking the middle mouse will paste it where needed.

We will also stop inserting here screenshots of the terminal from now on and paste only the output (though you should always run the command by yourself to see what it does as first-hand experience).

For pasting into our Forum enclose the text in the fenced block ``` to preserve the monospace font.

```
ls nonexistent
ls: cannot access 'nonexistent': No such file or directory
```

Navigating through the filesystem

We will start with simple navigation through the file system. Two basic commands will get you through.

Directory listing with ls

The ls command lists files in the current directory.

Executing ls shall produce something like this:

Desktop    Downloads  Music     Public     Videos
Documents  gif.md     Pictures  Templates

Now run ls -l. That is, ls and -l separated by a space. Here we are calling the program ls and giving it an extra argument, -l. Because the argument starts with a dash, it is actually a so-called option (or switch) that instructs ls to modify its behaviour. Now ls prints something like this:

total 4
drwxr-xr-x. 1 intro intro    0 Feb 10 13:43 Desktop
drwxr-xr-x. 1 intro intro    0 Feb 10 13:43 Documents
drwxr-xr-x. 1 intro intro    0 Feb 10 13:43 Downloads
-rw-r--r--. 1 intro intro 1022 Jan  9 18:13 gif.md
drwxr-xr-x. 1 intro intro    0 Feb 10 13:43 Music
drwxr-xr-x. 1 intro intro    0 Feb 10 13:43 Pictures
drwxr-xr-x. 1 intro intro    0 Feb 10 13:43 Public
drwxr-xr-x. 1 intro intro    0 Feb 10 13:43 Templates
drwxr-xr-x. 1 intro intro    0 Feb 10 13:43 Videos

The -l turned on the so-called long mode where more details about each file are printed.

We will return to the meaning of some of the columns later on, deciphering the columns for the last modification time and the file size is straightforward and sufficient for the moment.

Changing working directory with cd

The cd command allows us to change the working (current) directory (hence cd). It takes one argument – the directory we want to switch to.

Thus, cd Documents would move us to the Documents directory.

Execute ls here. What is the output? Answer.

How would you move back to the parent directory? Answer.

What will do the following command?

cd .

Notice that the command prompt changed whenever you switched to a different directory.

By default, it shows only the last component of the path. To show the full (absolute) path, we need to run pwd (print working directory).

It will show something like

/home/intro/Videos

Tab completion

Typing long filenames can be cumbersome and making typos is annoying. Shell offers tab completion to help you with this.

For this example, we assume you just launched your terminal and ls prints Desktop Documents Downloads Templates etc.

If we want to change to directory Templates, start typing cd Te and hit <Tab>. Unless there is another filename (directory) starting with Te, the name shall be completed for you and should read the full cd Templates/.

Submitting the command with <Enter> would switch you to the directory as we would expect. Try it and come back to this directory again.

Now, let us switch to Documents directory. For this example, type cd Do and press <Tab>. There are two directories with this prefix: Documents and Downloads. Because the shell cannot know which one you want, it does nothing.

However, pressing <Tab> for the second time shows the possible matches and after typing c (the next letter), <Tab> can finish the completion.

As an exercise, what happens if you type cd and hit <Tab>? Answer.

Type just c (as in cd) and hit <Tab>. What happens? Answer.

Home directory

You probably noticed that when you start your terminal, the directory name you see there is just a ~ even though it should read intro (or your username on that particular machine) as that is the last component from pwd.

However, the path /home/intro is your home directory and has a special shortcut of tilde ~.

Furthermore, if you just run command cd without any extra arguments, it will change the directory back to your home.

Text user interface tools

While the use of purely command-line tools such as uptime, ls or cd is cool and extremely useful for scripts, there are also occasions where a more interactive approach is faster.

In this sense, Linux typically offers three layers you can choose from. From a fully graphical one called Graphical User Interface (GUI), over a tool with a Text-based User Interface (TUI) to a pure Command-Line Interface (CLI). Every of these can be useful, depending on the circumstances.

By a textual user interface we mean what is offered by Midnight commander or Ranger.

Midnight commander

Run mc and navigate through the files as you have done with ls and cd.

The numbers at the bottom refer to your function keys for typical file operations (e.g., F5 copies the file).

Note that in a typical setup, MC offers two panels with file listing, you switch between them via <Tab> and, by default, copying is done to the directory in the other panel.

MC is a quite powerful tool as it can inspect file archives, show files on a remote machine, etc.

We will briefly mention the most important things that you can do with it. Do try them :-)

• <Insert> allows you to select multiple files for deletion/copying.
• <F3> displays file contents.
• <F4> offers a simple text editor with syntax highlighting.
• <+> allows you to enter a filename mask to select multiple files at once (we will talk about this more later in the Wildcards section).
• <Ctrl>-o hides the panels and temporarily switches you back to the shell. Perfect for running commands without leaving MC.

You can quit MC with <F10> or via a menu (activated by <F9>). Note that some terminals capture <F10> to activate their window menu (but this behaviour can be changed in Preferences of the terminal application).

Editing file contents

You probably noticed that the Development submenu contains several graphical text editors that you can use to edit the source code. However, it is also possible to edit files in TUI editors.

If you are asking why to learn another editor (if you are already happy with some of the graphical ones), here is the answer. On some machines, you may not have access to GUI at all. Recall that we talked about remote access earlier: in that case you will have only TUI available (and you will often need to edit files on the remote machine). Some users thus never use GUI editors at all, the reasoning is that it is much better to learn (and customize) one editor properly, and that editor is a TUI-based one.

On our disk, you will find Emacs, Joe, mcedit nano and Vim.

Each has its own advantages and it is up to you which one you will choose. Note that mcedit is probably the closest to an editor you may know from other systems. joe and nano are small ones, but perfectly suitable for script editing that we will be doing the most. Both emacs and vim are extremely powerful tools that can do much more than just edit files. However, they require a bit of time investment before you can start using them effectively.

If you are new to Linux, we would recommend you to use mcedit (either using it directly or when editing files in Midnight commander) and come back to the other ones later on for a final decision of THE text editor of your choice.

All of these editors can be launched from the command line, giving it the filename to edit as a parameter (e.g., mcedit quiz.md).

Setup exercise files

For the following, you will need to have the same list of files as we have.

Please, download this archive and unpack its contents. If you want to download it from the command line, you can use wget URL, otherwise use whatever browser you like. Use Midnight commander to copy the unpacked content to your home directory. Hint.

You should see a directory nswi177-lab02 on your disk.

Shell wildcards

We assume you have the directory nswi177-lab02 prepared as described in the previous section.

So far, we used ls to display all files in a directory. If we are interested in only a subset, we can specifically name them on the command line.

Move to the directory where you have unpacked the nswi177-lab02.tar.gz. You should see the following files:

a/ b/ c/ one.txt two.txt three.txt four.txt

If we want to list only details about the text files, we can execute

ls -l one.txt two.txt three.txt four.txt

-rw-r--r-- 1 intro intro 0 Mar  3 13:38 four.txt
-rw-r--r-- 1 intro intro 0 Mar  3 13:38 one.txt
-rw-r--r-- 1 intro intro 0 Mar  3 13:38 three.txt
-rw-r--r-- 1 intro intro 0 Mar  3 13:38 two.txt

Doing that for more files would not be very elegant, but the shell offers so called wildcards to specify multiple files at once. Thus, the same output can be obtained by running

ls -l *.txt

It is essential to note that ls (or any other program for that matter) will receive the expanded list of files – finding the matching files is done by the shell, not by individual programs. Thus for the above example, from inside ls there is no way of distinguishing whether the user used the full list or the *.txt wildcard. You will experiment with this in one of the next labs where we will talk about accessing these parameters in your favorite programming language. For developers, it means that they do not need to care about implementing the wildcard expansion themselves. The program would always receive a list of existing filenames, not a wildcard.

By the way – is the last sentence completely correct? What happens if we run ls -l *.txxxt? Answer.

How would you print all files starting with the letter t? Answer.

If we would like to print only information about files starting with either o or f with .txt extension, we would use.

ls [of]*.txt

If we want to print files that end with any of the letters from a to f, we could use

ls *[a-f].txt

Try it in the a subdirectory.

Note that the files are sorted alphabetically when specified via wildcards.

Switch back to your home directory. Hint.

And now list all files/directories starting with D (recall that Linux is case-sensitive). You might be surprised because a straightforward ls D* would actually list the contents in these directories. It is perfectly expectable, because ls Documents is supposed to print a list of files in that directory. If we do not want ls to descend into directories, we can add -d option to prevent that.

What happens when you specify a file that does not exist? And what if only some of the specified files do not exist?

Apart from * (that matches any part of the filename) and [list-of-characters] (that matches one letter in the filename) also exists ? that matches any single letter (character).

Hence x?.txt will match files where the filename is 6 letters (chars) long, starts with x and ends with .txt (i.e., two letter filename starting with x of plain text type).

More about hidden files

Recall that filenames starting with dot . are hidden. These are by default not listed by ls. If you want to see these files too, you have to either name them explicitly or use the -a option.

Try it in the nswi177-lab02 directory.

What hidden files are in your home directory? Answer.

Again: it is not a security measure, just a way to make the listing less cluttered.

Exploring file contents

We have already mentioned text editors and MC to look into files when working in the terminal. They are not the only options.

Text files

The simplest way to dump the contents of any file is to call a program called cat. Its arguments are filenames to print. The name cat has nothing to do with the mammal but refers to the middle of the word concatenate as it can be used to actually concatenate files.

Move to the b subdirectory. Executing cat 000.txt will show the contents of 000.txt on the screen.

How would you show the contents of all files in this directory? Answer.

Binary files

If we want to dump binary files (such as images), it is usually better to dump their bytes in hexadecimal.

hexdump utility can be used for that.

We will always use it with -C switch to print hexdump and ASCII characters next to each other. The dump of the GIF file looks like this:

hexdump -C c/sample.gif

00000000  47 49 46 38 39 61 0a 00  0a 00 91 00 00 ff ff ff  |GIF89a..........|
00000010  ff 00 00 00 00 ff 00 00  00 21 f9 04 00 00 00 00  |.........!......|
00000020  00 2c 00 00 00 00 0a 00  0a 00 00 02 16 8c 2d 99  |.,............-.|
00000030  87 2a 1c dc 33 a0 02 75  ec 95 fa a8 de 60 8c 04  |.*..3..u.....`..|
00000040  91 4c 01 00 3b                                    |.L..;|
00000045

Unprintable values (e.g., smaller than 32) are replaced with a dot.

Guessing file type

Even though the file extension is not mandatory, it is better to use it to explicitly identify file types.

If you are not sure about the file type, utility file can identify the file type for you.

Try running it on the sample GIF file.

file c/sample.gif

Manual pages

We have seen that the ls behaviour can be modified with -a, -d, and -l. hexdump has -C. Do you know that uptime accepts -s? And that cat takes -n to print line numbers?

It is virtually impossible to remember all of this. Luckily, Linux contains so-called manual pages (or just manpages) that describe the available options for (almost) each program that you have on your system.

Execute man cmd to access a manual for the cmd program (substitute cmd for the actual command name). Use arrows for scrolling and q to quit the manual. You can search inside the page with / (slash) key.

Manual pages are organized into sections and you can specify the section number as part of the man execution, e.g., man 3 printf opens a help page for printf() function in the C language because that is the contents of section 3. Note that man printf would show you the contents of printf manual from section 1, i.e., the shell command.

Open man man to see the full list of sections. Briefly, 1 is for shell commands, 3 is for library calls, and 4 and 5 are used for specific files (e.g., man 5 proc launches the manual page for the whole /proc directory).

Note that manual pages are also available on-line, hence you can study your favourite commands even without access to your Linux machine.

Typical options

Many of the options are more-or-less standardized across multiple programs and are worth remembering.

Almost all GNU programs that you will have on your machine will print a small help when executed with --help. Try it for ls or cd.

--version could be used to print the version and copyright information of the executed program. Sometimes -v or -V works as well.

--verbose or --debug (sometimes -v or -d) launch the program in verbose mode where the program prints in more detail what it is doing.

--dry-run (sometimes -n) executes the program without performing actual changes (e.g., it can print which files would be removed without actually deleting any of them).

--interactive (sometimes -i) will typically cause the program to ask for interactive confirmation of destructive actions.

-- could be used to terminate the list of options if you have filenames starting with a dash. For a classical example, move into the d subdirectory of nswi177-lab02 and list information about a file named -a. Then check your result and try again using the -- delimiter. Answer.

Option merging and ordering

Command ls -h -l *.txt will print detailed information about all text files in the current directory and will print their size in human-readable units (yes, unfortunately ls is an exception where -h does not print help but changes the output of -l).

You will sometimes encounter the following variants of the above.

ls -l -h *.txt
ls -lh *.txt
ls -hl *.txt
ls *.txt -l -h
ls -l *.txt -h

Unless you have some weirdly named file (e.g. -a.txt) all of the above would work the same.

The variant -lh is a shortcut supported by many utilities to save up some space. Our advice is avoid this as it complicates reading your scripts.

The variant where options are intermixed with filenames – such as in this version – ls -l *.txt -h – is also supported by most utilities: they will be able to separate the options from the filenames and treat -h as a switch instead of filenames and reorder the command-line reasonably.

But usually the command expect that the filenames come last: hence the safest variant is to use the following form.

ls -l -h -- *.txt

The above form (i.e. options first) will work for most of the utilities (and using the explicit delimiter of -- is a defensive approach against funny filenames).

But be careful that it’s up to the program and not up to the shell, to decide how to treat the arguments. Later we will learn about programs where the order of arguments matters and affects what the program does.

Caveats (file names with spaces in them)

If you create a file called file with spaces.txt and then execute

ls file with spaces.txt

you will receive

ls: cannot access 'file': No such file or directory
ls: cannot access 'with': No such file or directory
ls: cannot access 'spaces.txt': No such file or directory

because the space (or tab) is used as a delimiter between parameters. Hence, ls was actually looking for three files.

If you would have used tab completion, your command would be completed with escape characters.

ls file\ with\ spaces.txt

Note that the output would typically look like this:

'file with spaces.txt'

because using apostrophes (or quotes) is another way to specify that the space is a literal character and not a separator.

We will mention this again when talking about scripts, but it is something to remember: spaces in filenames can cause unexpected surprises and it is better to avoid such naming.

And yes, it is possible to create a file named ' ' (i.e., space) and show its contents with cat " " but it is not a very sensible idea to do so. It is similar to creating files starting with a dash – it is possible, there are ways to bypass the issues (e.g., using -- delimiter) but it is just simpler to avoid these issues.

Work efficiently

Running Python scripts from the command line

We will talk about this in greater detail in the following lab, for now you can use the following command to actually run your Python script:

python3 path_to_your_python_script.py

Tasks to check your understanding

We expect you will solve the following tasks before attending the labs so that we can discuss your solutions during the lab.

stats = {}
with open('/proc/meminfo', 'r') as f:
    for line in f:
        parts = line.split(":")
        stats[parts[0].strip()] = parts[1].split()[0].strip()
print(float(stats['MemFree'])/float(stats['MemTotal']))

Get this archive first:

https://d3s.mff.cuni.cz/f/teaching/nswi177/202324/labs/nswi177-lab02-wildcards.tar.gz

Get contents of all files in subdirectory `login` that
start with a decimal digit, ends with `z.txt` and the middle letter is
any letter (i.e., a to z, not numbers) from your GitLab login
(in lowercase).

For example, if your login is `johndoe`, you should paste contents from files
`0jz.txt`, `1ez.txt` but not from `ajz.txt` or `2wz.txt` or
`0jx.txt`.

Sort the list of files alphabetically before getting their content, duplicate
letters should be ignored (i.e., use wildcards naturally and you will be fine).


**Q1** Paste the contents of the files here
(do not insert any extra characters such as newlines, though).

Insert your answer between the markers below, replacing the three dots.
Leading and trailing whitespace in your answer will be ignored but
keep the starred A1 markers without changes (tests will check that).

**[A1]** ... **[/A1]**


**Q2** Insert here the wildcard pattern that you have used
(only the pattern without `ls` or any other command you have used).

**[A2]** ... **[/A2]**


**Q3** Paste here a full command (including `ls`) to print all
files in the `letters` directory that end with the letter `o`.
Please, assume that you are executing `ls`
from the `letters` subdirectory and **use a proper wildcard**.

**[A3]** ... **[/A3]**

/home/../usr/./share/./man/../../lib/../../etc/ssh/.././os-release

Learning outcomes

Learning outcomes provide a condensed view of fundamental concepts and skills that you should be able to explain and/or use after each lesson. They also represent the bare minimum required for understanding subsequent labs (and other courses as well).

Conceptual knowledge

Conceptual knowledge is about understanding the meaning and context of given terms and putting them into context. Therefore, you should be able to …

• list pros and cons of using a command-line interface vs a graphical one

• explain the difference between a terminal (emulator) and a shell

• explain what is a path to a file

• explain difference between a relative and an absolute file path

• explain what are shell (filename) wildcards

• explain what are command-line options (switches)

• explain usefulness of -- delimiter (when using wildcards on specifically named files)

Practical skills

Practical skills are usually about usage of given programs to solve various tasks. Therefore, you should be able to …

• start and close (exit) a terminal emulator

• customize a selected terminal emulator

• browse through a filesystem via text user interface tools (e.g. mc or ranger)

• browse through a filesystem using commands ls and cd

• use basic switches of the ls command such as -l, -h or -a

• use wildcards to apply commands to specific subsets of filenames

• run own Python programs from the command line

• view contents of text files using the cat utility

• view contents of binary files in hexadecimal using the hexdump utility

• identify file type by using file utility

• use (basic operations) of the built-in manual pages

• use clipboards available in a graphical interface on Linux

• use tab completion to effectively write file names and paths

• use irregularly named files

ls -a ~
ls -d ~/.*

cat *

cat [0-9][0-9][0-9].txt

ls -- -a

cd ..

cd ..

ls t*

ls -a ~
ls -d ~/.*

cat *

cat [0-9][0-9][0-9].txt

ls -- -a

ls t*