Cvičení #2: Střemhlav do příkazové řádky

Cvičení: 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14.

Na tomto cvičení se začneme učit jak ovládat náš Linuxový stroj tím nejefektivnějším způsobem – skrze rozhraní příkazové řádky.

Nezapomeňte, že Čtení před cvičením je povinné a je z něj kvíz, který musíte vyplnit před cvičením.

Tohle je čtení před druhým cvičením.

MemTotal:        7899128 kB
MemFree:          643052 kB
MemAvailable:    1441284 kB
Buffers:          140256 kB
Cached:          1868300 kB
SwapCached:            0 kB
Active:           509472 kB
Inactive:        5342572 kB
Active(anon):       5136 kB
Inactive(anon):  5015996 kB
Active(file):     504336 kB
Inactive(file):   326576 kB
...

Spusťte vaše terminály, prosím

For this (and almost all other labs) we will be working in the terminal. Please, locate this program in your environment and start it. Depending on your environment, it will be either Terminal, Console, or perhaps even Shell (although, technically, shell is the program running inside a terminal emulator).

We recommend you spend some time configuring the look of your terminal, such as having a nice font family and a reasonable font size. You will be spending quite a lot of time with it, so make the experience nice. Below are some possibilities of what you might get :-).

You will see something like [intro@localhost ~] and a blinking cursor after that. This is called a prompt and if you see it, it means you can enter your commands.

The prompt is displayed by your shell which is an interpreter of the commands you enter. The shell is actually a full-fledged programming language, but in this lab we will use it to launch very simple commands only.

Type uptime and start this command by submitting it with <Enter>. Until you hit <Enter>, you can easily edit the command. Shortcuts such as <Ctrl>-<Arrow> for jumping over words works too.

As we already mentioned, the experience is somewhat similar to an interactive Python session (editing etc.).

Rychlé kopírování/vkládání (a ukončování programů)

Whenever you select a text in the terminal with your mouse, it is automatically copied. This text then can be inserted by simply clicking the middle mouse-button (or the wheel).

Note that the well-known <Ctrl>-C and <Ctrl>-V combinations do not work in the shell as <Ctrl>-C is used to forcefully terminate a program. However, <Ctrl>-<Shift>-C usually works.

Note that these are actually two distinct clipboards – the special one bound to middle mouse-button and the one bound to <Ctrl>-C (<Ctrl>-<Shift>-C) and <Ctrl>-V. In graphical applications, <Ctrl>-C and <Ctrl>-V work as usual.

Ukončení terminálu

To close the terminal, you can simply close the whole window (e.g., via mouse) but you can also type exit or hit <Ctrl>-D on an empty line. Because we are moving away from needing mouse (in a sense), you should prefer <Ctrl>-D ;-).

Ladění chyb

When running programs in a terminal, never paste their output as a screenshot. Instead, select the text (including the command you have run) and paste where needed.

For pasting into our Forum enclose the text in the fenced block ``` to preserve the monospace font.

```
ls nonexistent
ls: cannot access 'nonexistent': No such file or directory
```

Průchod systémem souborů

We will start with simple navigation through the file system. Two basic commands will get you through.

Výpis souborů pomocí ls

The ls command lists files in the current directory.

Executing ls shall produce something like this:

Desktop    Downloads  Music     Public     Videos
Documents  gif.md     Pictures  Templates

Now run ls -l. That is, ls and -l separated by a space. Here we are calling the program ls and giving it an extra argument, -l. Because the argument starts with a dash, it is actually a so-called option (or switch) that instructs ls to modify its behaviour. Now ls prints something like this:

total 4
drwxr-xr-x. 1 intro intro    0 Feb 10 13:43 Desktop
drwxr-xr-x. 1 intro intro    0 Feb 10 13:43 Documents
drwxr-xr-x. 1 intro intro    0 Feb 10 13:43 Downloads
-rw-r--r--. 1 intro intro 1022 Jan  9 18:13 gif.md
drwxr-xr-x. 1 intro intro    0 Feb 10 13:43 Music
drwxr-xr-x. 1 intro intro    0 Feb 10 13:43 Pictures
drwxr-xr-x. 1 intro intro    0 Feb 10 13:43 Public
drwxr-xr-x. 1 intro intro    0 Feb 10 13:43 Templates
drwxr-xr-x. 1 intro intro    0 Feb 10 13:43 Videos

The -l turned on the so-called long mode where more details about each file are printed.

We will return to the meaning of some of the columns later on, deciphering the columns for the last modification time and the file size is straightforward and sufficient for the moment.

Změna pracovního adresáře pomocí cd

The cd command allows us to change the working (current) directory. It takes one argument – the directory we want to switch to.

Thus, cd Documents would move us to the Documents directory.

Execute ls here. What is the output? Answer.

How would you move back to the parent directory? Answer.

What will do the following command?

cd .

Notice that the command prompt changed whenever you switched to a different directory.

By default, it shows only the last component of the path. To show the full (absolute) path, we need to run pwd.

It will show something like

/home/intro/Videos

Doplňování tabulátorem

Typing long filenames can be cumbersome and making typos is annoying. Shell offers tab completion to help you with this.

For this example, we assume you just launched your terminal and ls prints Desktop Documents Downloads Templates etc.

If we want to change to directory Templates, start typing cd Te and hit <Tab>. Unless there is another filename (directory) starting with Te, the name shall be completed for you and should read the full cd Templates/.

Submitting the command with <Enter> would switch you to the directory as we would expect. Try it and come back to this directory again.

Now, let us switch to Documents directory. For this example, type cd Do and press <Tab>. There are two directories with this prefix: Documents and Downloads. Because the shell cannot know which one you want, it does nothing.

However, pressing <Tab> for the second time shows the possible matches and after typing c (the next letter), <Tab> can finish the completion.

Tab completion is an extremely powerful feature that saves hundreds of keystrokes and makes your interaction with the shell much faster.

As an exercise, what happens if you type cd and hit <Tab>? Answer.

Type just c (as in cd) and hit <Tab>. What happens? Answer.

Domovský adresář

You probably noticed that when you start your terminal, the directory name you see there is just a ~ even though it should read intro (or your username on that particular machine) as that is the last component from pwd.

However, the path /home/intro is your home directory and has a special shortcut of tilde ~.

Futhermore, if you just run command cd without any extra arguments, it will change the directory back to your home.

Nástroje s textovým uživatelským rozhraním

While the use of purely command-line tools such as uptime, ls or cd is cool and extremely useful for scripts, there are also occasions where a more interactive approach is faster.

In this sense, Linux typically offers three layers you can choose from. From a fully graphical one called Graphical User Interface (GUI), over a tool with a Text-based User Interface (TUI) to a pure Command-Line Interface (CLI). Every of these can be useful, depending on the circumstances.

By a textual user interface we mean what is offered by Midnight commander or Ranger.

Midnight commander

Run mc and navigate through the files as you have done with ls and cd.

The numbers at the bottom refer to your functional keys for typical file operations (e.g., F5 copies the file).

Note that in a typical setup, MC offers two panels with file listing, you switch between them via <Tab> and, by default, copying is done to the directory in the other panel.

MC is a quite powerful tool as it can inspect file archives, show files on a remote machine, etc.

We will briefly mention the most important things that you can do with it. Do try them :-)

• <Insert> allows you to select multiple files for deletion/copying.
• <F3> displays file contents.
• <F4> offers simple text editor with syntax highlighting.
• <+> allows you to enter a filename mask to select multiple files at once (we will talk about this more later in the Wildcards section).
• <Ctrl>-o hides the panels and temporarily switches you back to shell. Perfect for running commands without leaving MC.

You can quit MC with <F10> or via a menu (activated by <F9>). Note that some terminals capture <F10> to activate their window menu (but this behaviour can be tuned in Preferences of the terminal application).

Ranger

Ranger is a Vim-inspired file manager for the console. It brings some well-known key bindings from the Vim realm together with tabs pages.

Navigation

• j - Move down
• k - Move up
• h - Move to the parent directory
• l - Open file or move to directory
• gg - Go to the top of the list
• G - Go to the bottom of the list
• gh - cd ~
• gm - cd /media
• gr - cd /
• q - Quit Ranger

Working with Files

• zh - View hidden files
• cw - Rename current file
• <space> - Select current file
• yy - Yank (copy) file (or selected files)
• dd - Mark file (or selected files) for cut operation file
• pp - Paste yanked or cut file(s)
• dD - Delete file (or selected files)

See more on Ranger: A terminal file manager.

Úpravy souborů

You probably noticed that the Development submenu contains several graphical text editors that you can use to edit the source code. However, it is also possible to edit files in TUI editors.

If you are asking why to learn another editor (if you are already happy with some of the graphical ones), here is the answer. On some machines, you may not have access to GUI at all. Recall that we talked about remote access earlier: in that case you will have only TUI available (and you will often need to edit files on the remote machine). Some users thus never use GUI editors at all, the reasoning is that it is much better to learn (and customize) one editor properly, and that editor is a TUI-based one.

On our disk, you will find Emacs, Joe, mcedit and Vim.

Each has its own advantages and it is up to you which one you will choose. Note that mcedit is probably the closest to an editor you may know from other systems. joe is a small one, but perfectly suitable for script editing that we will be doing the most. Both emacs and vim are extremely powerful tools that can do much more than just edit files. However, they require a bit of time investment before you can start using them effectively.

If you are new to Linux, we would recommend you to use mcedit (either using it directly or when editing files in Midnight commander) and come back to the other ones later on for a final decision of THE text editor of your choice.

All of these editors can be launched from the command-line, giving it the filename to edit as a parameter (e.g., mcedit quiz.md).

Zástupné znaky v shellu (wildcards)

For the following you will need to have the same list of files as we have.

Please, download this archive and unpack its contents. If you want to download it from the command line, you can use wget URL, otherwise use whatever browser you like. Use Midnight commander to copy the unpacked content to your home directory.

You should see directory nswi177-lab02 on your disk.

So far, we used ls to display all files in a directory. If we are interested in only a subset, we can specifically name them on the command line.

Move to the directory where you have unpacked the nswi177-lab02.tar.gz. You should see the following files:

a/ b/ c/ one.txt two.txt three.txt four.txt

If we want to list only details about the text files, we can execute

ls -l one.txt two.txt three.txt four.txt

-rw-r--r-- 1 intro intro 0 Mar  3 13:38 four.txt
-rw-r--r-- 1 intro intro 0 Mar  3 13:38 one.txt
-rw-r--r-- 1 intro intro 0 Mar  3 13:38 three.txt
-rw-r--r-- 1 intro intro 0 Mar  3 13:38 two.txt

Doing that for more files would not be very elegant, but the shell offers so called wildcards to specify multiple files at once. Thus, the same output can be obtained by running

ls -l *.txt

It is essential to note that ls (or any other program for that matter) will receive the expanded list of files – finding the matching files is done by the shell, not by individual programs. Thus for the above example, from inside ls there is no way of distinguishing whether the user used the full list or the *.txt wildcard. You will experiment with this in one of the next labs where we will talk about accessing these parameters in your favorite programming language. For developers, it means that they do not need to care about implementing the wildcard expansion themselves. The program would always receive a list of existing filenames, not a wildcard.

By the way – is the last sentence completely correct? What happens if we run ls -l *.txxxt? Answer.

How would you print all files starting with the letter t? Answer.

If we would like to print only information about files starting with either o or f with .txt extension, we would use.

ls [of]*.txt

If we want to print files that end with any of the letters from a to f, we could use

ls *[a-f].txt

Try it in the a subdirectory.

Note that the files are sorted alphabetically when specified via wildcards.

Switch back to your home directory.

And now list all files/directories starting with D (recall that Linux is case-sensitive). You might be surprised because a straightforward ls D* would actually list the contents in these directories. It is perfectly expectable, because ls Documents is supposed to print a list of files in that directory. If we do not want ls to descend into directories, we can add -d option to prevent that.

What happens when you specify a file that does not exist? And what if only some of the specified files do not exist?

Recall that filenames starting with dot . are hidden. These are by default not listed by ls. If you want to see these files too, you have to either name them explicitly or use the -a option.

Try it in the nswi177-lab02 directory.

What hidden files are in your home directory? Answer.

Again: it is not a security measure, just a way to make the listing less cluttered.

Zkoumáme obsah souborů

We have already mentioned text editors and MC to look into files when working in the terminal. They are not the only options.

Textové soubory

The simplest way to dump the contents of any file is to call program called cat. Its arguments are filenames to print. The name cat has nothing to do with the mammal but refers to the middle of the word concatenate as it can be used to actually concatenate files.

Move to the b subdirectory. Executing cat 000.txt will show the contents of 000.txt on the screen.

How would you show the contents of all files in this directory? Answer.

Binární soubory

If we want to dump binary files (such as images), it is usually better to dump their bytes in hexadecimal.

hexdump utility can be used for that.

We will always use it with -C switch to print hexdump and ASCII characters next to each other. The dump of the GIF file looks like this:

hexdump -C c/sample.gif

00000000  47 49 46 38 39 61 0a 00  0a 00 91 00 00 ff ff ff  |GIF89a..........|
00000010  ff 00 00 00 00 ff 00 00  00 21 f9 04 00 00 00 00  |.........!......|
00000020  00 2c 00 00 00 00 0a 00  0a 00 00 02 16 8c 2d 99  |.,............-.|
00000030  87 2a 1c dc 33 a0 02 75  ec 95 fa a8 de 60 8c 04  |.*..3..u.....`..|
00000040  91 4c 01 00 3b                                    |.L..;|
00000045

Unprintable values (e.g., smaller than 32) are replaced with a dot.

Hierarchie souborových systémů

By now, you can navigate the file system in a TUI manager or via cd and ls. Open two terminals next to each other and perform the same actions in both.

Do not be afraid to actually display contents of the files we mention here. hexdump -C is really a great tool.

/boot contains the bootloader for loading the operating system. You would rarely touch this directory once the system is installed.

/dev is a very special directory where hardware devices have their file counterparts. You will probably see there a file sda or nvme0 that represents your hard (or SSD) drive. Unless you are running under a superuser (more about that later), you will not have access to these files, but if you would hexdump them, you would see the bytes as they are on the actual hard drive. And writing to this file would overwrite the data on your drive!

This directory also contains several special but very useful files for software development.

/dev/urandom returns random bytes indefinitely. It is probably internally used inside your favorite programming language to implement its random() function. Try to run hexdump on this file (and recall that <Ctrl>-C will terminate the program once you are tired of the randomness).

/etc/ contains system-wide configuration. Typically, most programs in unix systems are configured via text files. The reasoning is that an administrator needs to learn only one tool – a good text editor – for system management. The advantage is that most configuration files have support for comments and it is possible to comment even on the configuration. For an example of such a configuration file, you can have a look at /etc/systemd/system.conf to get the feeling.

Perhaps the most important file is passwd that contains a list of user accounts. Note that it is a plain text file where each row represents one record and individual attributes are simply separated by a colon :. Very simple to read, very simple to edit, and very simple to understand. In other words, the KISS principle in practice.

/home contains home directories for normal user accounts (i.e., accounts for real – human – users).

/lib and /usr contain dynamic libraries, applications, and system-wide data files.

/var is for volatile data. If you would install a database or a web server on your machine, its files would be stored here.

/tmp is a generic location for temporary files. This directory is automatically cleaned at each reboot, so do not use it for permanent storage.

/proc is a virtual file system that allows controlling and reading of kernel (operating system) settings. For example, the file /proc/meminfo contains quite detailed information about RAM usage.

Again, /proc/* are not normal files, but virtual ones. Until you read them, their contents do not exist physically anywhere.

Dokumentace (manual pages)

We have seen that the ls behaviour can be modified with -a, -d, and -l. hexdump has -C. Do you know that uptime accepts -s? And that cat takes -n to print line numbers?

It is virtually impossible to remember all of this. Luckily, Linux contains so-called manual pages (or just manpages) that describe the available options for (almost) each program that you have on your system.

Execute man cmd to access a manual for the cmd program (substitute cmd for the actual command name). Use arrows for scrolling and q to quit the manual. You can search inside the page with / (slash) key.

Manual pages are organized into sections and you can specify the section number as part of the man execution, e.g., man 3 printf opens a help page for printf() function in the C language because that is the contents of section 3. Note that man printf would show you the contents of printf manual from section 1, i.e., the shell command.

Open man man to see the full list of sections. Briefly, 1 is for shell commands, 3 is for library calls, and 4 and 5 are used for specific files (e.g., man 5 proc launches the manual page for the whole /proc directory).

Note that manual pages are also available on-line, hence you can study your favourite commands even without access to your Linux machine.

Obvyklé volby

Many of the options are more-or-less standardized across multiple programs and are worth remembering.

Almost all GNU programs that you will have on your machine will print a small help when executed with --help. Try it for ls or cd.

--version could be used to print the version and copyright information of the executed program. Sometimes -v or -V works as well.

--verbose or --debug (sometimes -v or -d) launch the program in verbose mode where the program prints in more detail what it is doing.

--dry-run (sometimes -n) executes the program without performing actual changes (e.g., it can print which files would be removed without actually deleting any of them).

--interactive (sometimes -i) will typically cause the program to ask for interactive confirmation of destructive actions.

-- could be used to terminate the list of options if you have filenames starting with a dash. For a classical example, move into the d subdirectory of nswi177-lab02 and list information about a file named -a. Then check your result and try again using the -- delimiter. Answer.

Zádrhele (soubory s mezerou v názvu)

If you create a file called file with spaces.txt and then execute

ls file with spaces.txt

you will receive

ls: cannot access 'file': No such file or directory
ls: cannot access 'with': No such file or directory
ls: cannot access 'spaces.txt': No such file or directory

because the space (or tab) is used as a delimiter between parameters. Hence, ls was actually looking for three files.

If you would use tab completion, your command would be completed with escape characters.

ls file\ with\ spaces.txt

Note that the output would typically look like this:

'file with spaces.txt'

because using apostrophes (or quotes) is another way to specify that the space is a literal character and not a separator.

We will mention this again when talking about scripts, but it is something to remember: spaces in filenames can cause unexpected surprises and it is better to avoid such naming.

And yes, it is possible to create a file named ' ' (i.e., space) and show its contents with cat " " but it is not a very sensible idea to do so. It is similar to creating files starting with a dash – it is possible, there are ways to bypass the issues (e.g., using -- delimiter) but it is just simpler to avoid these issues.

Jak pracovat efektivněji

Do not be afraid of running multiple terminals next to each other. Use one to navigate with ls and cd, use the other one for Midnight commander to mirror your actions.

Open another one with a manual page for the command you are using.

Most desktop environments allow you to create multiple workspaces or desktops. Then, each workspace has its own list of opened windows, windows opened on other workspaces are not visible. This can reduce the clutter significantly and – with proper keyboard shortcuts – speed up your work.

Spouštění Pythoních skriptů z příkazové řádky

O tomhle si budeme více povídat na dalším cvičení, prozatím můžete použít následující příkaz abyste spustili váš Pythoní program:

python3 cesta_k_vasemu_python_skriptu.py

Hodnocené úlohy (deadline: 6. březen)

02/tree.py (30 bodů)

Napište Pythoní skript, který vypíše strom souborů v aktuálním adresáři.

Předpokládejme, že existují následující soubory:

README.md
01/factor.py
02/wildcards.md
02/before.md
02/temp/file.txt
bin/run_tests.sh
.gitlab-ci.yml

Skript potom vypíše následující (všimněte si, jak se pracuje s adresáři a jak je výpis seřazen).

01/
    factor.py
02/
    before.md
    temp/
        file.txt
    wildcards.md
README.md
bin/
    run_tests.sh

Nápověda: zvažte použití os.scandir() jako výchozího bodu vaší implementace. Povšimněte si, že dokumentace dokonce obsahuje poměrně užitečný příklad pro odstartování vašeho řešení. Zkušení vývojáři si možná všimnou, že os.scandir je lepší volbou než starší os.walk kvůli rychlosti (PEP 471).

Zvláštní chování je vyžadováno pro následující případy:

• Skryté soubory nejsou vůbec tištěny.
• Symbolické odkazy jsou kompletně ignorovány (použijte follow_symlinks=False pro funkci is_file a jí podobné).

Můžete bezpečně předpokládat, že sorted() je tou správnou funkcí na třídění a že se adresářová struktura během skriptu nezmění.

Nezapomeňte zkopírovat první řádku z 01/factor.py (počkejte na další cvičení pro pořádné vysvětlení) a připravte váš kód na moduly (to je ta řádka s __name__).

02/wildcards.md (30 bodů)

Zkopírujte si následující fragment do GitLabu a vyplňte odpovědi.

Ani tento úkol není plně kontrolován GitLabem, protože bychom tím zveřejnili odpověď.

Stáhněte si nejdřív tento archiv:

https://d3s.mff.cuni.cz/f/teaching/nswi177/202122/labs/nswi177-task02.tar.gz

Není se podívejte na soubor v podadresáři `login`, které začínají desítkovou
číslicí, končí `z.txt` a prostřední písmeno je libovolné písmeno (tj. A-Z, nikoliv
čísla) z vašeho přihlašovacího jména na GitLab (malými písmeny).

Na příklad, pro login `johndoe` byste měli pracovat se soubory
`0jz.txt`, `1ez.txt` ale ne s `ajz.txt` nebo `2wz.txt` nebo
`0jx.txt`.

Seznam souborů setřiďte abecedně, opakovaná písmena nezahrnujte
(tj. použijte rozumně wildcardy a bude to rovnou fungovat).

**Q1** Vložte sem obsah těchto souborů.

**[A1]** ... **[/A1]**


**Q2** Vložte sem wildcard, který jste použili.

**[A2]** ... **[/A2]**

02/uptime.txt (20 bodů)

Vlastními slovy popište, co vytiskne následující příkaz.

cut -d ' ' -f 1 /proc/uptime

Nezapomeňte, že můžete využít příkaz man pro zjištění informací o daném souboru, informace o /proc souborech jsou v páté sekci.

GitLab jen kontroluje přítomnost souboru.

02/architecture.sh (20 bodů)

Do tohoto souboru vložte (jako jeho jediný obsah) příkaz, který vytiskne architekturu vašeho počítače (pro většinu z vás to bude x86_64).

Nápověda: podívejte se na příkaz uname.

Učební výstupy

Znalosti konceptů

Znalost konceptů znamená, že rozumíte významu a kontextu daného tématu a jste schopni témata zasadit do většího rámce. Takže, jste schopni …

• vysvětlit, kdy je rozhraní příkazového řádku lepší volbou než grafická aplikace

• vysvětlit, co je to terminál a co je to shell

• vysvětlit, co je to cesta (k danému souboru), co je to relativní a absolutní cesta

• vysvětlit, jak fungují tzv. wildcards na názvech souborů

• popsat hlavní adresáře a soubory na typickém Linuxovém systému

• vysvětlit, jak je strom adresářů vytvořen připojením jednotlivých subsystémů

• vysvětlit rozdíl mezi normálním (obyčejným) souborem, adresářem a symbolickým odkazem

Praktické dovednosti

Praktické dovednosti se obvykle týkají použití daných programů pro vyřešení různých úloh. Takže, dokážete …

• spustit a nastavit si oblíbený emulátor terminálu

• pohybovat se po souborovém systémů v terminálu pomocí TUI nástrojů nebo CLI příkazů

• zobrazit obsah textových soubor (použít cat)

• zobrazit obsah binárních souborů (jako hexadecimální výpis pomocí programu hexdump)

• využívat vestavěné manuálové stránky

• používat schránky dostupné na Linuxu

• používat doplňování tabelátorem pro rychlejší psaní názvů souborů

• používat soubory se neobvyklými názvy

• používat příkazy cd a ls (se základními volbami)

• používat mc nebo podobný správce souborů (na základní operace)

ls -- -a

cd ..

ls t*

ls -a ~
ls -d ~/.*

cat *

cat [0-9][0-9][0-9].txt

cd ..

ls -- -a

ls t*

ls -a ~
ls -d ~/.*

cat *

cat [0-9][0-9][0-9].txt