Session 06

Section 06¶

Notes from the previous session

The lecturer may have to reset passwords and VMS from time to time.
Please attempt to keep up.
If you are stuck at a point, read the instructions again, you may solve your own issue.
If you are able to complete a task, please assist students next to you if they struggle.

Instructions¶

See the Information Page to log into the jump node.

Note

We will all be working on this machine for today’s class.

Log into the jump node
Copy the /opt/example06/ directory to your home directory and cd into your home directory:
1 2
/bin/cp -vr /opt/example06/ ~ cd
Attempt to execute the commands that the instructor is executing

Overview¶

In this session, we will learn how to change file content using text manipulation tools like sed. After that, we will look at how to use two popular text editors named nano and vi.

Text manipulation using `sed`¶

In the previous session, we looked at regular expressions and searched for text in a file using the grep statement. We now need to look at how to modify text in a file.

We will look at the stream editor sed. Several tools, such as sed and awk, can be used for this purpose. However, we will only be looking at sed. To use awk, see the awk manual pages or online sources.

Using `sed` to modify output¶

The easiest way to use sed is to execute a command and modify the output using a regular expression (pattern) or specific text.

The sed command has various functions within the command that can be called. We will only use a few available options, which are helpful throughout this course.

Using `sed` to delete lines containing text/pattern¶

The first option is to delete lines containing specific text. The syntax is:
/search_string_we_are_looking_to_exclude/d

Note

The forward slash delimiter / is compulsory for the delete option.
For other options, such as the substitute hereafter, the delimiter can be changed to another character, such as % or |.
The same delimiter must be used throughout the command.

For instance, if we print 15 lines and we want to delete a line that contains the number three (3), we can execute the following:

# The following command will print numbers 01, 02 ... through 15:
for i in $(seq -w 1 15); do echo "Number: $i"; done

# Print the above text again, removing lines containing 3
for i in $(seq -w 1 15); do echo "Number: $i" | sed "/3/d"; done

If we want to use a regular expression, we can simply include the regular expression in our search.

For instance:

# Remove all lines that contain a zero (no regex):
for i in $(seq -w 1 15); do echo "Number: $i"   |  sed "/0/d"; done

# Now, only replace lines ending (anchor regex) in a zero:
for i in $(seq -w 1 15); do echo "Number: $i"   |  sed "/0$/d"; done

# Delete all lines containing a number (regex match - all lines):
for i in $(seq -w 1 15); do echo "Number: $i"   |  sed "/[0-9]*/d"; done

Using `sed` to replace specific text¶

We deleted complete lines in the above examples using the "/search_line_to_delete/d" syntax.
If you want to replace only specific values from a string, you need to use the following syntax:

Example:

sed "s/string_we_are_searching_for/replace_the_string_with_this/g" /etc/passwd

The search and replace function (substitute) starts with an s.
After that, a delimiter is used.
In this case, a forward slash /.
The delimiter is followed by the string we are looking for.
Another delimiter / is used to indicate that the replacement is to follow.
After the string that is replacing the search string, another delimiter / is used.
After the final delimiter is specified, a value is specified.
This value represents the occurrence at which the replacement should commence.
In the above example, a value of g is used, meaning a global replacement is done.
The g-option is the most common and replaces all occurrences of the text.
Note: the occurrence of a replacement string is per line, so if you specify:
"s/replace_me/with_me/2" then the second occurrence of replace_me in each line, will be replaced by with_me.

Note

Note that other special characters can be delimiters instead of the forward slash /.
Using a pipe character | instead when modifying a path will be easier.

Simple example

# Replace the first occurrence of name with Your-Name:
echo "Hello there, name is name. It is name?"  |   sed  "s/name/Your-Name/"

# The same as above, just specifying the first occurrence (implied):
echo "Hello there, name is name. It is name?"  |   sed  "s/name/Your-Name/1"

# Replace the second occurrence of name with Your-Name:
echo "Hello there, name is name. It is name?"  |   sed  "s/name/Your-Name/2"

# Replace all (globally) occurrences of name with Your-Name:
echo "Hello there, name is name. It is name?"  |   sed  "s/name/Your-Name/g"

# Execute the same command but with a different delimiter:
echo "Hello there, name is name. It is name?"  |   sed  "s|name|Your-Name|g"

Another option is to replace it from the n^th position to the last occurrence in the line, using the syntax "s/search/and_replace/ng".

# For instance, replace from the 3rd to the last occurrence:
echo "Replace from the third a character, up the last a in the line." | sed "s/a/@/3g"

Print a specific line using `sed`¶

The sed command can also print specific lines from a file. The print function is used and has the following syntax to print line three of /etc/passwd:

sed -n "3p" /etc/passwd

Ranges can also be specified; example:

# Print line 3 of the /etc/passwd file:
sed -n "3p" /etc/passwd

# Print lines 3 to 6 of the etc/passwd file:
sed -n "3,6p" /etc/passwd

# Print from line 18 to the end of the file:
sed -n "18,\$p" /etc/passwd

Changing a file in place, using `sed`¶

The above examples simply modified the output provided. The sed command can write modifications to a file using the in-place (-i) option.

Example

# Go to your home directory:
cd

# Copy the /etc/passwd file first so we can modify it:
/bin/cp /etc/passwd  $HOME/
# Change the first set of numbers (between the :: ) to XXX:
sed "s|[0-9]\+|XXX|1" passwd
#Note, \+ is regex indicating one or more occurrences of the preceding class

# Compare the output:
cat passwd

# Now, make the modification in place (permanently modifying the file):
sed -i  "s|[0-9]\+|XXX|1" passwd

# Compare the output again:
cat passwd

Note

Note using a different delimiter (other than / ) is recommended when using paths.
If a forward slash / is used as a delimiter, any forward slashes searched for must be escaped using a backslash \.

# Search for a string in a path and replace it.
# Search for /usr/bin/  in "/usr/bin/ls" and replace it with /bin/:
echo "ls lives as   /usr/bin/ls" | sed "s/\/usr\/bin\//\/bin\//g"

#The following reads easier but does the same replace:
echo "ls lives as   /usr/bin/ls" | sed "s|/usr/bin/|/bin/|g"

The in-place instruction is the most common use for the sed command to modify configuration or template files.

Exercise

Using the examples.log file, search for the following:

Copy the original /etc/passwd file over the current file in your home directory
Using sed, modify the file (in place) to change the nologin at the end of the lines to the word false
Replace all occurrences of /usr/sbin with /bin
Note using a different delimiter (other than / ) is recommended
If a forward slash / is used as a delimiter, any forward slashes searched for need to be escaped using a backslash \.

The `nano` text editor¶

In a GNU/Linux system, all configurations and data files are usually in plain text format. It is, therefore, valuable for modifying files directly in the terminal. The first and most user-friendly text editor we will look at is nano.
One drawback of the nano editor is that few GNU/Linux systems have nano installed by default.
However, it is available in their package managers, and thus, a simple dnf install nano will install nano on a RedHat-derived Linux system with Internet connectivity.

The nano editor can be used by executing the nano command, followed by the path and name of the file to edit.

Example

1	`nano /etc/passwd`

An empty file will be created if the file does not exist.
The following figure shows the primary nano interface.

Note the options at the bottom of the screen.
In this case, the most prominent line is the warning that the file is not writable.
For the other options, we see for instance, Ctrl+O will write the current file out (save).
To close nano, we press Ctrl+X

Nano file management¶

Task	Keystroke	Notes
Open a file from within nano	`Ctrl`+`R`	NOTE: tab completion is in effect; also, once this command has been entered, notice the new menu items are at the bottom of the screen. For example, `Ctrl`+`T` will allow you to browse the file system and look for a file to open.
Display the next file buffer	`Alt`+`>`
Display the previous file buffer	`Alt`+`<`
Save the current file buffer to disk	`Ctrl`+`O`
Close the current file buffer	`Ctrl`+`X`	NOTE: If the file hasn't been saved, you'll be asked if you want to save it. Also, if only one file buffer is open, closing it will exit from nano.

Nano copy and paste¶

Task	Keystroke	Notes
Select a region for a cut or paste operation	`Alt`+`A`	NOTE: After setting a mark with `Alt`+`A`, move the cursor to define the region; you should see it highlighted as you move the cursor. Also, to cancel the definition of the region, just press `Alt`+`A` again.
Copy a highlighted region into the clipboard	`Alt`+`^`
Cut a highlighted region into the clipboard	`Ctrl`+`K`
Paste the contents of the clipboard at the current cursor position	`Ctrl`+`U`
Cut from the current cursor position to the end-of-line (EOL)	`Ctrl`+`K`	NOTE: This command doesn't require highlighting of the region.

Task	Keystroke	Notes
Go to the beginning of the file	`Alt`+`\`
Go to end of file	`Alt`+`/`
Move forward one screenful	`Ctrl`+`V`
Move backwards one screenful	`Ctrl`+`Y`
Go to a target line number	`Alt`+`G`
Jump to matching open/close symbol	`Alt`+`]`	NOTE: Handy for finding mismatched brace compiler errors!
Window scrolling	`Alt`+`=`	To scroll down
Window scrolling	`Alt`+`-`	To scroll up
Indenting selected blocks	`Alt`+`A` to select a block	Then `Alt`+`}` will indent the selected block.
Outdenting selected blocks	`Alt`+`A` to select a block	Then `Alt`+`{` will outdent the selected block.

The `vi` text editor¶

The first release of vi was in 1976. The vi editor is installed by default in most GNU/Linux distributions. You can use' vi' to modify a repository or network configuration file to install nano on a clean installation.

The vi or newer vim (vi-improved) is a powerful text editor with various commands and shortcuts that are less intuitive and not displayed on screen like for nano. You will notice some syntax highlighting when opening a file such as a configuration or other Linux system file in vi.

Like with nano you can simply use the vi command followed by the path of the file to edit.

1	`vi /etc/passwd`

The above command will open the vi editor and will look like the following figure:

The most noticeable difference between the nano and vi interfaces, is the syntax/colour highlighting displayed in vi.

The different `vi` modes¶

The first concept in learning to use vi is identifying the different modes. Six modes exist, but you will most often only use two modes. Nonetheless, the following table lists and explains the six modes.

Name	Description	Help page
normal	For navigation and manipulation of text. `vim` usually starts in this mode, which you can usually get back to with `Esc`	:help Normal-mode
insert	For inserting new text. The main difference from `vi` is that many important "normal" commands are also available in insert mode - provided you have a keyboard with enough meta keys (such as `Ctrl`, `Alt`, `Win`, etc.).	:help Insert-mode
visual	For navigation and manipulation of text selections, this mode allows you to perform most normal commands, and a few extra commands, on selected text	:help Visual-mode
select	Similar to visual, but with a more MS Windows-like behaviour.	:help Select-mode
command-line	For entering editor commands - like the help commands in the 3^rd column.	:help Command-line-mode
Ex-mode	Similar to the command-line mode but optimized for batch processing.	:help Ex-mode

You will most likely only work in the insert and normal mode for 90% of the time.

Two visual modes are essential when working with vi.

Visual Mode	Key Stroke
Visual line mode	`Shift`+`V`
Visual block mode	`Alt`+`V`

See the Learning vi editor Wiki page for more information on the editor modes.

One-pager for `vi`¶

The following one pager cheat sheet is included as a quick reference guide.

Some humor

Demonstration using `vi`¶

The following demonstration will showcase the regular use of the vi editor.

The script below should be copied and pasted into a file, named example06.sh
The script will not work as expected, seeing that the two functions contained in the script, should be defined before they are called
There is also no indentation, which is fine in bash, but makes reading the code harder
Two variables (i01 and i02) are ambiguous and should be named better

#!/bin/bash
#
# This is a basic shell script that contains a function to calculate the average of two values provided.
# If this script is called without providing any options, a test function is performed.
#
#

# This if clause must move to the bottom of the script
if [ $# -eq 2 ]; then
echo "For the values: $1 $2"
echo "The average is: $(cal_avg $@)"
else
# The following two variables' names are non-descriptive, change them
i01=22
i02=11
avg=$(cal_avg $i01 $i02)
echo "You did not provide two values; the following will be used: $i01 & $i02"
echo "The average is: $avg"
fi

function cal_sum()
{
# Calculate the sum of two values
iResult=0
iResult=$(( $1 + $2 ))
iReturn=$?
# A function should echo the result that is returned
echo $iResult
# The return code is an integer to indicate a success or failure code
return $iReturn
}

function cal_avg()
{
# Get the number of values sent to this function
iNumber=$#
return_code=1
if [ $iNumber -eq 2 ]; then
iSum=$(cal_sum $1 $2)
val_return=$(( $iSum / $iNumber ))
return_code=$?
fi
echo $val_return
return $return_code
}

Note

One can also execute sed-like commands in vi.
To demonstrate, press Esc twice to go into normal mode.
Then press : (colon) to go into command-line mode.
Now, type the following:
%s/iNumber/myNumber/g
Press Enter to execute the sed command.
The above sed command will replace all iNumber entries with myNumber.

Example

More advanced commands in vi¶

In a previous challenge, you were asked to create a file with 99 entries in the format:

172.21.0.101 trn-usr01 trn-usr01.examplesdomain.com
172.21.0.102 trn-usr02 trn-usr02.examplesdomain.com
…
…
172.21.0.199 trn-usr99 trn-usr99.examplesdomain.com

This section is quite advanced and will most likely not be something you would often (or ever again) attempt in a text editor. This is only to showcase some of the powerful actions that can be performed in a text editor.

Let’s create this file in vi, carefully following the instructions:

Example

Execute the following command to create a file in your home directory:
1
vi ~/example06.hosts
Press 99 followed by I (lower case i)
This will instruct vi to execute the following insert statement 99 times.

Type (or paste) the following line:

172.21.0.101 trn-usr01 trn-usr01.examplesdomain.com

Press Enter to go to the following line
Press Esc
99 lines with the exact text will appear, followed by an empty line
Delete the last (empty) line:
Press D+D (lower case d, twice)
Go to the second line:
Press :+2+Enter (colon + 2 + enter)
Scroll to the zero of 101 (column 11), having the cursor blink on the zero
Press Ctrl+V to enter the visual block mode
Press Shift+G to go to the last line of the file
Note that the first columns will be highlighted
Press the Right (right arrow) until the last two digits (01) of the 101 column are selected
Press G to execute a global change
Press Ctrl+A to add one and auto increment each line in the column
Press :+2 (colon + 2) to go to the second line again
Move the cursor to the 0 of trn-usr01
Press . (period) to repeat the previous command
Move the cursor to the 0 of trn-usr01.examplesdomain.com
Press . (period) again to repeat the previous command
Press Shift+G to go to the last line in the file
You will notice that all the lines were modified
Press :+X (colon + x) to write and save the file

Timed Challenge

Execute the following code block (COPY AND PASTE)

export ch_result=/opt/challenge06/$USER/result.txt
echo "1" > $ch_result

# Check if your result is acceptable, if it was, it will be submited too:
c

Await further instructions.

Note

Execute the command: c to see the status of your results and submit it if it was correct.

Reminder

This time it is not necessary, but always remember to switch off your VM!

Session 06

Section 06¶

Instructions¶

Overview¶

Text manipulation using sed¶

Using sed to modify output¶

Using sed to delete lines containing text/pattern¶

Using sed to replace specific text¶

Print a specific line using sed¶

Changing a file in place, using sed¶

The nano text editor¶

Nano file management¶

Nano copy and paste¶

Nano navigation¶

The vi text editor¶

The different vi modes¶

One-pager for vi¶

Demonstration using vi¶

More advanced commands in vi¶

Text manipulation using `sed`¶

Using `sed` to modify output¶

Using `sed` to delete lines containing text/pattern¶

Using `sed` to replace specific text¶

Print a specific line using `sed`¶

Changing a file in place, using `sed`¶

The `nano` text editor¶

The `vi` text editor¶

The different `vi` modes¶

One-pager for `vi`¶

Demonstration using `vi`¶