what is the value?

The value is 123

#!/bin/sh
echo -n 'what is the value? '
read value
cat <<EOF
The value is $value
EOF

Multiple commands and order of execution

Addresses and Ranges of Text

Specifying a line by its number.

Specifying a range of lines by number.

All lines containing a pattern.

All lines from the beginning of a file to a regular expression

All lines from a regular expression to the end of the file.

All lines between two regular expressions.

restriction command

Restricting to a line number

sed '3 s/[0-9][0-9]*//' <file >new

Patterns

sed '/^#/ s/[0-9][0-9]*//'

sed ',^#, s/[0-9][0-9]*//'

sed '//usr/local/bin/ s//usr/local//common/all/'

sed '_/usr/local/bin_ s_/usr/local_/common/local_'

sed '/^g/s/g/s/g'

sed '/^g/ s_g_s_g'

Ranges by line number

sed '1,100 s/A/a/'

sed '101,532 s/A/a/'

sed '101,$ s/A/a/'

sed '200,300 s/A/a/' f1 f2 f3 >new

cat f1 f2 f3 | sed '200,300 s/A/a/' >new

Ranges by patterns

sed '/start/,/stop/ s/#.*//'

You can combine line numbers and regular expressions. This example will remove comments from the beginning of the file until it finds the keyword "start:"

sed -e '1,/start/ s/#.*//'

sed -e '1,/start/ s/#.*//' -e '/stop/,$ s/#.*//'

Before I start discussing the various commands, should show know that some commands cannot operate on a range of lines. I will let you know when I mention the commands. I will describe three commands in this secton. One of the three cannot operate on a range.

Delete with d

sed '11,$ d' <file

sed '1,/^$/ d' <file

sed '/^#/ d'

sed -e 's/#.*//' -e '/^$/ d'

sed -e 's/#.*//' -e 's/[ ^I]*$//' -e '/^$/ d'

This demonstrates the pattern space sed uses to operate on a line. The actual operation sed uses is:

Copy the input line into the pattern space.

Apply the first
sed command on the pattern space, if the address restriction is true.

Repeat with the next sed expression, again
operating on the pattern space.

When the last operation is performed, write out the pattern space
and read in the next line from the input file.

Printing with p

sed 'p'

sed '/^$/ p'

sed -n '1,10 p' <file

sed -n '/match/ p'

grep match

Reversing the restriction with !

sed -n '/match/ p'

sed -n '/match/ !p' </tmp/b

Relationships between d, p, and !

Relations between d, p, and !
  Sed	   Range   Command   Results
  --------------------------------------------------------
  sed -n   1,10	   p	     Print first 10 lines
  sed -n   11,$	   !p	     Print first 10 lines
  sed	   1,10	   !d	     Print first 10 lines
  sed	   11,$	   d	     Print first 10 lines
  --------------------------------------------------------
  sed -n   1,10	   !p	     Print last 10 lines
  sed -n   11,$	   p	     Print last 10 lines
  sed	   1,10	   d	     Print last 10 lines
  sed	   11,$	   !d	     Print last 10 lines
  --------------------------------------------------------
  sed -n   1,10	   d	     Nothing printed
  sed -n   1,10	   !d	     Nothing printed
  sed -n   11,$	   d	     Nothing printed
  sed -n   11,$	   !d	     Nothing printed
  --------------------------------------------------------
  sed	   1,10	   p	     Print first 10 lines twice,
			     Then next 10 lines once
  sed	   11,$	   !p	     Print first 10 lines twice,
			     Then last 10 lines once
  --------------------------------------------------------
  sed	   1,10	   !p	     Print first 10 lines once,
			     Then last 10 lines twice
  sed	   11,$	   p	     Print first 10 lines once,
			     then last 10 lines twice

sed -n '1,10 p'
sed -n '11,$ p'
sed '1,10 !d'
sed '11,$ d'

The q or quit command

sed '11 q'

The "q" command is the one command that does not take a range of addresses. Obviously the command

sed '1,10 q'

sed '1 q'

sed '10 q'

Grouping with { and }

Hardly worth the build up. All that prose and the solution is just matching squigqles. Well, there is one complication. Since each sed command must start on its own line, the curly braces and the nested sed commands must be on separate lines.

Previously, I showed you how to remove comments starting with a "#." If you wanted to restrict the removal to lines between special "begin" and "end" key words, you could use:

#!/bin/sh
# This is a Borne shell script that removes #-type comments
# between 'begin' and 'end' words.
sed -n '
/begin/,/end/ {
s/#.*//
s/[ ^I]*$//
/^$/ d
p
}
'

Click here to get file: sed_begin_end.sh
These braces can be nested, which allow you to combine address ranges. You could perform the same action as before, but limit the change to the first 100 lines:

#!/bin/sh
# This is a Borne shell script that removes #-type comments
# between 'begin' and 'end' words.
sed -n '
1,100 {
/begin/,/end/ {
s/#.*//
s/[ ^I]*$//
/^$/ d
p
}
}
'

Click here to get file: sed_begin_end1.sh
You can place a "!" before a set of curly braces. This inverts the address, which removes comments from all lines except those between the two reserved words:

#!/bin/sh
sed '
/begin/,/end/ !{
s/#.*//
s/[ ^I]*$//
/^$/ d
p
}
'

Click here to get file: sed_begin_end2.sh

Writing a file with the 'w' command

sed -n 's/^[0-9]*[02468] /&/w even' <file

sed -n '/^[0-9]*[02468]/ w even' <file

Reading in a file with the 'r' command

sed '$r end' <in>out

will append the file "end" at the end of the file (address "$)." The following will insert a file after the line with the word "INCLUDE:"

sed '/INCLUDE/ r file' <in >out

The order of the delete command "d" and the read file command "r" is important. Change the order and it will not work. There are two subtle actions that prevent this from working. The first is the "r" command writes the file to the output stream. The file is not inserted into the pattern space, and therefore cannot be modified by any command. Therefore the delete command does not affect the data read from the file.

The other subtlty is the "d" command deletes the current data in the pattern space. Once all of the data is deleted, it does make sense that no other action will be attempted. Therefore a "d" command executed in a curly brace also aborts all further actions. As an example, the substitute command below is never executed:

#!/bin/sh
# this example is WRONG
sed -e '1 {
d
s/.*//
}'

Click here to get file: sed_bad_example.sh

The earlier example is a crude version of the C preprocessor program. The file that is included has a predetermined name. It would be nice if sed allowed a variable (e.g "1)" instead of a fixed file name. Alas, sed doesn't have this ability. You could work around this limitation by creating sed commands on the fly, or by using shell quotes to pass variables into the sed script. Suppose you wanted to create a command that would include a file like cpp, but the filename is an argument to the script. An example of this script is:

% include 'sys/param.h' <file.c >file.c.new

A shell script to do this would be:

#!/bin/sh
# watch out for a '/' in the parameter
# use alternate search delimiter
sed -e '_#INCLUDE <'$1'>_{
r '$1'
d
}'

Click here to get file: sed_include1.sh

SunOS and the # Comment Command

Adding, Changing, Inserting new lines

Append a line with 'a'

You could eliminate two lines in the shell script if you wish:

#!/bin/sh
sed '/WORD/ a
Add this line after every line with WORD'

Click here to get file: sed_add_line_after_word1.sh

I prefer the first form because it's easier to add a new command by adding a new line and because the intent is clearer. There must not be a space after the "."

Insert a line with 'i'

Change a line with 'c'

A "d" command followed by a "a" command won't work, as I discussed earlier. The "d" command would terminate the current actions. You can combine all three actions using curly braces:

#!/bin/sh
sed '
/WORD/ {
i
Add this line before
c
Change the line to this one
a
Add this line after
'

Click here to get file: sed_insert_append_change.sh

Leading tabs and spaces in a sed script

#!/bin/sh
sed '
a
This line starts with a tab
'

Adding more than one line

#!/bin/sh
sed '
/WORD/ a
Add this line
This line
And this line
'

Adding lines and the pattern space

Address ranges and the above commands

#!/bin/sh
sed '
/begin/,/end/ c
***DELETED***
'

If you need to do this, you can use the curly braces, as that will let you perform the operation on every line:

#!/bin/sh
# add a blank line after every line
sed '1,$ {
a

}'

Multi-Line Patterns

Most UNIX utilities are line oriented. Regular expressions are line oriented. Searching for patterns that covers more than one line is not an easy task. (Hint: It will be very shortly.)

Sed reads in a line of text, performs commands which may modify the line, and outputs modification if desired. The main loop of a sed script looks like this:

1. The next line is read from the input file and places in the pattern space. If the end of file is found, and if there are additional files to read, the current file is closed, the next file is opened, and the first line of the new file is placed into the pattern space.
2. The line count is incremented by one. Opening a new file does not reset this number.
3. Each sed command is examined. If there is a restriction places on the command, and the current line in the pattern space meets that restriction, the command is executed. Some commands, like "n" or "d" cause sed to go to the top of the loop. The "q" command causes sed to stop. Otherwise the next command is examined.
4. After all of the commands are examined, the pattern space is output unless sed has the optional "-n" argument.

The restriction before the command determines if the command is executed. If the restriction is a pattern, and the operation is the delete command, then the following will delete all lines that have the pattern:

/PATTERN/ d

If the restriction is a pair of numbers, then the deletion will happen if the line number is equal to the first number or greater than the first number and less than or equal to the last number:

10,20 d

If the restriction is a pair of patterns, there is a variable that is kept for each of these pairs. If the variable is false and the first pattern is found, the variable is made true. If the variable is true, the command is executed. If the variable is true, and the last pattern is on the line, after the command is executed the variable is turned off:

/begin/,/end/ d

Whew! That was a mouthful. Hey, I said it was a review. If you have read the previous tutorials, you should have breezed through this. You may want to refer back to this review, because I covered several subtle points. My choice of words was deliberate. It covers some unusual cases, like:

# what happens if the second number
# is less than the first number?
sed -n '20,1 p' file

and

# generate a 10 line file with line numbers
# and see what happens when two patterns overlap
yes | head -10 | cat -n |
sed -n -e '/1/,/7/ p' -e '/5/,/9/ p'

AB
CD
EF
GH
IJ

+----------------+---------+------------------------------------------+
|Pattern   Next	 | Command | Output	       New Pattern   New Next |
|Space	   Input |	   |		       Space	     Input    |
+----------------+---------+------------------------------------------+
|AB	   CD	 | n	   | <default>   CD	     EF	      |
|AB	   CD	 | d	   | -		       CD	     EF	      |
|AB	   CD	 | p	   | AB		       CD	     EF	      |
+----------------+---------+------------------------------------------+

The "n" command may or may not generate output depending upon the existence of the "-n" flag.

That review is a little easier to follow, isn't it? Before I jump into multi-line patterns, I wanted to cover three more commands:

Print line number with =

# add line numbers first,
# then use grep,
# then just print the number
cat -n file | grep 'PATTERN' | awk '{print $1}'

The sed solution is:

sed -n '/PATTERN/ =' file

Earlier I used the following to find the number of lines in a file

#!/bin/sh
lines=`wc -l file | awk '{print $1}' `

Using the "=" command can simplify this:

#!/bin/sh
lines=`sed -n '$=' file `

The "=" command only accepts one address, so if you want to print the number for a range of lines, you must use the curly braces:

#!/bin/sh
# Just print the line numbers
sed -n '/begin/,/end/ {
=
d
}' file

Since the "=" command only prints to standard output, you cannot print the line number on the same line as the pattern. You need to edit multi-line patterns to do this.

Transform with y

sed 'y/abcdef/ABCDEF/' file

I could have used an example that converted all 26 letters into upper case, and while this column covers a broad range of topics, the "column" prefers a narrower format.

If you wanted to convert a line that contained a hexadecimal number (e.g. 0x1aff) to upper case (0x1AFF), you could use:

sed '/0x[0-9a-zA-Z]*/ y/abcdef/ABCDEF' file

Displaying control characters with a l

Working with Multiple Lines

The "n" command will print out the current pattern space (unless the "-n" flag is used), empty the current pattern space, and read in the next line of input. The "N" command does not print out the current pattern space and does not empty the pattern space. It reads in the next line, but appends a new line character along with the input line itself to the pattern space.

The "d" command deleted the current pattern space, reads in the next line, puts the new line into the pattern space, and aborts the current command, and starts execution at the first sed command. This is called starting a new "cycle." The "D" command deletes the first portion of the pattern space, up to the new line character, leaving the rest of the pattern alone. Like "d," it stops the current command and starts the command cycle over again. However, it will not print the current pattern space. You must print it yourself, as step 4 mentioned in the beginning of the article, is not executed. If the "D" command is executed with a group of other commands in a curly brace, commands after the "D" command are ignored. The next group of sed commands is executed, unless the pattern space is emptied. If this happens, the cycle is started from the top and a new line is read.

The "p" command prints the entire pattern space. The "P" command only prints the first part of the pattern space, up to the NEWLINE character.

Some examples might demonstrate "N" by itself isn't very useful. the filter

sed -e 'N'

#!/bin/sh
sed '
# look for a "#" at the end of the line
/#$/ {
# Found one - now read in the next line
N
# delete the "#" and the new line character,
s/#n//
}' file

You could search for two lines containing "ONE" and "TWO" and only print out the two consecutive lines:

#!/bin/sh
sed -n '
/ONE/ {
# found "ONE" - read in next line
N
# look for "TWO" on the second line
# and print if there.
/n.*TWO/ p
}' file

The next example would delete everything between "ONE" and "TWO:"

#!/bin/sh
sed '
/ONE/ {
# append a line
N
# search for TWO on the second line
/n.*TWO/ {
# found it - now edit making one line
s/ONE.*n.*TWO/ONE TWO/
}
}' file

You can either search for a particular pattern on two consecutive lines, or you can search for two consecutive words that may be split on a line boundary. The next example will look for two words which are either on the same line or one is on the end of a line and the second is on the beginning of the next line. If found, the first word is deleted:

#!/bin/sh
sed '
/ONE/ {
# append a line
N
# "ONE TWO" on same line
s/ONE TWO/TWO/
# "ONE
# TWO" on two consecutive lines
s/ONEnTWO/TWO/
}
}' file

Let's use the
"D" command, and if we find a line containing
"TWO" immediately after a line containing
"ONE," then delete the first line:


#!/bin/sh
sed '
/ONE/ {
# append a line
N
# if TWO found, delete the first line
/n.*TWO/ D
}' file

Click here to get file: sed_delete_line_after_word.sh

If we wanted to print the first line instead of deleting it, and not print every other line, change the "D" to a "P" and add a "-n" as an argument to sed:

#!/bin/sh
sed -n '
# by default - do not print anything
/ONE/ {
# append a line
N
# if TWO found, print the first line
/n.*TWO/ P
}' file

Click here to get file: sed_print_line_after_word.sh

It is very common to combine all three multi-line commands. The typical order is "N," "P" and lastly "D." This one will delete everything between "ONE" and "TWO" if they are on one or two consecutive lines:

#!/bin/sh
sed '
/ONE/ {
# append the next line
N
# look for "ONE" followed by "TWO"
/ONE.*TWO/ {
# delete everything between
s/ONE.*TWO/ONE TWO/
# print
P
# then delete the first line
D
}
}' file

Click here to get file: sed_delete_between_two_words.sh

Earlier I talked about the "=" command, and using it to add line numbers to a file. You can use two invocations of sed to do this (although it is possible to do it with one, but that must wait until next section. The first sed command will output a line number on one line, and then print the line on the next line. The second invocation of sed will merge the two lines together:

#!/bin/sh
sed '=' file |
sed '{
N
s/n/ /
}'

Click here to get file: sed_merge_two_lines.sh

If you find it necessary, you can break one line into two lines, edit them, and merge them together again. As an example, if you had a file that had a hexadecimal number followed by a word, and you wanted to convert the first word to all upper case, you can use the "y" command, but you must first split the line into two lines, change one of the two, and merge them together. That is, a line containing

0x1fff table2

will be changed into two lines:

0x1fff
table2

and the first line will be converted into upper case. I will use tr to convert the space into a new line:

#!/bin/sh
tr ' ' '012' file|
sed ' {
y/abcdef/ABCDEF/
N
s/n/ /
}'

Click here to get file: sed_split.sh

It isn't obvious, but sed could be used instead of tr. You can embed a new line in a substitute command, but you must escape it with a backslash. It is unfortunate that you must use "n" in the left side of a substitute command, and an embedded new line in the right hand side. Heavy sigh. Here is the example:

#!/bin/sh
sed '
s/ /
/' |
sed ' {
y/abcdef/ABCDEF/
N
s/n/ /
}'

Click here to get file: sed_split_merge.sh

Sometimes I add a special character as a marker, and look for that character in the input stream. When found, it indicates the place a blank used to be. A back slash is a good character, except it must be escaped with a backslash, and makes the sed script obscure. Save it for that guy who keeps asking dumb questions. The sed script to change a blank into a "" following by a new line would be:

#!/bin/sh
sed 's/ /
/' file

Click here to get file: sed_addslash_before_blank.sh

Yeah. That's the ticket. Or use the C shell and really confuse him!

#!/bin/csh -f
sed '
s/ /
/' file

Click here to get file: sed_addslash_before_blank.csh

A few more examples of that, and he'll never ask you a question again! I think I'm getting carried away. I'll summarize with a chart that covers the features we've talked about:

+----------------+---------+------------------------------------------+
|Pattern   Next	 | Command | Output	       New Pattern   New Next |
|Space	   Input |	   |		       Space	     Input    |
+----------------+---------+------------------------------------------+
|AB	   CD	 | n	   | <default>   CD	     EF	      |
|AB	   CD	 | N	   | -		       ABnCD	     EF	      |
|AB	   CD	 | d	   | -		       CD	     EF	      |
|AB	   CD	 | D	   | -		       CD	     EF	      |
|AB	   CD	 | p	   | AB		       CD	     EF	      |
|AB	   CD	 | P	   | AB		       CD	     EF	      |
+----------------+---------+------------------------------------------+
|ABnCD	   EF	 | n	   | <default>   EF	     GH	      |
|ABnCD	   EF	 | N	   | -		       ABnCDnEF    GH	      |
|ABnCD	   EF	 | d	   | -		       EF	     GH	      |
|ABnCD	   EF	 | D	   | -		       CD	     EF	      |
|ABnCD	   EF	 | p	   | ABnCD	       ABnCD	     EF	      |
|ABnCD	   EF	 | P	   | AB		       ABnCD	     EF	      |
+----------------+---------+------------------------------------------+

The Hold Buffer

So far we have talked about three concepts of sed: (1) The input stream or data before it is modified, (2) the output stream or data after it has been modified, and (3) the pattern space, or buffer containing characters that can be modified and send to the output stream.

There is one more "location" to be covered: the hold buffer or hold space. Think of it as a spare pattern buffer. It can be used to "copy" or "remember" the data in the pattern space for later. There are five commands that use the hold buffer.

Exchange with x

The "x" command eXchanges the pattern space with the hold buffer. By itself, the command isn't useful. Executing the sed command

sed 'x'

as a filter adds a blank line in the front, and deletes the last line. It looks like it didn't change the input stream significantly, but the sed command is modifying every line.

The hold buffer starts out containing a blank line. When the "x" command modifies the first line, line 1 is saved in the hold buffer, and the blank line takes the place of the first line. The second "x" command exchanges the second line with the hold buffer, which contains the first line. Each subsequent line is exchanged with the preceding line. The last line is placed in the hold buffer, and is not exchanged a second time, so it remains in the hold buffer when the program terminates, and never gets printed. This illustrates that care must be taken when storing data in the hold buffer, because it won't be output unless you explicitly request it.

Example of Context Grep

One use of the hold buffer is to remember previous lines. An example of this is a utility that acts like grep as it shows you the lines that match a pattern. In addition, it shows you the line before and after the pattern. That is, if line 8 contains the pattern, this utility would print lines 7, 8 and 9.

One way to do this is to see if the line has the pattern. If it does not have the pattern, put the current line in the hold buffer. If it does, print the line in the hold buffer, then the current line, and then the next line. After each set, three dashes are printed. The script checks for the existence of an argument, and if missing, prints an error. Passing the argument into the sed script is done by turning off the single quote mechanism, inserting the "$1" into the script, and starting up the single quote again:

#!/bin/sh
# grep3 - prints out three lines around pattern
# if there is only one argument, exit

case $# in
1);;
*) echo "Usage: $0 pattern";exit;;
esac;
# I hope the argument doesn't contain a /
# if it does, sed will complain

# use sed -n to disable printing
# unless we ask for it
sed -n '
'/$1/' !{
#no match - put the current line in the hold buffer
x
# delete the old one, which is
# now in the pattern buffer
d
}
'/$1/' {
# a match - get last line
x
# print it
p
# get the original line back
x
# print it
p
# get the next line
n
# print it
p
# now add three dashes as a marker
a
---
# now put this line into the hold buffer
x
}'

Click here to get file: grep3.sh

You could use this to show the three lines around a keyword, i.e.:

grep3 vt100 </etc/termcap

Hold with h or H

The "x" command exchanges the hold buffer and the pattern buffer. Both are changed. The "h" command copies the pattern buffer into the hold buffer. The pattern buffer is unchanged. An identical script to the above uses the hold commands:

#!/bin/sh
# grep3 version b - another version using the hold commands
# if there is only one argument, exit

case $# in
1);;
*) echo "Usage: $0 pattern";exit;;
esac;

# again - I hope the argument doesn't contain a /

# use sed -n to disable printing

sed -n '
'/$1/' !{
# put the non-matching line in the hold buffer
h
}
'/$1/' {
# found a line that matches
# append it to the hold buffer
H
# the hold buffer contains 2 lines
# get the next line
n
# and add it to the hold buffer
H
# now print it back to the pattern space
x
# and print it.
p
# add the three hyphens as a marker
a
---
}'

Click here to get file: grep3a.sh

Keeping more than one line in the hold buffer

The "H" command allows you to combine several lines in the hold buffer. It acts like the "N" command as lines are appended to the buffer, with a "n" between the lines. You can save several lines in the hold buffer, and print them only if a particular pattern is found later.

As an example, take a file that uses spaces as the first character of a line as a continuation character. The files /etc/termcap, /etc/printcap, makefile and mail messages use spaces or tabs to indicate a continuing of an entry. If you wanted to print the entry before a word, you could use this script. I use a "^I" to indicate an actual tab character:

#!/bin/sh
# print previous entry
sed -n '
/^[ ^I]/!{
# line does not start with a space or tab,
# does it have the pattern we are interested in?
'/$1/' {
# yes it does. print three dashes
i
---
# get hold buffer, save current line
x
# now print what was in the hold buffer
p
# get the original line back
x
}
# store it in the hold buffer
h
}
# what about lines that start
# with a space or tab?
/^[ ^I]/ {
# append it to the hold buffer
H
}'

Click here to get file: grep_previous.sh

You can also use the "H" to extend the context grep. In this example, the program prints out the two lines before the pattern, instead of a single line. The method to limit this to two lines is to use the "s" command to keep one new line, and deleting extra lines. I call it grep4:

#!/bin/sh

# grep4: prints out 4 lines around pattern
# if there is only one argument, exit

case $# in
1);;
*) echo "Usage: $0 pattern";exit;;
esac;

sed -n '
'/$1/' !{
# does not match - add this line to the hold space
H
# bring it back into the pattern space
x
# Two lines would look like .*n.*
# Three lines look like .*n.*n.*
# Delete extra lines - keep two
s/^.*n(.*n.*)$/1/
# now put the two lines (at most) into
# the hold buffer again
x
}
'/$1/' {
# matches - append the current line
H
# get the next line
n
# append that one also
H
# bring it back, but keep the current line in
# the hold buffer. This is the line after the pattern,
# and we want to place it in hold in case the next line
# has the desired pattern
x
# print the 4 lines
p
# add the mark
a
---
}'

Click here to get file: grep4.sh
You can modify this to print any number of lines around a pattern. As you can see, you must remember what is in the hold space, and what is in the pattern space. There are other ways to write the same routine.

Get with g or G

Instead of exchanging the hold space with the pattern space, you can copy the hold space to the pattern space with the "g" command. This deletes the pattern space. If you want to append to the pattern space, use the "G" command. This adds a new line to the pattern space, and copies the hold space after the new line.

Here is another version of the "grep3" command. It works just like the previous one, but is implemented differently. This illustrates that sed has more than one way to solve many problems. What is important is you understand your problem, and document your solution:

#!/bin/sh
# grep3 version c: use 'G' instead of H

# if there is only one argument, exit

case $# in
1);;
*) echo "Usage: $0 pattern";exit;;
esac;

# again - I hope the argument doesn't contain a /

sed -n '
'/$1/' !{
# put the non-matching line in the hold buffer
h
}
'/$1/' {
# found a line that matches
# add the next line to the pattern space
N
# exchange the previous line with the
# 2 in pattern space
x
# now add the two lines back
G
# and print it.
p
# add the three hyphens as a marker
a
---
# remove first 2 lines
s/.*n.*n(.*)$/1/
# and place in the hold buffer for next time
h
}'

Click here to get file: grep3c.sh

The "G" command makes it easy to have two copies of a line. Suppose you wanted to the convert the first hexadecimal number to uppercase, and don't want to use the script I described in an earlier column

#!/bin/sh
# change the first hex number to upper case format
# uses sed twice
# used as a filter
# convert2uc out
sed '
s/ /
/' |
sed ' {
y/abcdef/ABCDEF/
N
s/n/ /
}'

Click here to get file: convert2uc.sh

Here is a solution that does not require two invocations of sed:

#!/bin/sh
# convert2uc version b
# change the first hex number to upper case format
# uses sed once
# used as a filter
# convert2uc out
sed '
{
# remember the line
h
#change the current line to upper case
y/abcdef/ABCDEF/
# add the old line back
G
# Keep the first word of the first line,
# and second word of the second line
# with one humongeous regular expression
s/^([^ ]*) .*n[^ ]* (.*)/1 2/
}'

Click here to get file: convert2uc1.sh
This example only converts the letters "a" through "f" to upper case. This was chosen to make the script easier to print in these narrow columns. You can easily modify the script to convert all letters to uppercase, or to change the first letter, second word, etc.

Flow Control

As you learn about sed you realize that it has it's own programming language. It is true that it's a very specialized and simple language. What language would be complete without a method of changing the flow control? There are three commands sed uses for this. You can specify a label with an text string followed by a colon. The "b" command branches to the label. The label follows the command. If no label is there, branch to the end of the script. The "t" command is used to test conditions. Before I discuss the "t" command, I will show you an example using the "b" command.

This example remembers paragraphs, and if it contains the pattern (specified by an argument), the script prints out the entire paragraph.

#!/bin/sh
sed -n '
# if an empty line, check the paragraph
/^$/ b para
# else add it to the hold buffer
H
# at end of file, check paragraph
$ b para
# now branch to end of script
b
# this is where a paragraph is checked for the pattern
:para
# return the entire paragraph
# into the pattern space
x
# look for the pattern, if there - print
/'$1'/ p
'

Click here to get file: grep_paragraph.sh

Testing with t

You can execute a branch if a pattern is found. You may want to execute a branch only if a substitution is made. The command "t label" will branch to the label if the last substitute command modified the pattern space.

One use for this is recursive patterns. Suppose you wanted to remove white space inside parenthesis. These parentheses might be nested. That is, you would want to delete a string that looked like "( ( ( ())) )." The sed expressions

sed 's/([ ^I]*)/g'

would only remove the innermost set. You would have to pipe the data through the script four times to remove each set or parenthesis. You could use the regular expression

sed 's/([ ^I()]*)/g'

but that would delete non-matching sets of parenthesis. The "t" command would solve this:

#!/bin/sh
sed '
:again
s/([ ^I]*)//g
t again
'

Click here to get file: delete_nested_parens.sh

An alternate way of adding comments

There is one way to add comments in a sed script if you don't have a version that supports it. Use the "a" command with the line number of zero:

#!/bin/sh
sed '
/begin/ {
0i
This is a comment
It can cover several lines
It will work with any version of sed
}'

Click here to get file: sed_add_comments.sh

The poorly undocumented ;

There is one more sed command that isn't well documented. It is the ";" command. This can be used to combined several sed commands on one line. Here is the grep4 script I described earlier, but without the comments or error checking and with semicolons between commands:

#!/bin/sh
sed -n '
'/$1/' !{;H;x;s/^.*n(.*n.*)$/1/;x;}
'/$1/' {;H;n;H;x;p;a
---
}'

Click here to get file: grep4a.sh

Yessireebob! Definitely character building. I think I have made my point. As far as I am concerned, the only time the semicolon is useful is when you want to type the sed script on the command line. If you are going to place it in a script, format it so it is readable. I have mentioned earlier that many versions of sed do not support comments except on the first line. You may want to write your scripts with comments in them, and install them in "binary" form without comments. This should not be difficult. After all, you have become a sed guru by now. I won't even tell you how to write a script to strip out comments. That would be insulting your intelligence.

Passing regular expressions as arguments

In the earlier scripts, I mentioned that you would have problems if you passed an argument to the script that had a slash in it. In fact, and regular expression might cause you problems. A script like the following is asking to be broken some day:

#!/bin/sh
sed 's/'$1'//g'

If the argument contains any of these characters in it you may get a broken script: "/.*[]^$." One solution is to add a backslash before each of those characters:

#!/bin/sh
arg=`sed 's:[/.*[]^$]::g < $1
EndOfThisMess
`
sed 's/'$arg'//g'

Click here to get file: sed_with_regular_expressions.sh

or

#!/bin/sh
arg=`echo $1 | sed 's:[/.*[]^$]::g'`
sed 's/'$arg'//g

Click here to get file: sed_with_regular_expressions1.sh
If you were searching for the pattern "^../," the script would convert this into "^../" before passing it to sed.

Command Summary

As I promised earlier, here is a table that summarizes the different commands. The second column specifies if the command can have a range or pair of addresses (with a 2) or a single address or pattern (with a 1). The next four columns specifies which of the four buffers or streams are modified by the command. Some commands only affect the output stream, others only affect the hold buffer. If you remember that the pattern space is output (unless a "-n" was given to sed), this table should help you keep track of the various commands.

+--------------------------------------------------------+
 |Command   Address		Modifications to	  |
 |	    or Range   Input	Output	 Pattern   Hold	  |
 |		       Stream	Stream	 Space	   Buffer |
 +--------------------------------------------------------+
 |=	    -	       -	Y	 -	   -	  |
 |a	    1	       -	Y	 -	   -	  |
 |b	    2	       -	-	 -	   -	  |
 |c	    2	       -	Y	 -	   -	  |
 |d	    2	       Y	-	 Y	   -	  |
 |D	    2	       Y	-	 Y	   -	  |
 |g	    2	       -	-	 Y	   -	  |
 |G	    2	       -	-	 Y	   -	  |
 |h	    2	       -	-	 -	   Y	  |
 |H	    2	       -	-	 -	   Y	  |
 |i	    1	       -	Y	 -	   -	  |
 |l	    1	       -	Y	 -	   -	  |
 |n	    2	       Y	*	 -	   -	  |
 |N	    2	       Y	-	 Y	   -	  |
 |p	    2	       -	Y	 -	   -	  |
 |P	    2	       -	Y	 -	   -	  |
 |q	    1	       -	-	 -	   -	  |
 |r	    1	       -	Y	 -	   -	  |
 |s	    2	       -	-	 Y	   -	  |
 |t	    2	       -	-	 -	   -	  |
 |w	    2	       -	Y	 -	   -	  |
 |x	    2	       -	-	 Y	   Y	  |
 |y	    2	       -	-	 Y	   -	  |
 +--------------------------------------------------------+

The "n" command may or may not generate output, depending on the "-n" option. The "r" command can only have one address, despite the documentation.

In Conclusion

This concludes my tutorial on sed. It is possible to find shorter forms of some of my scripts. However, I chose these examples to illustrate some basic constructs. I wanted clarity, not obscurity. I hope you enjoyed it.

This document was translated by troff2html v0.21 on September 22, 2001.