You'll get your Mac news here from now on...

Help TMO Grow

Subscriber Login

Advertising Info


The Mac Observer Express Daily Newsletter


More Info

Site Navigation

Home
News
Tips
Columns & Editorials
Reviews
Reports
Archives
Search
Forums
Links
Mac Links
Software
Reports
Contact

Mac OS X Command Line 101
by Richard Burton


Of Pipes & More
Part VIII of this series...
June 14th, 2002

"Tiny differences in input could quickly become overwhelming differences in output."
-
James Gleick, "Chaos"

This series is designed to help you learn more about the Mac OS X command line. If you have any questions about what you read here, check out the earlier columns, write back in the comments below, or join us in the Hardcore X! forum.

In the last column, we looked at how input and output can be redirected to and from files. However, what if you want to run several commands on your data? Well, you could do something like this:

    [localhost:~] dr_unix% command1 sourcefile > tempfile1
    [localhost:~] dr_unix% command2 tempfile1 > tempfile2
    [localhost:~] dr_unix% rm tempfile1
    [localhost:~] dr_unix% command3 tempfile2 > tempfile3
    [localhost:~] dr_unix% rm tempfile2
    [localhost:~] dr_unix% command4 tempfile3 > out_file
    [localhost:~] dr_unix% rm tempfile3
    
That is a lot of typing; it discourages laziness and impatience. Worse, you might erase the wrong file along the way. And even worse, it is slow. Not only do you have to spend a lot of time typing and waiting, but constant I/O to a drive is generally the worst drag on speed.

If you think about it, you really don't need those tempfiles. Oh, a crusty old mainframer might hit you with a slide rule if you say this out loud, but think about it: all you need is a mechanism that will take the output from command1 and make it the standard input for command2, and so on.

In Unix shells, this is accomplished with a pipe, "|". We could have performed the previous task all in one line.

    [localhost:~] dr_unix%  command1 sourcefile | command2 | command3 | command4 > out_file
    
Recall the example from the previous column where we typed:
    [localhost:~] dr_unix% who -Hu > who_list
    [localhost:~] dr_unix% cat who_list
    USER     LINE     WHEN         IDLE     FROM
    dr_unix  console  May 10 16:09 04:16 
    dr_unix  ttyp1    May 10 19:53   .   
    dr_unix  ttyp2    May 10 20:20   .   
    [localhost:~] dr_unix% tr a-z A-Z < who_list > Cap_who_list
    [localhost:~] dr_unix% cat Cap_who_list
    USER     LINE     WHEN         IDLE     FROM
    DR_UNIX  CONSOLE  MAY 10 16:09 04:16 
    DR_UNIX  TTYP1    MAY 10 19:53   .   
    DR_UNIX  TTYP2    MAY 10 20:20   .   
    [localhost:~] dr_unix% 
    
That can now be replaced with:
    [localhost:~] dr_unix% who -Hu | tr a-z A-Z > Cap_who_list
    [localhost:~] cat Cap_who_list
    USER     LINE     WHEN         IDLE     FROM
    DR_UNIX  CONSOLE  MAY 10 16:09 22:24 
    DR_UNIX  TTYP1    MAY 11 13:30   .   
    DR_UNIX  TTYP2    MAY 11 13:43   .   
    DR_UNIX  TTYP3    MAY 10 22:34 15:36 
    [localhost:~] dr_unix% 
    
What this does is that, when tcsh is parsing the command line behind the scenes, it sees the "|" character and says "What ho! A pipe. The Unix genius who wrote this wants me to take the output from the command on the left, who -Hu, and make it the input for the command on the right, tr a-z A-Z." So it does.

The beauty of this is that you can take very simple tools, string them together in whatever order you wish, and create complex tools. This gives you great power. Given the many options that Unix commands can have, and the number of Unix commands, this also gives you great flexibility. And what's more, it makes things very simple for you. Anyone who has ever written code to do this in a language like C can tell you how great the shell is at this. (Actually, once they start they'll never shut up; so don't ask, just take my word for it.)

One of the more common uses of the pipe on the command line is to take a large amount of output and present it only one screen at a time using the more command (or page or less). Recall from an earlier column that we took a quick look at the /usr/bin directory where a lot of commands reside. First, using a pipe to connect commands, we can determine how many files exist in the directory:

    [localhost:/usr/bin] dr_unix% ls -l | wc -l
         457 
    [localhost:/usr/bin] dr_unix% 
    
That's a lot of files, far more than is standard for a Terminal.app console. So, what if we want to see all of the files, but not have them fly by at breakneck speed? Why, pipe the output through more (or page or less).
    [localhost:/usr/bin] dr_unix% ls | more
    CFInfoPlistConverter
    a2p
    addftinfo
    addr
    aexml
    afmtodit
    appleping
    appletviewer
    apply
    apropos
    ar
    arch
    as
    asa
    at
    at_cho_prn
    atlookup
    atos
    atprint
    atq
    atrm
    atstatus
    autoconf
    [localhost:/usr/bin] dr_unix% 
    
(I hit 'q' at this point to quit out of more.)

The pipe is only one way to put more than one command on one command line, because there are different ways that you might want to combine them. For example, we've already seen the use of backticks, `, to expand the results of a command into another command. To show the difference between the pipe and the backtick, let's look at how we can combine the ls and wc commands.

Consider my home directory:

    [localhost:~] dr_unix% ls
    Adam.txt   Documents  Movies     Pictures   Sites      temp.html  who_list
    Desktop    Library    Music      Public     personal   test_1.txt
    [localhost:~] dr_unix% 
    
[Consult local listings for home directories in your area.] Using backticks will take the results of ls above and use them as the arguments for wc, thus:
    [localhost:~] dr_unix% wc -l `ls`
           1 Adam.txt
    wc: Desktop: Is a directory
           0 Desktop
    wc: Documents: Is a directory
           0 Documents
    wc: Library: Is a directory
           0 Library
    wc: Movies: Is a directory
           0 Movies
    wc: Music: Is a directory
           0 Music
    wc: Pictures: Is a directory
           0 Pictures
    wc: Public: Is a directory
           0 Public
    wc: Sites: Is a directory
           0 Sites
    wc: personal: Is a directory
           0 personal
           0 temp.html
           1 test_1.txt
           4 who_list
           6 total
    [localhost:~] dr_unix% 
    
The pipe, on the other hand, takes the output from ls and makes it as the standard input for the wc command.
    [localhost:~] dr_unix% ls | wc -l
          13 
    [localhost:~] dr_unix% 
    
Thus, the backticks give you the line count for each non-hidden item in your home directory, whereas the pipe builds a command that tells you how many non-hidden items there are in the home directory. If this seems slightly confusing at first, don't worry. Try using pipes and backticks on your own to see the differences; after all, doing is the best way to learn.

A few other ways to combine commands exist. Generally, they are not used on the command line; they are more common in shell commands. However, they do exist, and we'll eventually want to write shell scripts anyway, so we may as well be aware of them. The first, and simplest, uses the ';' character to separate commands:

command1 ; command2

The command line simply executes the commands sequentially. command1 is run, and when it's finished, command2 is run.

    [localhost:~] dr_unix% who -Hu ; ls
    USER     LINE     WHEN         IDLE     FROM
    dr_unix  console  May 13 07:08  old  
    dr_unix  ttyp1    May 16 07:39   .   
    dr_unix  ttyp2    May 16 07:39   .   
    Adam.txt   Documents  Movies     Pictures   Sites      temp.html  who_list
    Desktop    Library    Music      Public     personal   test_1.txt
    [localhost:~] dr_unix% 
    
If you enclose commands in parentheses, they form a subshell. The commands are treated as a command group:

(command1 ; command2)

The idea behind this is to force commands to be grouped together so they are treated as one entity. We saw this when we looked at I/O redirection:

    [localhost:~] dr_unix% (ls who_list xiphoid > temp_out) >& temp_err
    [localhost:~] dr_unix% cat temp_out
    who_list
    [localhost:~] dr_unix% cat temp_err
    ls: xiphoid: No such file or directory
    [localhost:~] dr_unix%
    
When we use a command group by itself on the command line, the shell behaves as if the parentheses weren't there
    [localhost:~] dr_unix% (who -Hu; ls)
    USER     LINE     WHEN         IDLE     FROM
    dr_unix  console  May 13 07:08  old  
    dr_unix  ttyp1    May 16 07:39   .   
    dr_unix  ttyp2    May 16 07:39   .   
    Adam.txt   Documents  Movies     Pictures   Sites      temp.html  who_list
    Desktop    Library    Music      Public     personal   test_1.txt
    [localhost:~] dr_unix% 
    
Two related forms also allow you to string together commands in a conditional sense. (Yes, Ye Editor, this is getting close to shell scripting, but this is where it fits.) The first is known as the AND form:

command1 && command2

The shell will run command1. If command1 fails, the whole command ends and command2 is not run. However, if command1 succeeds, command2 is then run. First consider an example where both succeed.

    [localhost:~] dr_unix% ls && who -Hu
    Adam.txt   Documents  Movies     Pictures   Sites      temp.html  who_list
    Desktop    Library    Music      Public     personal   test_1.txt
    USER     LINE     WHEN         IDLE     FROM
    dr_unix  console  May 13 07:08  old  
    dr_unix  ttyp1    May 16 07:39   .   
    dr_unix  ttyp2    May 16 07:39   .   
    [localhost:~] dr_unix% 
    
tcsh finds the ls command, runs it, then runs the who -Hu command. As a counterexample, consider the following, which tries to run the non-existent command fred:
    [localhost:~] dr_unix% fred && who -Hu
    fred: Command not found.
    [localhost:~] dr_unix% 
    
tcsh cannot find the command fred, so it doesn't run the who -Hu command.

The second form is known as the OR form:

command1 || command2

The shell will run EITHER command1 OR command2. If command1 succeeds, the whole command ends and command2 is not run. However, if command1 fails, command2 is then run. First consider an example where command1 succeeds.

    [localhost:~] dr_unix% ls || who -Hu
    Adam.txt   Documents  Movies     Pictures   Sites      temp.html  who_list
    Desktop    Library    Music      Public     personal   test_1.txt
    [localhost:~] dr_unix% 
    
tcsh sees the command line ls || who -HU and, behind the scenes, splits the command line into the first command (ls) and the second command (who -Hu). It successfully runs the first command, notes that the two are connected by an OR (||), and ends without running the second command. And if the first command were to fail?
    [localhost:~] dr_unix% fred || who -Hu
    fred: Command not found.
    USER     LINE     WHEN         IDLE     FROM
    dr_unix  console  May 17 09:28 21:59 
    dr_unix  ttyp1    May 18 06:56   .   
    dr_unix  ttyp2    May 18 07:00   .   
    [localhost:~] dr_unix% 
    
Here, tcsh could not find the command fred, which results in a failure. So it looks at the second command, who -Hu, and attempts to run it (in this case, successfully).

---

[A final note for people looking ahead: the shell's definition of success and failure may not be quite the same as yours and mine might be. Consider:

    [localhost:~] dr_unix% ls not_there
    ls: not_there: No such file or directory
    [localhost:~] dr_unix% 
    
So we would think that is an error; after all, the file not_there is not found. So you and I might think that ls not_there && who -Hu will not run the who -Hu command. However:
    [localhost:~] dr_unix% ls not_there && who -Hu
    ls: not_there: No such file or directory
    USER     LINE     WHEN         IDLE     FROM
    dr_unix  console  May 13 07:08  old  
    dr_unix  ttyp1    May 16 07:39   .   
    dr_unix  ttyp2    May 16 07:39   .   
    [localhost:~] dr_unix% 
    
What happens is that ls runs, reports the relevant information about the list of files passed in the arguments, and exits successfully. So the shell says "What ho! The first command says it ran successfully; off to the second command, then." That may seem confusing. In fact, it is at first. You can probably get a good handle on how a command reacts from its man page or from considering how it would handle different situations. Examine the following:
    [localhost:~] dr_unix% ls who_list fred && who -Hu
    ls: fred: No such file or directory
    who_list
    USER     LINE     WHEN         IDLE     FROM
    dr_unix  console  May 17 09:28 21:34 
    dr_unix  ttyp1    May 18 06:56   .   
    dr_unix  ttyp2    May 18 07:00   .   
    [localhost:~] dr_unix% 
    
In this case, ls did find the file who_list but not fred, so ls did succeed in listing all of the existing files in its arguments. Clear as mud? Good.

[And as always in unix, there are ways around it. For this particular quirk, using File Inquiry Operators aka File Test Operators can do the trick. These are covered later.]

Again, just be aware of how things are supposed to work, and don't be afraid to try little things before building those methods into bigger things. This will teach you a lot about the command line, and indeed about anything for which you take that approach.]---

So far we've seen different ways to combine several commands into one, using many of the Unix shell's multitasking tools. But what if you want to run separate commands simultaneously? Tune in next week, same geek channel, same geek time.

You are encouraged to send Richard your comments, or to post them below.


Most Recent Mac OS X Command Line 101 Columns

Command Line History & Editing Your Commands
November 22nd

Pico: An Easy To Use Command Line Editor
November 1st

Understanding The "grep" Command In Mac OS X
October 4th

Command Line History & Editing Your Commands
September 6th

Mac OS X Command Line 101 Archives

Back to The Mac Observer For More Mac News!


Richard Burton is a longtime Unix programmer and a handsome brute. He spends his spare time yelling at the television during Colts and Pacers games, writing politically incorrect short stories, and trying to shoot the neighbor's cat (not really) nesting in his garage. He can be seen running roughshod over the TMO forums under the alias tbone1.



Today's Mac Headlines

[Podcast]Podcast - Apple Weekly Report #135: Apple Lawsuits, Banned iPhone Ad, Green MacBook Ad

We also offer Today's News On One Page!

Yesterday's News

 

[Podcast]Podcast - Mac Geek Gab #178: Batch Permission Changes, Encrypting Follow-up, Re-Enabling AirPort, and GigE speeds

We also offer Yesterday's News On One Page!

Mac Products Guide
New Arrivals
New and updated products added to the Guide.

Hot Deals
Great prices on hot selling Mac products from your favorite Macintosh resellers.

Special Offers
Promotions and offers direct from Macintosh developers and magazines.

Software
Browse the software section for over 17,000 Macintosh applications and software titles.

Hardware
Over 4,000 peripherals and accessories such as cameras, printers, scanners, keyboards, mice and more.

© All information presented on this site is copyrighted by The Mac Observer except where otherwise noted. No portion of this site may be copied without express written consent. Other sites are invited to link to any aspect of this site provided that all content is presented in its original form and is not placed within another .