You'll get your Mac news here from now on...

Help TMO Grow

Subscriber Login

Advertising Info


The Mac Observer Express Daily Newsletter


More Info

Site Navigation

Home
News
Tips
Columns & Editorials
Reviews
Reports
Archives
Search
Forums
Links
Mac Links
Software
Reports
Contact

Mac OS X Command Line 101
by Richard Burton


Input & Output From The Command Line
Part VII of this series...
June 7th, 2002

"Tiny differences in input could quickly become overwhelming differences in output."
-
James Gleick, "Chaos"

This series is designed to help you learn more about the Mac OS X command line. If you have any questions about what you read here, check out the earlier columns, write back in the comments below, or join us in the Hardcore X! forum.

So far, we have looked at examples where data is read from the keyboard and the results are printed to the screen. This is nice (in fact, this is vital), but it is not always appropriate. This column will examine how Unix I/O (input/output) can be redirected, a fancy way of saying "how to read and write data to and from files".

["And there was much rejoicing." Enough people have pestered me about this that I've moved this column up on the list. Actually, there is a lot more to I/O redirection than this, but, in the words of Johnny Cash, "One piece at a time."]

Many things happen behind the scenes when you open a window in Terminal.app. Among them, three file descriptors are automagically opened. A file descriptor is a low-level Unix I/O concept that is really only interesting to systems programmers. (What can I say, we're easily entertained.) The three are:

  • 0, which represents Standard Input or stdin. This is the keyboard.
  • 1, which represents Standard Output or stdout. This is the terminal window.
  • 2, which represents Standard Error or stderr. This is also the terminal window.
Now, you may wonder why there are two different file descriptors that both go to the terminal window. The reason for the two descriptors is that there are times when you want to keep the Standard Output separated from the Standard Error. After all, if you are running an application, you want the errors to go into a report to the developers, but they should be separated from the results to go to your boss. (Bosses don't like error messages; reality frightens them.) So having the standard output and the standard error distinct makes a lot of sense. At the same time, the "natural" place for error messages is the same as the "natural" place for output, the terminal window. So the default puts them together, but they can be separated quite easily.

Redirecting Standard Output is quite simple and by far the most common; it uses the > symbol to send the output to a file, thus:

command > outfile

This is best shown by example. The who command prints information on all the users who are currently logged into the system. (Two handy options are -H, which prints header information, and -u, which includes the idle time for each user.) At the command line, from your home directory, type:

    [localhost:~] dr_unix% who -Hu
    USER     LINE     WHEN         IDLE     FROM
    dr_unix  console  May 10 16:09 04:16 
    dr_unix  ttyp1    May 10 19:53   .   
    dr_unix  ttyp2    May 10 20:20   .   
    [localhost:~] dr_unix% who -Hu > who_list
    [localhost:~] dr_unix% ls -l who_list
    -rw-r--r--  1 dr_unix  staff  157 May 10 20:24 who_list
    [localhost:~] dr_unix% cat who_list
    USER     LINE     WHEN         IDLE     FROM
    dr_unix  console  May 10 16:09 04:16 
    dr_unix  ttyp1    May 10 19:53   .   
    dr_unix  ttyp2    May 10 20:20   .   
    [localhost:~] dr_unix% 
    
Yes, it really is that simple. The only pitfalls might be that 1) you could try to write the output to a nonexistent directory or one to which you are not permitted to write; or 2) you have the shell variable noclobber set so you don't overwrite a file that already exists. We'll cover permissions and shell variables like noclobber later. For now, we don't have to worry about them; noclobber can be turned off by typing set noclobber=no, and you can always write to your home directory; just use the tilde metacharacter (~): command > ~/outfile

But what if you want to append the data to a file, instead of overwriting it? Not a problem; use >> instead:

command >> out_file

[As an aside, if you want to force output to a file even with noclobber set, use >! instead of >:

command >! out_file or command >>! out_file

But I digress.]

By the same token, if you want to read input from a file, the < symbol can be used. This is less common and a little trickier; while you are learning the shell, you may be tempted to overuse input redirection. Partly that's because most commands will take a filename that you put on the command line. However, it does have its uses.

As an example, consider the tr command. tr string1 string2 will copy Standard Input to Standard Output after translating the characters in string1 to string2. (In other words, all occurrences of the first character in string1 are changed to the first in string2, the second in string1 to the second in string2, and so on. You can also use a '-' to denote a range of characters.) Let's say you want to make the file who_list all upper case so some addle-pated MCSE, if you'll pardon the redundancy, won't have a hissy-fit. You just need to redirect who_list to Standard Input, thus:

    [localhost:~] dr_unix% cat who_list
    USER     LINE     WHEN         IDLE     FROM
    dr_unix  console  May 10 16:09 04:16 
    dr_unix  ttyp1    May 10 19:53   .   
    dr_unix  ttyp2    May 10 20:20   .   
    [localhost:~] dr_unix% tr a-z A-Z < who_list
    USER     LINE     WHEN         IDLE     FROM
    DR_UNIX  CONSOLE  MAY 10 16:09 04:16 
    DR_UNIX  TTYP1    MAY 10 19:53   .   
    DR_UNIX  TTYP2    MAY 10 20:20   .   
    [localhost:~] dr_unix% 
    
And if you need to save the results to a file, you can combine the two redirection symbols thus:
    [localhost:~] dr_unix% tr a-z A-Z < who_list > Cap_who_list
    [localhost:~] dr_unix% cat Cap_who_list
    USER     LINE     WHEN         IDLE     FROM
    DR_UNIX  CONSOLE  MAY 10 16:09 04:16 
    DR_UNIX  TTYP1    MAY 10 19:53   .   
    DR_UNIX  TTYP2    MAY 10 20:20   .   
    [localhost:~] dr_unix% 
    
In days of old, the < was commonly used with the mail command. You could write a message to a file, rather than type and edit it all at one sitting, and then send the message:

mail user1 user2 user3 user4 user5 < message_file

These days, that is rare because of all the GUI mail utilities, but if you want to fire off a message or a log file from within a script, or you want to gain three geek points for having sent e-mail from the command line at least once, that is how you do it. And if I catch any of you using that to send SPAM, I will take you out back and shoot you. Slowly and painfully.

As you might guess, there is a corresponding << redirector. It is known as a "Here" document, and is used:

command << endtext

This means that command will read from Standard Input up to (but not including) a line identical to endtext:

    [localhost:~] dr_unix% cat << END
     -> Fred was here.
     -> Now he is gone.
     -> That would be okay.
     -> Except I think he stole my corkscrew.
     -> END
    Fred was here.
    Now he is gone.
    That would be okay.
    Except I think he stole my corkscrew.
    [localhost:~] dr_unix% 
    
Generally, this is only used in shell scripting or in some übergeek commands like cat, echo, ex, or sed. Still, there are times when it is handy to know. And kiddies, here is a warning from a grumpy old programmer: watch out for those trailing spaces. If I had put and extra space after cat << END so that it was really cat << END , the cat command would have said "Aha, 'END' is not the same as 'END ', so there must be more input coming". That can be a tough bug to find.

>, >>, <, and << will take you a long way, but there are other forms of redirection, less commonly used but needed from time to time. We have seen how to redirect Standard Output to a file; any error message still comes to the screen.

    [localhost:~] dr_unix% ls who_list xiphoid
    ls: xiphoid: No such file or directory
    who_list
    [localhost:~] dr_unix% ls who_list xiphoid > temp
    ls: xiphoid: No such file or directory
    [localhost:~] dr_unix% cat temp
    who_list
    [localhost:~] dr_unix% 
    
How do we make Standard Output and Standard Error go to the same file? With >&, thus:

command >& out_file

[or command >&! out_file if you want to overwrite out_file even if noclobber is set.]

    [localhost:~] dr_unix% ls who_list xiphoid
    ls: xiphoid: No such file or directory
    who_list
    [localhost:~] dr_unix% ls who_list xiphoid >& temp
    [localhost:~] dr_unix% cat temp
    ls: xiphoid: No such file or directory
    who_list
    [localhost:~] dr_unix% 
    
And if you want to append to a file, it's simply:

command >>& out_file

[or, you guessed it, command >>&! out_file if noclobber is set.]

And finally, if you want to send Standard Output to one file and Standard Error to another, this takes the form:

(command > out_file) >& error_file

    [localhost:~] dr_unix% (ls who_list xiphoid > temp_out) >& temp_err
    [localhost:~] dr_unix% cat temp_out
    who_list
    [localhost:~] dr_unix% cat temp_err
    ls: xiphoid: No such file or directory
    [localhost:~] dr_unix% 
    
Finally, this seems as good a place as any to bring up a wonderful little Unix feature, /dev/null. /dev/null is a truly handy piece of software that acts as a black hole for whatever you send to it. So it you type:
    (ls who_list xiphoid > temp_out) >& /dev/null
    
a wonderful thing will happen: the error messages will go away. Oh, the errors will still be there, but you don't see the error messages.

--

As an aside, I personally find that the Bourne and Korn shells redirectors to be a bit more flexible. For example, using the above command, ls who_list xiphoid, let's say that I want to redirect Standard Error to a file but keep Standard Output coming to the screen, note the steps I follow:

    [localhost:~] dr_unix% w
    10:01PM  up  5:53, 3 users, load averages: 0.16, 0.18, 0.00
    USER    TTY FROM              LOGIN@  IDLE WHAT
    dr_unix  co -                 4:09PM  5:53 -
    dr_unix  p1 -                 7:53PM     0 -
    dr_unix  p2 -                 8:20PM     0 -
    [localhost:~] dr_unix% ( ls who_list xiphoid > /dev/ttyp2 ) >& temp_err
    who_list
    [localhost:~] dr_unix% cat temp_err
    ls: xiphoid: No such file or directory
    [localhost:~] dr_unix% 
    
If anyone can tell me a better way to do this in tcsh, without involving /dev/whatever files or other shells, let me know. I'll pass it on and be sure you get credit.

...

If you look in your home directory now, you will find a lot of files that you need to clean up. Saving output to a file, only to use its contents as the input of another command, creates a mess; worse, it goes against laziness and impatience for you to have to rm these files. Even worse, you might rm one that you need. Fortunately, Unix allows you to redirect output from one command to another without saving to a temporary file; we'll look at that in the next column.

You are encouraged to send Richard your comments, or to post them below.


Most Recent Mac OS X Command Line 101 Columns

Command Line History & Editing Your Commands
November 22nd

Pico: An Easy To Use Command Line Editor
November 1st

Understanding The "grep" Command In Mac OS X
October 4th

Command Line History & Editing Your Commands
September 6th

Mac OS X Command Line 101 Archives

Back to The Mac Observer For More Mac News!


Richard Burton is a longtime Unix programmer and a handsome brute. He spends his spare time yelling at the television during Colts and Pacers games, writing politically incorrect short stories, and trying to shoot the neighbor's cat (not really) nesting in his garage. He can be seen running roughshod over the TMO forums under the alias tbone1.



Today's Mac Headlines

[Podcast]Podcast - Apple Weekly Report #135: Apple Lawsuits, Banned iPhone Ad, Green MacBook Ad

We also offer Today's News On One Page!

Yesterday's News

 

[Podcast]Podcast - Mac Geek Gab #178: Batch Permission Changes, Encrypting Follow-up, Re-Enabling AirPort, and GigE speeds

We also offer Yesterday's News On One Page!

Mac Products Guide
New Arrivals
New and updated products added to the Guide.

Hot Deals
Great prices on hot selling Mac products from your favorite Macintosh resellers.

Special Offers
Promotions and offers direct from Macintosh developers and magazines.

Software
Browse the software section for over 17,000 Macintosh applications and software titles.

Hardware
Over 4,000 peripherals and accessories such as cameras, printers, scanners, keyboards, mice and more.

© All information presented on this site is copyrighted by The Mac Observer except where otherwise noted. No portion of this site may be copied without express written consent. Other sites are invited to link to any aspect of this site provided that all content is presented in its original form and is not placed within another .