Bash sort... what the hell is up with tab separators?

UPDATE: I have updated the sort Wiki page to include an example for tab separated sorting.

In the world of Unix shells there exists a very common one called Bash. And within the Bash shell there are a whole host of commands that can be used both on the command line and in a script file. These commands run the gamut for what they can do, but there is a small subset that most people find their daily lives centered around when it comes to programming or hacking out solutions such as sed, cut, and sort.

Fairly common inputs to these commands are delimited-separated values such as CSVs and TSVs. Thankfully almost every command that you would use for these formats allows you to specify the delimiter.

For instance, say we have a TSV file called phonebook that contains the name and number for each contact:

$ cat phonebook 
Smith, Brett 555-4321
Doe, John 555-1234
Doe, Jane 555-3214
Avery, Cory 555-4132
Fogarty, Suzie 555-2314

With cut you could get just the names if you wanted:

$ cut -f1 phonebook 
Smith, Brett
Doe, John
Doe, Jane
Avery, Cory
Fogarty, Suzie

How did it know what the delimiter was? Luckily with cut the default delimiter is the tab character. What if it's not though? Looking at the man page for cut gives us:

$ man cut
...
     -d delim
             Use delim as the field delimiter character instead of the tab character.
...

So, say you wanted just the last name for everyone. Well, you can pipe the output from the first command to a second command to do just that! The only difference is now we are going to specify the delimiter to be a comma for the second command.

$ cut -f1 phonebook | cut -f1 -d ','
Smith
Doe
Doe
Avery
Fogarty

Nice! Now, lets check out sort. First, lets sort our phonebook by last name:

$ sort -k1,1 phonebook 
Avery, Cory 555-4132
Doe, Jane 555-3214
Doe, John 555-1234
Fogarty, Suzie 555-2314
Smith, Brett 555-4321

That works well enough. Now lets sort by phone numbers:

$ sort -k2,2 phonebook 
Smith, Brett 555-4321
Avery, Cory 555-4132
Doe, Jane 555-3214
Doe, John 555-1234
Fogarty, Suzie 555-2314

Well... that's not right. It sorted by first name instead. Hmmmmm. Looking at the man page we see:

$ man sort
...
       -t, --field-separator=SEP
              use SEP instead of non-blank to blank transition
...

OK, lets add our trusty tab character:

$ sort -k2,2 -t '\t' phonebook 
sort: multi-character tab `\\t'

Uhhhhh... multi-character? Looks like sort doesn't interpret '\t' as a tab character, but instead a literal '\' and 't'. In another way:

$ echo -n '\t' | hexdump -c
0000000   \   t                                                        
0000002

So, how do we set the separator to be a tab character? The beginner's bash guide provides some guidance on that:

3.3.5. ANSI-C quoting

Words in the form "$'STRING'" are treated in a special way. The word expands to a string, with backslash-escaped characters replaced as specified by the ANSI-C standard. Backslash escape sequences can be found in the Bash documentation.

Using our echo example again:

$ echo -n $'\t' | hexdump -c
0000000  \t                                                            
0000001

Yup, one character. Trying out phone number sort again:

$ sort -k2,2 -t $'\t' phonebook 
Doe, John 555-1234
Fogarty, Suzie 555-2314
Doe, Jane 555-3214
Avery, Cory 555-4132
Smith, Brett 555-4321

BINGO! Now we are truly sorting on the second column.

tl;dr

Turns out sort is similar to echo in that by default escaped characters are interpreted as two character literals rather than the intended escaped character. While echo -e does provide a means to do so, sort does not. So we must use the ANSI-C quoting (or some other means).

Passing bash arguments to an alias

A buddy of mine was trying to figure out why he could only pass $@ to the end of an alias and not somewhere before. His failed attempt looked something like this:

$ alias psgrep='ps wwax | grep $@ | grep -v grep'
$ psgrep Terminal
Usage: grep [OPTION]... PATTERN [FILE]...
Try `grep --help' for more information.
grep: Terminal: No such file or directory

His successful attempt was swapping the two greps as such:

$ alias psgrep='ps wwax | grep -v grep | grep $@'
$ psgrep Terminal
98893   ??  R      0:00.48 /Applications/Utilities/Terminal.app/Contents/MacOS/Terminal -psn_0_3666815

At first glance, this does seem odd, how could it properly interpret $@ in the second by not the first? Well, to answer that question, we first need to ask if it is really doing just that. Is $@ properly interpreted in the second attempt? Lets find out:

$ alias psgrep='echo "ps wwax | grep -v grep | grep [$@]"'
$ psgrep Terminal
ps wwax | grep -v grep | grep [] Terminal

Hmmmm, that definitely does not appear to be the case. So, what about the first attempt then:

$ alias psgrep='echo "ps wwax | grep [$@] | grep -v grep"'
$ psgrep Terminal
ps wwax | grep [] | grep -v grep Terminal

Now that is very enlightening. The error from the first attempt is not from the first grep as it appeared to be, but the second grep. Since grep can take a list of files, it assumed the now appended 'Terminal' argument was another file to search through.

That solves our question of whether or not it is interpreted at all, but not why it doesn't work. Well, it does, but only when it was first assigned to the alias. So, when the set alias command is executed, $@ is interpreted (correctly) as undefined, and therefore the alias is set as such. When the alias is then executed later, $@ has already been interpreted and any arguments are appended to the aliased command. That is why the second attempt appeared to work correctly.

So how would do something like this? One approach would be to use a bash function:

(first a little cleanup from the previous examples)

$ unalias psgrep

(now the actual example)

$ function psgrep() { ps wwax | grep $@ | grep -v grep ; }
$ psgrep Terminal
98893   ??  R      0:01.83 /Applications/Utilities/Terminal.app/Contents/MacOS/Terminal -psn_0_3666815

(finally, to unset the function if you wish)

$ unset psgrep

Enjoy!

UPDATE: In yet another case of RTFM, I decided to read up on the man page for sh. It literally states "There is no mechanism for using arguments in the replacement text. If arguments are needed, a shell function should be used (see FUNCTIONS below)". So, there you go. At least my conclusion was in line with the documentation. :-P