520
  • What is the difference between awk and sed ?
  • What kind of application are best use cases for sed and awk tools ?
Rachel
  • 91,207
  • 112
  • 255
  • 361
  • 2
    A related QA on unix.stackexchange: [Is there a basic tutorial for grep, awk and sed?](http://unix.stackexchange.com/questions/2434/is-there-a-basic-tutorial-for-grep-awk-and-sed) – Dani Oct 27 '09 at 16:40
  • https://pediaa.com/difference-between-sed-and-awk/#sed%20vs%20awk%20%20-%20Comparison%20of%20Key%20Differences – Hamza Belmellouki May 20 '19 at 01:48

3 Answers3

590

sed is a stream editor. It works with streams of characters on a per-line basis. It has a primitive programming language that includes goto-style loops and simple conditionals (in addition to pattern matching and address matching). There are essentially only two "variables": pattern space and hold space. Readability of scripts can be difficult. Mathematical operations are extraordinarily awkward at best.

There are various versions of sed with different levels of support for command line options and language features.

awk is oriented toward delimited fields on a per-line basis. It has much more robust programming constructs including if/else, while, do/while and for (C-style and array iteration). There is complete support for variables and single-dimension associative arrays plus (IMO) kludgey multi-dimension arrays. Mathematical operations resemble those in C. It has printf and functions. The "K" in "AWK" stands for "Kernighan" as in "Kernighan and Ritchie" of the book "C Programming Language" fame (not to forget Aho and Weinberger). One could conceivably write a detector of academic plagiarism using awk.

GNU awk (gawk) has numerous extensions, including true multidimensional arrays in the latest version. There are other variations of awk including mawk and nawk.

Both programs use regular expressions for selecting and processing text.

I would tend to use sed where there are patterns in the text. For example, you could replace all the negative numbers in some text that are in the form "minus-sign followed by a sequence of digits" (e.g. "-231.45") with the "accountant's brackets" form (e.g. "(231.45)") using this (which has room for improvement):

sed 's/-\([0-9.]\+\)/(\1)/g' inputfile

I would use awk when the text looks more like rows and columns or, as awk refers to them "records" and "fields". If I was going to do a similar operation as above, but only on the third field in a simple comma delimited file I might do something like:

awk -F, 'BEGIN {OFS = ","} {gsub("-([0-9.]+)", "(" substr($3, 2) ")", $3); print}' inputfile

Of course those are just very simple examples that don't illustrate the full range of capabilities that each has to offer.

Dennis Williamson
  • 303,596
  • 86
  • 357
  • 418
  • 7
    To see some examples of pushing the boundaries of `sed`: http://sed.sourceforge.net/#scripts – Dennis Williamson Oct 27 '09 at 21:58
  • 2
    @DennisWilliamson - Am I at a disadvantage if I only learn awk ? Is awk far more commonly used than sed ? – Steam Aug 20 '13 at 18:24
  • 6
    @blasto: My recommendation is to learn both but with more emphasis on awk. Lots of the regular expression stuff applies to both (and other tools and languages). Use sed for simpler stuff and try to avoid the complex things. It's really cool that you can do loops and branches in sed, but the resulting command lines are complex and hard to read. The answer to your question really depends on what you're doing. – Dennis Williamson Aug 20 '13 at 19:52
  • @DennisWilliamson - What would be most useful to an ETL developer ? ETL or Extract Transform and Load is a data-warehousing term. Put crudely, the job involves EXTRACTION of data from different disparate sources (such as DB's, excel files, csv files etc), TRANSFORMATION of the same and then LOADING into a datawarehouse (DW) for analysis, finding patterns in data, or just historical records. eg. End use of a DW - Algorithms applied to a DW of a grocery store which has data from the past 10 years might reveal that people who tend to buy apples also buy oranges or something similar. – Steam Aug 20 '13 at 19:56
  • A link says - http://www.vectorsite.net/tsawk_1.html - "There are, however, things that Awk is not. It is not really well suited for extremely large, complicated tasks." Now, how "large" a task can awk handle ? How does sed compare in that respect ? – Steam Aug 20 '13 at 20:09
  • That is a blanket statement with insufficient qualifiers. However, there is a point at which Python or Perl should be used instead of AWK. There are tasks at which a compiled language would be better. I would use AWK for textual data that appears in rows and columns. AWK is pretty much limited to one or two dimensional arrays (Python and Perl support much more complex data structures by comparison) and has really no library support. Not that it can't be done, but there just really isn't any established... – Dennis Williamson Aug 20 '13 at 21:07
  • library. OTOH, Python and Perl have vast core and 3rd part libs. Sed can handle quite large tasks within its capabilities and, although one could argue that it's Turing complete, it's not very well suited for algorithmic approaches. – Dennis Williamson Aug 20 '13 at 21:08
  • 1
    Just an FYI for anyone stumbling across this on a mac, try "sed -E 's/-([0-9]+.[0-9]*)/(\1)/g'" for the first sed example – Daniel Schmidt May 08 '18 at 21:05
  • @Steam Your example sounds like typical use case of awk, for example you can have a look at this [cool article](http://freesoftwaremagazine.com/articles/look_mum_no_database_thanks_to_awk/). – jena Nov 15 '18 at 14:09
  • @jena While there may be something cool about that, it's awful in so many ways I stopped counting. – Dennis Williamson Nov 15 '18 at 15:49
  • FYI, the example from @DanielSchmidt translates `Total losses were -1234.56 this month` into `Total losses were (1234.56) this month` for an answer to the "accountants brackets" mentioned in the answer. – pbarney Jan 12 '21 at 15:09
136

1) What is the difference between awk and sed ?

Both are tools that transform text. BUT awk can do more things besides just manipulating text. Its a programming language by itself with most of the things you learn in programming, like arrays, loops, if/else flow control etc You can "program" in sed as well, but you won't want to maintain the code written in it.

2) What kind of application are best use cases for sed and awk tools ?

Conclusion: Use sed for very simple text parsing. Anything beyond that, awk is better. In fact, you can ditch sed altogether and just use awk. Since their functions overlap and awk can do more, just use awk. You will reduce your learning curve as well.

ghostdog74
  • 286,686
  • 52
  • 238
  • 332
  • 7
    Nice point about the learning curve.. too many tools can mixup.. so I'd prefer learning grep and awk only.. lets forget about sed :) – Outlier May 07 '14 at 03:54
  • 191
    ^^ Enough sed. (sorry, I had to) – Greg M. Krsak Jun 16 '14 at 22:55
  • 8
    I find sed is much easier to learn though, so you need to account for that. As you learn to master awk, it might be useful to quickly learn sed to be able to use it faster for things you might not know how to do in awk yet. – Didier A. May 07 '16 at 08:03
  • 1
    @GregKrsak you made my day. (sorry, I had to) – Abel Callejo Sep 06 '16 at 01:38
  • 20
    Don't ditch `sed`, `'s/search/replace'` is way easier to type than `awk`'s syntax and is what you need most of the time. – sjas Jun 04 '17 at 12:10
  • 1
    Better to use a mixture. For instance, if you are looking for a pattern in a huge file, use grep to find the line number, and then sed to edit the line. Otherwise, sed is going to be much slower to process the entire file. – Konchog Oct 30 '18 at 11:27
  • 1
    If you're a craftsman, you learn how to use all your tools and you'll know which to use when. If you're just fixing stuff, a hammer and a screwdriver might just be all you need. – pbarney Jan 12 '21 at 17:20
67

Both tools are meant to work with text and there are tasks both tools can be used for.

For me the rule to separate them is: Use sed to automate tasks you would do otherwise in a text editor manually. That's why it is called stream editor. (You can use the same commands to edit text in vim). Use awk if you want to analyze text, meaning counting fields, calculate totals, extract and reorganize structures etc.

Also you should not forget about grep. Use grep if you only want to search/extract something in a text (file)

hek2mgl
  • 133,888
  • 21
  • 210
  • 235