Data Science at the Command Line
by Jeroen Janssens · first published 2014
53% recommend40% neutral7% critical
Why HN recommends it
Shows how to run data science workflows from the shell with tools like awk, cut, and datamash, and is recommended as a free online read for people who want to add command-line data work to their practice.
Summarised by Claude from recommending comments
Data Science at the Command Line demonstrates how the flexibility of the command line can help you become a more efficient and productive data scientist. You’ll learn how to combine small, yet powerful, command-line tools to quickly obtain, scrub, explore, and model your data.
Affiliate links. As an Amazon Associate I earn from qualifying purchases.
Mentions per year
What Hacker News says
Related books
2009–2026The AWK programming language
Alfred V. Aho, Brian W. Kernighan, Peter Weinberger
Recommended by 61 · mentioned by 7961 recommend · 79 mention61 recommend · 7961 recommend
2014–2026Designing Data-Intensive Applications
Martin Kleppmann
Recommended by 457 · mentioned by 659457 recommend · 659 mention457 recommend · 659457 recommend
I've seen this article before and still consider it valid, especially for one-offs. Another similar and valuable resource is "https://datascienceatthecommandline.com/" Data Science at the Command Line, by Jeronen Janssens. Free to read at the link.
on “Command-line Tools can be 235x Faster than your Hadoop Cluster (2014)” ↗See also Jeroen Janssens' Data Science at the Command Line: https://datascienceatthecommandline.com/2e/
on “Command-line data analytics” ↗…maintainable / supportable solution. For instance drake (https://github.com/Factual/drake) a 'Make for data' which does dependency checking would allow for sensible restarts of the pipeline. The O'Reilly Data Science at the Command Line book (linked elsewhere in the comments) has a good deal to say on the subject: turning one liners into extensible shell scripts, using drake, using Gnu Parallel.
on “Command-line tools can be faster than your Hadoop cluster” ↗I learnt a lot from the book Data Science at the Command Line, now free and online at https://www.datascienceatthecommandline.com/
on “Problem solving with Unix commands” ↗…the data science and analysis pipeline. I'm quick to learn, and I love learning how to use new tools, understanding their philosophy and what they're good for (or not good for!) I generally follow the "Data Science at the Command Line" approach to my work, and I like to build web systems for sharing my work from the lower level (d3.js over full-fat systems like observable, tableau, etc).
on “Ask HN: Who wants to be hired? (April 2023)” ↗There's a similar O'Reilly book, Data Science at the Command Line, which I can also recommend: https://datascienceatthecommandline.com/ Buffalo's book is great and I generally recommend it in my bioinformatics career, but it does go into some speciifc niches (especially…
on “Best Linux Books for Different Target Groups and Use Cases” ↗…`awk -F, ' {print $1}'`), you can also use `cut -d, -f1` GNU datamash (https://www.gnu.org/software/datamash/) provides features like groupby, statistical operations, etc. See also this free ebook: Data Science at the Command Line (https://jeroenjanssens.com/dsatcl/)
on “Working with CSV files on shell/terminal” ↗I usually recommend Data Science At The Command Line https://jeroenjanssens.com/dsatcl/ Not because it explains bash well, but because it fits with my data science/bioinformatics niche. It's all about these funky sort/uniq/rev/cut/uniq/awk/wc/sed patterns…
on “The bash book to rule them all” ↗…beyond just printing to stdout. Then add in something like Pry or Irb, where you are able to build castles in your sandbox. Most of my data exploration happens in Pry. A final book I’ll toss out is Data Science at the Command Line, in particular the first 40 or so pages. They highlight the amount of tooling that exists that’s just python shell scripts posing as bins. (Ruby of course has every bit of the same potential.) I had always…
on “Ruby 3.3” ↗The O'Reilly book "Data Science at the Command Line" by Jeroen Janssens is now in its 2nd edition. https://datascienceatthecommandline.com/
on “Ask HN: Comprehensive course/book data analysis” ↗…NYC: http://vimeo.com/79511496 Hopefully I can give an updated one covering the upcoming 1.0 release at this year's. Also, SciKit-Learn Laboratory (SKLL) was recently featured in the new O'Reilly book, Data Science at the Command Line, by Jeroen Janssens.
on “Show HN: SKLL, a utility to run scikit-learn experiments without writing code” ↗…about command-line tools for data science [1]. Thanks to HN, I received a lot of valuable comments and pointers to other great command-line tools! In the past 10 months, I have been writing a book titled Data Science at the Command Line [2]. Ever since that blog post, I've been discovering new tools. On the one hand, that's quite frustrating because it's difficult to keep up and include everything in the book. On the other hand, it's…
on “Ask HN: What's your favorite command-line tool for working with data?” ↗I'm flattered to see Data Science at the Command Line next to these great titles, but I'm not sure if I would recommend it to learn the basics of data science. DSATCL discusses the ideas behind various cleaning and visualization approaches and several machine…
on “Learn the basics of data science with these books” ↗…install, you'd be surprised how often people struggle installing Python and its packages on Mac OS X or Microsoft Windows! The initial reason I created DST was that I'm currently writing a book called Data Science at the Command Line. Installing the command-line tools discussed in this book is unfortunately not straightforward. I wanted to offer potential readers a Vagrant box (with a shell provisioner) that installs these command-line…
on “Data Science Toolbox” ↗Thanks for the link. Very interesting. In OP's article there is a link to a book "Data Science at the Command Line" which sounds quite relevant: https://www.datascienceatthecommandline.com
on “Using AWK and R to parse 25TB” ↗