The Shell

Modern Plain Text Social Science: Week 2

Kieran Healy

kieran.healy@duke.edu

September 8, 2023

Recap

Technical Computing is frustrating

Where technical computing lives

Windows and pointers.
Multi-tasking, multiple windows.
Exposes and leverages the file system.
Many specialized tools in concert.
Underneath, it’s the 1970s, UNIX, and the command-line.

Different Answers

Office model

Formatted documents are real.
Intermediate outputs are cut and pasted into documents.
Changes are tracked inside files.
Final output is often in the same format you’ve been working in, e.g. a Word file, or a PDF.

Engineering model

Plain-text files are real.
Intermediate outputs are produced via code, often inside documents.
Changes are tracked outside files, at the level of a project.
Final outputs are assembled programatically and converted to some desired format.

The File System

A tree-like hierarchy

You may never have actually used one of these.

File system hierarchy

Standard locations

/ : root. Everything lives inside or under the root.
/bin/ : For binaries. Core user executable programs and tools.
/sbin/ : System binaries. Essential executables for the super user (who is also called root)
/lib/ : Support files for executables.
/usr/ : Conventionally, stuff installed “locally” for users in addition to the core system. Will contain its own bin/ and lib/ subdirs.
/usr/local : Files that the local user has compiled or installed
/opt/ : Like /usr/, another place for locally installed software to go.
/etc/ : Editable text configuration. System-wide config files often go here.

Standard locations

/home/ or /Users/ : Where the files of individual system users live, like /Users/kjhealy/. This is the storage space associated with your user account

Standard locations

Your $PATH is an environment variable that tells the system where executable files can be found.

echo $PATH

/opt/homebrew/bin:/opt/homebrew/sbin:/usr/local/bin:/usr/local/sbin:/usr/bin:/bin:/usr/sbin:/sbin:/Library/TeX/texbin:/opt/X11/bin:/Users/kjhealy/bin

Delimited by : and searched in order from left to right.
To learn where a command is being executed from, use which

which R

/usr/local/bin/R

The Shell

What is it?

A shell is a way for you to tell the operating system to do things.
On Unix systems it’s the first user-facing thing to get off the ground during the startup/boot process.
The command line is where you type instructions. Shells come with a collection of standard utilities that let you do things.
These utilities can be chained or piped together for more complex tasks.
You can also write scripts that will run tasks for you.
Shell scripting languages are small programming languages that understand variables, command substitution, branching, and iteration.

There are many shells

Strictly speaking, GUI environments like Windows and the macOS Finder are shells too.
But “the shell” usually means a text-based interpreter that runs programs in response to typed commands.
The “original” Unix shell is sh.
Its most widely-used descendant is bash or the Bourne-Again Shell.
On macOS the default shell is the Z shell or zsh.
Windows has the Command shell and PowerShell, and possibly also e.g. Cygwin.

A command interpreter

A shell is an interpreter. It waits for commands. When you supply them, it does what you tell it.

echo "Hello there"

Hello there

The general name for this is a REPL or Read-Eval-Print Loop.
Programming languages like Python and R work this way as well. They are interpreted, meaning code is sent to an interpreter (the Python or R program) that runs the code directly.
This is distinct from languages whose programs are (at least originally) designed to be compiled into executable machine code before they are run. Languages like C, Go, and Rust are in this category.

Getting around the file system

Who and where

Who am I?

whoami

kjhealy

Where am I?

# Print working directory
pwd

/Users/kjhealy/Documents/courses/mptc

Listing files

What is in here?

# List files
ls

R
README.md
README.qmd
README_files
_extensions
_freeze
_quarto.yml
_site
_targets
_targets.R
_variables.yml
about
assets
assignment
content
data
deploy.sh
example
files
html
index.html
index.qmd
mptc.Rproj
renv
renv.lock
renv.lock.orig
schedule
site_libs
slides
syllabus

A Tree

## The kind of thing you can do ... just use <tree -d -L 2> at the console though
find . -type d -maxdepth 2 -print 2>/dev/null | awk '!/\.$/ {for (i=1;i<NF-1;i++){printf("│   ")}print "├── "$NF}'  FS='/'

├── schedule
├── README_files
│   ├── libs
├── example
│   ├── 04-example_files
├── R
├── content
├── assignment
├── html
│   ├── fonts
├── site_libs
│   ├── revealjs
│   ├── bootstrap
│   ├── quarto-html
│   ├── quarto-contrib
│   ├── quarto-nav
│   ├── quarto-search
│   ├── clipboard
├── about
├── slides
│   ├── 00-slides_files
├── syllabus
├── _extensions
│   ├── quarto-ext
│   ├── minion
│   ├── kjhealy
├── _site
├── files
│   ├── misc
│   ├── examples
│   ├── scripts
│   ├── bib
├── .git
│   ├── objects
│   ├── info
│   ├── logs
│   ├── hooks
│   ├── refs
│   ├── modules
├── _targets
│   ├── meta
│   ├── objects
│   ├── user
│   ├── workspaces
├── renv
│   ├── staging
│   ├── library
├── data
├── assets
│   ├── 03-editors
│   ├── 04-git
│   ├── 00-site
│   ├── 02-shell
│   ├── 01-file-system
│   ├── 05-build
├── _freeze
│   ├── schedule
│   ├── example
│   ├── content
│   ├── assignment
│   ├── site_libs
│   ├── slides
│   ├── index
├── .Rproj.user
│   ├── B6516D0D
│   ├── shared
├── .quarto
│   ├── xref
│   ├── idx
│   ├── preview
│   ├── _freeze

Changing directories

## Change directory and list files
cd files
ls  
cd ../slides

bib
examples
misc
schedule.ics
scripts

More detail

ls -l

total 352
drwxr-xr-x@  8 kjhealy  staff    256 Aug 15 16:35 R
-rw-r--r--@  1 kjhealy  staff   1967 Sep 18 21:38 README.md
-rw-r--r--@  1 kjhealy  staff   1764 Aug 30 12:28 README.qmd
drwxr-xr-x@  3 kjhealy  staff     96 Sep 18 21:38 README_files
drwxr-xr-x   6 kjhealy  staff    192 Jan  8 13:32 _extensions
drwxr-xr-x   9 kjhealy  staff    288 Jan  8 13:41 _freeze
-rw-r--r--@  1 kjhealy  staff   2946 Jan  8 13:38 _quarto.yml
drwxr-xr-x   2 kjhealy  staff     64 Jan  8 13:41 _site
drwxr-xr-x@  7 kjhealy  staff    224 Jan  8 13:38 _targets
-rw-r--r--@  1 kjhealy  staff   2286 Jan  8 13:37 _targets.R
-rw-r--r--@  1 kjhealy  staff    840 Jan  8 10:27 _variables.yml
drwxr-xr-x@  4 kjhealy  staff    128 Jan  8 13:41 about
drwxr-xr-x@ 11 kjhealy  staff    352 Sep 25 12:30 assets
drwxr-xr-x@  6 kjhealy  staff    192 Jan  8 13:41 assignment
drwxr-xr-x@ 14 kjhealy  staff    448 Jan  8 13:41 content
drwxr-xr-x@  4 kjhealy  staff    128 Jan  8 10:38 data
-rwxr-xr-x@  1 kjhealy  staff    295 Sep  7 14:38 deploy.sh
drwxr-xr-x@ 15 kjhealy  staff    480 Jan  8 13:41 example
drwxr-xr-x@  8 kjhealy  staff    256 Sep  8 11:14 files
drwxr-xr-x  14 kjhealy  staff    448 Jan  8 10:23 html
-rw-r--r--   1 kjhealy  staff  42894 Jan  8 13:41 index.html
-rw-r--r--@  1 kjhealy  staff   6379 Oct  4 16:22 index.qmd
-rw-r--r--@  1 kjhealy  staff    258 Jan  8 13:33 mptc.Rproj
drwxr-xr-x@  7 kjhealy  staff    224 Aug 15 17:04 renv
-rw-r--r--@  1 kjhealy  staff  46675 Jan  8 13:38 renv.lock
-rw-r--r--   1 kjhealy  staff  46717 Dec 11 18:14 renv.lock.orig
drwxr-xr-x@  4 kjhealy  staff    128 Jan  8 13:41 schedule
drwxr-xr-x   9 kjhealy  staff    288 Jan  8 13:41 site_libs
drwxr-xr-x@ 12 kjhealy  staff    384 Jan  8 13:42 slides
drwxr-xr-x   3 kjhealy  staff     96 Jan  8 10:39 syllabus

Note the idea of commands having options, or switches.

More detail

ls /

Applications
Library
System
Users
Volumes
bin
cores
dev
etc
home
opt
private
sbin
tmp
usr
var

The ~ character is a shortcut to the top of your home directory:

❯ cd ~
❯ ls
bin  certbot.log  logrotate.conf  old  projects  public  staging

Path rules

If the path name begins with /, it is an absolute path, starting from the root.
If the path name begins with ~, it will usually be expanded into an absolute path name starting at your home directory (~).

Path rules

If the pathname does not begin with a / or ~ then the path name is relative to the current directory.
If the path name begins with ./, the path is relative to the current directory, e.g., ./file.txt, though this can also execute the file if it is given executable file permissions.
If the path name begins with ../, the path is relative to the parent of the current directory. For example, if your current directory is /Users/kjhealy/Documents/papers then ../data means /Users/kjhealy/Documents/data
These conventions extend to things like URLs and paths associated with e.g. SSH or FTP servers, and APIs.

Some shell tools

`wc`, `cat`, `head`, and `tail`

wc files/examples/alice_in_wonderland.txt

    3761   29564  174392 files/examples/alice_in_wonderland.txt

We can ask for a count of lines only:

wc -l files/examples/alice_in_wonderland.txt

    3761 files/examples/alice_in_wonderland.txt

`wc`, `cat`, `head`, and `tail`

cat concatenates and prints the files given to it.

cat files/examples/jabberwocky.txt

’Twas brillig, and the slithy toves 
      Did gyre and gimble in the wabe: 
All mimsy were the borogoves, 
      And the mome raths outgrabe. 

“Beware the Jabberwock, my son! 
      The jaws that bite, the claws that catch! 
Beware the Jubjub bird, and shun 
      The frumious Bandersnatch!” 

He took his vorpal sword in hand; 
      Long time the manxome foe he sought— 
So rested he by the Tumtum tree 
      And stood awhile in thought. 

And, as in uffish thought he stood, 
      The Jabberwock, with eyes of flame, 
Came whiffling through the tulgey wood, 
      And burbled as it came! 

One, two! One, two! And through and through 
      The vorpal blade went snicker-snack! 
He left it dead, and with its head 
      He went galumphing back. 

“And hast thou slain the Jabberwock? 
      Come to my arms, my beamish boy! 
O frabjous day! Callooh! Callay!” 
      He chortled in his joy. 

’Twas brillig, and the slithy toves 
      Did gyre and gimble in the wabe: 
All mimsy were the borogoves, 
      And the mome raths outgrabe.

`wc`, `cat`, `head`, and `tail`

The top:

head files/examples/alice_in_wonderland.txt

The Project Gutenberg eBook of Alice's Adventures in Wonderland
    
This ebook is for the use of anyone anywhere in the United States and
most other parts of the world at no cost and with almost no restrictions
whatsoever. You may copy it, give it away or re-use it under the terms
of the Project Gutenberg License included with this ebook or online
at www.gutenberg.org. If you are not located in the United States,
you will have to check the laws of the country where you are located
before using this eBook.

The bottom:

tail files/examples/alice_in_wonderland.txt


Most people start at our website which has the main PG search
facility: www.gutenberg.org.

This website includes information about Project Gutenberg™,
including how to make donations to the Project Gutenberg Literary
Archive Foundation, how to help produce our new eBooks, and how to
subscribe to our email newsletter to hear about new eBooks.

`wc`, `cat`, `head`, and `tail`

There are 29 lines of boilerplate at the start of the book:

head -n 29 files/examples/alice_in_wonderland.txt

The Project Gutenberg eBook of Alice's Adventures in Wonderland
    
This ebook is for the use of anyone anywhere in the United States and
most other parts of the world at no cost and with almost no restrictions
whatsoever. You may copy it, give it away or re-use it under the terms
of the Project Gutenberg License included with this ebook or online
at www.gutenberg.org. If you are not located in the United States,
you will have to check the laws of the country where you are located
before using this eBook.

Title: Alice's Adventures in Wonderland


Author: Lewis Carroll

Release date: June 27, 2008 [eBook #11]
                Most recently updated: March 30, 2021

Language: English

Credits: Arthur DiBianca and David Widger


*** START OF THE PROJECT GUTENBERG EBOOK ALICE'S ADVENTURES IN WONDERLAND ***
[Illustration]

`wc`, `cat`, `head`, and `tail`

And 351 at the end:

tail -n 351 files/examples/alice_in_wonderland.txt | head -n 20

            *** END OF THE PROJECT GUTENBERG EBOOK ALICE'S ADVENTURES IN WONDERLAND ***
        

    

Updated editions will replace the previous one—the old editions will
be renamed.

Creating the works from print editions not protected by U.S. copyright
law means that no one owns a United States copyright in these works,
so the Foundation (and you!) can copy and distribute it in the United
States without permission and without paying copyright
royalties. Special rules, set forth in the General Terms of Use part
of this license, apply to copying and distributing Project
Gutenberg™ electronic works to protect the PROJECT GUTENBERG™
concept and trademark. Project Gutenberg is a registered trademark,
and may not be used if you charge for an eBook, except by following
the terms of the trademark license, including paying royalties for use
of the Project Gutenberg trademark. If you do not charge anything for
copies of this eBook, complying with the trademark license is very

`wc`, `cat`, `head`, and `tail`

We can use tail to skip the boilerplate at the top:

tail -n +29 files/examples/alice_in_wonderland.txt | head


Alice’s Adventures in Wonderland

by Lewis Carroll

THE MILLENNIUM FULCRUM EDITION 3.0

Contents

 CHAPTER I.     Down the Rabbit-Hole

`wc`, `cat`, `head`, and `tail`

We can use some shell variables along with tail twice to skip the boilerplate at the top and bottom, and put the result into a file of its own using > to redirect the output from STDOUT:

# This sets HEADSKIP to 29 and ENDSKIP to 351; 
# We can refer to them with $HEADSKIP and $ENDSKIP
HEADSKIP=29
ENDSKIP=351

# The backticks ` ` here mean "Evaluate this command"; then put the result in a variable
BOOKLINES=`cat files/examples/alice_in_wonderland.txt| wc -l | tr ' ' '\n' | tail -1`

# This line does the arithmetic using expr and makes the result a variable
GOODLINES=$(expr $BOOKLINES - $HEADSKIP - $ENDSKIP)

# Now we use $HEADKSIP and $GOODLINES and create a new file 
tail -n +$HEADSKIP files/examples/alice_in_wonderland.txt | 
  head -n $GOODLINES > files/examples/alice_noboiler.txt

`wc`, `cat`, `head`, and `tail`

Now our wc will be different:

wc files/examples/alice_in_wonderland.txt 

wc files/examples/alice_noboiler.txt

    3761   29564  174392 files/examples/alice_in_wonderland.txt
    3381   26524  154465 files/examples/alice_noboiler.txt

`uniq`, `sort`, and `cut`

A data file:

head files/examples/countries.csv

cname,iso3,iso2,continent
Afghanistan,AFG,AF,Asia
Algeria,DZA,DZ,Africa
Armenia,ARM,AM,Asia
Australia,AUS,AU,Oceania
Austria,AUT,AT,Europe
Azerbaijan,AZE,AZ,Asia
Bahrain,BHR,BH,Asia
Belarus,BLR,BY,Europe
Belgium,BEL,BE,Europe

How many lines?

wc -l files/examples/countries.csv

     214 files/examples/countries.csv

How many unique lines?

uniq files/examples/countries.csv | wc -l

`uniq`, `sort`, and `cut`

# Omit the header line 
tail -n +2 files/examples/countries.csv | sort -r | head

Zimbabwe,ZWE,ZW,Africa
Zambia,ZMB,ZM,Africa
Yemen,YEM,YE,Asia
Western Sahara,ESH,EH,Africa
Wallis and Futuna,WLF,WF,Oceania
Viet Nam,VNM,VN,Asia
Vanuatu,VUT,VU,Oceania
Uzbekistan,UZB,UZ,Asia
Uruguay,URY,UY,South America
United States,USA,US,North America

`uniq`, `sort`, and `cut`

This doesn’t quite work because of the way the data is coded:

tail -n +2 files/examples/countries.csv | sort -t , -k4 -k1

Algeria,DZA,DZ,Africa
Angola,AGO,AO,Africa
Benin,BEN,BJ,Africa
Botswana,BWA,BW,Africa
Burkina Faso,BFA,BF,Africa
Burundi,BDI,BI,Africa
Cabo Verde,CPV,CV,Africa
Cameroon,CMR,CM,Africa
Central African Republic,CAF,CF,Africa
Chad,TCD,TD,Africa
Comoros,COM,KM,Africa
Congo,COG,CG,Africa
Côte d'Ivoire,CIV,CI,Africa
Djibouti,DJI,DJ,Africa
Egypt,EGY,EG,Africa
Equatorial Guinea,GNQ,GQ,Africa
Eritrea,ERI,ER,Africa
Ethiopia,ETH,ET,Africa
Gabon,GAB,GA,Africa
Gambia,GMB,GM,Africa
Ghana,GHA,GH,Africa
Guinea,GIN,GN,Africa
Guinea-Bissau,GNB,GW,Africa
Kenya,KEN,KE,Africa
Lesotho,LSO,LS,Africa
Liberia,LBR,LR,Africa
Libya,LBY,LY,Africa
Madagascar,MDG,MG,Africa
Malawi,MWI,MW,Africa
Mali,MLI,ML,Africa
Mauritania,MRT,MR,Africa
Mauritius,MUS,MU,Africa
Morocco,MAR,MA,Africa
Mozambique,MOZ,MZ,Africa
Namibia,"NAM",NA,Africa
Niger,NER,NE,Africa
Nigeria,NGA,NG,Africa
Rwanda,RWA,RW,Africa
Sao Tome and Principe,STP,ST,Africa
Senegal,SEN,SN,Africa
Seychelles,SYC,SC,Africa
Sierra Leone,SLE,SL,Africa
Somalia,SOM,SO,Africa
South Africa,ZAF,ZA,Africa
South Sudan,SSD,SS,Africa
Sudan,SDN,SD,Africa
Swaziland,SWZ,SZ,Africa
Togo,TGO,TG,Africa
Tunisia,TUN,TN,Africa
Uganda,UGA,UG,Africa
Western Sahara,ESH,EH,Africa
Zambia,ZMB,ZM,Africa
Zimbabwe,ZWE,ZW,Africa
Afghanistan,AFG,AF,Asia
Armenia,ARM,AM,Asia
Azerbaijan,AZE,AZ,Asia
Bahrain,BHR,BH,Asia
Bangladesh,BGD,BD,Asia
Bhutan,BTN,BT,Asia
Brunei Darussalam,BRN,BN,Asia
Cambodia,KHM,KH,Asia
China,CHN,CN,Asia
Georgia,GEO,GE,Asia
India,IND,IN,Asia
Indonesia,IDN,ID,Asia
Iraq,IRQ,IQ,Asia
Israel,ISR,IL,Asia
Japan,JPN,JP,Asia
Jordan,JOR,JO,Asia
Kazakhstan,KAZ,KZ,Asia
Kuwait,KWT,KW,Asia
Kyrgyzstan,KGZ,KG,Asia
Lao People's Democratic Republic,LAO,LA,Asia
Lebanon,LBN,LB,Asia
Malaysia,MYS,MY,Asia
Maldives,MDV,MV,Asia
Mongolia,MNG,MN,Asia
Myanmar,MMR,MM,Asia
Nepal,NPL,NP,Asia
Oman,OMN,OM,Asia
Pakistan,PAK,PK,Asia
Philippines,PHL,PH,Asia
Qatar,QAT,QA,Asia
Saudi Arabia,SAU,SA,Asia
Singapore,SGP,SG,Asia
Sri Lanka,LKA,LK,Asia
Syrian Arab Republic,SYR,SY,Asia
Tajikistan,TJK,TJ,Asia
Thailand,THA,TH,Asia
Turkey,TUR,TR,Asia
United Arab Emirates,ARE,AE,Asia
Uzbekistan,UZB,UZ,Asia
Viet Nam,VNM,VN,Asia
Yemen,YEM,YE,Asia
"Bolivia, Plurinational State of",BOL,BO,South America
"Bonaire, Sint Eustatius and Saba",BES,BQ,North America
"Congo, the Democratic Republic of the",COD,CD,Africa
Albania,ALB,AL,Europe
Andorra,AND,AD,Europe
Austria,AUT,AT,Europe
Belarus,BLR,BY,Europe
Belgium,BEL,BE,Europe
Bosnia and Herzegovina,BIH,BA,Europe
Bulgaria,BGR,BG,Europe
Croatia,HRV,HR,Europe
Cyprus,CYP,CY,Europe
Czech Republic,CZE,CZ,Europe
Denmark,DNK,DK,Europe
Estonia,EST,EE,Europe
Faroe Islands,FRO,FO,Europe
Finland,FIN,FI,Europe
France,FRA,FR,Europe
Germany,DEU,DE,Europe
Gibraltar,GIB,GI,Europe
Greece,GRC,GR,Europe
Guernsey,GGY,GG,Europe
Holy See (Vatican City State),VAT,VA,Europe
Hungary,HUN,HU,Europe
Iceland,ISL,IS,Europe
Ireland,IRL,IE,Europe
Isle of Man,IMN,IM,Europe
Italy,ITA,IT,Europe
Jersey,JEY,JE,Europe
Kosovo,XKV,NA,Europe
Latvia,LVA,LV,Europe
Liechtenstein,LIE,LI,Europe
Lithuania,LTU,LT,Europe
Luxembourg,LUX,LU,Europe
Malta,MLT,MT,Europe
Monaco,MCO,MC,Europe
Montenegro,MNE,ME,Europe
Netherlands,NLD,NL,Europe
Norway,NOR,NO,Europe
Poland,POL,PL,Europe
Portugal,PRT,PT,Europe
Romania,ROU,RO,Europe
Russian Federation,RUS,RU,Europe
San Marino,SMR,SM,Europe
Serbia,SRB,RS,Europe
Slovakia,SVK,SK,Europe
Slovenia,SVN,SI,Europe
Spain,ESP,ES,Europe
Sweden,SWE,SE,Europe
Switzerland,CHE,CH,Europe
Ukraine,UKR,UA,Europe
United Kingdom,GBR,GB,Europe
"Iran, Islamic Republic of",IRN,IR,Asia
"Korea, Republic of",KOR,KR,Asia
"Moldova, Republic of",MDA,MD,Europe
"Macedonia, the former Yugoslav Republic of",MKD,MK,Europe
Anguilla,AIA,AI,North America
Antigua and Barbuda,ATG,AG,North America
Aruba,ABW,AW,North America
Bahamas,BHS,BS,North America
Barbados,BRB,BB,North America
Belize,BLZ,BZ,North America
Bermuda,BMU,BM,North America
Canada,CAN,CA,North America
Cayman Islands,CYM,KY,North America
Costa Rica,CRI,CR,North America
Cuba,CUB,CU,North America
Curaçao,CUW,CW,North America
Dominica,DMA,DM,North America
Dominican Republic,DOM,DO,North America
El Salvador,SLV,SV,North America
Greenland,GRL,GL,North America
Grenada,GRD,GD,North America
Guatemala,GTM,GT,North America
Haiti,HTI,HT,North America
Honduras,HND,HN,North America
Jamaica,JAM,JM,North America
Mexico,MEX,MX,North America
Montserrat,MSR,MS,North America
Nicaragua,NIC,NI,North America
Panama,PAN,PA,North America
Puerto Rico,PRI,PR,North America
Saint Kitts and Nevis,KNA,KN,North America
Saint Lucia,LCA,LC,North America
Saint Vincent and the Grenadines,VCT,VC,North America
Sint Maarten (Dutch part),SXM,SX,North America
Trinidad and Tobago,TTO,TT,North America
Turks and Caicos Islands,TCA,TC,North America
United States,USA,US,North America
Australia,AUS,AU,Oceania
Fiji,FJI,FJ,Oceania
French Polynesia,PYF,PF,Oceania
Guam,GUM,GU,Oceania
Marshall Islands,MHL,MH,Oceania
New Caledonia,NCL,NC,Oceania
New Zealand,NZL,NZ,Oceania
Northern Mariana Islands,MNP,MP,Oceania
Papua New Guinea,PNG,PG,Oceania
Solomon Islands,SLB,SB,Oceania
Timor-Leste,TLS,TL,Oceania
Vanuatu,VUT,VU,Oceania
Wallis and Futuna,WLF,WF,Oceania
"Palestine, State of",PSE,PS,Asia
Argentina,ARG,AR,South America
Brazil,BRA,BR,South America
Chile,CHL,CL,South America
Colombia,COL,CO,South America
Ecuador,ECU,EC,South America
Falkland Islands (Malvinas),FLK,FK,South America
Guyana,GUY,GY,South America
Paraguay,PRY,PY,South America
Peru,PER,PE,South America
Suriname,SUR,SR,South America
Uruguay,URY,UY,South America
"Taiwan, Province of China",TWN,TW,Asia
"Tanzania, United Republic of",TZA,TZ,Africa
"Venezuela, Bolivarian Republic of",VEN,VE,South America
"Virgin Islands, British",VGB,VG,North America
"Virgin Islands, U.S.",VIR,VI,North America

`uniq`, `sort`, and `cut`

cut slices out columns defined by a delimiter (by default \t or tab)

cut -d , -f 2,4 files/examples/countries.csv

iso3,continent
AFG,Asia
DZA,Africa
ARM,Asia
AUS,Oceania
AUT,Europe
AZE,Asia
BHR,Asia
BLR,Europe
BEL,Europe
BRA,South America
KHM,Asia
CAN,North America
CHN,Asia
HRV,Europe
CZE,Europe
DNK,Europe
DOM,North America
ECU,South America
EGY,Africa
EST,Europe
FIN,Europe
FRA,Europe
GEO,Asia
DEU,Europe
GRC,Europe
ISL,Europe
IND,Asia
IDN,Asia
 Islamic Republic of",IR
IRQ,Asia
IRL,Europe
ISR,Asia
ITA,Europe
JPN,Asia
KWT,Asia
LBN,Asia
LTU,Europe
LUX,Europe
MYS,Asia
MEX,North America
MCO,Europe
NPL,Asia
NLD,Europe
NZL,Oceania
NGA,Africa
 the former Yugoslav Republic of",MK
NOR,Europe
OMN,Asia
PAK,Asia
PHL,Asia
QAT,Asia
ROU,Europe
RUS,Europe
SMR,Europe
SGP,Asia
 Republic of",KR
ESP,Europe
LKA,Asia
SWE,Europe
CHE,Europe
 Province of China",TW
THA,Asia
ARE,Asia
GBR,Europe
USA,North America
VNM,Asia
AND,Europe
JOR,Asia
LVA,Europe
MAR,Africa
PRT,Europe
SAU,Asia
SEN,Africa
SXM,North America
TUN,Africa
ARG,South America
CHL,South America
POL,Europe
UKR,Europe
HUN,Europe
LIE,Europe
SVN,Europe
BTN,Asia
BIH,Europe
FRO,Europe
 State of",PS
ZAF,Africa
CMR,Africa
COL,South America
CRI,North America
VAT,Europe
MLT,Europe
PER,South America
SRB,Europe
SVK,Europe
TGO,Africa
BGR,Europe
MDV,Asia
 Republic of",MD
PRY,South America
ALB,Europe
BGD,Asia
BRN,Asia
CYP,Europe
MNG,Asia
PAN,North America
BFA,Africa
 the Democratic Republic of the",CD
 Plurinational State of",BO
CIV,Africa
CUB,North America
HND,North America
JAM,North America
TUR,Asia
ABW,North America
CUW,North America
GAB,Africa
GHA,Africa
GUY,South America
VCT,North America
TTO,North America
ETH,Africa
GIN,Africa
KEN,Africa
XKV,Europe
SDN,Africa
ATG,North America
GNQ,Africa
SWZ,Africa
GTM,North America
KAZ,Asia
MRT,Africa
"NAM",Africa
RWA,Africa
LCA,North America
SYC,Africa
SUR,South America
URY,South America
 Bolivarian Republic of",VE
BHS,North America
CAF,Africa
COG,Africa
UZB,Asia
BEN,Africa
LBR,Africa
MMR,Asia
SOM,Africa
 United Republic of",TZ
BRB,North America
GMB,Africa
MNE,Europe
DJI,Africa
SLV,North America
PYF,Oceania
GUM,Oceania
KGZ,Asia
NIC,North America
ZMB,Africa
BMU,North America
CYM,North America
TCD,Africa
FJI,Oceania
GIB,Europe
GRL,North America
GGY,Europe
HTI,North America
JEY,Europe
MUS,Africa
CPV,Africa
IMN,Europe
MDG,Africa
MSR,North America
NCL,Oceania
NER,Africa
PNG,Oceania
ZWE,Africa
AGO,Africa
ERI,Africa
TLS,Oceania
UGA,Africa
DMA,North America
GRD,North America
MOZ,Africa
SYR,Asia
BLZ,North America
 U.S.",VI
LAO,Asia
LBY,Africa
TCA,North America
MLI,Africa
KNA,North America
AIA,North America
 British",VG
GNB,Africa
PRI,North America
MNP,Oceania
BWA,Africa
BDI,Africa
SLE,Africa
 Sint Eustatius and Saba",BQ
MWI,Africa
FLK,South America
SSD,Africa
STP,Africa
YEM,Asia
ESH,Africa
TJK,Asia
COM,Africa
LSO,Africa
SLB,Oceania
WLF,Oceania
MHL,Oceania
VUT,Oceania

Again in this case it doesn’t quite behave as you might think!

Finding files and finding text

`find` and `grep`

find is for locating files and directories by name:

# Everything in the `files/` subdirectory
find files

files
files/misc
files/misc/home-tree.txt
files/misc/root-tree.txt
files/.DS_Store
files/schedule.ics
files/examples
files/examples/jabberwocky.txt
files/examples/01_ibm_card.png
files/examples/ulysses.txt
files/examples/_make-example
files/examples/_make-example/mypaper.md
files/examples/_make-example/fig1.r
files/examples/_make-example/Makefile
files/examples/_make-example/README.md
files/examples/_make-example/.gitignore
files/examples/_make-example/.RData
files/examples/.DS_Store
files/examples/countries.csv
files/examples/bashrc.txt
files/examples/apple_mobility_daily_2021-04-12.csv
files/examples/alice_in_wonderland.txt
files/examples/fruit.txt
files/examples/shalott_1832.txt
files/examples/01_ibm_operator.png
files/examples/01_ibm_system_360.png
files/examples/pride_and_prejudice.txt
files/examples/words.txt
files/examples/mortality.txt
files/examples/sentences.txt
files/examples/shalott_1842.txt
files/examples/alice_noboiler.txt
files/scripts
files/scripts/hello-world.sh
files/scripts/make-thumbnail.sh
files/bib
files/bib/samplesyllabus.csl
files/bib/american-political-science-association.csl
files/bib/references.bib
files/bib/chicago-fullnote-bibliography-no-bib.csl
files/bib/chicago-fullnote-bibliography.csl
files/bib/chicago-syllabus-no-bib.csl
files/bib/apa.csl
files/bib/chicago-author-date.csl
files/bib/.auctex-auto
files/bib/.auctex-auto/references.el
files/bib/chicago-note-bibliography.csl

`find` and `grep`

We can use globbing (or wildcards) to narrow our search:

# Everything underneath the `files/` subdirectory 
# whose name ends in `.csl`
find files -name "*.csl"

files/bib/samplesyllabus.csl
files/bib/american-political-science-association.csl
files/bib/chicago-fullnote-bibliography-no-bib.csl
files/bib/chicago-fullnote-bibliography.csl
files/bib/chicago-syllabus-no-bib.csl
files/bib/apa.csl
files/bib/chicago-author-date.csl
files/bib/chicago-note-bibliography.csl

`find` and `grep`

Here we use the . to mean “Search in the current folder”

find . -name "*.xlsx"

./data/schedule.xlsx
./data/~$schedule.xlsx

`find` and `grep`

The -exec option lets us do things with each result.
The {} expands to each found file in turn.
Here we use echo to see what the rm (remove) command would do.
The quoted semicolon ";" or \; is required to end the line

find files -name "*.png" -exec echo rm {} ";"

rm files/examples/01_ibm_card.png
rm files/examples/01_ibm_operator.png
rm files/examples/01_ibm_system_360.png

If we omitted the echo here the found files would be deleted one at a time.

`find` and `grep`

We can also use xargs to act on search results:

# Everything underneath the `files/` subdirectory 
# whose name ends in `.png`
find files -name "*.png"

files/examples/01_ibm_card.png
files/examples/01_ibm_operator.png
files/examples/01_ibm_system_360.png

Convert all these png files to jpg:

# Everything underneath the `files/` subdirectory 
# whose name ends in `.png`
find files -name '*.png' -print0 | xargs -0 -r mogrify -format jpg

`find` and `grep`

Check:

find files -name '*.png' -o -name '*.jpg'

files/examples/01_ibm_card.jpg
files/examples/01_ibm_card.png
files/examples/01_ibm_operator.png
files/examples/01_ibm_operator.jpg
files/examples/01_ibm_system_360.png
files/examples/01_ibm_system_360.jpg

Delete them (with another method of deletion):

find files  -name '*.jpg' -type f -delete

`find` and `grep`

find searches file and folder names only. To search inside files we use grep.

grep Stately files/examples/ulysses.txt

Stately, plump Buck Mulligan came from the stairhead, bearing a bowl of

Search more than one file:

## -i for case-insensitive search
grep -i jabber files/examples/*.txt

files/examples/jabberwocky.txt:“Beware the Jabberwock, my son! 
files/examples/jabberwocky.txt:      The Jabberwock, with eyes of flame, 
files/examples/jabberwocky.txt:“And hast thou slain the Jabberwock? 
files/examples/ulysses.txt:jabber on their girdles: roguewords, tough nuggets patter in their

`find` and `grep`

Grep and its derivatives are very powerful thanks to their ability to use regular expressions. We will be learning more about that next week. For now, we’ll introduce rg or ripgrep, which is a modern version of grep that has some nice additional features. For more details see the ripgrep project page

# On a Mac
brew install rg

Getting around in the shell

Your command history

Eventually you will accumulate a history of shell commands you have typed, many of which you may want to reuse from time to time.

Go to the previous command with the up arrow, ↑
Search your command history with control-R, ^R
^R will also work for history search at the RStudio console and in many other places.

Aside: Standard Modifier Key Symbols

Symbol	Key	Unicode	Symbol	Key	Unicode
⎋	Escape	`U+238B`	⌫	Backspace	`U+232B`
⇥	Tab	`U+21E5`	⌦	Delete	`U+2326`
⇪	Caps Lock	`U+21EA`	⇱	Home	`U+21F1`
⇧	Shift	`U+21E7`	⇲	End	`U+21F2`
⌃	Control	`U+2303`	⇞	Page Up	`U+21DE`
⌥	Option/Alt	`U+2325`	⇟	Page Down	`U+21DF`
⌘	Command	`U+2318`	⌤	Enter	`U+2324`
⏎	Return	`U+23CE`	␣	Space	`U+2423`

Integrate `rg` and `fzf`

fzf is a command-line fuzzy-finder. It makes ^R really powerful and convenient. For details see the fzf project page.

# On a Mac
brew install fzf

Naming things

Naming files

The better your names for things, the easier they will be to find (and programmatically work with)
In civilized operating systems, names containing spaces and special characters (such as ? ! , . # $ * <space> and the like) are not a problem.
However, the more you work programatically, the more you will want to avoid them.
Jenny Bryan’s 5 minute Normconf talk is a good overview of good habits

Naming files

Names should tell you something about what the file is
Names should avoid spaces and punctuation
Names should follow some reasonable convention
Names with numbers should sort in useful ways
Names should not be used to track the versions of files

Naming files

Find all files in or below the project directory that end in .qmd:

find . -name "*.qmd"

./schedule/index.qmd
./example/04-example.qmd
./example/01-example.qmd
./example/02-example.qmd
./example/index.qmd
./example/05-example.qmd
./example/03-example.qmd
./content/05-content.qmd
./content/03-content.qmd
./content/index.qmd
./content/04-content.qmd
./content/01-content.qmd
./content/02-content.qmd
./assignment/02-problem-set.qmd
./assignment/index.qmd
./about/index.qmd
./index.qmd
./slides/05-slides.qmd
./slides/00-slides.qmd
./slides/02-slides.qmd
./slides/01-slides.qmd
./slides/04-slides.qmd
./slides/03-slides.qmd
./syllabus/index.qmd
./README.qmd

Naming files

Find all files in or below the current directory that start with two characters followed by -example and end with any other number of characters:

find . -name "??-example*"

./example/04-example.qmd
./example/01-example.qmd
./example/02-example.qmd
./example/04-example.html
./example/03-example.html
./example/04-example_files
./example/02-example.html
./example/05-example.html
./example/01-example.html
./example/05-example.qmd
./example/03-example.qmd
./_freeze/example/03-example
./_freeze/example/02-example
./_freeze/example/01-example
./_freeze/example/04-example
./_freeze/example/05-example
./.quarto/idx/example/03-example.qmd.json
./.quarto/idx/example/02-example.qmd.json
./.quarto/idx/example/01-example.qmd.json
./.quarto/idx/example/05-example.qmd.json
./.quarto/idx/example/04-example.qmd.json
./.quarto/_freeze/example/03-example
./.quarto/_freeze/example/02-example
./.quarto/_freeze/example/01-example
./.quarto/_freeze/example/04-example
./.quarto/_freeze/example/05-example

Sort order

mkdir tmp
touch tmp/{1..15}.txt

See how these sort:

ls tmp/

1.txt
10.txt
11.txt
12.txt
13.txt
14.txt
15.txt
2.txt
3.txt
4.txt
5.txt
6.txt
7.txt
8.txt
9.txt

Not what we want.

Sort order

rm -f tmp/*.txt
touch tmp/{01..15}.txt
ls tmp/

01.txt
02.txt
03.txt
04.txt
05.txt
06.txt
07.txt
08.txt
09.txt
10.txt
11.txt
12.txt
13.txt
14.txt
15.txt

Sort order

rm -f tmp/*.txt
touch tmp/{a..d}{01..03}.txt
ls -l tmp/
rm -rf tmp/
rm -rf ../tmp/

total 0
-rw-r--r--  1 kjhealy  staff  0 Jan  8 13:42 a01.txt
-rw-r--r--  1 kjhealy  staff  0 Jan  8 13:42 a02.txt
-rw-r--r--  1 kjhealy  staff  0 Jan  8 13:42 a03.txt
-rw-r--r--  1 kjhealy  staff  0 Jan  8 13:42 b01.txt
-rw-r--r--  1 kjhealy  staff  0 Jan  8 13:42 b02.txt
-rw-r--r--  1 kjhealy  staff  0 Jan  8 13:42 b03.txt
-rw-r--r--  1 kjhealy  staff  0 Jan  8 13:42 c01.txt
-rw-r--r--  1 kjhealy  staff  0 Jan  8 13:42 c02.txt
-rw-r--r--  1 kjhealy  staff  0 Jan  8 13:42 c03.txt
-rw-r--r--  1 kjhealy  staff  0 Jan  8 13:42 d01.txt
-rw-r--r--  1 kjhealy  staff  0 Jan  8 13:42 d02.txt
-rw-r--r--  1 kjhealy  staff  0 Jan  8 13:42 d03.txt

In general keep your names lower-case.

Dates

Use the one true YMD format, ISO 8601:

YYYY-MM-DD

Naming files

Be consistent in your use of naming conventions
No need to get too clever, but …

data_clean/
data_raw/
docs/
figures/
R/01_clean-data.R
R/02_process-data.R
R/03_descriptive-figs-tables.R
R/04_brms-model.R
paper/
README.md

Unix naming conventions

Dotfiles and underscores

ls -l

total 352
drwxr-xr-x@  8 kjhealy  staff    256 Aug 15 16:35 R
-rw-r--r--@  1 kjhealy  staff   1967 Sep 18 21:38 README.md
-rw-r--r--@  1 kjhealy  staff   1764 Aug 30 12:28 README.qmd
drwxr-xr-x@  3 kjhealy  staff     96 Sep 18 21:38 README_files
drwxr-xr-x   6 kjhealy  staff    192 Jan  8 13:32 _extensions
drwxr-xr-x   9 kjhealy  staff    288 Jan  8 13:41 _freeze
-rw-r--r--@  1 kjhealy  staff   2946 Jan  8 13:38 _quarto.yml
drwxr-xr-x   2 kjhealy  staff     64 Jan  8 13:41 _site
drwxr-xr-x@  7 kjhealy  staff    224 Jan  8 13:38 _targets
-rw-r--r--@  1 kjhealy  staff   2286 Jan  8 13:37 _targets.R
-rw-r--r--@  1 kjhealy  staff    840 Jan  8 10:27 _variables.yml
drwxr-xr-x@  4 kjhealy  staff    128 Jan  8 13:41 about
drwxr-xr-x@ 11 kjhealy  staff    352 Sep 25 12:30 assets
drwxr-xr-x@  6 kjhealy  staff    192 Jan  8 13:41 assignment
drwxr-xr-x@ 14 kjhealy  staff    448 Jan  8 13:41 content
drwxr-xr-x@  4 kjhealy  staff    128 Jan  8 10:38 data
-rwxr-xr-x@  1 kjhealy  staff    295 Sep  7 14:38 deploy.sh
drwxr-xr-x@ 15 kjhealy  staff    480 Jan  8 13:41 example
drwxr-xr-x@  8 kjhealy  staff    256 Sep  8 11:14 files
drwxr-xr-x  14 kjhealy  staff    448 Jan  8 10:23 html
-rw-r--r--   1 kjhealy  staff  42894 Jan  8 13:41 index.html
-rw-r--r--@  1 kjhealy  staff   6379 Oct  4 16:22 index.qmd
-rw-r--r--@  1 kjhealy  staff    258 Jan  8 13:33 mptc.Rproj
drwxr-xr-x@  7 kjhealy  staff    224 Aug 15 17:04 renv
-rw-r--r--@  1 kjhealy  staff  46675 Jan  8 13:38 renv.lock
-rw-r--r--   1 kjhealy  staff  46717 Dec 11 18:14 renv.lock.orig
drwxr-xr-x@  4 kjhealy  staff    128 Jan  8 13:41 schedule
drwxr-xr-x   9 kjhealy  staff    288 Jan  8 13:41 site_libs
drwxr-xr-x@ 12 kjhealy  staff    384 Jan  8 13:42 slides
drwxr-xr-x   3 kjhealy  staff     96 Jan  8 10:39 syllabus

Unix naming conventions

ls -la

total 440
drwxr-xr-x@ 41 kjhealy  staff   1312 Jan  8 13:42 .
drwxr-xr-x@ 23 kjhealy  staff    736 Jan  3 11:40 ..
-rw-r--r--@  1 kjhealy  staff  10244 Sep  6 18:18 .DS_Store
-rw-r--r--@  1 kjhealy  staff  12691 Jan  8 10:44 .Rhistory
-rw-r--r--@  1 kjhealy  staff     26 Aug 15 16:55 .Rprofile
drwxr-xr-x@  4 kjhealy  staff    128 Aug 10 16:06 .Rproj.user
drwxr-xr-x@ 16 kjhealy  staff    512 Jan  8 13:42 .git
-rw-r--r--@  1 kjhealy  staff    330 Sep 19 15:41 .gitignore
-rw-r--r--   1 kjhealy  staff    163 Dec 11 18:17 .gitmodules
-rw-r--r--@  1 kjhealy  staff    821 Aug 16 10:34 .luarc.json
drwxr-xr-x@  6 kjhealy  staff    192 Jan  8 13:41 .quarto
drwxr-xr-x@  8 kjhealy  staff    256 Aug 15 16:35 R
-rw-r--r--@  1 kjhealy  staff   1967 Sep 18 21:38 README.md
-rw-r--r--@  1 kjhealy  staff   1764 Aug 30 12:28 README.qmd
drwxr-xr-x@  3 kjhealy  staff     96 Sep 18 21:38 README_files
drwxr-xr-x   6 kjhealy  staff    192 Jan  8 13:32 _extensions
drwxr-xr-x   9 kjhealy  staff    288 Jan  8 13:41 _freeze
-rw-r--r--@  1 kjhealy  staff   2946 Jan  8 13:38 _quarto.yml
drwxr-xr-x   2 kjhealy  staff     64 Jan  8 13:41 _site
drwxr-xr-x@  7 kjhealy  staff    224 Jan  8 13:38 _targets
-rw-r--r--@  1 kjhealy  staff   2286 Jan  8 13:37 _targets.R
-rw-r--r--@  1 kjhealy  staff    840 Jan  8 10:27 _variables.yml
drwxr-xr-x@  4 kjhealy  staff    128 Jan  8 13:41 about
drwxr-xr-x@ 11 kjhealy  staff    352 Sep 25 12:30 assets
drwxr-xr-x@  6 kjhealy  staff    192 Jan  8 13:41 assignment
drwxr-xr-x@ 14 kjhealy  staff    448 Jan  8 13:41 content
drwxr-xr-x@  4 kjhealy  staff    128 Jan  8 10:38 data
-rwxr-xr-x@  1 kjhealy  staff    295 Sep  7 14:38 deploy.sh
drwxr-xr-x@ 15 kjhealy  staff    480 Jan  8 13:41 example
drwxr-xr-x@  8 kjhealy  staff    256 Sep  8 11:14 files
drwxr-xr-x  14 kjhealy  staff    448 Jan  8 10:23 html
-rw-r--r--   1 kjhealy  staff  42894 Jan  8 13:41 index.html
-rw-r--r--@  1 kjhealy  staff   6379 Oct  4 16:22 index.qmd
-rw-r--r--@  1 kjhealy  staff    258 Jan  8 13:33 mptc.Rproj
drwxr-xr-x@  7 kjhealy  staff    224 Aug 15 17:04 renv
-rw-r--r--@  1 kjhealy  staff  46675 Jan  8 13:38 renv.lock
-rw-r--r--   1 kjhealy  staff  46717 Dec 11 18:14 renv.lock.orig
drwxr-xr-x@  4 kjhealy  staff    128 Jan  8 13:41 schedule
drwxr-xr-x   9 kjhealy  staff    288 Jan  8 13:41 site_libs
drwxr-xr-x@ 12 kjhealy  staff    384 Jan  8 13:42 slides
drwxr-xr-x   3 kjhealy  staff     96 Jan  8 10:39 syllabus

Unix naming conventions

Files and folders beginning with a period, ., are “hidden”
They won’t show up via ls
By convention they are often used for configuration information
Files or folders beginning with an underscore, _, are often “generated” (though this is a weak convention)
The structure of plain-text config files will depend on the thing they are configuring. It might just a list of words or options, or it might be a structured file based on a Markup language like YAML or TOML, or it might be written to be parsed in R or Python, etc.

Unix naming conventions

Here’s the .gitignore file for this project:

.Rproj.user
.Rhistory
.RData
.Ruserdata

/.quarto/
/_site/
/renv/

/_freeze/
/_targets/

about/*.pdf
about/*.html
assignment/*.html
example/*.html
schedule/*.html
syllabus/*.html

slides/*.pdf
slides/*.html
slides/**/*_cache/*
slides/libs/*
projects/*.zip

# knitr and caching
**/*_files/*
**/*_cache/*

README.html

/.luarc.json

Unix naming conventions

A .bashrc file to configure non-login shells for Bash:

# ~/.bashrc: executed by bash(1) for non-login shells.
# see /usr/share/doc/bash/examples/startup-files (in the package bash-doc)
# for examples

# If not running interactively, don't do anything
case $- in
    *i*) ;;
      *) return;;
esac

# don't put duplicate lines or lines starting with space in the history.
# See bash(1) for more options
HISTCONTROL=ignoreboth

# append to the history file, don't overwrite it
shopt -s histappend

# for setting history length see HISTSIZE and HISTFILESIZE in bash(1)
HISTSIZE=1000
HISTFILESIZE=2000

# check the window size after each command and, if necessary,
# update the values of LINES and COLUMNS.
shopt -s checkwinsize

# If set, the pattern "**" used in a pathname expansion context will
# match all files and zero or more directories and subdirectories.
#shopt -s globstar

# make less more friendly for non-text input files, see lesspipe(1)
#[ -x /usr/bin/lesspipe ] && eval "$(SHELL=/bin/sh lesspipe)"

# set variable identifying the chroot you work in (used in the prompt below)
if [ -z "${debian_chroot:-}" ] && [ -r /etc/debian_chroot ]; then
    debian_chroot=$(cat /etc/debian_chroot)
fi

# set a fancy prompt (non-color, unless we know we "want" color)
case "$TERM" in
    xterm-color) color_prompt=yes;;
esac

# uncomment for a colored prompt, if the terminal has the capability; turned
# off by default to not distract the user: the focus in a terminal window
# should be on the output of commands, not on the prompt
force_color_prompt=yes

if [ -n "$force_color_prompt" ]; then
    if [ -x /usr/bin/tput ] && tput setaf 1 >&/dev/null; then
        # We have color support; assume it's compliant with Ecma-48
        # (ISO/IEC-6429). (Lack of such support is extremely rare, and such
        # a case would tend to support setf rather than setaf.)
        color_prompt=yes
    else
        color_prompt=
    fi
fi

if [ "$color_prompt" = yes ]; then
#    PS1='${debian_chroot:+($debian_chroot)}\[\033[01;32m\]\u@\h\[\033[00m\]:\[\033[01;34m\]\w\[\033[00m\]\$ '
     PS1='${debian_chroot:+($debian_chroot)}\[\033[01;32m\]\H\[\033[00m\]:\[\033[01;34m\]\w\[\033[00m\] \$ '
else
    PS1='${debian_chroot:+($debian_chroot)}\u@\h:\w\$ '
fi
unset color_prompt force_color_prompt

# If this is an xterm set the title to user@host:dir
case "$TERM" in
xterm*|rxvt*)
    PS1="\[\e]0;${debian_chroot:+($debian_chroot)}\u@\h: \w\a\]$PS1"
    ;;
*)
    ;;
esac

# enable color support of ls and also add handy aliases
if [ -x /usr/bin/dircolors ]; then
    test -r ~/.dircolors && eval "$(dircolors -b ~/.dircolors)" || eval "$(dircolors -b)"
    alias ls='ls --color=auto'
    #alias dir='dir --color=auto'
    #alias vdir='vdir --color=auto'

    alias grep='grep --color=auto'
    alias fgrep='fgrep --color=auto'
    alias egrep='egrep --color=auto'
fi

# some more ls aliases
#alias ll='ls -l'
#alias la='ls -A'
#alias l='ls -CF'

# Alias definitions.
# You may want to put all your additions into a separate file like
# ~/.bash_aliases, instead of adding them here directly.
# See /usr/share/doc/bash-doc/examples in the bash-doc package.

if [ -f ~/.bash_aliases ]; then
    . ~/.bash_aliases
fi

# enable programmable completion features (you don't need to enable
# this, if it's already enabled in /etc/bash.bashrc and /etc/profile
# sources /etc/bash.bashrc).
if ! shopt -oq posix; then
  if [ -f /usr/share/bash-completion/bash_completion ]; then
    . /usr/share/bash-completion/bash_completion
  elif [ -f /etc/bash_completion ]; then
    . /etc/bash_completion
  fi
fi

The Shell

Recap

Technical Computing is frustrating

Where technical computing lives

Different Answers

Office model

Engineering model

The File System

A tree-like hierarchy

File system hierarchy

Standard locations

Standard locations

Standard locations

The Shell

What is it?

There are many shells

A command interpreter

Getting around the file system

Who and where

Listing files

A Tree

Changing directories

More detail

More detail

Path rules

Path rules

Some shell tools

wc, cat, head, and tail

wc, cat, head, and tail

wc, cat, head, and tail

wc, cat, head, and tail

wc, cat, head, and tail

wc, cat, head, and tail

wc, cat, head, and tail

wc, cat, head, and tail

uniq, sort, and cut

uniq, sort, and cut

uniq, sort, and cut

uniq, sort, and cut

Finding files and finding text

find and grep

find and grep

find and grep

find and grep

find and grep

find and grep

find and grep

find and grep

Getting around in the shell

Your command history

Aside: Standard Modifier Key Symbols

Integrate rg and fzf

Naming things

Naming files

Naming files

Naming files

Naming files

Sort order

Sort order

Sort order

Dates

Use the one true YMD format, ISO 8601:

YYYY-MM-DD

Naming files

Unix naming conventions

Unix naming conventions

Unix naming conventions

Unix naming conventions

Unix naming conventions

`wc`, `cat`, `head`, and `tail`

`wc`, `cat`, `head`, and `tail`

`wc`, `cat`, `head`, and `tail`

`wc`, `cat`, `head`, and `tail`

`wc`, `cat`, `head`, and `tail`

`wc`, `cat`, `head`, and `tail`

`wc`, `cat`, `head`, and `tail`

`wc`, `cat`, `head`, and `tail`

`uniq`, `sort`, and `cut`

`uniq`, `sort`, and `cut`

`uniq`, `sort`, and `cut`

`uniq`, `sort`, and `cut`

`find` and `grep`

`find` and `grep`

`find` and `grep`

`find` and `grep`

`find` and `grep`

`find` and `grep`

`find` and `grep`

`find` and `grep`

Integrate `rg` and `fzf`