Understanding Environment Variables and the Unix Path

Published 2013-03-26

When I was new to Mac and Linux, environment variables and the Unix PATH were mysterious and confusing. Documentation for various tools would frequently make references to "add this to your path" or "set this environment variable" and I had no idea what they meant.

Fortunately, this seemingly-magic behavior is relatively transparent once you know where to look and understand some of the basic assumptions in Unix. By the time you're done reading you'll be on your way to mastering environment variables.

PATH and ENV In 5 Minutes or Less

When you type the name of a program cat or grep, your shell looks in all the directories specified in your PATH to try to find a matching program. PATH itself is an environment variable (other common ones include EDITOR and JAVA_HOME). Environment variables are global variables that live in your shell session, and help your shell fill in shortcuts or specify preferences.

You can dynamically set (or change) environment variables using export, or persistently set the values in your ~/.bash_profile or ~/.bashrc file. For example, we set EDITOR to vim, then use $EDITOR to invoke vim on our .bash_profile so we can persist some other environment variables:

$ export EDITOR=vim
$ $EDITOR ~/.bash_profile
export PATH=$PATH:/something/i/need/to/add
export NEW_ENVIRONMENT_VARIABLE=value

Environment variables are read from .bash_profile at the start of the shell session, so after you save the file you'll need open a new terminal window before the changes will take effect.

If the extremely terse example above doesn't satisfy you, keep reading. (Also, if you got stuck in vim, press ESC type :quit! and press ENTER to exit.)

A Primer on bash

bash is the standard shell on OS X and most Linux distros. bash is both an interactive shell and a scripting language. Examples of the latter case are called "bash scripts".

~/.bash_profile is actually a bash script which, by convention, is executed every time bash starts up (i.e. whenever you open a new terminal). So before we go further, let's get familiar with a bit of bash syntax.

bash Prompt Conventions

$ followed by a space is used in documentation to indicate the bash prompt, so you will frequently see this indicating that you should type something at the terminal. (Don't type the $, obviously.) E.g:

$ export EDITOR=vim

When you see a line in this context that does not start with $ it generally refers to output from the preceding command.

Also, all exported variable names should be UPPERCASED, or Bad Things will happen.

Basic bash Syntax

You can play around with these on your terminal:

$ FOO=cake
$ echo $FOO
cake

In bash alone, this has essentially the same effect:

$ export BAR=pie
$ echo $BAR
pie

If you want BAR to become an environment variable and be available to other programs you must use export.

Notice that you can't add spaces around the = when doing variable assignment:

$ BAR = pie
BAR: command not found

Also, if you don't use the dollar sign when referencing a variable you'll just output a string:

$ echo $BAR
pie
$ echo BAR
BAR

Finally, let's put it all together to modify an existing variable:

$ FOO="I like $FOO$BAR"
$ echo $FOO
I like cakepie

Yes, cakepie is delicious. Anyhow, this is the basis of how we'll modify our PATH environment variable. (Because someone smarter than us put a lot of useful things in there, and if we override PATH instead of appending things we'll break our shell. Which is a Bad Thing.)

bash Documentation

If you want to learn more about bash, export, etc., type man bash at your terminal to read the man page, or read it online.

Environment Variables

As in the examples above, an environment variable is simply a variable that has been exported into an environment (usually your shell session) and can be referenced by other programs.

export EDITOR=vim
export PATH=/home/cbednarski/pythons/27/bin:/usr/local/bin:/usr/bin:/bin

Or exported from the terminal:

$ export EDITOR=vim

If you have php installed on your system you can run this:

$ php -r "echo getenv('EDITOR');"
vim

Or if you have python installed:

$ python -c "import os; print os.environ['EDITOR']"
vim

Which demonstrates that the EDITOR environment variable we exported is now available to php and python during runtime. This is a common pattern for configuring command-line tools, since the environment variable is easy to change in terminal session or on a user-by-user basis across a single system.

PATH to Success

PATH is a special environment variable. The "full path" to a file represents its exact location on the filesystem, such as /usr/bin/bash. Any time you type the name of a program without a full path, bash searches your PATH to find it. Here's a simple example of my PATH:

/home/cbednarski/pythons/27/bin:/usr/local/bin:/usr/bin:/bin

The PATH is searched from left to right, with : as a separator. So when I type python in my terminal, bash will look in the following places (in order) to find it:

If it finds an executable named python, it will run that program. If bash doesn't find it, we'll see python: command not found.

Wait! What if bash finds more than one python program?

It won't. As soon as bash finds python in one of those paths, it executes it and stops searching.

Which Program did bash Find?

If you have multiple pythons in your path you can use which python to see which one is being executed, or which -a python to enumerate all of them. The left-to-right precedence in PATH is a key feature, since it allows you to override system defaults.

which is a great way to see whether your path is configured correctly.

Modifying your PATH

The safest way to add something to your path is to append it. Note that you do not name the executable itself in the path, just the folder that it sits in.

export PATH=$PATH:/opt/python33

If you want to override the system default for a program, you should prepend instead:

export PATH=/opt/python:$PATH

If you want to make persistent modifications to PATH, you'll need to make these changes in ~/.bash_profile on your computer. If you just want to mess around and throw away the changes later you can run these commands in your terminal (as we did earlier in the "Primer on bash" section).

Also, remember that you'll need to open a new terminal before changes made to ~/.bash_profile take effect.

Stepping into the Path

Sometimes it's more convenient to place something in the PATH instead of changing the PATH itself. You can accomplish this easily using symlinks. The format is:

$ ln -s TARGET ALIAS

/usr/local/bin is the conventional place to make customizations like this, so I could do this and have many versions of python on my system:

$ ln -s /home/cbednarski/pythons/27/bin/python /usr/local/bin/python27
$ ln -s /home/cbednarski/pythons/33/bin/python /usr/local/bin/python33

Then I invoke them using:

$ python27
$ python33

Normally, bash would only find one of these programs because they're all named python, but with the symlink trick I can make their names unique and still run them easily without having to type in the full path each time.

Appendix A: The Executable Bit

Unix file permissions include a special flag that indicates whether or not you're allowed to run a script as an executable. If you placed something in your path but you can't run it by typing the name, make sure the executable bit is set:

$ chmod +x /home/cbednarski/pythons/27/bin/python

You can verify this by running ls -l and looking for x in the file permissions:

$ ls -l /home/cbednarski/pythons/27/bin/python
-rwxrwxr-x 1 cbednarski cbednarski 2989480 Dec 16 17:28 /home/cbednarski/pythons/27/bin/python

You may need elevated permissions (e.g. sudo) to change this.

Appendix B: The shebang, or "Which program do I use to run this file?"

You may have noticed in some files a pattern that looks like this:

#!/usr/bin/python

This is called a "shebang" or "hashbang". When you write an executable script you need to instruct bash where to find the interpreter to run it. For example, bash doesn't understand python, and the python interpreter can't run php code, so if you want to write a script in python or php and make it executable, you'll need the proper shebang line.

The most common form of the shebang, #!/usr/bin/python, is actually not correct. You might observe that it doesn't look in our PATH, and instead is hard-coded to look in a particular place on the file system. What if system python on our computer is version 2.6, but we want to use 2.7? Or what if python doesn't live in /usr/bin?

For optimum portability, the best way to write this is:

#!/usr/bin/env python

The /usr/bin/env bit itself is still a hard-coded convention, but it's more flexible since it allows us to use PATH expansion to find which python interpreter to use without needing to modify the original script.

Related

path bash linux osx environment env