Brad Conte

A Letter to My Congressmen Regarding SOPA and PIPA

Brad Conte — Wed, 18 Jan 2012 21:45:16 +0000

I wrote my three congressmen today to voice my opposition to a well-known pair of bills called SOPA and PIPA that are under consideration by the United States House of Representatives and Senate (respectively). These bills were drafted with strong support from the multimedia industry and they bring a very heavy hand into the legal realm of copyright enforcement and are very unpopular with Internet-based companies and most Internet users in general. As I write this, many websites are in the middle of a day-long self-imposed blackout in protest.

I strongly oppose these bills. I do sympathize with the fact that multimedia companies have their legal and moral rights that are violated by mass-piracy, but these bills take far too drastic action to protect the multimedia sector. I won’t re-iterate the reasons why these bills are bad, even in spite of recent tamer modifications.

So I wrote my three congressmen today to voice my opposition. I’d like to share the letter publicly, as a wider proclamation of my position on this issue and hopefully as an aid for anyone writing their congressmen on a political issue. It’s not a master-piece, but I thought it might be helpful. I’ll explain some of the reasoning and structure below. Here it is:

To the Honorable [insert congressman's full name],

I would like to add my voice of support to the millions of people who oppose [SOPA/PIPA].

I am sympathetic to the motivation of the bill. I understand that multimedia companies have proper motivation to protect their intellectual property. I support their moral and legal rights to own their content.

However, I do not believe that their efforts to protect their property should be at the expense of humanity’s convenience and technological development. We the people do not exist to listen to music, we listen to music while we live our lives. Similarly, technology does not exist to play music, it exists to enable us to do what we want, some of which is to listen to music. Legislation like [SOPA/PIPA] takes a multimedia-centered viewpoint of the universe, assuming that we should hinder productivity and change the dynamics of an entire industry just to protect the convenience of one sector of that industry.

The multimedia industry has a history of resisting technological development due to their reluctance to change business models. They tried to legally combat cassette tapes, video tapes, and CDs under the pretense of fighting piracy that would hurt them. Yet those very mediums enabled them to distribute their content in better, more widely-reaching ways than before. They resist change, yet change and progress is what technology, and humanity itself, is about. Every sector of an industry faces critical changes over time, and while it can be difficult for them to adjust they should not seek legal aid pass their difficulties onto us, the common citizen. This is a capitalist society, the multimedia sector needs to adjust, not be pampered.

My request, congressman, is that your position be to protect the progress of the general public. I support the multimedia sector’s desire to protect its content – once again, I sympathize with their motivation – but not at the expense of all our development. Their history and the current [SOPA/PIPA] legislation show that they are not seeking to provide us with multimedia enjoyment while we live our lives, they prefer to limit our lives to fit their existing business models.

Thank you for your time. For what it’s worth, I voted for you. To protect me.

–Brad Conte

Some thoughts on this letter, in no particular order:

I used SOPA or PIPA where appropriate for the recipient. One bill is in one house, the other bill in the other house. Writing a blanket statement like “SOPA and PIPA”, or worse yet just the popular one “SOPA”, sounds more like a form letter. Who wants to get a letter that mentioning a bill that they can’t even vote on?
I wanted to paint a big picture perspective. 1) I gave a very quick history of similar actions in the industry and their outcomes. 2) I noted that the multimedia industry is just a sub-sector of the larger industry that this bill effects. 3) I remembered international concerns; the U.S. is the country most impacted immediately, but this likely has implications for the whole world, hence I slipped in the word “humanity” a couple times. The goal wasn’t to be dramatic, but to always remind them of how wide-reaching the implication would be. The overall point was that this was far too invasive an action to protect one specific sector.
As a general rule for opinions and debates, you should always separate general intent from specific implementation and you should separate analysis of the consequences from debate over what the consequences are. If you want to get your point across to someone, decide which perspective you’re arguing and make it clear, too many conversations are completely wasted due to a missing of the minds on those simple topics and jumping confusingly between perspectives. In my letter, I opposed this implementation of copyright and laid out the consequences for what happened. If the congressmen doesn’t agree that the assessments that I asserted, that’s the subject for a separate, much longer e-mail. I would suggest that it’s best to go for lots of detail or very little, because few things are as weak as an argument that uses just a few details strewn out.
It’s easy to read – about 370 words and about 2/3 of a page when printed, so easily readable in a few minutes. There are 5 paragraphs (the closing line isn’t really a paragraph), but only 3 have the bulk of the content; it looks very digestible at a quick glance. And that’s all you need to give a summary of a position. This makes it also easily skim-able: a) while each paragraph is unique, you could omit any one and get the majority of the argument, b) you could read the first sentence of each paragraph and still get the message (that’s actually a good rule in general), and c) you could read the first and last paragraphs and understand it. Given the popularity of this topic, it’s likely that if the congressman is reading my letter, it is just one of hundreds on the same topic.
I emphasized that my position didn’t spell doom and gloom for the opposition. In fact, I trivialized their position as simply being for “convenience”. It isn’t the annihilation of multimedia companies it’s simply them finding and adjusting to a new business model, just like they’ve done several times in the past. It’s important to know what’s at stake in a situation like this, and I wanted to contrast humanity’s technological stifling against their convenient business model. (In retrospect, trivializing it as “convenience” was a pretty strong statement to make without supporting evidence, maybe it should’ve been a little tamer.)
Similar to the above, the secondary focus of the letter was on priorities. “We the people” are not befitted in general by this bill. I said and hinted at that a couple of times.
The tone is unemotional, yet a little grave. It’s a serious subject but it won’t kill my grandmother, so there’s no need to sound like it would.
The closing line (“…I voted for you. To protect me.”) might come off a bit snarky, but it conveys a strong and valid point. Congressmen are put in place by “the people” and those people are who will lose (and lose very badly) should the legislation pass. Obviously there are times where a congressmen needs to put aside the individual or short-term benefits of each person for the big picture, but I wanted to remind them that my hope is that their voice will do its best to echo mine. If I would be screaming in opposition to it, that should count for something. I didn’t make any threats (ie, “I won’t vote for you if you don’t oppose this”), I just reminded them of established fact. And, for what it’s worth, I did actually vote for them. (I shouldn’t have to point that out, but it’s an easy statement to lie about and I’ve seen it done elsewhere.)

Hopefully the letters will do some good. I’m sure that some of you can do better.

Redirect input and output for an existing process on Linux

Brad Conte — Tue, 01 Mar 2011 23:51:17 +0000

Redirecting input and output of an executable is a standard and trivial practice in Linux operations. When launching a process, the user can trivially redirect the output of the process away from stdout via the “>” operator and can redirect input away from stdin using the “<” operator.

But that only works if stdin or stdout are switching before the process is launched. What if we have a pre-existing process that we would like to change a file handles for, but we would like to avoid restarting the process? For example, say you have a cron script execute “sudo my_command” from an environment where you can’t provide input (perhaps you meant to use gksudo instead). You might be able to kill the sudo process, but perhaps when sudo exits the script will proceed with undesirable results. You could kill the script too, but assume that you very badly don’t want to abort the script in a semi-completed state. (Obviously a well-written script shouldn’t have this sort of behavior, but the assumption is that we are in an unusual situation.) The ideal solution would be to redirect input into the hanging sudo process allowing it to succeed and your script to continue. Another example would be if you needed to redirect the output of an existing process to a file on

Thankfully, we can perform redirection on existing processes by explicitly manipulating the existing file descriptors. The method for doing so is fairly straight forward:

Determine which file descriptor you need to change.
Attach to the target process via gdb.
Create a file (to redirect output) or named pipe (to redirect input).
Use gdb to point the desired file descriptor to the file or pipe.
If applicable, send the necessary content through the pipe.

In terminal A, find the PID of the process, call it “TARGET_PID”. First, list the target’s existing file descriptors:

$ ls -l /proc/TARGET_PID/fd/

When we are done we will double check this list to ensure we made the changes we wanted to.

Now you need to determine which file descriptor (hereon “FD”) you want to change. Remember, we can only manipulate existing FDs, not add new ones. (For those who don’t know: FD 0 is stdin (standard input), FD 1 is stdout (standard output), FD 2 is stderr (standard error output). These are the the base FDs that every process will have, your target process may or may not have more.) Examples:

To change the input for a typical terminal program you likely need to to change stdin.
To change output file X to a different file Y, you need to find which FD on the list is linked to X.
For sudo, to change the input that accepts the user password you actually need to change FD 4, which should point to /dev/tty or something similar.

We’ll call the the FD number that you want to change “TARGET_FD”.

For using a named pipe: First create the pipe using

$ mkfifo /my/file/name

We’ll call this path TARGET_FILE. Then provide input to the pipe, or else gdb will not be able to open it in a later step. Provide the content by, for example, “echo MyContent > TARGET_FILE” from a separate terminal or as a backgrounded process. “MyContent” should be the content you want to send the process.

For using a normal file: Create an output file called TARGET_FILE.

Attach gdb to the process:

$ gdb -p TARGET_PID

Within gdb, close the file descriptor using the “close()” system call:

(gdb) p close(TARGET_FD) $1 = 0

The right-hand side of the output is the return value of the call we just executed. A value of 0 means that all went well.

Now create a FD using the “open()” system call. This must be done after “close()”, because file descriptors are issued sequentially from the lowest unused non-negative integer and we are making use of the fact that once we delete TARGET_FD it is now the lowest unused file descriptor, so the next one created will use the same number.

(gdb) p open(“TARGET_FILE”,0600) $2 = TARGET_FD

If the right-hand side number is equal to TARGET_FD, that means we just successfully created an FD and it got the same FD that we just closed, which is perfect. Remember, if you are using a named pipe, this step may (will?) hang if there is no output going into the named pipe.

Now quit gdb:

(gdb) q

At this point, you should be done. If you are redirecting output, the redirection should be under way. If you are redirecting input, the first input should be consumed from the pipe and you can continue to provide input as necessary by sending it into the pipe; when you are done simply delete the pipe using “rm”.

We can verify hat we were successful by checking the target process’s FDs. Run “ls -l /proc/TARGET_PID/fd/” again and compare the output against the output from the first time. If all went well then TARGET_FD should be changed to point at TARGET_FILE.

FoxyProxy, Firefox 3.5, and DNS Leaking

Brad Conte — Sat, 14 Nov 2009 04:18:30 +0000

[Update: Jan. 24, 2010] The DNS leaking problem described in this article applied to FoxyProxy v2.14. On Jan. 12, FoxyProxy v2.17 fixed the problem.

FoxyProxy is a popular Firefox extension that enables users to, setup, easily manage, and quickly switch between multiple proxy configurations. One of the most common uses of a proxy server is for security/privacy. By establishing an encrypted connection (usually via SSH) with a proxy server on a trusted network, you can have your web traffic go through an encrypted “pipe” to that server and have that server send and receive web requests on your behalf, relaying data to and from you only through that pipe. By doing this you eliminate the risk that someone on your current network could see your HTTP traffic. Maybe you don’t trust other clients on the network, maybe you don’t trust the gateway, it doesn’t matter — your HTTP(S) traffic is encrypted and shielded from prying eyes. (Readers unfamiliar with the concept of using HTTP proxies through SSH tunnels are encouraged to research the matter, there are many well-written tutorials available.)

There are many other popular uses of proxy servers, but the application of encrypted web traffic is of concern in this case for the following reason: A key problem that arises when using web proxy servers is the issue of handling DNS requests. DNS does not go through the HTTP protocol, so even if HTTP is being forwarded to a proxy the DNS requests may not be. If DNS requests are sent out over the network like normal then eavesdroppers can still read them. So although the actual traffic may be encrypted, the DNS queries behave normally and may cause the same problems that using an encrypted tunnel was designed to avoid in the first place. A situation in which HTTP traffic is encrypted but DNS is not is referred to as “DNS leaking”. When using a proxy for the benefit of security or privacy, DNS leaking may be just as bad as non-encrypted traffic.

Solving the DNS leaking problem is simple. One type of proxy, SOCKS5, can forward DNS requests as well as HTTP(S) data. A user simply needs to tell their browser to use the SOCKS5 proxy for both HTTP(S) and DNS and then both will be routed through the encrypted stream, allowing the user to surf the web with greatly strengthened security and privacy.

However, Firefox users who use FoxyProxy at the moment will encounter a problem when using DNS forwarding to a SOCKS5 proxy. When using FoxyProxy, DNS leaking occurs even when it is configured not to, which has made many users very upset. Initially many people thought the problem was with Firefox 3.5, but others confirmed it was only present with FoxyProxy installed. Unfortunately, however, not everyone is convinced that this is FoxyProxy-related behavior and I have not found anyone who has presented a solution yet. I plan to do both.

This is the basic setup for my tests:

I set up an SSH server.
I established an SSH connection and used the built-in SOCKS5 functionality of the SSH server daemon:
$ ssh username@myserver -D localhost:8080
(For the non-SSH inclined: This command forwarded all traffic my client sends to itself on port 8080 through the SSH connection to the SSH server, which then acts as a SOCKS5 proxy and sends the data on to the destination.)
I used Wireshark to monitor all packets, specifically DNS requests, sent or received my network’s interface. Note that DNS requests tunneled over the SSH connection to the SOCKS5 proxy are not visible to the packet sniffer.
I monitored my Firefox configuration in about:config. (All network proxy-related settings are under the filter network.proxy.)
I used Firefox v3.5.5 and FoxyProxy v2.14.

Using this I was able to monitor all DNS requests while I experimented with Firefox and FoxyProxy using a SOCKS5 proxy. I did a base test with no proxy configuration, a test using Firefox’s included proxy management, and a test using Foxyproxy for proxy management.

Using no proxy

Starting with a default configuration (SSH SOCKS connection established but no proxy settings configured to use it) I visited several websites such as google.com, yahoo.com, and schneier.com. This was the simple base test.

I checked showmyip.com to get my IP address.

The relevant about:config settings:

network.proxy.socks “” network.proxy.socks_port 0 network.proxy.socks_remote_dns false network.proxy.type 0

Via Wireshark I watched as the websites generated normal DNS requests over the standard network.

Using Firefox to configure proxy settings

I restarted Firefox to avoid any cached DNS entries. Then, without FoxyProxy installed, I setup my SOCKS5 proxy. (Note that FoxyProxy replaces the standard Firefox proxy editor, so it is impossible to not use FoxyProxy when it is installed.)

Under Firefox’s Preferences/Tools (depending on your operating system) I went to the “Advanced” tab, “Network” sub-tab, and opened “Settings”. I chose “Manual proxy configuration” and entered “localhost” for the SOCKS Host and “8080″ for the port.

Unfortunately, Firefox v3.5 does not support a GUI method of enabling DNS SOCKS5 proxying, so I had to manually go to about:config and enable it by setting network.proxy.socks_remote_dns to “true”.

I checked showmyip.com to ensure that my IP address displayed as coming from the server and not my client. It did show as coming from the server, so Firefox was using the proxy.

The final about:config settings were:

network.proxy.socks localhost network.proxy.socks_port 8080 network.proxy.socks_remote_dns true network.proxy.type 1

I visited the same websites. Via Wireshark, I did not see any DNS requests sent over the standard network. Firefox channeled both the HTTP and DNS data through the SSH tunnel perfectly.

Using FoxyProxy to configure proxy settings

I reset all the about:config settings back to their defaults. Then I installed FoxyProxy Standard v2.14. I went to FoxyProxy’s options and, under the “Proxies” tab, created a new proxy entry whch I named “SSH SOCKS5″. I set it to connect to “localhost” on port 8080. As well, I check-marked the “SOCKS proxy?” box and selected “SOCKS v5″. I went to the “Global Options” tab and checked the box “Use SOCKS proxy for DNS lookups”. To let this take effect, I had to restart Firefox.

When Firefox had restarted, I went to the Tools > FoxyProxy menu and selected to “Use proxy SSH SOCKS5 for all URLS”. I checked showmyip.com to ensure that my IP address displayed as coming from the server and not my client. It did show as coming from the server, so Firefox was using the proxy.

I checked about:config:

network.proxy.socks “” network.proxy.socks_port 0 network.proxy.socks_remote_dns false network.proxy.type 0

The configuration was the same as the default, so apparently FoxyProxy does not adjust about:config to do its work.

Watching the DNS requests via Wireshark, I watched as all the website visits generated DNS requests over the normal network. Complete and thorough DNS leaking. And I would like to emphasize that I had selected “Use SOCKS proxy for DNS lookups”, which is FoxyProxy’s option to address the DNS leaking issue.

Fixing DNS leaking

There was no question about it, FoxyProxy caused the DNS leaking in my test. I wanted to solve the problem so I fiddled with about:config.

In about:config I manually set network.proxy.type to 1. I verified my IP address was from the server via showmyip.com.

The new about:config:

network.proxy.socks network.proxy.socks_port 0 network.proxy.socks_remote_dns false network.proxy.type 1

I watched for DNS requests again via Wireshark. I saw none. It seemed that just manually setting network.proxy.type to 1 fixed the FoxyProxy DNS leaking problem.

I also tried other about:config settings, such as manually changing network.proxy.socks_remote_dns to “true”, but that didn’t work. The above was the only change in about:config that I found that fixed the problem.

Summary

I repeated the results above three times in different orders on different computers on both Linux and Windows to ensure I made no configuration mistakes and to ensure that the behavior was consistent and cross-platform. All the tests yielded the same results. Here is the final summary:

Firefox v3.5 does not suffer from DNS leaking by itself.
DNS leaking occurs when FoxyProxy is managing the proxies.
FoxyProxy does not suffer from DNS leaking when network.proxy.type is manually set to 1.

It is obvious that FoxyProxy does not adjust about:config in order to configure proxy settings, but I do not know why. Many Firefox extensions adjust about:config in order to accomplish their goals and I know of no reason they should not. It’s possible that FoxyProxy has not had a need to do so before, but in light of this serious problem that may need to change. The quickest/simplest solution for FoxyProxy may to set network.proxy.type to 1 if the currently enabled proxy is SOCKS5 and if the global options for FoxyProxy (or the about:config for Firefox) are set to enable DNS forwarding.

However, although this seems to indicate that FoxyProxy has made a mistake, I don’t know that FoxyProxy is the party at fault. Clearly FoxyProxy does not have to alter about:config in order to change the other proxy settings, so why must network.proxy.type be set in about:config in order for DNS forwarding to work? Note that network.proxy.type isn’t related to DNS forwarding, it just specifies which type of proxy is enabled. For all I know someone implemented a hack in Firefox that checks about:config when it shouldn’t. Of course, I don’t know that and I don’t know if this is expected behavior from Firefox or not. It could be that FoxyProxy isn’t setting whatever hidden configuration for DNS forwarding that exists on the same plane as the other invisible proxy settings it uses. Or maybe FoxyProxy is relying on an unreliable hack in order to avoid changing about:config. I don’t know about any of that. What I do know is that Firefox by itself does not have this DNS leaking problem, FoxyProxy does, and a simple solution exists.

Again, I am certainly not the first person to note this problem, but a) I have seen many people blame Firefox for this bug, and b) I have not yet seen anyone else mention the solution that I noted above.

I leave it to someone with more time and knowledge about these software projects to determine which project should have which bug report filed. This needs to be fixed permanently.

Three Tips for the Linux Terminal

Brad Conte — Sun, 11 Jan 2009 10:11:47 +0000

The power of Linux lies in the tools it uses. If you spend a lot of time in a terminal, you likely value anything that makes using it smoother. Here are a few tips to help make the terminal experience as smooth as possible. These tips are all shell-independent.

Change directories with the directory stack
The Bash shell (as well as others, like Zsh) have a built-in “back” feature for navigating directories. The built-in “pushd” command will put your current working path at the top of a shell-maintained stack and allow you to change to another directory. To go back to the saved path you can use “popd”.

Example:
user@host : /some/long/path/you/don’t/want/to/retype$ pushd /some/other/path/ /some/long/path/you/don’t/want/to/retype user@host : /some/other/path/$ [do stuff] user@host : /some/other/path/$ popd /some/big/long/directory/you/don’t/want/to/retype user@host : /some/long/path/you/don’t/want/to/retype$
Since “pushd” stores the directories in a stack, you can push multiple directories onto the stack and later pop them off in the reverse order you pushed them. It’s basically the standard “cd” only with a “back” feature. Speaking of which, the command “$ cd -” will always take you back to the previous directory you were in.
Interact with the X clipboard
Before I discovered “xclip”, one of the most annoying things about being in a terminal was my lack of access to the X clipboard. Most shells (all the ones I’ve come across) have a simple means of pasting text into the terminal, but not all have an elegant means of getting text out of the terminal. Yes, highlight-and-middle-click is often an option, but it’s a) not always feasible, and b) not always convenient. Remember, this is the terminal, you don’t necessarily want to be using your mouse. “xclip” removes the annoyance of using the X clipboard.

xclip can read and store to the clipboard from the standard input. As well, it can output the current contents of the clipboard. Since xclip can manipulate multiple clipboard buffers, the argument “-selection c” must be used to select the standard X clipboard. The “-i” and “-o” arguments tell xclip whether you are inputting or outputting clipboard contents, respectively.

Example:
$ pwd /some/long/path/you/don’t/want/to/retype $ ls /some/path | xclip -selection c -i
You may now paste (standard Ctrl-v, or right-click -> paste) the directory listing where ever you choose. Another example:
xclip -selection c -o | cat >> /some/file
This will concatenate the contents of the clipboard to “/some/file”.

If you only use yourself using xclip to manage the X clipboard, you might want to consider aliasing “xclip” to “xclip -selection c” to make it simpler to type. Or you could take it one step further and alias “xcopy” to “xclip -selection c -i” and “xpaste” to “xclip -selection c -o”, or something similar.
Use “head” and “tail” more powerfully
You probably know how to use “head” to extract the first N lines from a file and “tail” to extract the last N lines of a file. While this is useful, it’s often just as useful to extract the complement of those selections, namely, everything except the first N lines or everything except the last N lines.

Thankfully, “head” and “tail” are powerful enough to accommodate those needs. “head” allows you to extract either a finite quantity of text from the top or everything but a finite quantity of text from the bottom, and “tail” allows you to extract a finite quantity of text from the bottom or everything but a finite quantity of text from the top.

• head -n N — by default outputs the first N lines. Using -N outputs all but the last N lines.

• tail -n N — by default outputs the last N lines. Using +N outputs lines starting on the Nth.

Example:
/tmp $ cat example 1 2 3 4 5 /tmp $ head -n 2 example 1 2 /tmp $ tail -n 2 example 4 5 /tmp $ head -n -2 example 1 2 3 /tmp $ tail -n +3 example 3 4 5
Note that tail’s complement mode requires you to specify the first line number to include in the output. Thus if you want all but the top N lines, actually specify N+1.

LoginMSG

Brad Conte — Wed, 26 Nov 2008 03:36:06 +0000

About

I wrote LoginMSG to address a very simple convenience that I was without in my day-to-day life in Linux, namely, a simple way to leave myself a one-time message for the next time I logged in.

Keeping to-do lists is one thing, but quick “first thing tomorrow” reminders are another. After writing my fair share of such reminders, I got tired of going through post-it notes and decided to write a quick script that would allow me to automatically display myself a message the next time I logged in.

Thus LoginMSG was born. LoginMSG is a simple shell script (less than 50 lines long) that stores a text message and displays it. You can add a message, append to the existing message, remove the message, and view the existing message. When you view the message, a simple, no-frills xmessage box is displayed center-screen with your message, allowing you to read it and then remove it or keep it.

Download

• LoginMSG v.1 – Jan 6, 2009

Usage

LoginMSG v.1 – Display yourself a message (the next time X starts) usage: loginmsg [ options ] –add MSG : Create a message with contents MSG, or append MSG to the existing message contents. Message contents are stored in ~/.loginmsg. –show : Display the message. –remove : Remove any existing message.

Place the script somewhere in your $PATH. Add (or append) a message from a terminal using “loginmsg –add”. To display any existing messages when you login, have your desktop environment run “loginmsg –show” when you log in. LoginMSG does not have a feature to make itself automatically start when a user logs in — you will have to see to that manually. There are many ways to do this. For example, in Gnome you could add it to the “Session” list in System -> Preferences -> Session. In Openbox, you could add it to ~/.config/openbox/autostart.sh. Reference the help pages for your specific desktop environment.

ArchLinux AUR Contributions

Brad Conte — Sat, 27 Sep 2008 21:16:57 +0000

I’m a user of ArchLinux, a lightweight and flexible Linux distribution that tries to Keep It Simple.

There are thousands of packages officially maintained in Arch’s package repositories. However, to ensure the robustness and completeness of available packages, a supplementary user-administrated repository exists, where users can create and maintain their own packages. This is called the Arch User Repository. The AUR allows users to add lesser-known packages that are not widely used enough to warrant maintenance time from an official developer, create spin-off packages of packages in the official repositories, contribute their own scripts, etc.

I’ve made a few contributions to the AUR. To see them, I refer you to:
• My AUR profile
• My AUR List of Packages

Bash Path Hashing

Brad Conte — Fri, 05 Sep 2008 22:41:29 +0000

Bash is a Unix shell of many features. One of those features, which is both useful and potentially annoying, is the path hash table.

The environment variable we are probably most familiar with is PATH — it contains all the paths that should be searched when we specify an executable to run. Bash starts with the first path specified in PATH and looks to see if there is an executable present, if there is not it checks the second directory, and so on until it either finds an executable with the specified name or it runs out of paths to search.

The path of an executable practically never changes. “ls” is always in the same place, as is “cat”, and every other executable you might have. Thus, to save time, the first time Bash executes an executable it saves the path it finds the executable in, this way Bash can avoid searching for the executable again. The first time you execute “ls”, Bash has to search for it. The second through bajillionth times you execute it, Bash will just look up the full path from the table — much more efficient. It is important to note that every instance of bash maintains its own path hash table — thus if you have multiple terminals open they will not share path tables.

However, if the actual desired path of an executable changes, problems can arise. By default, Bash will not look for an executable again. If an executable changes paths, Bash will not find it. If an executable is duplicated to a new path, Bash will execute the original executable even if the new one is in a path of higher priority (namely, listed first in PATH). As can be imagined, this can cause quite a headache in these rare scenarios if one is unfamiliar with Bash path hashing. If you create a script, execute it, then edit it and copy it somewhere else (say from your personal “bin” to a global “bin”), the old version will still execute. If you simply move an executable between paths, bash will not find the new executable despite the fact that it seemingly should. Observe the following example, in which I create “myscript” in my home bin, run it, copy it to a global bin, run “myscript” again, and execute the one in my home bin again.

(This image can suffice stand-alone — feel free to redistribute it.)
Thankfully, however, the hash table is easily manipulated — by the builtin shell command “hash” no less. There are four switches that are likely of the most interest to you:

“-d [name]” will delete a specific entry from the table
“-r” will remove all entries from the table
“-t [name]” will list the save path for the given executable name
“-l” will list all existing executable -> path entries in the table

Using this tool you can view existing hashes and remove old stale ones as you see fit. To solve an out-of-date path hash problem, simple removing it will suffice. The next time the command is run the path hash will be re-updated. For more details, I refer you to the following key excerpts from the Bash man page:

If the name is neither a shell function nor a builtin, and contains no slashes, bash searches each element of the PATH for a directory con- taining an executable file by that name. Bash uses a hash table to remember the full pathnames of executable files (see hash under SHELL BUILTIN COMMANDS below). A full search of the directories in PATH is performed only if the command is not found in the hash table. If the search is unsuccessful, the shell prints an error message and returns an exit status of 127. [...] hash [-lr] [-p filename] [-dt] [name] For each name, the full file name of the command is determined by searching the directories in $PATH and remembered. If the -p option is supplied, no path search is performed, and filename is used as the full file name of the command. The -r option causes the shell to forget all remembered locations. The -d option causes the shell to forget the remembered location of each name. If the -t option is supplied, the full pathname to which each name corresponds is printed. If multiple name arguments are supplied with -t, the name is printed before the hashed full pathname. The -l option causes output to be displayed in a for- mat that may be reused as input. If no arguments are given, or if only -l is supplied, information about remembered commands is printed. The return status is true unless a name is not found or an invalid option is supplied.