Working With Open Source Code

This page describes just a little bit about how to download, compile, and run open-source code that you can find on the Internet.

A good example to look at is the Vim source code, which is the editor we use in class. Point your browser at the Vim website: http://www.vim.org and use the guidelines below to find the code.

Getting The Source

Finding Code on the Internet

If you're looking for the source for a particular program or just looking for a category or an application that will solve a task, there are several places you can look:

Source Tarballs

The first thing you need to do is find the code. This will typically come in a source tarball. A "tarball" is a file that has been archived using the tar utility and then compressed with a compresion program, usually gzip or bzip2.

Look for a link that says "Download" or "Latest Version" or some such. This will probably take you to a web page or FTP directory that contains a listing of various versions of the program.

A note on stable / unstable branches

It is very common for an open-source project to have two branches: a stable (or "release) branch, and an unstable (or "development") branch. Typically, the stable one is the one you want to use for production. If you want to live on the bleeding edge or see how the latest features are being implemented, you might want to try the unstable branch.

Downloading the Tarball to Gautama

To download the code, you cannot just click on the link and save it to the lab workstations, that will save it to the local hard drive. You want to save it on gautama in your home directory, or preferably a subdirectory (called "src" or "dls" or "tmp" or some such). To save the tarball to gautama, right click on the link and select "Copy Link Address" or "Copy Link Location" (or whatever your browser happens to call it).

Then, in a putty window, type wget at the command line, followed by a space, and then paste in the link that you copied from the browser. The link should start with http:// or ftp:// and end with a file that looks like filename.src.tar.gz. (Every tarball will end with the .tar.gz extension.) When you're sure that the link looks correct, hit enter and wget will download the file.

Extracting The Source

Viewing The Tarball's Contents

Before you extract it, you want to look at the contents of the tarball. You can do this by typing:

tar tzvf filename.src.tar.gz
or (depending on the extension)
tar tjvf filename.src.tar.bz2
("man tar").

The reason why we look at it first is to find out whether this tarball will create its own subdirectory where the source code will live. A well-behaved tarball will do this, but not all tarballs are well-behaved. If the tarball doesn't make its own subdirectory, you should make one by hand.

Un-Tarring the Tarball

Having verified that the tarball looks good, you can type:

tar xzvf filename.src.tar.gz
or (depending on the extension)
tar xjvf filename.src.tar.bz2
which will uncompress and extract the files in the tarball, creating directories and subdirectories as needed.

Now you can cd into the directory and look around.

Compiling The Code

Before You Compile

Most source tarballs will include a file named README or INSTALL. You should probably look at those first just to verify if there are any details that you need to be aware of.

As a general rule, any filename in all caps should be read first (i.e. BUILDING).

Building Conventions Used on Unix

There is a semi-well-observed convention for compiling source code on Unix / Linux systems, You need to type just a few commands:

  1. ./configure
  2. make
  3. make install (if you plan on installing it -- we won't do that in class)

We'll examine each of these steps in greater detail.

Configuring with ./configure

Most well-behaved source code tarballs use a system called "autoconf" to automatically determine what libraries and features are available on a given system. Running the ./configure script probes this system for information and generates a Makefile. This is the makefile that will be used by the "make" program in step #2.

As mentioned earlier, not all source tarballs are well-behaved. Some of them will make you edit files by hand, enabling or disabling options to your taste. Typically the README file will tell you how to do this.

Something that can go wrong at this point is that if you don't have the required libraries on your system, the configure script will choke and tell you that you need to install some other library before you can install this program. (This is why package management programs were invented.)

Building An Executable With make

After ./configure runs sucessfully, you should be able to type make to build the program. This is just like using make in the labs for class.

Sometimes typing make without any arguments will print some lines that list the vaild targets.

As before, make may generate an error if you don't have a required library or build tool. You then have to track down everything necessary before you can build it.

Running The Program

Typically, running make will produce an executable in the same directory as the README, configure, and Makefile files. If it's not there, you can probably find it in a directory called src. To execute it, type:

./progname
Note that you need to use the ./ in front, otherwise you will probably execute the one installed on the system.

Installing the Program

Some programs will not run until they've been installed onto the system somewhere. The system administrator will typically install programs built from source into the /usr/local directory. Since you're not the system administrator, in order to install and use the program, you will need to rebuild it with make, specifying a prefix where it should install. You do this by typing:

make PREFIX=$HOME install
which will install the program into your home directory, making subdirectories (like "bin", "lib", and "etc") as needed.

Note that you do not have to install the program right into your home directory; you can set the PREFIX variable to any path on the system that you have write access to. (Like $HOME/tmp for example.)

Most programs do not need to be installed in order to run, which is good news.

Navigating Through the Source Code

grep

If you're looking for something particular in the source code, grep is your friend. Grep will find strings (actually, regular expressions) in files (or the output of other commands). Some handy flags to grep are "r" (recurse subdirectorys), "i" (case insensetive matching), "s" (supress error messages), and "w" (match the complete word; do not match words that include the search string as a substring). ("man grep")

ctags

The ctags program can be used to help you navigate source code when using vim. In the base directory type:

ctags -R .
This will generate a file called "tags" in the current directory. You can then type:
vi -t main
to go to the main() function (where the program starts). As you are navigating through the code, when you want to jump to the definition of a function you can type CNTRL-] to jump to the funciton. CNTRL-t will take you back to where you were before.

In Vim you can type :help tags for more information on navigating with the tags file.