Make me a Millionaire part II: Co-dependents

by John Fehr, QNX Software Systems Ltd.

They say your first million is the hardest to make. Using the QNX Neutrino makefile system is quite similar. Once you've figured out how to make your first makefile, the next one is a lot easier.

In part I, we learned how to turn our x86 application project into a multi-CPU project complete with packaging. But what if we want to distribute shared libraries (again for all the CPU's we can) and perhaps a development package as well? How do we set the appropriate dependencies in our applications that require those shared libs, so that the correct libraries will automatically be installed when that application's package is installed?

Todays victim:

Let's use the bzip2 distribution as our example. The bzip2 binary already comes with QNX Neutrino, but the library does not.

This is a good example, because it contains both a library (libbz2) and an application. (bzip2) Once we've downloaded it, we can extract and build it with:

 tar -zxvf bzip2-1.0.1.tar.gz
 cd bzip2-1.0.1
 make

We notice that a typical compile looks like:

gcc -Wall -Winline -O2 -fomit-frame-pointer -fno-strength-reduce
   -D_FILE_OFFSET_BITS=64 -c decompress.c

Let's remember those options for later.

Two projects, and I love em both the same.

The problem with using the QNX Neutrino makefile system in this case, is that we want to make two projects: the libbz2 library and the bzip2 application. With the QNX Neutrino makefile system, we usually have a single project.

The best solution is to separate them into different directories. Instead of moving the sourcecode around, we'll just create two subdirectories, one called lib, and the other called app, in which we'll create the appropriate Makefile's and common.mk files:

 $ mkdir app
 $ cd app
 $ addvariant -i OS
 $ addvariant nto arm o.le
 $ addvariant nto sh o.le
 $ addvariant nto mips o.le
 $ addvariant nto mips o.be
 $ addvariant nto ppc o.be
 $ addvariant nto x86 o
 $ cd ..
 $ mkdir lib
 $ cd lib
 $ addvariant -i OS
 $ addvariant nto arm so.le
 $ addvariant nto arm a.le
 $ addvariant nto sh so.le
 $ addvariant nto sh a.le
 $ addvariant nto mips so.le
 $ addvariant nto mips so.be
 $ addvariant nto mips a.le
 $ addvariant nto mips a.be
 $ addvariant nto ppc so.be
 $ addvariant nto ppc a.be
 $ addvariant nto x86 so
 $ addvariant nto x86 a

If we try to build either of these projects now, not much happens. This is because we haven't told the makefile system where our sourcecode files are!

Hope you have a library card.

Since we're already there, let's start with the lib. Its common.mk file already contains the default lines:

 ifndef QCONFIG
 QCONFIG=qconfig.mk
 endif
 include $(QCONFIG)

include $(MKFILES_ROOT)/qtargets.mk

Let's add a few more lines just before the qtargets.mk include. First, we'll add the compile options it used originally:

 CCFLAGS+=-Wall -Winline -O2 -fomit-frame-pointer -fno-strength-reduce
 -D_FILE_OFFSET_BITS=64

Next, let's tell it where to find the source files. The PRODUCT_ROOT directory is the parent directory of the PROJECT_ROOT directory, which is where our sourcecode is located. Let's use that to specify where our sourcecode is:

 EXTRA_SRCVPATH=$(PRODUCT_ROOT)

Since the parent directory also contains the sourcecode for the bzip2 app, and we only want the object files for the libbz library, lets weed out the object files we don't need:

 EXCLUDE_OBJS=bzip2recover.o bzip2.o dlltest.o spewG.o unzcrash.o

We're planning on packaging this up, so we should add some PINFO definitions to automate the packaging as much as possible. We should also specify where we're going to want the package to be installed (usr/lib in this case):

 define PINFO
 PINFO DESCRIPTION=bzip2 data compressions library
 endef
 INSTALLDIR=usr/lib

Finally, lets make sure the library has the correct name. By default, it'll use the directory name of the PROJECT_ROOT directory. Since we don't want to call our library 'lib', lets change it:

 NAME=bz2

Looks like we're all set to go! Let's just type 'make' at the terminal, and watch all of our libraries being built!

NOTE: You may notice that there are libbz2S.a libraries being built in the so directories. If you're curious, Darrin Fry has a wonderful article describing how you can use these to make customizable shared libs called 'Customizable Shared Libs' located at http://www.qnx.com/developers/articles/article_308_1.html. These libraries can also be used if you want to create other shared libraries that require code from this library.

I'll have that to go, please.

All of our libraries are built now, but how do we package them? It's actually quite similar to how we packaged our lha binary in the previous article. We start with a simple package.qpg file that contains the location of the libraries we want to package:

 <QPG:Generation>
  <QPG:Values>
   <QPG:Files>
    <QPG:Add file="nto/*/*/libbz2.a" pinfo="$"/>
    <QPG:Add file="nto/*/*/libbz2.so" pinfo="$"/>
   </QPG:Files>
  </QPG:Values>
 </QPG:Generation>

Now we can run packager like we did in the previous article with:

 $ packager -p

Use the defaults for all the questions except for the following:

 What type of package is this? Library
 What is the full name of your product? () libbz2
 What is your product identifier? () libbz2
 What is the short description of this product? () libbz2 libraries
 What is the version of this product? (1.0) 1.0.1
 What type of license applies? Custom License
 What is the package license URL? () ../LICENSE
 What topic best describes this product? System/Libraries and Extensions/C
Libs
 Enter comma-delimited keywords: () libbz2
 What display requirements does your software have? None

Woohoo! We now have a file called libbz2-1.0.1-public.qpr that's seems to be ready to ship! Unfortunately, we forgot to include the header file, so it's not much of a development release. We'll add it in a few minutes. Lets make a full package.qpg file, so we can automate the creation the package file again:

 $ packager -x -m libbz2-1.0.1-public.qpr package.qpg

Unfortunately, when the packager built the package, it stores the license file itself, not where the original file was located. This means we'll have to add it manually. We also need to change the QPG: Add file lines to match what we put in originally, and add the bzlib.h header file, which will automatically become part of a development package:

    <QPG:Add file="../bzlib.h" install="/usr/include/"/>
    <QPG:Add file="nto/*/*/libbz2.a" pinfo="$"/>
    <QPG:Add file="nto/*/*/libbz2.so" pinfo="$"/>
    <QPG:Add file="../LICENSE" install="LicenseUrl/" handling="repdata"/>

Lets change the 'QPG:Repository generate=' line to yes instead of no, so that we can build (and maintain) a single package. Now whenever we need to rebuild this package, we just run packager with:

 $ packager -p -u -m libbz2-1.0.1-public.qpr

Let's install our new package with:

 $ pkg-installer -u file://`pwd`/libbz2-1.0.1-public.qpr

(Our Host and Target lists should both be set to 'All' when we install.)

Once we've finished installing, we notice that there is a bzlib.h file in /usr/include, and libbz2.a and libbz2.so files in /x86/usr/lib, /armle/usr/lib, etc. It worked! This also sets it up nicely for getting our app compiled and linked!

How do you like them apples?

What about our application, bzip2? Lets change into the app directory we build before, and set up our common.mk file. This time, though, we exclude everything but the bzip2.o from our objects, and add a new line:

 LIBS+=bz2

I won't re-explain each line again, but here's our complete common.mk file:

 ifndef QCONFIG
 QCONFIG=qconfig.mk
 endif
 include $(QCONFIG)

 CCFLAGS+=-Wall -Winline -O2 -fomit-frame-pointer -fno-strength-reduce
    -D_FILE_OFFSET_BITS=64
 EXTRA_SRCVPATH=$(PRODUCT_ROOT)
 EXCLUDE_OBJS= blocksort.o bzip2recover.o bzlib.o compress.o crctable.o
     decompress.o dlltest.o huffman.o randtable.o spewG.o unzcrash.o
 LIBS+=bz2
 define PINFO
 PINFO DESCRIPTION=bzip2 file compressor/decompressor
 endef
 INSTALLDIR=usr/bin
 NAME=bzip2

 include $(MKFILES_ROOT)/qtargets.mk   

We can easily create our bzip2.use file by getting help on our previously created bzip2 executable:

 $ ../bzip2 --help 2> bzip2.use
   

Now we can build our binaries, and make sure they exist:

 $ make
 $ ls -l nto/*/*/bzip2   

They look great! Let's get to the packaging.

We'll start with a very simple package.qpg file:

<QPG:Generation> <QPG:Values> <QPG:Files> <QPG:Add file="nto/*/*/bzip2" pinfo="$"/> </QPG:Files> </QPG:Values> </QPG:Generation>

Let's build our package as before:

 $ packager -p

Lets use the default answers for everything except:

 What is the full name of your product? () bzip2
 What is your product identifier? () bzip2
 What is the short description of this product? () bzip2 file (de)compressor
 What is the version of this product? (1.0) 1.0.1
 What type of license applies? Custom License
 What is the package license URL? () ../LICENSE
 What topic best describes this product? System/File Utilities/Compression
 Enter comma-delimited keywords: () bzip2
 Force installation (O)ver, (U)nder, (N)one, or (S)how details? (N) o
 Does this package depend on any external packages [Y/N] ? (N) y
 Please enter a requirement note. () Requires the bz2 shared library.
 What is the path to your dependent file? () /usr/lib/libbz2.so.1
 What is the home repository of the dependent product? ()
   

Now we can build our complete package.qpg file with:

 $ packager -x -m bzip2-1.0.1-public.qpr package.qpg

Again, we change the Repository generate line in package.qpg to read yes instead of no. We also add the license file once more, since the package doesn't contain the location of the original license file:

 <QPG:Add file="nto/*/*/bzip2" pinfo="$"/>
 <QPG:Add file="../LICENSE" install="LicenseUrl/" handling="repdata"/>

Now if we change the sourcecode the our bzip2 application, we can simply run:

 $ packager -p -u -m bzip2-1.0.1-public.qpr

and this recreates the package. Let's try installing it with:

$ pkg-installer -u file://`pwd`/bzip2-1.0.1-public.qpr   

When you've finished installing, you should have bzip2 binaries in /x86/usr/bin, /armle/usr/bin, etc. Yay! By sure to put the binary package on your repository, along with your libraries!

Is codependency good?

We kind of cheated. We installed the library first, and then the binary. That doesn't really show that our dependencies work. Lets verify it by using package manager to remove both our bzip2 binaries and bz2 libraries, and trying to install only the bzip2 binaries. First we remove our packages:

Run pkg-installer
Click on 'View My Software'
Double click on 'User Repository'
Click on each libbz2 entry in System->Libraries and Extensions->C Libraries and then the Remove button.
Click on each bzip2 entry in System->File Utilities->Compression and then the Remove button.
Click on Apply.

Now we should have none of our packages installed. Let's try installing just our bzip2 binaries:

Run pkg-installer
Add your repository if its not already there.
Double click on your repository
Click on the box next to the bzip2 packages under System->File Utilities->Compression
Click on Install.

Ta-da! It automagically figured out that it needed to install the libbz packages as well! It doesn't install the development packages though, which are not needed to run the binaries. Oops. Did I say easier?

Ok, it wasn't really easier than the first time, but wasn't it worth it? Unfortunately, simply building the various CPU versions of our apps doesn't mean they'll work on those CPU's, but it's a big step in the right direction! Be sure to check out Jerry Chappell's article, http://www.qnx.com/developers/articles/article_883_1.html, on the packager, for tons more information on how to use this great tool!

Now get out there, and build multi-CPU packages for all your projects! :)