The Hidden Dependency Problem

Johnny C. Lam

jlam@pkgsrc.org

The NetBSD Project

Outline

• Problem description
• Brief description of two different approaches
  • FreeBSD Ports' WITH_*/WITHOUT_*
  • pkgsrc buildlink
• New and improved solution

The Problem

Typical command sequence:

• 10 Install OpenSSL package.
• 20 Build and install Mutt.
• 30 Remove OpenSSL package.

/usr/pkg/bin/mutt no longer works.

The Problem (cont.)

What went wrong?

• Mutt was built after OpenSSL was installed.
• /usr/pkg/bin/mutt is linked against libssl.so
• There is a missing dependency relationship between Mutt and OpenSSL.
• In pkgsrc, we say OpenSSL is a hidden dependency of Mutt.

The Problem (cont.)

Revised command sequence:

• 10 Install OpenSSL package.
• 20 Build and install Mutt.
• 25 Register fact that Mutt depends on OpenSSL.
• 30 Remove OpenSSL package. ERROR!

Removing OpenSSL fails because Mutt depends on OpenSSL.

The Problem (cont.)

Problem solved, right?

No.

• What if you didn't know OpenSSL was installed?
• What if you don't want the SSL/TLS features of Mutt?

How do you configure Mutt to not use OpenSSL?

The FreeBSD Ports Way

Teach packages to honor WITH_* and WITHOUT_* knobs.

• WITH_SSL causes various --with-ssl* options to be passed to configure script.
• WITHOUT_SSL causes --without-ssl to be passed to configure script.
• Red Hat Source RPMs do something similar -- user edits spec file to add or remove configure options.

The FreeBSD Ports Way (cont.)

Major configure script options are promoted into WITH_* and WITHOUT_* knobs.

Problems:

• Tedious to discover every configure script option.
• What if the Mutt configure script doesn't offer --without-ssl?
• Have you seen the way configure scripts try to detect Kerberos? Oh, the horror!

The pkgsrc way

Manipulate the environment so that the configure script only sees what we want it to see and nothing more.

• Doesn't matter if --without-ssl is available or not.
• Doesn't matter if OpenSSL is installed or not.

In either case, the Mutt configure script shouldn't see any of the OpenSSL files.

The pkgsrc way (cont.)

chroot is the obvious implementation.

• Install dependencies directly in the chroot environment.
• Build the package sources within the chroot.
• If it's not in the chroot, it can't be found.

Problems:

• chroot command is not portable across every platform.
• Requires root privileges to chroot.

The pkgsrc way (cont.)

The infamous buildlink:

1. Create buildlink directory, e.g. /tmp/buildlink.
2. Symlink all required libraries and headers into that directory, e.g., /tmp/buildlink/lib/libreadline.so, /tmp/buildlink/include/readline/readline.h.
3. Intercept calls to cc so that the compiler never looks in /usr/pkg, only in /tmp/buildlink.

This works for all packages that are compiled.

pkgsrc has been using buildlink for over 5 years now with good success.

buildlink

Benefits of buildlink over The FreeBSD Ports Way:

• Don't need to force configure scripts to not find libraries.
• Rarely need to patch configure scripts or Makefiles.

Simplifies package creation and maintenance.

buildlink (cont.)

Current implementation is mixture of make, shell, awk and sed code.

• /tmp/buildlink/bin/cc, /tmp/buildlink/bin/ld, /tmp/buildlink/bin/cpp shell scripts wrap calls to the actual toolchain.
• Generated by complex process per package build.
• Transform -I and -L options by replacing /usr/pkg with /tmp/buildliink.

buildlink (cont.)

Problems with current implementation:

• Quoting hell when mixing make/shell/awk/sed -- difficult to code correctly
• Slow -- many, many fork/exec
• Tightly bound to pkgsrc -- cannot be easily ported to another packaging system.

New wrapper design

• Goal is to put all the complexity into a standalone buildlink package.
  • /usr/pkg/bin/cc
  • /usr/pkg/bin/ld
  • /usr/pkg/bin/cpp
• Written in C for portability.
• Provides GCC frontend to backend compiler.

To integrate into other packaging systems, Force using cc, etc. in /usr/pkg/bin in place of ones in /usr/bin.

New wrapper design (cont.)

• buildlink wrappers should call gcc with -nostdlib and -nostdinc.
• Ensures that only libraries and headers in the buildlink directory are used.
• Useful on systems that have huge collections of libraries in /usr/lib.

Closing Thoughts

• buildlink technology is basically emulating a chroot by using wrapper trickery that's portable across platforms.
• The new wrapper design should be easy to use within other packaging systems, e.g. FreeBSD Ports.