Do it with pkgsrc

Hello, I'm a humble netBSD user, and I like to discover new packages to do more and more things with my computer. Here I'll post how I do things with pkgsrc. Feel free to make suggestions and comments about it.

2006-09-27

Manage your packages in a sandbox with...

pkgtools/pkglint: Verifier for NetBSD packages and complete pkgsrc tree
pkgtools/pkgdepgraph: Visual representation of installed NetBSD packages
pkgtools/pkg_tarup: Generates binary package(s) from installed pkg(s)
wip/pkg_rmleaves: Interactive script for deinstalling 'leaf' packages
mk/bulk/mksandbox: Script used to build a sandbox

As you may know if you've read the previous posts of this blog, I used to trust wip/pkgmanager to handle all of my pkgsrc admistration tasks. Recently, I ran into a few problems that really disappoint me, and I decided to look for some other way to update and manage my packages.

What i wanted was a clean and secure way, with simple commands using scripts I could understand. I must give credit to WIntellect from BSDnexus, which provided me a clever way to make the upgrade. Clever, yes, but no secure enough ;)

My method use a sandbox to build all the packages that need to be upgraded, and once you've checked that everything is ok in the chrooted sandbox, you just create all the binaries you need and use them to upgrade your packages in the real environment. Let's see how to do that.

1) Update you pkgsrc trees (/usr/pkgsrc and /usr/pkgsrc/wip) in the way you want.

2) You may want to check and delete unwanted "leaf" packages. pkg_rmleaves is a curse based application that can help you to do that. Recompiling packages can be time consuming, and you don't want to waste time on packages you don't care about anymore, do you ?

3) You should know build your sandbox, once and for all. Doing that is pretty easy, you just have to use /usr/pkgsrc/mk/bulk/mksandbox. If you're using xorg instead of Xfree86, edit it to change this line (line n°115) "need_xsrc=yes" to "need_xsrc=no". Then you just have to launch the script to create your sandbox in the directory of tour choice :

root@YOURbox [root]# cd /usr/pkgsrc/mk/bulk/
root@YOURbox [bulk]# ./mksandbox /usr/sandbox/

That should do the trick. Type df. See how the script null mounted in read-only mode all the directories the chroot will need, and in read-write, all the directories he will work in. Here's the part of the script that give details about that :

/bin /bin ro
/sbin /sbin ro
/lib /lib ro
/libexec /libexec ro
/usr/X11R6 /usr/X11R6 ro
/usr/bin /usr/bin ro
/usr/games /usr/games ro
/usr/include /usr/include ro
/usr/lib /usr/lib ro
/usr/libdata /usr/libdata ro
/usr/libexec /usr/libexec ro
/usr/lkm /usr/lkm ro
/usr/share /usr/share ro
/usr/sbin /usr/sbin ro
/var/mail /var/mail ro
/usr/src /usr/src ro
/usr/pkgsrc /usr/pkgsrc rw
/usr/pkgsrc/packages /usr/pkgsrc/packages rw
/usr/pkg_distfiles /usr/pkg_distfiles rw

Notice that my distfiles are stored in /usr/pkg_distfiles, to avoid losing them when I decide to update the trees with my barbarian script (see pkgmanager's post).

The script that controls those null mounts is /usr/sandbox/sandbox. It is invoked when you first create you sandbox, but next time you'll have to use it yourself.

root@YOURbox [root] #/usr/sandbox/sandbox mount
to null mount the directories

root@YOURbox [root] #/usr/sandbox/sandbix umount
to unmount them


OK, now your sanbox is created. You'll have to add a few packages in it to begin with. Blame me if you want to, but I like to use bash, so this one will be part of what I need. You will also need pkgtools/pkg_tarup. So you have to chroot in your sandbox and install those.

root@YOURbox [root]# chroot /usr/sandbox /bin/sh
root@SANDbox [/]# cd /usr/pkgsrc/shells/bash && make install
root@SANDbox [bash]# cd /usr/pkgsrc/pkgtools/pkg_tarup && make install

Here you are, get out of the sanbox with Ctrl-D, or by typing exit.

root@SANDbox [pkg_tarup]# exit

4) Now, we are going to use WIntellect's method to build the needed packages :

root@YOURbox [root] # cd /usr/sandbox/tmp
root@YOURbox [tmp]# lintpkgsrc -i > out_of_date

That command will generate the list of outdated packages

root@YOURbox [tmp]# pkgdepgraph -D out_of_date > to_delete

That one will create the list of packages that need to be deleted before hands

root@YOURbox [tmp]# pkgdepgraph -R out_of_date > rebuild.sh

And finally, this one generate a script which will be used to build all the packages needed.

5) It's now time to chroot and to perform the build :

root@YOURbox [tmp]# chroot /usr/sandbox/ /usr/pkg/bin/bash
root@SANDbox [/]# cd /tmp
root@SANDbox [tmp]# pkg_delete -r `cat to_delete`
root@SANDbox [tmp]# sh rebuild.sh

If the building process encounters a problem on a package, he will stop. Then you can see which package caused the issue, and solve it with your bare hands, then fire rebuild.sh up again. This script won't rebuild packages already done, don't worry, it's just perfect ;)

Note that the process can be longer in this virgin chrooted environment, you may need to build a lot of packages that you already have in your real environment. But with time passing, your sandbox should be almost in sync with the real one and that won't be an issue anymore. I think it's a cheap price for the serenity provided by the fact that you don't have to worry about half broking your box if there's a problem during the build !

6) Once everything is done, you have to create the binary packages for later use in your real environment. To be sure, we will create every single packages installed in the sandbox :


root@SANDbox [tmp]# pkg_tarup `pkg_info | cut -d ' ' -f 1 -s`

This command will create all the binary packages in /tmp (or /usr/sandbox/tmp, as you're still chrooted). Once this is done, un-chroot yourself.


root@SANDbox [tmp]# exit

7) You're now in you real environment, and you don't want to make any mess. Trust me, there won't be any harm.


root@YOURbox [tmp]# cd /usr/sandbox/tmp
root@YOURbox [tmp]# pkg_add -uu `cut -d "'" -f 2 -s out_of_date`

This should install everything. You can check that your system is up to date by running lintpkgsrc -i and verifying that you don't see anymore mismatching package.

8) Time for housekeeping

root@YOURbox [tmp]# rm -rf /usr/sandbox/tmp/*
root@YOURbox [tmp]# /usr/sandbox/sandbox umount

You're done.

Let's sum up all the commands needed, considering that the sandbox is already created :

root@YOURbox [root]# cd /usr/sandbox/tmp
root@YOURbox [tmp]# lintpkgsrc -i > out_of_date
root@YOURbox [tmp]# pkgdepgraph -D out_of_date > to_delete
root@YOURbox [tmp]# pkgdepgraph -R out_of_date > rebuild.sh
root@YOURbox [tmp]# /usr/sandbox/sandbox mount
root@YOURbox [tmp]# chroot /usr/sandbox/ /usr/pkg/bin/bash
root@SANDbox [/]# cd /tmp
root@SANDbox [tmp]# pkg_delete -r `cat to_delete`
root@SANDbox [tmp]# sh rebuild.sh
root@SANDbox [tmp]# pkg_tarup `pkg_info | cut -d ' ' -f 1 -s`
root@SANDbox [tmp]# exit
root@YOURbox [tmp]# pkg_add -uu `cut -d "'" -f 2 -s out_of_date`
root@YOURbox [tmp]# rm -rf /usr/sandbox/tmp/*
root@YOURbox [tmp]# /usr/sandbox/sandbox umount

This, in my opinion, is the ultimate way to upgrade your packages :) I hope you like it too, don't hesitate to give me comments about it, if you think about ways to improve it.