Do it with pkgsrc

Hello, I'm a humble netBSD user, and I like to discover new packages to do more and more things with my computer. Here I'll post how I do things with pkgsrc. Feel free to make suggestions and comments about it.

2006-09-27

Manage your packages in a sandbox with...

pkgtools/pkglint: Verifier for NetBSD packages and complete pkgsrc tree
pkgtools/pkgdepgraph: Visual representation of installed NetBSD packages
pkgtools/pkg_tarup: Generates binary package(s) from installed pkg(s)
wip/pkg_rmleaves: Interactive script for deinstalling 'leaf' packages
mk/bulk/mksandbox: Script used to build a sandbox

As you may know if you've read the previous posts of this blog, I used to trust wip/pkgmanager to handle all of my pkgsrc admistration tasks. Recently, I ran into a few problems that really disappoint me, and I decided to look for some other way to update and manage my packages.

What i wanted was a clean and secure way, with simple commands using scripts I could understand. I must give credit to WIntellect from BSDnexus, which provided me a clever way to make the upgrade. Clever, yes, but no secure enough ;)

My method use a sandbox to build all the packages that need to be upgraded, and once you've checked that everything is ok in the chrooted sandbox, you just create all the binaries you need and use them to upgrade your packages in the real environment. Let's see how to do that.

1) Update you pkgsrc trees (/usr/pkgsrc and /usr/pkgsrc/wip) in the way you want.

2) You may want to check and delete unwanted "leaf" packages. pkg_rmleaves is a curse based application that can help you to do that. Recompiling packages can be time consuming, and you don't want to waste time on packages you don't care about anymore, do you ?

3) You should know build your sandbox, once and for all. Doing that is pretty easy, you just have to use /usr/pkgsrc/mk/bulk/mksandbox. If you're using xorg instead of Xfree86, edit it to change this line (line n°115) "need_xsrc=yes" to "need_xsrc=no". Then you just have to launch the script to create your sandbox in the directory of tour choice :

root@YOURbox [root]# cd /usr/pkgsrc/mk/bulk/
root@YOURbox [bulk]# ./mksandbox /usr/sandbox/

That should do the trick. Type df. See how the script null mounted in read-only mode all the directories the chroot will need, and in read-write, all the directories he will work in. Here's the part of the script that give details about that :

/bin /bin ro
/sbin /sbin ro
/lib /lib ro
/libexec /libexec ro
/usr/X11R6 /usr/X11R6 ro
/usr/bin /usr/bin ro
/usr/games /usr/games ro
/usr/include /usr/include ro
/usr/lib /usr/lib ro
/usr/libdata /usr/libdata ro
/usr/libexec /usr/libexec ro
/usr/lkm /usr/lkm ro
/usr/share /usr/share ro
/usr/sbin /usr/sbin ro
/var/mail /var/mail ro
/usr/src /usr/src ro
/usr/pkgsrc /usr/pkgsrc rw
/usr/pkgsrc/packages /usr/pkgsrc/packages rw
/usr/pkg_distfiles /usr/pkg_distfiles rw

Notice that my distfiles are stored in /usr/pkg_distfiles, to avoid losing them when I decide to update the trees with my barbarian script (see pkgmanager's post).

The script that controls those null mounts is /usr/sandbox/sandbox. It is invoked when you first create you sandbox, but next time you'll have to use it yourself.

root@YOURbox [root] #/usr/sandbox/sandbox mount
to null mount the directories

root@YOURbox [root] #/usr/sandbox/sandbix umount
to unmount them


OK, now your sanbox is created. You'll have to add a few packages in it to begin with. Blame me if you want to, but I like to use bash, so this one will be part of what I need. You will also need pkgtools/pkg_tarup. So you have to chroot in your sandbox and install those.

root@YOURbox [root]# chroot /usr/sandbox /bin/sh
root@SANDbox [/]# cd /usr/pkgsrc/shells/bash && make install
root@SANDbox [bash]# cd /usr/pkgsrc/pkgtools/pkg_tarup && make install

Here you are, get out of the sanbox with Ctrl-D, or by typing exit.

root@SANDbox [pkg_tarup]# exit

4) Now, we are going to use WIntellect's method to build the needed packages :

root@YOURbox [root] # cd /usr/sandbox/tmp
root@YOURbox [tmp]# lintpkgsrc -i > out_of_date

That command will generate the list of outdated packages

root@YOURbox [tmp]# pkgdepgraph -D out_of_date > to_delete

That one will create the list of packages that need to be deleted before hands

root@YOURbox [tmp]# pkgdepgraph -R out_of_date > rebuild.sh

And finally, this one generate a script which will be used to build all the packages needed.

5) It's now time to chroot and to perform the build :

root@YOURbox [tmp]# chroot /usr/sandbox/ /usr/pkg/bin/bash
root@SANDbox [/]# cd /tmp
root@SANDbox [tmp]# pkg_delete -r `cat to_delete`
root@SANDbox [tmp]# sh rebuild.sh

If the building process encounters a problem on a package, he will stop. Then you can see which package caused the issue, and solve it with your bare hands, then fire rebuild.sh up again. This script won't rebuild packages already done, don't worry, it's just perfect ;)

Note that the process can be longer in this virgin chrooted environment, you may need to build a lot of packages that you already have in your real environment. But with time passing, your sandbox should be almost in sync with the real one and that won't be an issue anymore. I think it's a cheap price for the serenity provided by the fact that you don't have to worry about half broking your box if there's a problem during the build !

6) Once everything is done, you have to create the binary packages for later use in your real environment. To be sure, we will create every single packages installed in the sandbox :


root@SANDbox [tmp]# pkg_tarup `pkg_info | cut -d ' ' -f 1 -s`

This command will create all the binary packages in /tmp (or /usr/sandbox/tmp, as you're still chrooted). Once this is done, un-chroot yourself.


root@SANDbox [tmp]# exit

7) You're now in you real environment, and you don't want to make any mess. Trust me, there won't be any harm.


root@YOURbox [tmp]# cd /usr/sandbox/tmp
root@YOURbox [tmp]# pkg_add -uu `cut -d "'" -f 2 -s out_of_date`

This should install everything. You can check that your system is up to date by running lintpkgsrc -i and verifying that you don't see anymore mismatching package.

8) Time for housekeeping

root@YOURbox [tmp]# rm -rf /usr/sandbox/tmp/*
root@YOURbox [tmp]# /usr/sandbox/sandbox umount

You're done.

Let's sum up all the commands needed, considering that the sandbox is already created :

root@YOURbox [root]# cd /usr/sandbox/tmp
root@YOURbox [tmp]# lintpkgsrc -i > out_of_date
root@YOURbox [tmp]# pkgdepgraph -D out_of_date > to_delete
root@YOURbox [tmp]# pkgdepgraph -R out_of_date > rebuild.sh
root@YOURbox [tmp]# /usr/sandbox/sandbox mount
root@YOURbox [tmp]# chroot /usr/sandbox/ /usr/pkg/bin/bash
root@SANDbox [/]# cd /tmp
root@SANDbox [tmp]# pkg_delete -r `cat to_delete`
root@SANDbox [tmp]# sh rebuild.sh
root@SANDbox [tmp]# pkg_tarup `pkg_info | cut -d ' ' -f 1 -s`
root@SANDbox [tmp]# exit
root@YOURbox [tmp]# pkg_add -uu `cut -d "'" -f 2 -s out_of_date`
root@YOURbox [tmp]# rm -rf /usr/sandbox/tmp/*
root@YOURbox [tmp]# /usr/sandbox/sandbox umount

This, in my opinion, is the ultimate way to upgrade your packages :) I hope you like it too, don't hesitate to give me comments about it, if you think about ways to improve it.

10 Comments:

At 2:11 PM, Anonymous dynek said...

What do you mean with "no secure enough" ?

 
At 3:19 PM, Blogger wilouf said...

Secure may not be the right word, you know I'm quite weak in english :)

What I meant is that with the basic method, if a build fails at some point, you may have a half updated system, which is not something enjoyable.

When you use a sandbox, you system stay clean whatever happen during the build process.

Thanks for your comment

 
At 3:14 PM, Anonymous Anonymous said...

aaaaaaaaaahhh ben je vais essayer la prochaine fois alors (dynek).

 
At 3:16 PM, Anonymous Anonymous said...

Ah encore "root@SANDbox [tmp]# pkg_delete -r `cat to_delete`" ca supprime bien les packages ? Je vois pas en quoi ca change de le faire dans un sandbox la.. (dynek)

 
At 3:30 PM, Blogger wilouf said...

En fait, pour chaque package que rebuild.sh va reconstruire, il vérifie qu'il n'est pas déjà installé, sinon il passe au suivant. Il est donc indispensable de desinstaller tous les packages contenus dans to_delete, et ce même dans le sandbox, sinon tout ne va pas se mettre à jour.

 
At 3:26 AM, Anonymous maple said...

best article on updating pkgs yet! worked perfect.

may want to note that you should manually upgrade pkgtools/pkg_install first as it writes to directories below /usr/pkg which are mounted read-only via sandbox.

 
At 8:08 AM, Anonymous Anonymous said...

wilouf, ce que je voulais dire c'est que ça supprime les packages dans le monde réél. Comme ça le ferait en dehors d'un sandbox. Donc quel est l'avantage ?

 
At 8:58 AM, Blogger wilouf said...

Maple : Thanks for your comment, I truly appreciate !
Regarding pkg_install, I don't understand why I should upgrade it manually. /usr/pkg is just not mounted. /usr/sandbox/usr/pkg is independant of /usr/pkg. Plus, pkg_install is not used by rebuild.sh.

Dynek (je suppose que c'est toi) : Comme je l'ai dit à Maple, /usr/pkg et /usr/sandbox/usr/pkg sont différents, et le pkg_delete que je fais dans la sandbox affecte /usr/sandbox/usr/pkg uniquement. J'aurais pu faire un "pkg_delete -K /usr/sandbox/var/db/pkg" depuis YOURbox, ça aurait eu le même effet. Du coup, quand tu fais cette commande, tu ne touches pas à tes pacquets réels, c'est ça qui est bon justement ! Ca vient encore de me sauver la vie hier soir ;)

 
At 12:44 AM, Anonymous Anonymous said...

Author, respect 2 you. Bookmark! :)


replica louis vuitton handbags
paycheck cash advance

fax loan payday quick

loan personal quick small

approval loan personal quick

advance cash loan paycheck payday

advance loan online payday

lender loan online payday

emergency loan payday

home equity loan calculation

free credit report and credit score online

Good Bye!

 
At 11:32 AM, Blogger wilouf said...

To Maple : You were right about pkg_install, when i got time, i will update the article.

 

Post a Comment

<< Home