Do it with pkgsrc

Manage your packages in a sandbox with...

2006-09-27T20:29:00.000+02:00

pkgtools/pkglint: Verifier for NetBSD packages and complete pkgsrc tree
pkgtools/pkgdepgraph: Visual representation of installed NetBSD packages
pkgtools/pkg_tarup: Generates binary package(s) from installed pkg(s)
wip/pkg_rmleaves: Interactive script for deinstalling 'leaf' packages
mk/bulk/mksandbox: Script used to build a sandbox

As you may know if you've read the previous posts of this blog, I used to trust wip/pkgmanager to handle all of my pkgsrc admistration tasks. Recently, I ran into a few problems that really disappoint me, and I decided to look for some other way to update and manage my packages.

What i wanted was a clean and secure way, with simple commands using scripts I could understand. I must give credit to WIntellect from BSDnexus, which provided me a clever way to make the upgrade. Clever, yes, but no secure enough ;)

My method use a sandbox to build all the packages that need to be upgraded, and once you've checked that everything is ok in the chrooted sandbox, you just create all the binaries you need and use them to upgrade your packages in the real environment. Let's see how to do that.

1) Update you pkgsrc trees (/usr/pkgsrc and /usr/pkgsrc/wip) in the way you want.

2) You may want to check and delete unwanted "leaf" packages. pkg_rmleaves is a curse based application that can help you to do that. Recompiling packages can be time consuming, and you don't want to waste time on packages you don't care about anymore, do you ?

3) You should know build your sandbox, once and for all. Doing that is pretty easy, you just have to use /usr/pkgsrc/mk/bulk/mksandbox. If you're using xorg instead of Xfree86, edit it to change this line (line n°115) "need_xsrc=yes" to "need_xsrc=no". Then you just have to launch the script to create your sandbox in the directory of tour choice :

root@YOURbox [root]# cd /usr/pkgsrc/mk/bulk/
root@YOURbox [bulk]# ./mksandbox /usr/sandbox/

That should do the trick. Type df. See how the script null mounted in read-only mode all the directories the chroot will need, and in read-write, all the directories he will work in. Here's the part of the script that give details about that :

/bin /bin ro
/sbin /sbin ro
/lib /lib ro
/libexec /libexec ro
/usr/X11R6 /usr/X11R6 ro
/usr/bin /usr/bin ro
/usr/games /usr/games ro
/usr/include /usr/include ro
/usr/lib /usr/lib ro
/usr/libdata /usr/libdata ro
/usr/libexec /usr/libexec ro
/usr/lkm /usr/lkm ro
/usr/share /usr/share ro
/usr/sbin /usr/sbin ro
/var/mail /var/mail ro
/usr/src /usr/src ro
/usr/pkgsrc /usr/pkgsrc rw
/usr/pkgsrc/packages /usr/pkgsrc/packages rw
/usr/pkg_distfiles /usr/pkg_distfiles rw

Notice that my distfiles are stored in /usr/pkg_distfiles, to avoid losing them when I decide to update the trees with my barbarian script (see pkgmanager's post).

The script that controls those null mounts is /usr/sandbox/sandbox. It is invoked when you first create you sandbox, but next time you'll have to use it yourself.

root@YOURbox [root] #/usr/sandbox/sandbox mount
to null mount the directories
root@YOURbox [root] #/usr/sandbox/sandbix umount
to unmount them

OK, now your sanbox is created. You'll have to add a few packages in it to begin with. Blame me if you want to, but I like to use bash, so this one will be part of what I need. You will also need pkgtools/pkg_tarup. So you have to chroot in your sandbox and install those.

root@YOURbox [root]# chroot /usr/sandbox /bin/sh
root@SANDbox [/]# cd /usr/pkgsrc/shells/bash && make install
root@SANDbox [bash]# cd /usr/pkgsrc/pkgtools/pkg_tarup && make install

Here you are, get out of the sanbox with Ctrl-D, or by typing exit.

root@SANDbox [pkg_tarup]# exit

4) Now, we are going to use WIntellect's method to build the needed packages :

root@YOURbox [root] # cd /usr/sandbox/tmp
root@YOURbox [tmp]# lintpkgsrc -i > out_of_date

That command will generate the list of outdated packages

root@YOURbox [tmp]# pkgdepgraph -D out_of_date > to_delete

That one will create the list of packages that need to be deleted before hands

root@YOURbox [tmp]# pkgdepgraph -R out_of_date > rebuild.sh

And finally, this one generate a script which will be used to build all the packages needed.

5) It's now time to chroot and to perform the build :

root@YOURbox [tmp]# chroot /usr/sandbox/ /usr/pkg/bin/bash
root@SANDbox [/]# cd /tmp
root@SANDbox [tmp]# pkg_delete -r `cat to_delete`
root@SANDbox [tmp]# sh rebuild.sh

If the building process encounters a problem on a package, he will stop. Then you can see which package caused the issue, and solve it with your bare hands, then fire rebuild.sh up again. This script won't rebuild packages already done, don't worry, it's just perfect ;)

Note that the process can be longer in this virgin chrooted environment, you may need to build a lot of packages that you already have in your real environment. But with time passing, your sandbox should be almost in sync with the real one and that won't be an issue anymore. I think it's a cheap price for the serenity provided by the fact that you don't have to worry about half broking your box if there's a problem during the build !

6) Once everything is done, you have to create the binary packages for later use in your real environment. To be sure, we will create every single packages installed in the sandbox :

root@SANDbox [tmp]# pkg_tarup `pkg_info | cut -d ' ' -f 1 -s`

This command will create all the binary packages in /tmp (or /usr/sandbox/tmp, as you're still chrooted). Once this is done, un-chroot yourself.

root@SANDbox [tmp]# exit

7) You're now in you real environment, and you don't want to make any mess. Trust me, there won't be any harm.

root@YOURbox [tmp]# cd /usr/sandbox/tmp
root@YOURbox [tmp]# pkg_add -uu `cut -d "'" -f 2 -s out_of_date`

This should install everything. You can check that your system is up to date by running lintpkgsrc -i and verifying that you don't see anymore mismatching package.

8) Time for housekeeping

root@YOURbox [tmp]# rm -rf /usr/sandbox/tmp/*
root@YOURbox [tmp]# /usr/sandbox/sandbox umount

You're done.

Let's sum up all the commands needed, considering that the sandbox is already created :

root@YOURbox [root]# cd /usr/sandbox/tmp
root@YOURbox [tmp]# lintpkgsrc -i > out_of_date
root@YOURbox [tmp]# pkgdepgraph -D out_of_date > to_delete
root@YOURbox [tmp]# pkgdepgraph -R out_of_date > rebuild.sh
root@YOURbox [tmp]# /usr/sandbox/sandbox mount
root@YOURbox [tmp]# chroot /usr/sandbox/ /usr/pkg/bin/bash
root@SANDbox [/]# cd /tmp
root@SANDbox [tmp]# pkg_delete -r `cat to_delete`
root@SANDbox [tmp]# sh rebuild.sh
root@SANDbox [tmp]# pkg_tarup `pkg_info | cut -d ' ' -f 1 -s`
root@SANDbox [tmp]# exit
root@YOURbox [tmp]# pkg_add -uu `cut -d "'" -f 2 -s out_of_date`
root@YOURbox [tmp]# rm -rf /usr/sandbox/tmp/*
root@YOURbox [tmp]# /usr/sandbox/sandbox umount

This, in my opinion, is the ultimate way to upgrade your packages :) I hope you like it too, don't hesitate to give me comments about it, if you think about ways to improve it.

Monitor your network activity with...

2006-06-15T13:59:00.000+02:00

net/bmon: Bmon is an interface bandwidth monitor
net/slurm: Realtime traffic statistics
net/wmnet: Dockable network monitor
sysutils/pftop: Utility for real-time display of statistics for PF
wip/tcptrack: Watch TCP connections
sysutils/pfstat: Utility to render graphical statistics for PF

I like to always have an eye on my network activity. When I detect a strange behaviour, I also like to be able to track it down, to know who the hell is trying to hack my box! Most of the time it's actually just a cron job I forgot :)

Here are the tools I use to feed my madness:

bmon is a text based application, which gives you a lot of information about traffic on each of your network interfaces, including some kind of graph. Very useful when you're not at home.

slurm serves the same purpose than bmon, to chose one or another is a matter of taste. You can only track one interface at a time, but ergonomics may seem cleaner... (though I prefer bmon)

wmnet is a dockapp I use when I'm at home. I tried a lot of dockapps but this one is definitely the best, because you can use a logarithmic scale and thus see little traffic as well as bandwidth tsunamis. You can also customize the colours to adapt it to your desktop's style.

I use it like that :
wmnet --logscale --maxrate=10000000 -t grey -r green -w --device vr0 &

Now let's see the tools I use to have a deeper understanding of what's happening.

If like me you use pf as firewall, you may try pftop. It gives you a very detailed view on every connections currently used on your computer. You can sort connection by address, port, rate, age, number of packets. Not an easy to read interface, but there's really a lot of information.

tcptrack is also very handy. The interface looks like top's one, and allows you to easily spot the most active connections. I prefer pftop but if you don't use pf (though I don't understand why, since it's so great), I guess tcptrack is a good tool.

pfstat, for pf users once again, allows you to generate graphs from your network activity. The man page explains pretty well how to set it up, but I will summarize it here.

Add the following line to your pf.conf (of course, replace vr0 by the name of the interface you wanna log) and restart pf.

set loginterface vr0

Add a cron task as root (crontab -e) to feed your log file

* * * * * /usr/pkg/bin/pfstat -q >>/var/log/pfstat

You should consider setting up another cron task to occasionally truncate the log file

1 1 * * 1 tail -n 50000 /var/log/pfstat >/tmp/pfstat && mv /tmp/pfstat /var/log/pfstat

Write a pfstat.conf file and save it in /etc for example. Here's the one I use, see man page to write your own :

image "/home2/stats/pfstat-week.png" {
from 1 weeks to now
width 960 height 300
left
graph bytes_v4_in label "incoming" color 0 192 0 filled,
graph bytes_v4_out label "outgoing" color 0 0 255
right
graph states_searches label "states searches" color 192 192 0
}
image "/home2/stats/pfstat-day.png" {
from 1 days to now
width 960 height 300
left
graph bytes_v4_in label "incoming" color 0 192 0 filled,
graph bytes_v4_out label "outgoing" color 0 0 255
right
graph states_entries label "states" color 255 0 0
}
image "/home2/stats/pfstat-hour.png" {
from 1 hours to now
width 960 height 300
left
graph bytes_v4_in label "incoming" color 0 192 0 filled,
graph bytes_v4_out label "outgoing" color 0 0 255
right
graph states_entries label "states" color 255 0 0
}

Now add a cron task to refresh your graphs every five minutes for example

*/5 * * * * /usr/pkg/bin/pfstat -c /etc/pfstat.conf -d /var/log/pfstat > /dev/null

You can do whatever you want with those graphs, but printing them on a web page hosted on your computer may be a good idea.

Here's the kind of graphs you can obtain:

Of course, I didn't mention netstat because it's not a pkgsrc package, but it's the most useful tool when it comes to monitor your network activity.

Bypass your company firewall/proxy with...

2006-06-09T14:41:00.000+02:00

net/corkscrew: Tool for tunneling SSH through HTTP proxies
net/tsocks: Transparent SOCKS proxying library
www/tinyproxy: Lightweight HTTP/SSL proxy

Most company restrict the internet access by forcing users to use an HTTP proxy. If that access isn't enough for you to endure the pain of a whole day of (pseudo) work, if you want to be able to browse any website, to listen to any streaming radio, to connect to your favorite poker online site or anything else, know that there IS a solution.

1) Host an openSSH server on your home computer, and make it listen on the port 443

If you're office computer is running windows

2) You should also install an HTTP proxy on your home computer, because you won't be able to use openssh and thus to benefit of the socks proxy feature (-D), tinyproxy is easy to configure and works very well.

pkgmanager install www/tinyproxy
cp /usr/pkg/share/examples/rc.d/tinyproxy /etc/rc.d/
echo 'tinyproxy=YES' >> /etc/rc.conf
cp /usr/pkg/share/examples/tinyproxy/tinyproxy.conf.default /usr/pkg/etc/tinyproxy/tinyproxy.conf
vi /usr/pkg/etc/tinyproxy/tinyproxy.conf

Edit the tinyproxy.conf file, make sure you find the following lines

Port 8888
Bind 127.0.0.1
ConnectPort 443

That way, your http proxy will listen only on localhost, on the port 8888, and will allow you to browse https sites.

3) You'll have to use putty. Go in the Connection/Proxy section and give the HTTP proxy details here. Go in the Connection/SSH/Tunnels section and use port forwarding to redirect local port 8888 to localhost:8888 (on the remote host, your home computer). You can redirect any port you need, for example, if you want to play Go on KGS, redirect local port 2379 to goserver.igoweb.org:2379 and tell your Go client (cgoban) to connect to localhost:2379.

You should also go in the Connection section and tell putty to send null packets every 10 seconds, to keep the connection alive. Once this is done, go in the Session section, enter the hostname of your home computer, port 443, SSH, and save the session for later use, then try to connect.

4) Once you are logged on your server, you may check that port redirection is OK. Launch cmd.exe and make netstat -an to check that you're listening on the port 8888. Launch your web browser and tell him to use an http proxy on localhost:8888, and you should be able to browse any web site, enjoy.

If your office computer is running an unix-like, let's say NetBSD ;)

2) Install corkscrew. It allows you to use ssh through an HTTP proxy.

pkgmanager install net/corkscrew

3) Write a file that you will name auth_proxy for example, and which should contain your credentials to connect on the HTTP proxy, if needed.

echo 'login:pass' >> ~/.ssh/auth_proxy

4) Write a file that you will name config_out for example (192.168.3.10 is the address of the proxy I have to use)

echo 'ProxyCommand /usr/pkg/bin/corkscrew 192.168.3.10 80 %h %p ~/.ssh/auth_proxy' >> ~/.ssh/config_out

5) Add an alias in your .bashrc

alias ssh_out='ssh -F ~/.ssh/config_out'

6) If you're not always on the same network, then create as many auth_proxy files, config_out files, and alias that you need.

7) Now just connect to your home computer with the -D option.

ssh_out -D 8080 yourcomputer.hostname.org -p 443

You now have a SOCKS4/5 proxy listening on your port 8080.

8) What you want is to redirect all your TCP resquests to this socks proxy. For that purpose, you'll use tsocks. It's a great tool.

pkgmanager install net/tsocks
echo 'local = 192.168.0.0/255.255.255.0' >> /usr/pkg/etc/tsocks.conf
echo 'server = 127.0.0.1' >> /usr/pkg/etc/tsocks.conf
echo 'server_port = 8080' >> /usr/pkg/etc/tsocks.conf

192.168.0.0/255.255.255.0 is my local network, use yours.

9) To activate tsocks, you have to export the LD_PRELOAD variable

export LD_PRELOAD=/usr/pkg/lib/libtsocks.so

Then, every application you will run through this shell will use the proxy, which means that everything will go through the ssh tunnel right to your home computer wich will redirect all that stuff to the right directions. Everything will be as if you were at home.

Be aware that suid binaries or binaries launched through sudo won't use tsocks, so consider logging into your root account to use them.

Masterize file deletion with...

2006-06-09T13:46:00.000+02:00

sysutils/wipe: Secure data destruction
wip/neb-wipe: Secure disk partition eraser (NetBSD only)
sysutils/fatback: Recover deleted files from FAT filesystems

Some people are paranoid when it comes to the security and the privacy of their data, even of the erased ones, I guess that it is my case... Since erasing in a secure way is really easy with wipe, there's no reason to ignore it.

wipe interface is the same as rm for the main flags :

-i interactive - prompt whether to remove each file explicitly checks file permissions
-f forces file wiping and suppresses permission warnings
-r or -R recursion - traverse subdirectories

So you can easily write an alias in your .bashrc file or whatever shell you use, to replace rm by wipe.

alias rm='wipe -i'

neb-wipe allows you to erase your whole netBSD partition, before you give your crappy disk back to your reseller or sell it to your worst enemy. Usage is pretty simple :

neb-wipe - Erase harddisk partitions in a very secure manner,
using the 35-pass Gutmann method
Version 1.0

Usage is: neb-wipe [ -r count ] device
where device is the desired partition, e.g. "sd0a"
If the -r option is given, no Gutmann patterns are used;
Instead, neb-wipe writes the given amount of random
patterns to the disk.

If you are in the opposite situation, you want to explore the disk your worst enemy sold you, and it happens this sucker was using fat32, then you can use fatback, it is made to undelete files from FAT filesystems and it works pretty well.

Manage your packages with ...

2006-06-09T11:15:00.000+02:00

wip/pkgmanager: Package manager for pkgsrc
devel/cpuflags: Determine compiler flags to best target current cpu

pkgmanager is the greatest tool I know to manage my package collection from pkgsrc. It would never leave your system in inconsistent state as pkg_chk could. Check the website to understand how it works, and give it a try. Here’s my method to handle package installation, removal and update :

Of course you first need a pkgsrc tree. If you don't already have one, you can grab it and extract it with :

wget -c -P /tmp ftp://ftp.netbsd.org/pub/pkgsrc/current/pkgsrc.tar.gz
tar xfz /tmp/pkgsrc.tar.gz -C /usr

You may also grab the last wip tree :

wget -c -P /tmp http://pkgsrc-wip.sourceforge.net/snapshots/pkgsrc-wip-\

$(date+%Y%m%d)-snapshot.tar.gz
tar xfz /tmp/pkgsrc-wip-$(date +%Y%m%d)-snapshot.tar.gz -C /usr/pkgsrc

Now, the first thing you have to do is to install pkgmanager

cd /usr/pkgsrc/wip/pkgmanager
make install clean clean-depends

Let's see how to handle the main tasks :

a) Build and Install a new package

Let's say you want to install pkgfind

pkgmanager install pkgtools/pkgfind

pkgtools/pkgfind is then written in your want-list, compiled and installed, with all dependencies needed of course

b) Remove a package

If you want to remove pkgfind, just type

pkgmanager uninstall pkgtools/pkgfind

pkgtools/pkgfind will be removed from your want-list, and uninstalled, as well as every unneeded packages (former dependencies) its uninstallation may produce.

c) Update your packages

You first need to update your pkgsrc tree.

cd /usr/pksrc/
cvs update
cd /usr/pkgsrc/wip
cvs update

If you don't like cvs and don't care about netBSD servers bandwidth, which is bad, then you may use this crappy script :

#!/usr/pkg/bin/bash

echo "Fetching the trees"

echo -n "--Fetching pkgsrc-wip-"
echo -n $(date +%Y%m%d)
echo -n "-snapshot.tar.gz : "
wget --quiet -c -P /tmp http://pkgsrc-wip.sourceforge.net/snapshots/pkgsrc-wip-$(date +%Y%m%d)-snapshot.tar.gz
if [ "$?" -ne 0 ]
then
echo "Can't fetch it, trying archive from yesterday"
echo -n "--Fetching pkgsrc-wip-"
echo -n $(date -r $(expr `date +%s` - 86400) +%Y%m%d)
echo -n "-snapshot.tar.gz : "
wget --quiet -c -P /tmp http://pkgsrc-wip.sourceforge.net/snapshots/pkgsrc-wip-$(date -r $(expr `date +%s` - 86400) +%Y%m%d)-snapshot.tar.gz
if [ "$?" -ne 0 ]
then
echo "Can't fetch it, exiting"
exit 1
fi
fi
echo "OK"

echo -n "--Fetching pkgsrc.tar.gz : "
wget --quiet -c -P /tmp ftp://ftp.netbsd.org/pub/pkgsrc/current/pkgsrc.tar.gz
if [ "$?" -ne 0 ]
then
echo "Can't fetch it, exiting"
exit 1
fi
wget --quiet -c -P /tmp ftp://ftp.netbsd.org/pub/pkgsrc/current/pkgsrc.tar.gz.MD5
wget --quiet -c -P /tmp ftp://ftp.netbsd.org/pub/pkgsrc/current/pkgsrc.tar.gz.SHA1
echo "OK"

echo -n "Removing the old tree : "
rm -rf /usr/pkgsrc/
echo "OK"

echo -n "Extracting the trees : "
tar xfz /tmp/pkgsrc.tar.gz -C /usr
if [ "$?" -ne 0 ]
then
echo "NOK"
echo "Problems during extraction, check your pkgsrc tree"
exit 1
fi
tar xfz /tmp/pkgsrc-wip-$(date +%Y%m%d)-snapshot.tar.gz -C /usr/pkgsrc
if [ "$?" -ne 0 ]
then
echo "NOK"
echo "Problems during extraction, check your wip tree"
exit 1
fi
echo " OK"

echo -n "Fetching vulnerability list : "
/usr/pkg/sbin/download-vulnerability-list >> /dev/null
if [ "$?" -ne 0 ]
then
echo "NOK"
else
echo "OK"
fi

echo
echo "All done"
echo
exit 0

Once your trees are up to date, just type

pkgmanager sync

That's it !

Now if you wish to compile your packages with the appropriate flags, I suggest you to use devel/cpuflags. You just have to install it and to add those lines at the beginning of your /etc/mk.conf

.ifdef BSD_PKG_MK
.sinclude "/usr/pkg/share/mk/cpuflags.mk"
.endif