Do it with pkgsrc

Hello, I'm a humble netBSD user, and I like to discover new packages to do more and more things with my computer. Here I'll post how I do things with pkgsrc. Feel free to make suggestions and comments about it.

2006-06-15

Monitor your network activity with...

net/bmon: Bmon is an interface bandwidth monitor
net/slurm: Realtime traffic statistics
net/wmnet: Dockable network monitor
sysutils/pftop: Utility for real-time display of statistics for PF
wip/tcptrack: Watch TCP connections
sysutils/pfstat: Utility to render graphical statistics for PF

I like to always have an eye on my network activity. When I detect a strange behaviour, I also like to be able to track it down, to know who the hell is trying to hack my box! Most of the time it's actually just a cron job I forgot :)

Here are the tools I use to feed my madness:

bmon is a text based application, which gives you a lot of information about traffic on each of your network interfaces, including some kind of graph. Very useful when you're not at home.

slurm serves the same purpose than bmon, to chose one or another is a matter of taste. You can only track one interface at a time, but ergonomics may seem cleaner... (though I prefer bmon)

wmnet is a dockapp I use when I'm at home. I tried a lot of dockapps but this one is definitely the best, because you can use a logarithmic scale and thus see little traffic as well as bandwidth tsunamis. You can also customize the colours to adapt it to your desktop's style.

I use it like that :
wmnet --logscale --maxrate=10000000 -t grey -r green -w --device vr0 &


Now let's see the tools I use to have a deeper understanding of what's happening.

If like me you use pf as firewall, you may try
pftop. It gives you a very detailed view on every connections currently used on your computer. You can sort connection by address, port, rate, age, number of packets. Not an easy to read interface, but there's really a lot of information.

tcptrack is also very handy. The interface looks like top's one, and allows you to easily spot the most active connections. I prefer pftop but if you don't use pf (though I don't understand why, since it's so great), I guess tcptrack is a good tool.

pfstat, for pf users once again, allows you to generate graphs from your network activity. The man page explains pretty well how to set it up, but I will summarize it here.

  • Add the following line to your pf.conf (of course, replace vr0 by the name of the interface you wanna log) and restart pf.

set loginterface vr0

  • Add a cron task as root (crontab -e) to feed your log file

* * * * * /usr/pkg/bin/pfstat -q >>/var/log/pfstat

  • You should consider setting up another cron task to occasionally truncate the log file

1 1 * * 1 tail -n 50000 /var/log/pfstat >/tmp/pfstat && mv /tmp/pfstat /var/log/pfstat

  • Write a pfstat.conf file and save it in /etc for example. Here's the one I use, see man page to write your own :

image "/home2/stats/pfstat-week.png" {
from 1 weeks to now
width 960 height 300
left
graph bytes_v4_in label "incoming" color 0 192 0 filled,
graph bytes_v4_out label "outgoing" color 0 0 255
right
graph states_searches label "states searches" color 192 192 0
}
image "/home2/stats/pfstat-day.png" {
from 1 days to now
width 960 height 300
left
graph bytes_v4_in label "incoming" color 0 192 0 filled,
graph bytes_v4_out label "outgoing" color 0 0 255
right
graph states_entries label "states" color 255 0 0
}
image "/home2/stats/pfstat-hour.png" {
from 1 hours to now
width 960 height 300
left
graph bytes_v4_in label "incoming" color 0 192 0 filled,
graph bytes_v4_out label "outgoing" color 0 0 255
right
graph states_entries label "states" color 255 0 0
}

  • Now add a cron task to refresh your graphs every five minutes for example
*/5 * * * * /usr/pkg/bin/pfstat -c /etc/pfstat.conf -d /var/log/pfstat > /dev/null

You can do whatever you want with those graphs, but printing them on a web page hosted on your computer may be a good idea.

Here's the kind of graphs you can obtain:




Of course, I didn't mention netstat because it's not a pkgsrc package, but it's the most useful tool when it comes to monitor your network activity.

2006-06-09

Bypass your company firewall/proxy with...

net/corkscrew: Tool for tunneling SSH through HTTP proxies
net/tsocks: Transparent SOCKS proxying library
www/tinyproxy: Lightweight HTTP/SSL proxy

Most company restrict the internet access by forcing users to use an HTTP proxy. If that access isn't enough for you to endure the pain of a whole day of (pseudo) work, if you want to be able to browse any website, to listen to any streaming radio, to connect to your favorite poker online site or anything else, know that there IS a solution.

1) Host an openSSH server on your home computer, and make it listen on the port 443

If you're office computer is running windows

2) You should also install an HTTP proxy on your home computer, because you won't be able to use openssh and thus to benefit of the socks proxy feature (-D), tinyproxy is easy to configure and works very well.

pkgmanager install www/tinyproxy
cp /usr/pkg/share/examples/rc.d/tinyproxy /etc/rc.d/
echo 'tinyproxy=YES' >> /etc/rc.conf
cp /usr/pkg/share/examples/tinyproxy/tinyproxy.conf.default /usr/pkg/etc/tinyproxy/tinyproxy.conf
vi /usr/pkg/etc/tinyproxy/tinyproxy.conf

Edit the tinyproxy.conf file, make sure you find the following lines

Port 8888
Bind 127.0.0.1
ConnectPort 443

That way, your http proxy will listen only on localhost, on the port 8888, and will allow you to browse https sites.

3) You'll have to use putty. Go in the Connection/Proxy section and give the HTTP proxy details here. Go in the Connection/SSH/Tunnels section and use port forwarding to redirect local port 8888 to localhost:8888 (on the remote host, your home computer). You can redirect any port you need, for example, if you want to play Go on KGS, redirect local port 2379 to goserver.igoweb.org:2379 and tell your Go client (cgoban) to connect to localhost:2379.

You should also go in the Connection section and tell putty to send null packets every 10 seconds, to keep the connection alive. Once this is done, go in the Session section, enter the hostname of your home computer, port 443, SSH, and save the session for later use, then try to connect.

4) Once you are logged on your server, you may check that port redirection is OK. Launch cmd.exe and make netstat -an to check that you're listening on the port 8888. Launch your web browser and tell him to use an http proxy on localhost:8888, and you should be able to browse any web site, enjoy.

If your office computer is running an unix-like, let's say NetBSD ;)

2) Install corkscrew. It allows you to use ssh through an HTTP proxy.

pkgmanager install net/corkscrew

3) Write a file that you will name auth_proxy for example, and which should contain your credentials to connect on the HTTP proxy, if needed.

echo 'login:pass' >> ~/.ssh/auth_proxy

4) Write a file that you will name config_out for example (192.168.3.10 is the address of the proxy I have to use)

echo 'ProxyCommand /usr/pkg/bin/corkscrew 192.168.3.10 80 %h %p ~/.ssh/auth_proxy' >> ~/.ssh/config_out

5) Add an alias in your .bashrc

alias ssh_out='ssh -F ~/.ssh/config_out'

6) If you're not always on the same network, then create as many auth_proxy files, config_out files, and alias that you need.

7) Now just connect to your home computer with the -D option.

ssh_out -D 8080 yourcomputer.hostname.org -p 443

You now have a SOCKS4/5 proxy listening on your port 8080.

8) What you want is to redirect all your TCP resquests to this socks proxy. For that purpose, you'll use tsocks. It's a great tool.

pkgmanager install net/tsocks
echo 'local = 192.168.0.0/255.255.255.0' >> /usr/pkg/etc/tsocks.conf
echo 'server = 127.0.0.1' >>
/usr/pkg/etc/tsocks.conf
echo 'server_port = 8080' >>
/usr/pkg/etc/tsocks.conf

192.168.0.0/255.255.255.0 is my local network, use yours.

9) To activate tsocks, you have to export the LD_PRELOAD variable

export LD_PRELOAD=/usr/pkg/lib/libtsocks.so

Then, every application you will run through this shell will use the proxy, which means that everything will go through the ssh tunnel right to your home computer wich will redirect all that stuff to the right directions. Everything will be as if you were at home.

Be aware that suid binaries or binaries launched through sudo won't use tsocks, so consider logging into your root account to use them.

Masterize file deletion with...

sysutils/wipe: Secure data destruction
wip/neb-wipe: Secure disk partition eraser (NetBSD only)
sysutils/fatback: Recover deleted files from FAT filesystems

Some people are paranoid when it comes to the security and the privacy of their data, even of the erased ones, I guess that it is my case... Since erasing in a secure way is really easy with
wipe, there's no reason to ignore it.

wipe interface is the same as rm for the main flags :

-i interactive - prompt whether to remove each file explicitly checks file permissions
-f forces file wiping and suppresses permission warnings
-r or -R recursion - traverse subdirectories

So you can easily write an alias in your .bashrc file or whatever shell you use, to replace rm by wipe.

alias rm='wipe -i'


neb-wipe allows you to erase your whole netBSD partition, before you give your crappy disk back to your reseller or sell it to your worst enemy. Usage is pretty simple :

neb-wipe - Erase harddisk partitions in a very secure manner,
using the 35-pass Gutmann method
Version 1.0

Usage is: neb-wipe [ -r count ] device
where device is the desired partition, e.g. "sd0a"
If the -r option is given, no Gutmann patterns are used;
Instead, neb-wipe writes the given amount of random
patterns to the disk.


If you are in the opposite situation, you want to explore the disk your worst enemy sold you, and it happens this sucker was using fat32, then you can use
fatback, it is made to undelete files from FAT filesystems and it works pretty well.

Manage your packages with ...

wip/pkgmanager: Package manager for pkgsrc
devel/cpuflags: Determine compiler flags to best target current cpu

pkgmanager is the greatest tool I know to manage my package collection from pkgsrc. It would never leave your system in inconsistent state as pkg_chk could. Check the website to understand how it works, and give it a try. Here’s my method to handle package installation, removal and update :

Of course you first need a pkgsrc tree. If you don't already have one, you can grab it and extract it with :

wget -c -P /tmp ftp://ftp.netbsd.org/pub/pkgsrc/current/pkgsrc.tar.gz
tar xfz /tmp/pkgsrc.tar.gz -C /usr

You may also grab the last wip tree :

wget -c -P /tmp http://pkgsrc-wip.sourceforge.net/snapshots/pkgsrc-wip-\

$(date+%Y%m%d)-snapshot.tar.gz
tar xfz /tmp/pkgsrc-wip-$(date +%Y%m%d)-snapshot.tar.gz -C /usr/pkgsrc


Now, the first thing you have to do is to install pkgmanager

cd /usr/pkgsrc/wip/pkgmanager
make install clean clean-depends

Let's see how to handle the main tasks :

a) Build and Install a new package

Let's say you want to install pkgfind

pkgmanager install pkgtools/pkgfind

pkgtools/pkgfind is then written in your want-list, compiled and installed, with all dependencies needed of course

b) Remove a package

If you want to remove pkgfind, just type

pkgmanager uninstall pkgtools/pkgfind

pkgtools/pkgfind will be removed from your want-list, and uninstalled, as well as every unneeded packages (former dependencies) its uninstallation may produce.

c) Update your packages

You first need to update your pkgsrc tree.

cd /usr/pksrc/
cvs update
cd /usr/pkgsrc/wip
cvs update

If you don't like cvs and don't care about netBSD servers bandwidth, which is bad, then you may use this crappy script :

#!/usr/pkg/bin/bash

echo "Fetching the trees"

echo -n "--Fetching pkgsrc-wip-"
echo -n $(date +%Y%m%d)
echo -n "-snapshot.tar.gz : "
wget --quiet -c -P /tmp http://pkgsrc-wip.sourceforge.net/snapshots/pkgsrc-wip-$(date +%Y%m%d)-snapshot.tar.gz
if [ "$?" -ne 0 ]
then
echo "Can't fetch it, trying archive from yesterday"
echo -n "--Fetching pkgsrc-wip-"
echo -n $(date -r $(expr `date +%s` - 86400) +%Y%m%d)
echo -n "-snapshot.tar.gz : "
wget --quiet -c -P /tmp http://pkgsrc-wip.sourceforge.net/snapshots/pkgsrc-wip-$(date -r $(expr `date +%s` - 86400) +%Y%m%d)-snapshot.tar.gz
if [ "$?" -ne 0 ]
then
echo "Can't fetch it, exiting"
exit 1
fi
fi
echo "OK"

echo -n "--Fetching pkgsrc.tar.gz : "
wget --quiet -c -P /tmp ftp://ftp.netbsd.org/pub/pkgsrc/current/pkgsrc.tar.gz
if [ "$?" -ne 0 ]
then
echo "Can't fetch it, exiting"
exit 1
fi
wget --quiet -c -P /tmp ftp://ftp.netbsd.org/pub/pkgsrc/current/pkgsrc.tar.gz.MD5
wget --quiet -c -P /tmp ftp://ftp.netbsd.org/pub/pkgsrc/current/pkgsrc.tar.gz.SHA1
echo "OK"

echo -n "Removing the old tree : "
rm -rf /usr/pkgsrc/
echo "OK"

echo -n "Extracting the trees : "
tar xfz /tmp/pkgsrc.tar.gz -C /usr
if [ "$?" -ne 0 ]
then
echo "NOK"
echo "Problems during extraction, check your pkgsrc tree"
exit 1
fi
tar xfz /tmp/pkgsrc-wip-$(date +%Y%m%d)-snapshot.tar.gz -C /usr/pkgsrc
if [ "$?" -ne 0 ]
then
echo "NOK"
echo "Problems during extraction, check your wip tree"
exit 1
fi
echo " OK"

echo -n "Fetching vulnerability list : "
/usr/pkg/sbin/download-vulnerability-list >> /dev/null
if [ "$?" -ne 0 ]
then
echo "NOK"
else
echo "OK"
fi

echo
echo "All done"
echo
exit 0


Once your trees are up to date, just type

pkgmanager sync

That's it !

Now if you wish to compile your packages with the appropriate flags, I suggest you to use devel/cpuflags. You just have to install it and to add those lines at the beginning of your /etc/mk.conf

.ifdef BSD_PKG_MK
.sinclude "/usr/pkg/share/mk/cpuflags.mk"
.endif