Table of Contents

Exploring the communication from your citadel server

There are various nice tools around, that enable you listening to whatever ip connection you like. This is called “sniffing”. You should be carefull about what you're doing and respect other peoples private sphere.


Tcpdump is the root of all tools described here. All of them use the libpcap, which was split out of tcpdump, most of them accept the same filterlanguage tcpdump accepts.

Ommitting Nameserver Lookup

A problem often experienced is, that packages won't appear though you're know they're on the wire. This may be caused by an unreachable nameserver, whichs awaiting queries hold the Package back from being printed. Avoid this using:

tcpdump -n

listening on a specific interface

Sometimes you might just be interested on the traffic on one interface, you just need to tell tcpdump so:

tcpdump -ni eth0

(you see, we can combine the -i with the -n from the last sample)

Create files to explore them later

tcpdump  -s0 -w /tmp/out.pcap

move out.pcap to the Box you're equipped with Wireshark, and Mouse it! This is also the suggested way to post to the support mailinglist. Please remember that you might post usernames and passwords here, so try to demonstrate your behaviour with a dummy user and password.

Reducing Packages later

tcpdump can read, filter, and reoutput packages from above .pcap files.

tcpdump -r infile.pcap -w outfile.pcap


you can filter the output of tcpdump using a neat filter language. For example you just want to see http traffic:

tcpdump host and port 80

Here is a list of Ports and which kind of traffic they cary. if you want to filter for mac-addresses you can do that by using 'host ether 00:….' explore more of this neat filter language on man tcpdump


ngrep is a unix commandline tool, that might show you running traffic better than wireshark or tcpdump, because of it focuses of printing the payload of the traffic. It allso uses the tcpdump backend library, and thus takes the same expressions as tcpdump:

ngrep port 80 and not host -W byline

the -W byline stops it from stripping linebreaks from the traffic. this will make it show you requests and stuff in a better readable manner. The output of ngrep is somewhat comparable to a tail -f.


Wireshark was formerly known as ethereal. It “understands” most of the protocols around in the internet today, and will display you structures in a tree view, or enable you just to view the tcp payload of a text oriented protocol as HTTP or SMTP. It runs under allmost any os with a GUI arround these days. If you want to analyze traffic spoken by a remote box see the above tcpdump howto capture streams for later analysis.

You might also have a look at this Wikipedia article on Wireshark.


Ethereape just displays you load graphs. But its neat to analyze which connections eat up most bandwith.


NTop can give you neat statistics about whats on the line. Searching for who has been using the most bandwith in the last 24 hours? This is your tool. Be warned, it eats truckloads of memory.

Sniffing several simultaneous Connections

If you want to watch webcits HTTP Traffic in conjunction with its Citadel Traffic like you were

tail -f 

ing several files you can use this Script:

# choose one of them, depending on your webcit; it will choose 2000 by default.
#export HTTP_PORT=80
export HTTP_PORT=2000
#export HTTP_PORT=8504 
# make shure you call webcit with 505 so it doesn't use Unix domain sockets
export CITADEL_PORT=504 
tcpdump -U -l -s0 -i lo -n -A port $CITADEL_PORT or port $HTTP_PORT | \
sed --unbuffered \
-ne "
    # parse out important fields
    s/^[0-9][0-9]:[0-9][0-9]:[0-9][0-9]\.[0-9][0-9]* IP \([^:> ]*\)\.\([^:> .]*\) > \([^:> ]*\)\.\([^:> .]*\): \(.\) [^ ]* \([^ ]*\)/\1=\2>\3=\4 \5 \6 /;
    # if it wasn't a new packet, skip to the end.
      # annote packets according to connect
      s/^\([^\n]*=$HTTP_PORT>[^=]*=\([^ ]*\) [^\n]*\n\)/\1\2 HTTP RESPONSE::: /;
      s/^\([^\n]*=\([^>]*\)>[^=]*=$HTTP_PORT [^\n]*\n\)/\1\2 HTTP REQUEST:::  /;
      s/^\([^\n]*=$CITADEL_PORT>[^=]*=\([^ ]*\) [^\n]*\n\)/\1\2 CITADEL RESPONSE::: /;
      s/^\([^\n]*=\([^>]*\)>[^=]*=$CITADEL_PORT [^\n]*\n\)/\1\2 CITADEL REQUEST:::  /;
      # spot new connections
      s/^\([^\n]* S [^\n]*\n\)/\1NEW /;
      s/^\([^\n]* S ack [^\n]*\n\)/\1ACCEPT /;
      s/^\([^\n]* F [^\n]*\n\)/\1CLOSE /;
      # save header
      # clear
      # read the next line, it's packet header
      # ignore line 1, read-in line 2, there always is one for tcp
      # erase 56 bytes header altogether
      # if there is nothing left, it was probably an ack packet, 
      # so we will skip it unless NEW or FIN
        / [FS] /bok;
      #get rid of parsed fields
      # attach rest of packet payloads