Table of Contents

How do I recover from a spam attack?

So it has happened. One of your users account was hacked, and its abused to send spam via it. Since these guys don't care about your systems health, your system is probably under heavy load right now. This is what best to do:

Stop spam from being sent

Since you want to be a nice guy to others, and don't want to end being enlisted on some RBL DNS stopping the outbound mailstack is the first thing to do. This is only possible via restarting citserver with a special environment varable set:

export CITSERVER_n_session_max=0

you could add this to i.e. /etc/init.d/citadel and restart citserver.

Remember this, since you have to remove it after the situation is cleared and you want to resume normal operations.

Stop spam from comming in

Now you closed the outbound floodgates, its time to close the inbound one. Most probably the spammer is sending mails in via SMTP. Closing the SMTP ports (25 / 465 / 587) via your firewall is a good idea, but you also can close them using webcit - Administration/ Edit site-wide configuration / SMTP → and now set MTA/MSA/SMTP + SSL to -1; Save it and restart citserver again. If you run another mailer in front of citadel, its probably a good idea to stop this one too. If you've got a secondary mx running on another host, stop it too. Now you're offline; which is not as bad as you may think; Remote systems will try to reach you again later and you won't loose mail which is sent to you by others.

Analyze the Situation

This is probably a good time to tell your users that the mailsystem is unavailable, and you're working on the situation.
Now you need to find out whose account was hacked. therefore you need to inspect your outbound mailqueue.
As an aide user, Goto Administration / View the outbound SMTP queue/
The Jobs waiting for further processing: table shows you probably a huge list of mail jobs being processed. The Sender Column should show you who is the one that was hacked. Contact this person, and tell him he has to clear the situation on his system. Right now its probably a good idea to change his password.

Clearing the Situation

And now you're in the need for a way to delete a huge number of jobs from the Queue, right? Since webcit dynamicaly decides which representation to provide for a room, and the mailqueue is just another (hidden) room, you simply need to chose another representation by changing the view parameter in the URL in your browser from 11 to 1 and press enter. Now you've got the compfy mailbox view, you will get a better view of the situation. There are two sorts of messages in this room: those with regular subjects, and those with QMSG as subject. its always pairs of them; the QMSG with the queueing information, and the regular mail. We assume that NONE of the mails currently in the Queue will be sent. Select all of them and press <del> to move them into your personal trashbin. You can later check, whether there are mails from your users that they should send again. Please note that you should respect the privacy of your users.

Clearing the situation - without webcit

If your situation is that worse, that webcit can't help you anymore, we need to go a little more low-tech. Instead of webcit talking to citserver, we write a little shellscript utilizing netcat (Note: some distros call the Unix-Domain-Socket enabled netcat we need 'openbsd-netcat', test with nc -h whether 'U' is in the possible arguments).

SIDENOTE: this script will delete messages permanentely, you won't be able to look at them later.

You've probably already read about citadel scripting; more details can be found at the application protocol documentation; eventualy you may want to implement the MOVE command…

Please adjust the location of your unix domain socket according to your installation.

You need to run this script with write permissions to the admin-socket.

# get all message IDs from the spool folder: 
(printf 'GOTO __CitadelSMTPspoolout__\nMSGS\n'; sleep 1; ) |nc -U /var/run/citadel/citadel-admin.socket  |grep -v '^200 .*' |grep -v '^000$'  |grep -v '^100 *$'> /tmp/msgids
# (grep out status-codes etc.)
# check the file /tmp/msgids for validity, then remove this: 
# now we know all of them, chunksize items per command.
count=`wc -l < /tmp/msgids`
chunks=$(($((count/${chunksize})) + 1))
printf 'GOTO __CitadelSMTPspoolout__\n'
for i in `seq 1 $chunks`; do 
    messages=`head /tmp/msgids -n${start}|tail -n ${chunksize}`
    #printf "${messages}"
    for msgid in ${messages}; do 
	if test -n "${line}"; then 
    printf "DELE $line\n"
    start=$((${start} + ${chunksize}))
# after we sent all commands, give citserver a while to catch up,
# before we close the connection.
sleep 1000000
)|nc -U /var/run/citadel/citadel-admin.socket

getting back to normal business

Now that you

you need to