Log files, which are of great of importance for troubleshooting, are typically found in /var/log. Some very common log files and their content (apart from log files or directories created by some applictions like /var/log/apache2 are:

• /var/log/auth.log authorization log: use of sudo, ssh,…
• /var/log/daemon.log daemon log
• /var/log/debug.log syslog messages with priotity DEBUG (& Ubuntu debugging messages)
• /var/log/kern.log kernel log
• /var/log/messages.log syslog messages with priority INFO
• /var/log/syslog most syslog messages

The kernel ring buffer is important, because it allows logging even in the startup phase when it is not yet possible to log to regular files in /var/log. It can be read directly with the command dmesg. Ubuntu also writes these messages to a log file if possible:

• /var/log/dmesg logs from kernel ring buffer

More Ubuntu specific info.

The original syslogd is nowadays (2014) mostly replaced by rsyslogd (e.g. Debian) and syslog-ng (e.g. SuSe).

The original config file for logging rules was /etc/syslog.conf. For rsyslogd it is /etc/rsyslog.conf, in Ubuntu configuration is further split into several files: logging rules are moved to /etc/rsyslog.d/.

Original syslogd-style selectors (are still supported in rsyslogd) and consist of a facility (e.g. auth, cron, syslog, user1,..) and priority (e.g. debug, warning, err, alert,..), which together form a selector, and an action, e.g. logfile.

The selector is written in the form 'facility.priority', where either can be replaced by a star (*) - meaning all. The meaning of a selector is, that all logs from the facility with the given priority or higher will be logged. To select exactly one priority you can add an equals sign as such: 'facility.=priority'.

The action (or target) can be a regular file, named pipes, terminals or consoles, remote machines,.. Prepending the target with a minus (-) will make it buffered.

# print kernel warnings (and worse) on a konsole
kern.warn               /dev/tty10

# print kernel warnings (and only warnings) on a konsole
kern.=warn               /dev/tty10

# log warnings (or worse) of several facilities
local0,local1.warn      /var/log/localwarnings.log
local0,local1.warn      -/var/log/bufferedlocalwarnings.log

# log everything but auth & authpriv to syslog file
*.*;auth,authpriv.none  -/var/log/syslog

# print emergency messages on the terminals of all users
*.emerg                 :omusrmsg:*

Note, that if syslog should write to terminals or consoles it must have root privileges. Therefore no privilege drops must be activated in the config file, e.g. $PrivDropToUser syslog (which is default in Ubuntu).

See also rsyslog documentation on filters.

klogd is similar to syslogd but only logs messages from the kernel and can do so in more detail. See this page for more info (German!)

logger can be used to conveniently log with syslogd, rsyslogd,.. e.g. in bash scripts. In addition to syslog it can also log to stderr, files and more.

logger "message"                        # log to syslog (user.noticy by default)
logger -p local0.warn "message"         # explicitly specify facility
logger -s "message"                     # also log to stderr
logger -f /tmp/extralog "message"       # also log to file (which must exist!)

logrotate allows automatic rotation, compression, removal, and mailing of traditional log files. Each log file may be handled daily, weekly, monthly, or when it grows too large. It is started regularly (typically daily) via a cron job and should help to avoid running out of disk space due to old log files.

Which logs should be rotated is configured in /etc/logratate.conf (or separate files in /etc/logrotate.d)

An example taken from /etc/logrotate.d/rsyslog:

/var/log/debug
/var/log/messages
{
        rotate 4
        weekly
        missingok
        notifempty
        compress
        delaycompress
        sharedscripts
        postrotate
                reload rsyslog >/dev/null 2>&1 || true
        endscript
}

syslogd is part of systemd and changes the way logging works drastically. syslogd collects logs from the early boot process, the kernel, the initrd, and even application standard error and out. Configuration is done in /etc/systemd/journald.conf.

All logs are stored in binary format (typically below /var/log/journal) and can be viewed in syslog-style or exported in other formats with journalctl:

journalctl --utc              # view logs with UTC timestamps
journalctl -k                 # view kernel messages (similar to dmesg)
journalctl --since "2015-01-10" --until "2015-01-11 03:00"
journalctl -b -o json-pretty  # get log since last boot in JSON format
journalctl -f                 # print logs and follow (like tail -f)

Check out this tutorial for more.

Via procfs exhaustive information about each process is available in /proc/<process_number>. For must use cases these programs are commonly used:

ps extracts relevant information about current processes. It is more useful in scripts than the interactive programs top or htop, which provide about the same information.

ps                     # print all processes with same effective current user in current terminal
ps axu                 # print all processes (a=for all users, x=also if no tty, u=user format)
ps -l ax               # print all processes in long format (with e.g. nice value)
ps -u user -U user u   # print all processes for user in user format
ps axu --sort %mem     # print all proceses sorted by memory consumptin (cpu: %cpu)

Process trees give a good overview of parent-child relationships:

ps axjf                # print process tree
pstree                 # print even nicer process tree

This is a good replacement for using first ps and then piping its output into grep.

pgrep java             # prints all ids of processes with java in its name
pgrep -f java -u user  # prints all ids of processes with java in its full command line and belong to user

Bash normaly executes commands in the foreground. Commands ending with an ampersand (&) are started in the background. A currently executed foreground-process can be stopped (SIGSTOP) by pressing CTRL-z.

Apart from that the following bash builtins can be used for execution control:

jobs                   # list jobs (and their numbers)
bg                     # put last process into background (highest nr in 'jobs')
bg <nr>                # put process into background (same as start with &)
fg                     # put last process into foreground (highest nr in 'jobs')
fg <nr>                # put process into foreground
kill -s TERM %<nr>     # send signal to job (-s 15 and -s TERM are equivalent)

In addition to the bash builtins kill is also a program (see type -a kill). It can only work with process ids and not job ids. Actually kill can send any signal to a process. The most used signals are

• SIGTERM (15): politely ask a process to terminate gracefully and clean up all resources (default for kill, killall)
• SIGINT (2): ask a process to terminage (ungracefully) - often used for programs running from the terminal (CTRL-c)
• SIGQUIT (3): same as SIGQUIT, but also writes a core dump on some systems (CTRL-\)
• SIGKILL (9): force-kill unresponsive processes - only use if less drastic signals had no effect
• SIGSTOP (17,19,23): stop (pause) a process (CTRL-z)
• SIGHUP (1): tell a program to re-read its configuration files

In cases where the job or process id is unknown or multiple instances of the same program must be killed (or supplied with a signal) these two (more or less equivalent) programs can be used:

killall -s KILL command  # kill all processes of command (exact name required, or use -r to interpret as regex)
pgrep -l under           # list all processes whose *program* contains the string under, e.g. thunderbird
pkill under              # send all matching processes SIGTERM

nohup starts commands and protects them from signals sent when the shell it was created within is terminated (e.g. when the remote session on a server is closed when you log out). Output is automatically redirected to nohup.out. In combination with the ampersand to execute a command in the background this is quite useful for long-running tasks on remote machines (when not using 'screen' or 'byobu'):

nohup command            # start command detached from console
nohup command &          # start command detached from console (in background)

nice values represent the niceness of a process from from -20 (most favorable scheduling) over 0 (default) to 19 (least favorable).

nice -n -20 program      #start program with a nice-value of -20 (values < 0 may require root!)
nice --20 program        #same result, but not very readable
renice -n -20 <pid>      #give a process a new nice-value

at and batch schedule the one-time execution of a command at a given time (at) or when the system load is low enough (batch). The job is read from stdin (or a file provided with -f).

at <time> (<day>)  # interactively schedule execution for a given time
batch              # interactively schedule execution when system load allows (same as at -b)
atq                # view queue of user (root sees all jobs)
atrm <jobid>       # remove scheduled job

Start at with a time (and optional day) and interactively type commands (may span several lines) and then Ctrl+d when finished.

Various time and date formats can be used, e.g.

at 18:10 2014-10-13
at midnight

To see the contents of a job, which are stored in /var/spool/cron/atjobs, use

at -c <jobid>

With /etc/at.allow and /etc/at.deny you can configure who is allowed to schedule jobs. Add one user per line (without whitespace).

cron is a deamon used to schedule the regular execution of programs / scripts / commands. A scheduled task is typically called (cron) job.

The maximum time resolution is minutes, a job can therefore be executed at maximum once a minute.

Cron jobs can be configured in two ways.

The system-wide approach is to put an executable (script) in one of these self-explanatory directories:

/etc/cron.hourly
/etc/cron.daily
/etc/cron.weekly
/etc/cron.monthly

The second way is via crontrab-entries into the global crontab file /etc/crontab with the following format (must end with a newline!):

  0   2   12  *   *   user /usr/bin/false
# ┬   ┬   ┬   ┬   ┬   ┬    ┬
# │   │   │   │   |   │    └ command
# │   │   │   │   |   └───── user executing the command
# │   │   │   │   └───────── day of week (0 - 7) (0 to 7 are Sunday to Sunday, or use names such as MON-FRI)
# │   │   │   └───────────── month (1 - 12) (or names such as JAN-DEC)
# │   │   └───────────────── day of month (1 - 31)
# │   └───────────────────── hour (0 - 23)
# └───────────────────────── min (0 - 59)

nifty online editor

Also possible, but not encouraged in order to not put cron jobs into too many different places and overcomplicate administration, are the following options:

Put files in the same format as /etc/crontab into /etc/cron.d.

Or use the program crontab, which edits a special crontab file for each user and validates the file at each save. The cron job will be executed by the respective user (with the according rights).

crontab -l     # view (list) crontab file for current user
crontab -e     # edit crontab file  for current user
crontab -r     # remove crontab file for current user

The crontab files created by the command crontab for each user reside in /var/spool/cron/crontabs (and omit the user-column!).

Minutes, days,… can be specified as follows

Gotcha: when specifying e.g. 2/5 in minutes and the system boots at 18:03 the first job is executed at 19:03 because 18:02 was missed!

The following special strings can be used instead of the first five fields of a row:

To configure which user can use crontab add respective entries (one user name per line) to

/etc/cron.allow # takes precedence
/etc/cron.deny

Ubuntu Tutorial, In-depth explanations

To disable a cron job in /etc/cron.* either remove the executable flag from the script / program (chmod -x script) or move the script out of the folder.

Jobs in crontab are disabled by prepending the line with a hash (#).

Cron passes only a minimal set of environment variables to your jobs. E.g. the PATH may be different than what you expect, things set in .bashrc may not be present,…

More info. Ubuntu CronHowTo

For computers not running 24/7 or in cases when missed jobs due to downtime must be run as soon as the server is up again use anacron.

Jobs for anacron are specified in /etc/anacrontab, which only root can edit (there is no equivalent to the command crontab usable by every user as for regular cron jobs)

# environment variables
SHELL=/bin/sh
PATH=/sbin:/bin:/usr/sbin:/usr/bin
RANDOM_DELAY=30
START_HOURS_RANGE=3-22

# jobs
  1   15  backup  /usr/local/bin/my_backup.sh
# ┬   ┬   ┬       ┬   
# │   │   │       └── command (may span several lines if line is terminated with '\'
# │   │   └────────── job-identifier string (should be unique)
# │   └────────────── delay in minutes (e.g. after system startup)
# └────────────────── period in days (>=1) or @period_name (e.g. @daily,..)

Two environment variables can be used by putting them before the job definition in /etc/anacrontab:

• START_HOURS_RANGE=3-22 (job may be run between 03:00 and 22:00)
• RANDOM_DELAY=30 (a random delay up to 30 added to 'delay in minutes')

The timestamp (YYYYMMDD) of the last execution for each job is stored in /var/spool/anacron/<job-identifier>.

The X Window System (X11) is a windowing system that handles user data input (e.g. keyboard) and graphical output. It uses a client server model, where one server can handle several clients.

A complete graphical environment consists of four layers:

1. Desktop Environment Manager: collection of programs designed for a good user experience, look&feel,.. (KDE, Gnome,..)
2. Window Manager: draw and manager windows (KWin, Metaciy, Enlightenment,..)
3. Display Manager: handle user login and start window manager (lightdm, kdm, gdm, xdm,..)
4. Display Server: interface to hardware (X-Server, wayland,..)

X can be started with the command X.

The central configuration file to configure all aspects of X11 is /etc/X11/xorg.conf, although in 2015 this file is seldomly used. Instead auto-configuration (and hotplugging) through xrandr, HAL, udev,… is used.

The file is structured in sections, the most important being:

• Monitor: properties of physical monitors (e.g. monitor modelines)
• Screen: configuration of screens (that may span several monitors)
• Device: configuration of video card
• Files: locations of fonts (FontPath) or modules (ModulePath)
• InputDevice: configuration of keyboard, mouse, touchpad,..

The parameter -geometry defines width, height, x offset (from left side) and y offset (from top) in that order. The unit for width and height depend on the program (e.g. pixels or characters)

xterm -geometry 120x20+100+100  # 120 characters wide, 20 lines tall, 100 pixels offset to top left corner
xterm -geometry 120x20-0-0      # same as above, but in the right lower corner of the screen

With -display the display where the started program should be displayed is chosen. Because of the client server architecture of X it is possible to run a program one computer and display it on another.

The syntax is: hostname:XServer.Display, but hostname and display can be omitted.

xeyes -display :0             # display xeyes on the first display of the first local X-Server
xeyes -display remotehost:1:2 # display xeyes on the third display of the second X-Server on remotehost

The default display is stored in the environment variable DISPLAY.

export DISPLAY=":0"

As mentioned before the DISPLAY variable can contain other hosts - meaning the window will be shown on a different host.

To configure an X-Server to receive connections via TCP/IP:

• X must not be started with -nolisten tcp (as is the case in e.g. Ubuntu: see /etc/X11/xinit/xserverrc)
• the display manager must be configured accordingly (e.g. lightdm by default disables tcp listening, it must be turned on by adding xserver-allow-tcp=true in /etc/lightdm/lightdm.conf
• the remote host must have been whitelisted with xhost

xhost +             # allow connections from everywhere
xhost -             # disallow connections from non-listed IPs
xhost +localhost    # allow TCP/IP connections from localhost
xhost -localhost    # disallow TCP/IP connections from localhost
xhost               # show current status

If in doubt check if X11 is listening on port 6000 (nmap -p 6000 localhost)

When connected to a remote machine over SSH it is often useful to redirect the graphical output to the local machine. This can be done with the parameter -X.

ssh -X remotehost

However, the remote host must allow this: X11Forwarding yes must specified in /etc/ssh/sshd_config. (more explanations)

The X display manager control protocol (XDMCP) is used by display managers (e.g. xdm) to look for X servers where the client can log in. However XDMCP is insecure (no encryption) and slow (no compression) - alternatives are recommended. (Ubuntu documentation)

On Linux systems fonts typically can be found in /usr/share/fonts. In /etc/fonts/fonts.conf these locations are defined.

Fonts must reside in one of the font directories and can only be used after mkfontdir and mkfontscale have been used on the respective directory. These tools create the helper files fonts.dir and fonts.scale.

With the X-Font-Server xfs a central server can be used to maintain a collection of fonts served to its clients. This server can be configured in /etc/X11/xfs.conf.

Clients must add an entry for the server into /etc/X11/xorg.conf:

Section "Files"
    FontPath "tcp/192.168.0.2:7100"
EndSection

To provide better accessibility for the physically disabled you can use tools like on-screen keyboards, text to speech (e.g. Orca, emacspeak), high contrast desktop themes with large fonts, magnifiers, output for Braille displays, or adapted keyboard sensitivity.

Most of the use cases are covered by various programs of your favourite desktop environment, but for adapting the keyboard we can use core functionality of X: AccessX.

With xkbset keyboard-related features can be turned off (with -) or on (without -):

xkbset -repeatkeys        # pressing a key once (no matter how long) counts as one key press
xkbset slowkeys 500       # a key counts as pressed after 500ms 
xkbset bouncekeys 500     # pressing a key twice within 500ms only counts as one key press

Some useful tools in an X environment:

xwininfo         # prints details about a program after klicking it (or supplying its id)
xkill            # kills a program after klicking its window (or supplying its id)
xdpyinfo         # prints information about the X server
xrandr           # configure the X-server on the fly

Networking is based on the ISO/OSI model consisting of seven layers.

1. Physical (network hardware)
2. Data Link (MAC)
3. Network (IP)
4. Transport (TCP, UDP)
5. Session (sockets)
6. Presentation (SSL)
7. Application (HTTP)

TCP (connection-based) and UDP (connectionless) are very common transport protocols based on the IP network protocol. Their differences are explained here

The internet protocol (IP) is currently in use in v4 and v6, see this infographic for detailed comparison and the most important facts here:

For IPv6 addresses leading zeros can be omitted (000A = A or 0000 = 0) and one group of consecutive zeros can be shortened with a double-colon (::). IPv4 requires network address translation (NAT) to map several hosts in a local network to one public IP. NAT is no longer required with IPv6.

Port numbers (16 bit with 65536 possible ports) map to single services behind an IP. Conventions for port assignments are managed by IANA (Internet Assigned Numbers Authority) who provide an up-to-date list of known ports. On Linux systems /etc/services contains the known ports.

The port range of 0 - 65535 is split into three regions:

1. 0 - 1023: system / well-known ports (ports registered with IANA, root required to open a port)
2. 1014 - 49151: user / registered ports (ports registered with IANA, a normal user can open a port)
3. 49152 - 65535: dynamic / private ports (no rules)

IP4v netmasks are 32 bits long. They consist of a consecutive number of set bits and then only unset bits. The Classless Inter Domain Routing (CIDR) notation gives the number of set bits. The number of hosts in a network is defined as 2^nr-of-0-bits minus 2. The lowest (network address) and highest (broadcast address) addresses have to be deducted.

List of class C networks (with only the last 8 bit block available):

Note, that /31 is missing because it would only consist of a network and a broadcast address but no host. /32 is a special case identifying a single address. The n

Physical network interfaces were typically named ethN and wlanN. For virtual interfaces (several interfaces over one physical interface) a colon is added, e.g. eth0:0.

Modern systems using systemd use predictable network interface names in the form en (ethernet), wl (WLAN), or ww (WWAN), e.g. enp0s25. The advantage of this new naming schema is, that the names will not change on reboots, which could happen with the old scheme.

Inspection of the current state and temporary configuration can be done with ifconfig, route and ip.

ifconfig -a      # view all interfaces (also disabled ones)
netstat -i -e    # same..

Basic configuration:

ifconfig eth0 192.168.0.1  # minimum
ifconfig eth0 192.168.0.1 netmask 255.255.255.240 broadcast 192.168.0.16

Turning interfaces on and off:

ifconfig eth0 up
ifconfig eth0 down

Virtual interfaces can be simply created if the underlying physical interface exists

ifconfig eth0:0 192.168.0.1  # create
ifconfig eth0:0 down         # disable / remove

The gateway address is the address packets not destined for our current subnet are sent (and then e.g. sent into the internet through a router). It is set via route in the kernel's IP routing table.

route            # view current routing table 
route -n         # same, but with numeric IP addresses instead of names

Set a default route (to a gatewy) for a network

route add default gw 10.0.0.138

Add route for a specific network (network address and netmask must match!) and a specific device:

route add -net 10.0.1.0 netmask 255.255.255.0 gw 10.0.0.138 dev eth0 

For removing routes the exact same command has to be executed, but with add replaced by del:

route del -net 10.0.1.0 netmask 255.255.255.0 gw 10.0.0.138 dev eth0

To permanently configure the network configuration (also after a reboot) use /etc/network/interfaces. All interfaces configured here can be en/disabled with ifup and ifdown.

Use DHCP:

auto eth0
iface eth0 inet dhcp

Manual configuration:

auto eth0
iface eth0 inet static
address 192.168.0.100
netmask 255.255.255.0
gateway 192.168.0.1

Note, that since at least Ubuntu 12.04 desktop versions use NetworkManager instead of /etc/network/interfaces. Use the respective GUI tools or nmcli and nm-tool for configuration.

Simple static name resolution can be configured in /etc/hosts, which contains a simple name to IP mapping.

Usage of the Domain Name Service (DNS) is configured in /etc/resolv.conf. The keyword nameserver specifies DNS servers:

nameserver 8.8.8.8
nameserver 8.8.4.4

In Ubuntu 14.04 or later you should edit /etc/resolvconf/resolv.conf.d/head and then run resolvconf -u, since /etc/resolv.conf is generated.

In /etc/nsswitch.conf you can configure if and in which order local files (/etc/hosts) and DNS should be used:

hosts:    files dns
networks: files dns

DNS (reverse) lookups can be done with host or the 'domain information groper' dig (nslookup is obsolete due to security issues!).

host -a <hostname>   # look up DNS entry for a hostname
dig <hostname>       # same, but more verbose (and more options available)

ethtool              # query or control network driver and hardware settings
nm-tool              # report NetworkManager state and devices

tracepath <host>     # trace the path to a network host (which servers / hops are on the way)
tracepath6 <host>    # for IPv6
traceroute <host>    # same as tracepath, but requires root
traceroute6 <host>   # for IPv6

netstat can provide very detailed information about open connections and statistics:

netstat -a           # print open & listening internet connections and Unix domain sockets
netstat -a -t -p     # print only tcp connections (-u for udp) and the associated pid
netstat -s           # print statistics about e.g. nr of received packets
netstat -r           # print the kernel routing table

nmap is an advanced port scanner (see some examples here). A simple use case is to check which services are available / ports are open on a server:

nmap localhost       # print open ports on current machine (only a selection of common ports)
nmap -p 1-9999 host  # print open ports on a host (with a defined port range)
nmap -Pn -p 1-9999 host  # as above, but skip host discovery (helpful if above probe fails) 
nmap -A -T4 host     # print open ports with enabled version detection (-A) and don't wait too long (-T4)

Advanced guide for Ubuntu network configuration

The deprecated inetd and the modern (and icompatible) xinetd are super-daemons that listen for incoming requests on several ports and start the according services only on demand. This can be useful to save resources by running rarely used services only when needed.

inetd was configured via one entry for each service in /etc/inetd.conf (where the service name should be present in /etc/services or a port number):

# service     type     protocol  wait    user    programm            arguments
telnet        stream   tcp       nowait  root    /usr/sbin/telnetd   telnetd -a
pop3          stream   tcp       nowait  root    /usr/sbin/tcpd      ipop3d
3000          stream   tcp       nowait  nobody  /bin/echo           echo Hello World

xinetd provides additional features such as access control mechanisms (e.g. with TCP Wrapper ACLs), extensive logging capabilities, the ability to make services available based on time or the maximum number of started services (to counter DoS attacks). It is configured via /etc/xinetd.conf and one file for each service in /etc/xinetd.d/. An example configuration for the TCP version of daytime that prints the current date and time:

service daytime
{
        disable         = yes
        type            = INTERNAL
        id              = daytime-stream
        socket_type     = stream
        protocol        = tcp
        user            = root
        wait            = no
}

tcpd is a TCP wrapper which first performs security checks and then starts a program (see pop3 service in the inetd-example above). tcpd checks if the incoming request is allowed through the settings in /etc/hosts.allow and /etc/hosts.deny. The check if access is granted works as follows: first allow is checked. If not explicitly allowed it is checked if it is explicitly denied. If not it is allowed. The lines in each are processed in order of appearance and search terminates when a match is found (so put more specific rules first).

Syntax: daemon : client [:option1:option2:…] (see man hosts_access), here are some simple examples:

vsftpd: 192.168.1.,.abc.com
sshd: ALL EXCEPT 192.168.0.15
ALL: ALL                 

Email basically works as follows.

The sender uses a Mail User Agent (MUA), or email client, to send the message through one or more Mail Transfer Agents (MTA), the last of which will hand it off to a Mail Delivery Agent (MDA) for delivery to the recipient's mailbox, from which it will be retrieved by the recipient's email client, usually via a POP3 or IMAP server.

Commonly used programs are:

• MUA: Mozilla Thunderbird, Microsoft Outlook,..
• MTA: sendmail (the original), postfix, exim, postmaster, qmail, smail
• MDA: procmail, maildrop

apt-get install postfix
dpkg-reconfigure postfix  (basically edits /etc/postfix/main.cf)

No matter which MTA is used, they all share:

• sendmail as command to send emails
• /etc/aliases

With /etc/aliases emails to local recipients can be redirected to one or multiple recipients.

The format for each line is:

#name: value1, value2, ...
postmaster: alice
root: alice, bob

After changes to /etc/aliases the command newaliases or sendmail -bi must be called. It updates the alias-database, which is used by sendmail et al instead of the raw text file.

Each user can create redirections for only his/her account in ~/.forward using the same format.

More in the Ubuntu server guide

An email consists of a header and a body, separated by a new line. A very simple Email:

Subject: Hello World

This is the body of my first email.

To send it the command sendmail can be used (no matter which MTA is actually used):

sendmail recipient@example.com < email.txt
sendmail -bm -f sender@example.com recipient@example.com < email.txt

Incoming emails are stored in the mail spool in /var/mail/username. The spool can e.g. be inspected with the command line tool mail.

Typically MUAs move emails from the spool to a different location. MUAs that support mail spools are e.g. mutt and pine (command line) or Mozilla Thunderbird (create a 'Unix Mailspool (Movemail)' account).

TODO

find package of suspect file

dpkg -S <filename>

basic check:

debsums <package_name>

a little more paranoid check:

aptitude download <package_name>
ar x <package.deb>
tar xfvz control.tar.gz
grep $(md5sum <filename>) md5sums