pf, pfsync and CARP on FreeBSD 6.2


Redundant Failover firewall with pf, pfsync and CARP on FreeBSD 6.2

Posted by Chris Barnes on Friday, June 15, 2007

This is a step by step tutorial that should take most of a day. I’m posting this here mostly as a reminder so that I can come back and read it when I need to build another firewall but hopefully it will be helpful to someone else also. If you find this tutorial useful or if you find anything wrong with it, send me an email hammockintahiti@gmail.com.

Install FreeBSD

Download the disc 1 and disc 2 from ftp://ftp.freebsd.org/pub/FreeBSD/releases/i386/ISO-IMAGES/6.2

Burn the iso images to cds and boot the target machine with disc 1.

Start a Standard installation.

Highlight any partitions and hit “d” to delete them, then hit “a” to use the entire disk, then hit “q” to continue.

Choose Standard for no boot manager.

Create partitions. You can adjust the sizes of the partitions based on the size of your drive. The /usr/local and /usr/home partitions can go as low as 128MB since this won’t be a common-user system and there won’t be a lot of user-specific files or binaries…but the /usr partition should never go below 2,000MB since that’s where all of your kernel source code and ports tree is located. Here’s a partition scheme if you have a 6GB drive:

486MB swap partition (or at least 2x your RAM)
512MB file system mounted as /
512MB file system mounted as /tmp
1267MB file system mounted as /var
3115MB file system mounted as /usr
128MB file system mounted as /usr/local
128MB file system mounted as /usr/home

Press q to continue.

Highlight Kern-Developer and press space bar.

When asked about installing the ports collection choose yes.

Highlight exit and press enter.

Choose CD/DVD as the install media.

Last Chance, are you sure? Yes

When you see Congradulations, hit ok.

FreeBSD Post-Install configuration

Would you like to configure any ethernet or SLIP/PPP network devices? Yes

Highlight your device that is connected to the internet and press enter.

Do you want to try IPv6? No

Do you want to try DHCP? Yes

Verify network info and update if necessary.

Do you want this machine to function as a gateway? Yes

Do you want to configure inetd and the network services that it provides? No

Would you like to enable SSH login? Yes

Do you want to have anonymous FTP access to this machine? No

Do you want to configure this machine as an NFS Server: No

Do you want to configure this machine as an NFS Client: No

Would you like to customize your system console settings? No

Would you like to set this machine’s time zone now? Yes

Is your machine’s CMOS clock is set to UTC? No

Select the appropriate time zone - by region, country, and then the applicable time zone.

Would you like to enable Linux Binary compatibility? No

Does your system have a PS/2, serial, or bus mouse? Yes (unless, of course, it doesn’t…)

Would you like to browse the ports collection? Yes

Install cvsup and bash

cvsup will be used to keep your system up to date.

Highlight net and press enter.

Highlight cvsup-without-gui and press space bar.

Tab to ok and press enter.
bash is a shell that we will use instead of the default sh

Highlight shells and press enter.

Highlight bash-2 and press space bar.

Tab to ok and press enter.
Tab to Install and press enter.

Review your selections and press enter to install.

Would you like to add any user accounts? Yes (we need at least one so we can dis-allow root ssh access and require you to login as an unprivilaged user and su to root)

Highlight User and press enter.

Type the username, password, full name, and make the member group 0. (this is important so that your new user will be in the ‘wheel’ group and they will be able to su to root. Also make the login shell /usr/local/bin/bash (I will create a user with the username admin so if you do the same you can follow the rest of this tutorial verbatim, otherwise just change admin to your username every time I use it later.

Set the root password.

Would you like to visit the general configuration menu for a chance to set any last options? Yes

Go to network then ntupdate and choose a server near you.

Highlight exit and press enter. (do this twice to get to the main menu)

Tab over to Exit Install and hit enter.

(System Reboots)

Make “bash” the default shell for ‘root’ and perform an initial set up of root’s bash environment.

Log in as your non-privaleged user account. You should see a ‘bash-2.05b$’ prompt…indicating that bash was successfully installed. After you log in, then type ’su’ to switch user to root. Enter the root password.

Type vipw and press enter.

Change the default shell from /bin/csh (at the end of the first line) to /usr/local/bin/bash.

If you like, you can change root’s unoffical name from ‘Charlie &’ to ‘Super-User’ or whatever you like. When you get mail from root, it will be marked as being from the name that you enter here.

Verify that your changes are successful. Press -F2 to get another terminal, then log in as root. Verify that you’re presented with the ‘bash-2.05b#’ prompt. If you are, then log out and go back to the 1st virtual terminal to continue working. If you don’t see the bash prompt, then you need to go back to the previous step and figure out what you did wrong.
Create a file named .bashrc in /root that contains the following.

umask 077
PS1="[u@h W]\$ "
alias ls='ls -alFG'

Also create a file named .bash_profile in /root that contains the following.

PATH=/sbin:/bin:/usr/sbin:/usr/bin:/usr/local/bin:/usr/local/sbin:$HOME/bin; export PATH
umask 077
PS1="[u@h W]\$ "
alias ls='ls -alFG'

Change the permissions on both files.

chmod 600 .bashrc
chmod 600 .bash_profile

Test your settings again at the 2nd virtual terminal. Log in as root, verify you’re using the bash shell, your cursor line looks different (it has your userid and current working directory), and that you have colorized directory listings. Close that session, return to the 1st virtual terminal, log out, and log back in and su to root.

Copy the files to your non privilaged user change the ownership of the new copies.

cp /root/.bashrc /usr/home/admin/.bashrc
cp /root/.bash_profile /usr/home/admin/.bash_profile
chown admin /usr/home/admin/.bashrc
chown admin /usr/home/admin/.bash_profile

Redirect root’s email to your email account.

vi /etc/aliases

Uncomment # root: me@my.domain and change me@mydomain to your email address.

Save and exit.

Update the email alias database.

newaliases

Configure cvsup and update your source tree & ports collection.

After you configure cvsup and update your source and ports collection, you will want to re-run cvsup every once in a while to ensure your sources are up to date, then recompile your kernel & system binaries to ensure you are using the latest versions with security patches applied.

cp /usr/share/examples/cvsup/stable-supfile /etc
chmod 600 /etc/stable-supfile
vi /etc/stable-supfile

Type “:set num” in vi to see line numbers

Change line 68 to point to a CVS server near you. Here you’ll find a list of CVSup servers http://www.freebsd.org/doc/en_US.ISO8859-1/books/handbook/cvsup.html#CVSUP-MIRRORS. I ususlly go with cvsup2 or 3 because the main site reaches the maximum number of simultaneous users often.

Add these lines at the bottom of the file:

ports-net tag=CHANGE_THIS.FreeBSD.org
ports-shells tag=CHANGE_THIS.FreeBSD.org

Syncronize your source tree with the CVS server…should take 30-60 minutes

cvsup /etc/stable-supfile

Configure a custom kernel with support for ph, phsync, pflog, and carp

cd /usr/src/sys/i386/conf
cp GENERIC FIREWALL
vi FIREWALL

In line 2 of the file (part of the main comment block) change the word, GENERIC, to your hostname, FIREWALL.

On line 19 of the file (still part of the main comment block), change the word, GENERIC, to your hostname, FIREWALL

On lines 22-24, comment out the “cpu” lines so that only the one for your specific chip is left.

On line 25, change the value of the ident parameter so that it’s your hostname, FIREWALL

Add the following lines to the bottom of the file.

# pf support
device pf
device pfsync
device pflog
device carp

Recompile your kernel to the updated stable version. (make buildworld will take about an hour and make buildkernel will take about 20 minutes)

[root@firewall /]# cd /usr/src
[root@firewall src]# echo "KERNCONF=FIREWALL" >> /etc/make.conf
[root@firewall src]# make buildworld
[root@firewall src]# make buildkernel
[root@firewall src]# make installkernel

Configure the SSH daemon and your user’s DSA key files.

Modify the SSH daemon configuration file, /etc/ssh/sshd_config, so that it reads as follows. The modified lines are in red. I’m using port 8081 for ssh access instead of the usual port 21. In the past I have had issues with hackers trying to use brute force tactics on my standard ports if they were open.

Port 8081
Protocol 2
#AddressFamily any
ListenAddress 192.168.1.1 # Put your internal interface’s address here
#ListenAddress :: 

# HostKey for protocol version 1
#HostKey /etc/ssh/ssh_host_key
# HostKeys for protocol version 2
#HostKey /etc/ssh/ssh_host_dsa_key

# Lifetime and size of ephemeral version 1 server key
#KeyRegenerationInterval 1h
#ServerKeyBits 768

# Logging
# obsoletes QuietMode and FascistLogging
#SyslogFacility AUTH
LogLevel VERBOSE

# Authentication:

#LoginGraceTime 2m
PermitRootLogin no
StrictModes yes
#MaxAuthTries 6

RSAAuthentication yes
PubkeyAuthentication yes
AuthorizedKeysFile .ssh/authorized_keys

# For this to work you will also need host keys in /etc/ssh/ssh_known_hosts
RhostsRSAAuthentication no
# similar for protocol version 2
HostbasedAuthentication no
# Change to yes if you don’t trust ~/.ssh/known_hosts for
# RhostsRSAAuthentication and HostbasedAuthentication
IgnoreUserKnownHosts yes
# Don’t read the user’s ~/.rhosts and ~/.shosts files
IgnoreRhosts yes

# Change to yes to enable built-in password authentication.
PasswordAuthentication no
PermitEmptyPasswords no

# Change to no to disable PAM authentication
ChallengeResponseAuthentication no

# Kerberos options
#KerberosAuthentication no
#KerberosOrLocalPasswd yes
#KerberosTicketCleanup yes
#KerberosGetAFSToken no

# GSSAPI options
#GSSAPIAuthentication no
#GSSAPICleanupCredentials yes

# Set this to ‘no’ to disable PAM authentication, account processing,
# and session processing. If this is enabled, PAM authentication will
# be allowed through the ChallengeResponseAuthentication and
# PasswordAuthentication. Depending on your PAM configuration,
# PAM authentication via ChallengeResponseAuthentication may bypass
# the setting of “PermitRootLogin without-password”.
# If you just want the PAM account and session checks to run without
# PAM authentication, then enable this but set PasswordAuthentication
# and ChallengeResponseAuthentication to ‘no’.
UsePAM no

AllowTcpForwarding no
GatewayPorts no
X11Forwarding no
#X11DisplayOffset 10
#X11UseLocalhost yes
#PrintMotd yes
PrintLastLog yes
#TCPKeepAlive yes
#UseLogin no
#UsePrivilegeSeparation yes
#PermitUserEnvironment no
#Compression delayed
#ClientAliveInterval 0
#ClientAliveCountMax 3
#UseDNS yes
#PidFile /var/run/sshd.pid
#MaxStartups 10
#PermitTunnel no

# no default banner path
#Banner /some/path

# override default of no subsystems
Subsystem sftp /usr/libexec/sftp-server

# Example of overriding settings on a per-user basis
#Match User anoncvs
# X11Forwarding no
# AllowTcpForwarding no
# ForceCommand cvs server
AllowUsers admin # Substitute your non-privilaged user for admin

Generate an SSH key (version 2) for your user, by performing the following steps:

[root@firewall /root]# su - admin          *** substitute your non-privileged userid for 'admin'
[admin@firewall admin]$ ssh-keygen -d   *** then accept the default DSA key name & enter a passphrase (twice)

Add the public copy of your user’s version 2 key to their own authorized_keys file by typing the following steps:

[admin@firewall admin]$ cd .ssh
[admin@firewall .ssh]$ cat id_dsa.pub > authorized_keys

Copy your user’s private & public keys to other systems that you’ll be using to SSH to your new firewall from. By default, the private & public key go into a user’s ‘.ssh’ directory on those systems. Without the private key on those remote systems, your firewall will not accept connections from them. If you’re new to FreeBSD and need to know how to access the floppy drive, follow the following steps.

[root@firewall root]# mkdir /mnt/floppy                     # This will make an empty mount point to mount the floppy to
[root@firewall root]# mount -t msdosfs /dev/fd0 /mnt/floppy # Insert a DOS-formatted floppy before you do this
[root@firewall root]# cd /mnt/floppy
[root@firewall floppy]# cp /home/admin/.ssh/id_dsa* .       # Copies all of your user's ssh key info to the floppy
[root@firewall floppy]# ls                                  # List the contents of the floppy to verify the files are there
[root@firewall floppy]# cd ..
[root@firewall mnt]# umount /mnt/floppy                     # Unmount the floppy

Now that you’ve copied your user’s private & public keys to another system, remove them from your user’s .ssh directory on the firewall. This is only a precaution so that it can’t be stolen by a hacker and compromised.

Open up your /etc/hosts.allow file, delete all of the lines, and ensure that it reads as follows. Note that 192.168.1.0 is the address space of your internal network in this example. If you’re using a different internal address space (e.g. 10.10.10.0), then make the appropriate modifications.

# hosts.allow access control file for "tcp wrapped" applications.
ALL : localhost 127.0.0.1 : allow
sshd : 192.168.1.0/255.255.255.0 : allow
ALL : ALL : deny
# If you want to allow a specific computer on the Internet to SSH into your
# system, replace the 'sshd' line above with one like this...but subsitute
# the X.X.X.X and subnet mask to suit your needs (e.g. one computer, entire subnet
# etc.). Also, make sure you allow inbound SSH from that same host/subnet
# in your /etc/ipf.rules file.
# sshd : 192.168.1.0/255.255.255.0 X.X.X.X/255.255.255.255 : allow

Install and configure AIDE (Advanced Intrusion Detection Environment)

[root@firewall /root]# cd /usr/ports/security/aide
[root@firewall aide]# make install clean
[root@firewall aide]# cp /usr/local/etc/aide.conf.sample /var/db/aide/aide.conf
[root@firewall aide]# cd /var/db/aide
[root@firewall aide]# aide --init
[root@firewall aide]# mv databases/aide.db.new databases/aide.db

Create a cron job to check the integrity of your system every Sunday at 4AM:

[root@firewall /root]# cd /etc
[root@firewall /etc]# vi crontab

Add the following line to the file:

0   4   *   *   7    root   /usr/local/bin/aide --check

Restrict crontab access/usage

Create the file /var/cron/allow and add the following lines to it. Be sure to substitute ‘newuser’ for whatever your non-privileged user account is.

root
newuser

Edit /etc/crontab and comment out the ‘at’ job that runs every 5 minutes.

*/5 * * * * root /usr/libexec/atrun

Chmod your /etc/crontab file so that it is only readable by root.

[root@firewall /etc]# chmod 600 /etc/crontab

Edit your /etc/rc.conf to look like this

This is my rc.conf file. Your network interfaces will be named differently if you have different cards (e.g. xl0 and xl1 are 3com network cards in my machine and rl0 is an integrated NE2000 network card)

icmp_drop_redirects="YES"
ntpdate_enable="YES"
ntpdate_flags="north-america.pool.ntp.org"
sshd_enable="YES"
usbd_enable="YES"
syslogd_flags="-ss"
sshd_flags="-4"
pf_enable="YES"
pf_rules="/etc/pf.conf"
pf_flags=""
pflog_enable="YES"
pflog_logfile="/var/log/pflog"
pflog_flags=""
kern_securelevel_enable="YES"
kern_securelevel="2"
dhcpd_enable="YES"
gateway_enable="YES"
defaultrouter="208.180.xxx.xxx"
hostname="dell_firewall"
network_interfaces="xl0 xl1 rl0 lo0 pfsync0"
cloned_interfaces="carp0 carp1 carp2"

# Loopback Interface
ifconfig_lo0="inet 127.0.0.1"

# External Public Interface (for the secondary firewall use a different public ip.)
ifconfig_xl0="208.180.xxx.xxx"

# External Public Carp Interface
ifconfig_carp0="208.180.xxx.xxx vhid 1 pass foo"
ifconfig_carp0_alias0="208.180.xxx.xxx vhid 1 pass foo"
ifconfig_carp0_alias1="208.180.xxx.xxx vhid 1 pass foo"

# Internal Interface (for the secondary firewall change the ip address to 192.168.1.251)
ifconfig_xl1="192.168.1.250"

# Internal Carp Interface
ifconfig_carp1="192.168.1.1 vhid 2 pass foo"

# Heartbeat Interface (for the secondary firewall, change the ip address to 10.10.10.251)
ifconfig_rl0="10.10.10.250"

#Heartbeat Carp Interface
ifconfig_carp2="10.10.10.1 vhid 3 pass foo"

# PFSync Interface
ifconfig_pfsync0="up syncif rl0"

Add the following lines to the bottom of /etc/sysctl.conf

Type ifconfig with no parameters to view your network configuration. Each real interface should have a line that says status: active when the ethernet cable is plugged in. If any of them do not, then preempt will not work. Note that the changes we made to rc.conf do not show up because we have not rebooted yet.

net.inet.tcp.blackhole=2
net.inet.udp.blackhole=1

# if one interface fails then all will fail over
net.inet.carp.preempt=1

net.inet.tcp.sendspace=65536
net.inet.tcp.recvspace=65536

Configure the main firewall /etc/pf.conf

################################################################################
# Macro and lists
################################################################################
lop_if  ="lo0"
pfs_if  ="dc0"
ext_if  ="xl0"
int_if  ="xl1"
ext_carp ="carp0"

dns_srv  = "{ 208.180.xxx.xxx, 208.180.xxx.xxx }"
int_fw   = "{ 208.180.xxx.xxx 192.168.1.1 }"
chat_srv = "{ 192.168.1.119 }"

# Allowed incoming ICMP types
icmp_types = "{ echorep, echoreq, timex, paramprob, unreach code needfrag }"

# Private networks (RFC 1918)
priv_nets = "{ 127.0.0.0/8, 192.168.0.0/16, 172.16.0.0/12, 10.0.0.0/8 }"

mail_ports         = "{ 25, 110 }"
web_ports          = "{ www, https }"
ftp_ports          = "{ 8084 9000 9001 }"
firewall_ssh_ports = "{ 8081 }"
web_ssh_ports      = "{ 8082 }"
chat_ports         = "{ 8080 }"
dns_ports          = "{ 53 }"

################################################################################
# Options, scrub and NAT
################################################################################
set block-policy drop
set skip on $lop_if

scrub in

# NAT outgoing connections
nat on $ext_if from $int_if:network to any -> $ext_if

################################################################################
# Redirection
################################################################################
rdr on $ext_if proto tcp from any to $ext_carp port $chat_ports -> $chat_srv
rdr on $ext_if proto tcp from any to $ext_carp port $ftp_ports -> 192.168.1.80
rdr on $ext_if proto tcp from any to $ext_carp port $mail_ports -> 192.168.1.80
rdr on $ext_if proto tcp from any to $ext_carp port $web_ssh_ports -> 192.168.1.80
rdr on $ext_if proto tcp from any to 208.180.xxx.xxx port $web_ports -> 192.168.1.80
rdr on $ext_if proto tcp from any to 208.180.xxx.xxx port $web_ports -> 192.168.1.81
rdr on $ext_if proto tcp from any to 208.180.xxx.xxx port $web_ports -> 192.168.1.82

################################################################################
# Filtering Rules
################################################################################
block log all
antispoof quick for $int_if

pass in quick on $ext_if inet proto tcp from 208.180.xxx.xxx to $int_fw port $firewall_ssh_ports keep state

pass in quick on $int_if inet proto udp from any to any port 123 keep state
pass out quick on $ext_if inet proto udp from any to any port 123 keep state

pass quick on $pfs_if proto pfsync
pass quick proto carp

block in quick on $ext_if from $priv_nets to any
block out quick on $ext_if from any to $priv_nets

pass in quick on $ext_if inet proto tcp from any to $int_if:network port $mail_ports 
     synproxy state
pass out quick on $int_if inet proto tcp from any to $int_if:network port $mail_ports 
     keep state
pass in quick on $int_if inet proto tcp from $int_if:network port $mail_ports to any 
     keep state
pass out quick on $ext_if inet proto tcp from $int_if:network port $mail_ports to any 
     modulate state

pass in  quick on $ext_if inet proto tcp from any to $int_if:network port $web_ports 
     synproxy state
pass out quick on $int_if inet proto tcp from any to $int_if:network port $web_ports 
     keep state

pass in quick on $ext_if inet proto tcp from any to $chat_srv port $chat_ports 
     synproxy state
pass out quick on $int_if inet proto tcp from any to $chat_srv port $chat_ports 
     keep state

pass in  quick inet proto icmp all icmp-type $icmp_types
pass out quick inet proto icmp all

pass in quick on $int_if inet proto { tcp, udp } from $int_if:network to $dns_srv port domain keep state

pass out quick on $ext_if inet proto { tcp, udp } from { $ext_if $int_if:network} to $dns_srv port domain keep state

pass in  quick on $int_if inet proto tcp from $int_if:network to any port $web_ports
pass out quick on $ext_if inet proto tcp from { $ext_if $int_if:network } to any port $web_ports 
     modulate state

Configure the system to mount partitions restrictive

Modify the /etc/fstab file with vi so that we can change how each partition is mounted…to ensure that hackers can do at little as possible if they (by chance alone) hack the box. Essentially, we’re restricting some of the partitions so that they are ‘nosuid’, ‘noexec’, and ‘ro’. The original /etc/fstab should look something like this. Yours might look a little different…the first column (device names) might be a little different, but that’s OK. The stuff we’ll be modifying is in the 4th column.

# Device    Mountpoint  FStype  Options   Dump  Pass#
/dev/ad0s1b none        swap    sw        0     0
/dev/ad0s1a /           ufs     rw        1     1
/dev/ad0s1d /tmp        ufs     rw        2     2
/dev/ad0s1f /usr        ufs     rw        2     2
/dev/ad0s1h /usr/home   ufs     rw        2     2
/dev/ad0s1g /usr/local  ufs     rw        2     2
/dev/ad0s1e /var        ufs     rw        2     2
/dev/acd0   /cdrom      cd9660  ro,noauto 0     0

First, copy the original /etc/fstab file to /etc/fstab.original
Then, make another copy of the /etc/fstab file and call it /etc/fstab.restrictive
Then, modify the /etc/fstab.restrictive file so that it reads as follows:

# Device    Mountpoint  FStype  Options                 Dump  Pass#
/dev/ad0s1b none        swap    sw                      0     0
/dev/ad0s1a /           ufs     rw,nosuid               1     1
/dev/ad0s1d /tmp        ufs     rw,noexec,nosuid,nodev  2     2
/dev/ad0s1f /usr        ufs     ro                      2     2
/dev/ad0s1h /usr/home   ufs     rw,noexec,nosuid        2     2
/dev/ad0s1g /usr/local  ufs     ro,nosuid               2     2
/dev/ad0s1e /var        ufs     rw,noexec,nosuid        2     2
/dev/acd0   /cdrom      cd9660  ro,noauto               0     0

Next, copy your new /etc/fstab.restrictive file and over-write the original /etc/fstab…so that your “real” fstab file has the restrictive settings, and you have the two other config files available (the original and restrictive one).

[root@firewall etc]# cp /etc/fstab.restrictive /etc/fstab

This will make adding new software, etc. much more difficult since /usr and /usr/local are mounted read-only. This means that programs which try to install their user-land programs in /usr/local/bin will fail during their install programs. And cvsup…which will try to update the kernel’s source code in /usr/src and the ports in /usr/ports…well, they’re now read-only because they fall under /usr. So, mounting your partitions in a very restrictive way will limit what the hacker can do on your system, but it makes software installs and kernel upgrades more difficult (or impossible…if the partitions are still mounted in a restrictive way).

Given that, if you want to add new software or upgrade the kernel & ports tree source code, you’ll need to

  1. Change the partition’s mounting in /etc/fstab back to their original values by copying your /etc/fstab.original file to /etc/fstab.
  2. Bump the kernel security level back down to “1″ by setting the kern_securelevel paramater in your /etc/rc.conf file, and then
  3. Reboot the machine (this will not cause downtime because the secondary firewall will take over while this machine is rebooting)
  4. Update your sources with cvsup, then make buildworld, make buildkernel, and make installworld

Then when you’re done upgrading, recompiling, and installing, do the steps in reverse:

  1. Change the partition’s mounting in /etc/fstab to their restrictive values by copying your /etc/fstab.restrictive file to /etc/fstab.
  2. Bump the kernel security level back up to “2″ by setting the kern_securelevel paramater in your /etc/rc.conf file, and then
  3. Reboot the machine

This is the price you pay for a VERY, VERY secure machine. Reboot the machine so we can finish the job…

[root@firewall /etc]# shutdown -r now

If the system doesn’t reboot, it means that you probably made an error in the kernel configuration file…possibly setting the wrong type of CPU. DON’T PANIC. We can still boot the machine so that you can fix the error. To boot into the original version of the kernel, following the steps, below:

  1. Reboot the machine (power off, then on)
  2. When you reboot the machine and get to the bootloader screen, select option 6, “Escape to loader prompt”. This will give you an “OK” prompt at the bottom of the screen. Boot from your “old” kernel.
    OK unload
    OK boot /boot/kernel.old/kernel
  3. After the old kernel boots, you’ll want to copy the “old” kernel to a safe place before you recompile a new kernel. This is an important thing to do since “kernel.old” is overwritten when you install a new kernel. To do this, type the following commands:
    [root@firewall /]# cp -R /boot/kernel.old /boot/kernel.good

    If subsequent kernel compiles still don’t work, you can always manually boot off your good kernel from the “OK” prompt until you resolve the problem…just substitute “kernel.good” for “kernel.old” in the commands listed above.

  4. Now that you’ve saved a copy of your “good” kernel, modify your kernel configuration file and fix whatever was causing the problems, recompile & install, and then reboot and continue with the next step.

After the system comes back up, you’ll want to re-generate the AIDE database and replace the old one. Since you updated the kernal and all of the system binaries, the AIDE database signatures of those files is out of date. If you don’t update the AIDE database, AIDE will find thousands of “changes” to the system binaries when it runs for the first time at 4AM in the morning. To update the database so that it has signatures for the newest kernel & system binaries, etc, just type the following commands:

[root@firewall /]# cd /var/db/aide
[root@firewall aide]# aide --init
[root@firewall aide]# mv databases/aide.db.new databases/aide.db

After you do this you should have a completely working firewall.



Freebsd 7.0下通过CARP+PF+PFSYNC实现服务器群集笔记


硬件: 
服务器 5台(实际只用了4台, 2台做LoadBalance(主/备各一,每台服务器至少需要三块网卡), 2台做应用服务.)
cisco2950 2台

系统: FreeBSD 7.0 release(最小化安装)

实现目标: 
1. 通过pfsync, 实现LoadBalance 的 高可用性(HA);
2. 实现服务器池的负载均衡;

我部署环境的测试地址(不能长期保留, 群集部署是在内网中,所以外网通过重定向来carp0上的): 

一 网络拓扑:
外网: 192.168.1.0/24
内网: 192.168.10.0/24
HA专用: 10.10.10.250 / 10.10.10.251
[attach]250672[/attach]

二 Load Balance配置:

A.内核设置: 
master/slave的内核配置是一样的
device carp
device pf        #启动虚拟网络设备来记录流量(经由 bpf)
device pflog     #启动虚拟网络设备来监视网络状态
device pfsync

options         ALTQ
options         ALTQ_CBQ        # 基于分类的排列 (CBQ)
options         ALTQ_RED        # 随机先期检测 (RED)
options         ALTQ_RIO        # 对进入和发出的包进行 RED
options         ALTQ_HFSC       # 带等级的包调度器 (HFSC)
options         ALTQ_PRIQ       # 按优先级的排列 (PRIQ)
options         ALTQ_NOPCC      # 在联编 SMP 内核时必须使用,禁止读时钟

重编内核. 

B.配置rc.conf
这里只写与本章节相关的配置项了. 
master部分:
gateway_enable=”YES”
defaultrouter=”192.168.1.1″
hostname=”master.cluster.org”
cloned_interfaces=”carp0 carp1″

# External Public Interface (for the secondary firewall use a different public ip.)
ifconfig_em0=”inet 192.168.1.52 netmask 255.255.255.0″
# External Public Carp Interface
#ifconfig_carp0=”vhid 1 pass 11111 192.168.1.51/24″
ifconfig_carp0=”vhid 1 pass 11111 192.168.1.51/24 advskew 10″

# Internal Interface (for the secondary firewall change the ip address to 192.168.10.11)
ifconfig_em1=”inet 192.168.10.10 netmask 255.255.255.0″
# Internal Carp Interface
ifconfig_carp1=”vhid 1 pass 22222 192.168.10.100/24 advskew 10″

# Heartbeat Interface (for the secondary firewall, change the ip address to 10.10.10.251)
ifconfig_vr0=”10.10.10.250 netmask 255.255.255.0″

# PFSync Interface
ifconfig_pfsync0=”up syncif vr0″

pf_enable=”YES”
pf_rules=”/etc/pf.conf”
pf_flags=”"
pflog_enable=”YES”
pflog_logfile=”/var/log/pflog”
pflog_flags=”"

slave部分:
[color=Red]#slave的配置与master配置大概相近了. 改动的主要是本机IP和advskew的优先值. 注意: carp 的IP是公共的必须一致. [/color]
gateway_enable=”YES”
defaultrouter=”192.168.1.1″
hostname=”slave.cluster.org”
cloned_interfaces=”carp0 carp1″

# External Public Interface (for the primary firewall use a different public ip.)
ifconfig_em0=”inet 192.168.1.53 netmask 255.255.255.0″
# External Public Carp Interface
#ifconfig_carp0=”vhid 1 pass 11111 192.168.1.51/24″
ifconfig_carp0=”vhid 1 pass 11111 192.168.1.51/24 advskew 20″

# Internal Interface (for the primary firewall change the ip address to 192.168.10.10)
ifconfig_em1=”inet 192.168.10.11 netmask 255.255.255.0″
# Internal Carp Interface
ifconfig_carp1=”vhid 1 pass 22222 192.168.10.100/24 advskew 20″

# Heartbeat Interface (for the primary firewall, change the ip address to 10.10.10.250)
ifconfig_fxp0=”10.10.10.251 netmask 255.255.255.0″

# PFSync Interface
ifconfig_pfsync0=”up syncif fxp0″

pf_enable=”YES”
pf_rules=”/etc/pf.conf”
pf_flags=”"
pflog_enable=”YES”
pflog_logfile=”/var/log/pflog”
pflog_flags=”"

C.pf.conf规则
master和slave除了网卡标识不一样.其他是一致的. 
################################################################################
# Macro and lists
################################################################################
lop_if = ”lo0″
ext_if = ”em0″
int_if = ”em1″
sync_if= ”vr0″
ext_carp = ”carp0″

web_ports = ”{ 80, 443 }”
#web_servers = ”{ 192.168.10.20, 192.168.10.21, 192.168.10.22 }”
#web_servers = ”{ 192.168.10.20 }”
web_servers = ”{ 192.168.10.20, 192.168.10.21 }”

################################################################################
# Options, scrub and NAT
################################################################################
set block-policy drop
set skip on $lop_if

scrub in

nat on $ext_if from $int_if:network to any -> $ext_if

################################################################################
# Redirection
################################################################################
#rdr on $ext_if proto tcp from any to any port 80 -> $web_servers round-robin sticky-address
rdr on $ext_if proto tcp from any to $ext_carp port $web_ports -> $web_servers round-robin sticky-address

################################################################################
# Filtering Rules
################################################################################

pass quick on { $sync_if } proto pfsync keep state (no-sync)
pass on { $ext_if,$int_if } proto carp keep state

D. sysctl.conf的相关配置
#master和slave 一样. 
net.inet.carp.preempt=1

net.inet.tcp.blackhole=2
net.inet.udp.blackhole=1

net.inet.tcp.sendspace=65536
net.inet.tcp.recvspace=65536

E. 配置好后重启服务器.检查系统状况
master上
master# ifconfig
vr0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=8<VLAN_MTU>
        ether 00:05:5d:85:84:d8
        inet 10.10.10.250 netmask 0xffffff00 broadcast 10.10.10.255
        media: Ethernet autoselect (100baseTX <full-duplex>)
        status: active
em0: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=9b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM>
        ether 00:c0:9f:31:25:a2
        inet 192.168.1.52 netmask 0xffffff00 broadcast 192.168.1.255
        media: Ethernet autoselect (100baseTX <full-duplex>)
        status: active
em1: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=9b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM>
        ether 00:c0:9f:31:25:a3
        inet 192.168.10.10 netmask 0xffffff00 broadcast 192.168.10.255
        media: Ethernet autoselect (100baseTX <full-duplex>)
        status: active
pflog0: flags=141<UP,RUNNING,PROMISC> metric 0 mtu 33204
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
        inet 127.0.0.1 netmask 0xff000000 
pfsync0: flags=41<UP,RUNNING> metric 0 mtu 1460
        pfsync: syncdev: vr0 syncpeer: 224.0.0.240 maxupd: 128
carp0: flags=49<UP,LOOPBACK,RUNNING> metric 0 mtu 1500
        inet 192.168.1.51 netmask 0xffffff00 
        carp: [color=Red]MASTER[/color] vhid 1 advbase 1 [color=Red]advskew 10[/color]
carp1: flags=49<UP,LOOPBACK,RUNNING> metric 0 mtu 1500
        inet 192.168.10.100 netmask 0xffffff00 
        carp: [color=Red]MASTER[/color] vhid 1 advbase 1 [color=Red]advskew 10[/color]

slave上:  
slave# ifconfig
fxp0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=8<VLAN_MTU>
        ether 00:07:e9:1b:4b:cd
        inet 10.10.10.251 netmask 0xffffff00 broadcast 10.10.10.255
        media: Ethernet autoselect (100baseTX <full-duplex>)
        status: active
em0: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=9b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM>
        ether 00:c0:9f:38:bd:af
        inet 192.168.1.53 netmask 0xffffff00 broadcast 192.168.1.255
        media: Ethernet autoselect (100baseTX <full-duplex>)
        status: active
em1: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=9b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM>
        ether 00:c0:9f:38:bd:b0
        inet 192.168.10.11 netmask 0xffffff00 broadcast 192.168.10.255
        media: Ethernet autoselect (100baseTX <full-duplex>)
        status: active
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
        inet 127.0.0.1 netmask 0xff000000 
pflog0: flags=141<UP,RUNNING,PROMISC> metric 0 mtu 33204
pfsync0: flags=41<UP,RUNNING> metric 0 mtu 1460
        pfsync: syncdev: fxp0 syncpeer: 224.0.0.240 maxupd: 128
carp0: flags=49<UP,LOOPBACK,RUNNING> metric 0 mtu 1500
        inet 192.168.1.51 netmask 0xffffff00 
        carp: [color=Red]BACKUP[/color] vhid 1 advbase 1 [color=Red]advskew 20[/color]
carp1: flags=49<UP,LOOPBACK,RUNNING> metric 0 mtu 1500
        inet 192.168.10.100 netmask 0xffffff00 
        carp: [color=Red]BACKUP[/color] vhid 1 advbase 1 [color=Red]advskew 20[/color]

二. 服务器池的配置.
统一配置,把网关指向192.168.10.100就可以了. 
defaultrouter=”192.168.10.100″

我在二台服务器上部署了相同的应用(lighttpd + php-fcgi), 配置我就不写了. 大家GG一下. 
由于服务器不够,mysql我部署在其中的s2上.让两台服务器连到他上. 

三 测试. 
A.静态页面: 
在s1和s2上分别与建立个index.htm页面. 
s1上index.htm的内容是: 
“hi, this is No.1 server”

s2上index.htm的内容是: 
“hi, this is No.2 server”

访问地址:  http://192.168.1.51/index.htm
返回的内容分别s1和s2的. 注意: 调度上做的状态保持的. 所以需要关闭浏览器再开才分别看得到不同的内容. 建议最好是几个人一起访问来返回不同的结果. 

B. 动态页面:

在s1和s2上简单部署了phpwind.  假如在./bbs下.
访问地址: http://192.168.1.51/bbs/
返回的内容保持了一致性, 
这里要提到的是文件的共享我没做, 现实的方法可以是通过nfs和iscsi. 暂时调不出iscsi的存储, 又不想用nfs所以就不作了. :)

master# pfctl -s state
all carp 192.168.1.52 -> 224.0.0.18       SINGLE:NO_TRAFFIC
all pfsync 224.0.0.240 <- 10.10.10.251       NO_TRAFFIC:SINGLE
all pfsync 10.10.10.250 -> 224.0.0.240       SINGLE:NO_TRAFFIC
all carp 224.0.0.18 <- 192.168.10.10       NO_TRAFFIC:SINGLE
all carp 224.0.0.18 <- 192.168.1.52       NO_TRAFFIC:SINGLE
all carp 192.168.10.10 -> 224.0.0.18       SINGLE:NO_TRAFFIC
all tcp 192.168.10.21:65346 -> 192.168.1.52:61575 -> 130.104.5.67:25  FIN_WAIT_2:FIN_WAIT_2
all tcp 192.168.10.20:80 <- 192.168.1.51:80 <- 192.168.1.89:62414   TIME_WAIT:TIME_WAIT
all tcp 192.168.10.20:80 <- 192.168.1.51:80 <- 192.168.1.89:62416   FIN_WAIT_2:FIN_WAIT_2
all tcp 192.168.10.20:80 <- 192.168.1.51:80 <- 192.168.1.89:62423   FIN_WAIT_2:FIN_WAIT_2
all tcp 192.168.10.20:80 <- 192.168.1.51:80 <- 192.168.1.89:62425   FIN_WAIT_2:ESTABLISHED

C. HA的测试
拨掉master的网线, slave将自动接管. 状态保持.
ifconfig你可以看到
carp0: flags=49<UP,LOOPBACK,RUNNING> metric 0 mtu 1500
        inet 192.168.1.51 netmask 0xffffff00 
        carp: [color=Red]MASTER[/color] vhid 1 advbase 1 [color=Red]advskew 20[/color]
大家可以做些些测试了. 

最后能有些优化:)

文献参考: 
http://www.countersiege.com/doc/pfsync-carp/
http://blog.randomutterings.com/articles/2007/06/15/redundant-failover-firewall-with-pf-pfsync-and-carp-on-freebsd
原文链接:http://bbs.chinaunix.net/viewthread.php?tid=1099236



freebsd 常用命令


01.查看网络流量
   a.systat -if 1 (1表示1s刷新屏幕一次)
   b.netstat 1
   # Traffic 流量 peak 峰值 average 平均值

02.查看硬盘信息
   diskinfo -vt /dev/ad0

03.查看核心信息
   vmstat 1

04.断电后系统无法启动
   a.启动到sing user模式
   b.fsck
   c.reboot

05.硬盘分区时-shift+F –>NO可以使用全部空间

06.ctrl+alt+backspace 杀掉xwindows

07.alt+F1…F8 或 printscreensysr循环切换控制台

08.shutdown now 切换到单用户模式
   shutdown -p now shutdown -hp now 关闭电源
   shutdown -r now reboot 重新启动机器

09.升级内核后无法启动
   a.重启机器到10秒读秒时按enter以外的任意键
   b.unload
   c.boot kernel.old 或其他可以启动的内核

10.挂载ntfs,fat32分区和u盘
   ntfs: mount_ntfs -C eucCN /dev/ad0s5 /mnt/d
   fat32:mount_msdos -L zh_CN.eucCN /dev/ad0s6 /mnt/e
   u盘:  mount_msdos -L zh_CN.eucCN /dev/da0s1 /mnt/u

11.卸载分区
   umount /dev/ad0s5
   umount -f /dev/ad0s5 (强制卸载)

12.查看进程
   top   ps auxww  ps -ef |grep xxx

13.查看硬盘详细分区信息&读写状况
   gstat

14.声卡
   vi /boot/defaults/loader.conf
   更改snd_ich_load=”NO”->”YES” #intel ich
   检测声卡 kldstat 或 dmesg |grep pcm

15.ls -G 显示颜色

16.sysctl sysctl -a 查看变量

17.丢失root密码
   启动到单用户模式,执行 /sbin/mount -a,
   然后拥passwd更改密码

18.查找程序或文件
   whereis 程序名
   locate 文件名
   whatis xxx 要找东西但不知道它是什么
   grep bpf /usr/src/sys/i386/conf/GENERIC 在文件中找指定文本
   #只能对文本进行grep,如是命令,则
   command |grep text
   grep text filename

19.中文locale
   zh_CN.eucCN zh_CN.GBK zh_CN.GB18030 zh_CN.euc
   #语言代码_国家代码.字符编码
   #查看环境变量 locale

20.xmms,mplayer 乱码问题
   在用户目录下建立 .gtkrc.mine 文件.代码如下
   style “gtk-default-zh-cn”{
        f
  }
   class “GtkWidget” style “gtk-default-zh-cn”
  #前提是安装 simsun 字体,可参照以下方法安装。
  #拷贝 Windows系统下的 simsun.ttc 字体文件到 /usr/X11R6/lib/X11/fonts/TrueType/,没有的话创建。
   # mkfontdir /usr/X11R6/lib/X11/fonts/TrueType
   # fc-cache -fv

21.停止sendmail
   vi /etc/rc.conf 加入
   sendmail_enable=”NONE”

22.you have mail
   mail 回车,输入1看第一封邮件…输入?看帮助

23.traceroute &ping的整合工具my traceroute /mtr
   /usr/ports/net/mtr
   使用: mtr www.163.com

24.本地化设置,以系统默认的csh为例
   vi /etc/csh.cshrc 加入
   setenv LANG zh_CN.eucCN (语言设为中文)
   setenv LC_ALL zh_CN.eucCN (所有本地子集设为中文)
   setenv XMODIFIERS @im=fcitx (X下的中文输入法)
   set prompt = “%B%m<%/>”(个性化控制台%B颜色%m主机%/目录)
   set autolist  (自动补齐扩展)
   set nobeep    (告诉应用程式避免发出哔哔声)
   set filec     (使档案名称在命令列被完成)
   alias ls        ls -FG (彩色化ls)

25.在/etc/fstab中挂载fat32分区
   /dev/ad0s5  /mnt/d  msdos  rw,-L=zh_CN.eucCN 2 2

26.使用gnome2的gdm做桌面管理器
   vi /etc/rc.conf 加入
   gdm_enable=”YES”
   vi /root/.dmrc 更改Language=zh_CN.UTF8–>zh_CN.eucCN

27.远程连接windows终端
   rdesktop [-a 16 -f] server’IP[:port]

28.播放APE格式音乐
   /usr/ports/multimedia/xmms
   /usr/ports/audio/xmms-mac

29.看图
   /usr/ports/graphics/xnview

30.fcitx小企鹅输入法(有关步骤见24)
   /usr/ports/chineses/fcitx
   vi /root/.xinitrc 添加
   exec fcitx& 到最前一行

31.apache22+php5+mysql+postgresql+ZendOptimizer
   a,apache22:
     cd /usr/ports/www/apache22
     make install clean
   b,php5:
     cd /usr/ports/www/mod_php5
     make install clean (勿选debug,否则不能装ZendOptimizer)
   c,php5模块:
     cd /usr/ports/lang/php5-extesions
     make install [FORCE_PKG_REGISTER=yes] 强迫安装
     make clean
     (除预设项目还应选中CTYPE,EXIF,GD,ICONV,IMAP,PCRE,SESSION,ZLIB)
   d,cp /usr/local/etc/php.ini.recommended /usr/local/etc/php.ini
   e,ZendOptimizer
     cd /usr/ports/devel/ZendOptimizer
     make install clean
   f,mysql:
     cd /usr/ports/databases/mysql41-server
     make WITH_CHARSET=gbk WITH_XCHARSET=all SKIP_DNS_CHECK=yes install clean
     cd /usr/ports/databases/php5-mysql
     make install clean
   g,postgresql
     cd /usr/ports/databases/postgresql81-server
     make install clean
     cd /usr/ports/databases/php5-pqsql
     make install clean

32.怎样刷新环境变量值,而不用推出再登陆?
   a,比如要重新设置环境变量ABC的值为abcdef
     bash$ export ABC=abcdef
     csh% setenv ABC abcdef
   b,更新了PATH变量指定目录里可执行命令
     rehash
   c,更新配置文件
     比如你编辑了.cshrc等文件,就需要用source命令
     source .cshrc

33.磁盘空间使用情形    df 和 du
   指令df用来查看整个档案系统的使用情形,如果您需要知道某个目录的使用情形用指令du。
   df -h
   du -h /etc
   du -sh /etc
   -s 参数来省略指定目录下的子目录
   -h 表示使用GB、MB等易读的格式

34.检视软件间的关联
   cd /usr/ports/sysutils/pkg_tree
   make install clean
   之后我们就可以使用 pkg_tree | more 来看各个软件之间的关系了

35.使用 portupgrade 更新软件
   cd /usr/ports/sysutils/portupgrade
   make install clean
   如果您只想要升级某一个软件本身,而非所有相依的软件,只要使用 portupgrade “pkgname” 即可。
   如果您要升级所有和该软件相依的其它软件,则可以加上参数 -r。如果您要更新包含相依软件的相依软件,
   则可以使用参数 -R。另外,您可以使用参数 -m 来要求编译 (make) ports 时多传入一些参数,还可以
   使用参数 -f 要求就算版本一样也强制更新。portupgrade 在更新软件时,会保留您原本的设定档。例如,
   在更新 Apache 时,它会保留 httpd.conf 的设定。
   假设我们要更新的软件是 perl-5.8,我们可以使用下列指令来更新:
   # portupgrade -rf “perl-5.8.*” -m “ENABLE_SUIDPERL=yes”

36.ports安装失败提示
   stop in /usr/ports/lang/ruby18/work
   删除work目录下所有文件&文件夹再次安装即可.

37.ntop–网络监控软件
   cd /usr/ports/net/ntop
   make install clean
   vi /etc/rc.conf 加入
   ntop_enable=”YES”
   在地址栏中输入http://x.x.x.x:3000 即出来管理界面.

38.修改/etc/rc.conf不重启生效
   1.sh /etc/rc
   2./etc/netstart

39.freebsd启动的时候停到blanktime的解决办法
   可以试试 这个参数:blanktime=”NO”
   etc/rc.conf 中把 default_router 注释掉
   #未验证!!!

40.Bsd 下的QQ
   eva不错,运行速度飞快。
   /usr/ports/net-im/eva

41.一次性把所有安装的ports删除掉
   a.pkg_delete -a
   b.安装pkg_cutleaves它会告诉你,哪些package没有人依赖,并让你决定要不要删除

42.pure-ftp安装
   cd /usr/ports/ftp/pure-ftpd/
   make WITH_LANG=simplified-chinese install clean
   [X] BANNER   Show pure-ftpd welcome upon session start
   cd /usr/local/etc/
   cp pure-ftpd.conf.sample pure-ftpd.conf
   vi /etc/rc.conf –>加入pureftpd_enable=”YES”

43.开机自动校准时间
   vi /etc/rc.conf—>加入
   ntpdate_enable=”YES”             # Run ntpdate to sync time on boot (or NO).
   ntpdate_flags=”207.46.232.189″   # time.windows.com

44.用win2000的引导器启动FreeBSD
   我在笔记本上装了双系统,win2000 和freebsd,装完 freebsd后 系统自动设置如下:
   f1 dos
   f2 freebsd
   先起动到dos下,fdisk /mbr,再把FreeBSD光盘上的boot\boot1复制到c:\,再编辑c:\boot.ini
   加一行c:\boot1=”FreeBsd”

45.安装mplayer
make WITH_GUI=yes WITH_LANG=zh_CN WITH_FREETYPE=yes WITHOUT_RUNTIME_CPUDETECTION=yes install clean


Frsky’s blog . is proudly powered by WordPress and themed by Mukkamu