Categories
FreeBSD/Unix

Enabling SSH access to WD My Book World Edition

The WD My Book World edition that can be connected to LAN actually runs Linux inside. Log in to the NAS using the web interface. Next go to the “Advanced Mode” -> “Advanced”, tick the tick box in front of “SSH Access” and press submit. Now you can ssh to the NAS as root. The default password is welc0me. Password can be changed using passwd command.

Categories
FreeBSD/Unix

Recovering GPT partitions with gpart

If one manages to destroy the GPT scheme partition tables you’ll still have few options to recover them. Since FreeBSD 9.x the default used by the installation CD is GPT. Just grab the FreeBSD installation CD and boot the system using that. When it boot’s up it gives you options to install, shell, and live CD. Select Live CD and log in as root.

Now if you have previously made a backup of the partition table using gpart backup command then you are lucky and most likely few shell commands away from getting the system back online. If not then you have more things to do.

The backup created by gpart backup command looks like this (backup_partition_table.ada0):

GPT 128
1   freebsd-boot         40        128  
2    freebsd-ufs        168 5851053944  
3   freebsd-swap 5851054112    8388607 

Restoring partitions and bootcode from backup:

gpart restore -F ada0 < backup_partition_table.ada0
gpart bootcode -b /boot/pmbr -p /boot/gptboot -i 1 ada0

In the restore command -F option first flushes completely the existing data from the partition table and then recovers the table from the file.

In case you haven’t saved the partition table then if you have used the installation defaults then the first partition should be the freebsd-boot and the size is most likely 64k then second partition is the root partition and the third is the swap. By default the boot partition is 64K size other partitions depend on your configuration.

gpart create -s gpt ada0
gpart add -s 64k -t freebsd-boot ada0
gpart add -s 2.7T -t freebsd-ufs ada0
gpart add -t freebsd-swap ada0
gpart bootcode -b /boot/pmbr -p /boot/gptboot -i 1 ada0
Categories
FreeBSD/Unix

Dovecot: imap-login: Cannot map anonymous memory

Encountered today a problem on one of the mail servers running dovecot that after upgrade wasn’t able to login using imap and dovecot only printed in to maillog that dovecot: imap-login: Cannot map anonymous memory. It seems that this error is produced if in the dovecot.conf file the login_process_size is set to too low value. The explanation and default for the value is:

# Set max. process size in megabytes. If you don't use
# login_process_per_connection you might need to grow this.
# login_process_size = 64

So what I did is that in the dovecot.conf file I uncommented the login_process_size line and increased it to 256:

# Set max. process size in megabytes. If you don't use
# login_process_per_connection you might need to grow this.
login_process_size = 256
Categories
FreeBSD/Unix

PHP 5.4.x and UTF-8

As the PHP 5.4 release note says the default character encoding has been changed from ISO-8859-1 (latin1) to UTF-8. Although the encoding can be controlled by setting default_charset = “iso-8859-1” in php.ini. This change doesn’t seem to have an effect on functions like htmlspecialchars/htmlentities which have their own default encoding (UTF-8) and own function argument for character encoding. If you want to continue using 5.4 the only way to solve this issue is to modify all the places where this function is called to following:


$str_to_be_coded = "äÄöÖüÜ";
$str = htmlspecialchars($str_to_be_coded, ENT_COMPAT | ENT_HTML401 | ENT_SUBSTITUTE, ISO-8859-1);
echo $str;

$str = htmlentities($str_to_be_coded, ENT_COMPAT | ENT_HTML401 | ENT_SUBSTITUTE, ISO-8859-1);
echo $str;

If you omit the ENT_SUBSTITUTE then the htmlspecialchars may return null string if the string contains umlauts (äÄöÖüÜ).

The alternate solution is of course to use UTF-8 encoding but this might require (if your system wide setting is ISO-8859-1) you to configure the Apache server to use UTF-8 encoding and you to update your PHP scripts to use UTF-8 encoding inside. In addition if you are using MySQL you should set the character set with MySQL as well:


/* change character set to utf8 */
if (!$mysqli->set_charset("utf8")) {
printf("Error loading character set utf8: %s\n", $mysqli->error);
} else {
printf("Current character set: %s\n", $mysqli->character_set_name());
}

Finally the solution to which I did with one of the third party software was that I installed from FreeBSD ports the package called lang/php53 which contains the latest 5.3 branch PHP interpreter.

Categories
FreeBSD/Unix

Creating GUID Partition Table (GPT) in FreeBSD

Just wanted to share a very short how to on creating GPT partition table using part command. The reasons for using GPT instead of MBR are quite obvious as disk spaces increase. The MBR partition table restricts partition sizes to a maximum of 2.19 terabytes (2.19 × 1012 bytes) or almost exactly 2 TB (2,199,023,255,040 bytes or 4,294,967,295 (232?1) sectors × 512 (29) bytes per sector). GPT supports partition sizes up to 9.3 ZB or 8 ZiB?512 bytes (9,444,732,965,739,290,426,880 bytes or 18,446,744,073,709,551,615 (264?1) sectors × 512 (29) bytes per sector). MBR disks support only four partition table entries. If more partitions are wanted, a secondary structure known as an extended partition is necessary. Extended partitions can then be subdivided into one or more logical disks. Similarly GPT support 128 partitions on a single disk allowing much larger number of filesystems on a single device without doing extension structures.

To create an empty GPT partition table run:

# gpart create -s gpt mfid1

You can check the result:

# gpart show mfid1
=> 34 5857345469 mfid1 GPT (2.7T)
34 5857345469 - free - (2.7T)

Now create first a 50G partition:

# gpart add -t freebsd-ufs -s 50G mfid1
mfid1p1 added

Then second partition that uses rest of the drive

# gpart add -t freebsd-ufs mfid1
mfid1p2 added

And the result:

# gpart show mfid1
=> 34 5857345469 mfid1 GPT (2.7T)
34 104857600 1 freebsd-ufs (50G)
104857634 5752487869 2 freebsd-ufs (2.7T)

Then just initialize the file systems:

# newfs /dev/mfid1p1
...
# newfs /dev/mfid1p2
...

Categories
FreeBSD/Unix

Apache & Subversion mod_dav_svn configuration

First one should naturally install subversion and apache in to the system and while installing make sure that mod_dav option is enabled in apache package and that mod_dav_svn option is enabled in subversion.

There is two ways to setup the mod_dav_svn with apache depending on whether you plan to have one or multiple repositories within single URI. For single repository the you should add in to httpd.conf the following:

## SVN WebDAV Repository Setup
<Location /svn>
     DAV svn
     SVNPath /home/svn/repos
     SVNIndexXSLT "http://www.myservername.com/svnindex.xsl"
     
     # authenticating them valid ones
     Satisfy All
     Require valid-user
     AuthType Basic
     AuthName "Subversion Repositories"
     AuthUserFile /home/svn/access/users
</Location>

For multiple repositories add following in to httpd.conf:

## SVN WebDAV Repository Setup
<Location /svn>
     DAV svn
     SVNParentPath /home/svn/repos
     SVNIndexXSLT "http://www.myservername.com/svnindex.xsl"
     
     # authenticating them valid ones
     Satisfy All
     Require valid-user
     AuthType Basic
     AuthName "Subversion Repositories"
     AuthUserFile /home/svn/access/users
</Location>

The default svnindex.xsl and svnindex.css can be found and copied from the subversion source package under subversion-<VERSION-NUMBER>/tools/xslt/.

Create the users using the htpasswd utility when adding the first user add -c after the htpasswd command in order to create the password file and for the following users you should remove the -c option:

# htpasswd /home/svn/access/users <username>

Building repository in the single repository configuration. The repository needs to be writable by the user or group to which the apache process belongs to.

# svnadmin create /home/svn/repos
# svn import /home/<where_ever_you_have_initial_files> \
    file:///home/svn/repos -m "initial import"
# cd /home/svn
# chown -R www repos

Building repository in the multiple repository configuration. The repository needs to be writable by the user or group to which the apache process belongs to.

# svnadmin create /home/svn/repos/<PROJECT_NAME>
# svn import /home/<where_ever_you_have_initial_files> \ 
    file:///home/svn/repos/<PROJECT_NAME> -m "initial import"
# cd /home/svn/repos
# chown -R www <PROJECT_NAME>

In the single repository configuration you can access the repository through web browser simply by writing in to the location bar the http://www.myservername.com/svn, but in the multiple repository configuration you must also add the project name after e.g. http://www.myservername.com/svn/<PROJECT_NAME> simply accessing the parent directory will result in to permission denied page to displayed so listing of repositories is not possible.

Categories
FreeBSD/Unix

pf rtables and setfib in FreeBSD

If one has multiple outgoing links to which one would like to use different routing tables the FreeBSD provides possibility through the setfib command but in order to have multiple routing tables one has to first compile a custom kernel with option ROUTETABLES in example simple kernel config:


include GENERIC
options         ROUTETABLES=4

After the kernel has been built and rebooted the different routing tables can be accessed as shown in the setfib(1) man page by issuing command setfib 0 netstat -rn. 0 is the default routing table.

After this one has to create the second routing table by prepending every route add command with setfib 1 route add… e.g:

# setfib 1 route add -net default 10.0.0.1

With packet filter one can control how the routing table is selected by using rtable option but it should be noted that this selection can only be done on the input of the packets as the routing decision is done at the input not at the output. Here is an example of very simple pf.conf that uses rtable rules and NATs everything to the external interface address:

#
# Macros
#
INT_IF = "em0"
EXT_IF = "bge0"
EXT_IF2 = "bge1"

table <private_nets> persist { 127/8, 172.16/12, 192.168/16, 169.254/16 }

#
# Options and default policy
#
set block-policy drop
set state-policy if-bound

#
# Packet normalization
#
scrub in                          all
scrub out on $EXT_IF all random-id
scrub        on $EXT_IF all reassemble tcp

#
# NAT/redirects
#

# NAT
nat on $EXT_IF from <private_nets> to any -> ($EXT_IF)
nat on $EXT_IF2 from <private_nets> to any -> ($EXT_IF2)

#
# Filter rules
#
pass all
pass in from 192.168.100.0/24 to any rtable 0
pass in from 192.168.150.0/24 to any rtable 1
Categories
FreeBSD/Unix

Gmirror device and S.M.A.R.T. Current_Pending_Sector

I woke up this morning on that the smartd was reporting an error: “smartd[1209]: Device: /dev/ad3, 1 Currently unreadable (pending) sectors” from one of the physical devices that was connected to my mirrored device. When I googled I found explanation:

Current count of unstable sectors (waiting for remapping). The raw value of
this attribute indicates the total number of sectors waiting for remapping. 
Later, when some of these sectors are read successfully, the value is 
decreased. If errors still occur when reading some sector, the hard drive will 
try to restore the data, transfer it to the reserved disk area (spare area) and 
mark this sector as remapped. If this attribute value remains at zero, it 
indicates that the quality of the corresponding surface area is low.

smartctl -a /dev/ad3 showed the following:

...
196 Reallocated_Event_Count 0x0032   200   200   000    Old_age   Always       -       0
197 Current_Pending_Sector  0x0032   200   200   000    Old_age   Always       -       1
198 Offline_Uncorrectable   0x0030   200   200   000    Old_age   Offline      -       0
...

So what needed to be done was to force the drive to write the area with new data and thus getting the sector either remapped or cleared from the error. As the disk was a mirrored drive and the other drive was working properly (+ I have backups in case of horror) I decided that I can remove the drive from the mirror using gmirror remove. Then I ran a long test with smartctl -t long /dev/ad3 and it reported no errors. After that I inserted the drive back to the mirror and as the drive was resynchronized S.M.A.R.T. Current_Pending_Sector value was decreased:

...
196 Reallocated_Event_Count 0x0032   200   200   000    Old_age   Always       -       0
197 Current_Pending_Sector  0x0032   200   200   000    Old_age   Always       -       0
198 Offline_Uncorrectable   0x0030   200   200   000    Old_age   Offline      -       0
...

The drive didn’t increase the Reallocated_Sector_Ct.

——

Update: Although I managed to correct the pending sectors couple of days after the first error occurred the drive started to fail totally. So tweaking and fixing was not very helpful as the errors started to escalate. In conclusion I decided to change the drive.

Smartd status just before I removed the drive:
smartd[1209]: Device: /dev/ad3, 348 Currently unreadable (pending) sectors
smartd[1209]: Device: /dev/ad3, 257 Offline uncorrectable sectors

Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME          FLAG     VALUE WORST THRESH TYPE      UPDATED  WHEN_FAILED RAW_VALUE
  1 Raw_Read_Error_Rate     0x002f   200   200   051    Pre-fail  Always       -       1
  3 Spin_Up_Time            0x0027   239   238   021    Pre-fail  Always       -       1050
  4 Start_Stop_Count        0x0032   100   100   000    Old_age   Always       -       54
  5 Reallocated_Sector_Ct   0x0033   200   200   140    Pre-fail  Always       -       0
  7 Seek_Error_Rate         0x002e   200   200   000    Old_age   Always       -       0
  9 Power_On_Hours          0x0032   084   084   000    Old_age   Always       -       12022
 10 Spin_Retry_Count        0x0032   100   253   000    Old_age   Always       -       0
 11 Calibration_Retry_Count 0x0032   100   253   000    Old_age   Always       -       0
 12 Power_Cycle_Count       0x0032   100   100   000    Old_age   Always       -       52
192 Power-Off_Retract_Count 0x0032   200   200   000    Old_age   Always       -       51
193 Load_Cycle_Count        0x0032   200   200   000    Old_age   Always       -       2
194 Temperature_Celsius     0x0022   124   107   000    Old_age   Always       -       23
196 Reallocated_Event_Count 0x0032   200   200   000    Old_age   Always       -       0
197 Current_Pending_Sector  0x0032   196   196   000    Old_age   Always       -       348
198 Offline_Uncorrectable   0x0030   197   197   000    Old_age   Offline      -       257
199 UDMA_CRC_Error_Count    0x0032   200   200   000    Old_age   Always       -       0
200 Multi_Zone_Error_Rate   0x0008   001   001   000    Old_age   Offline      -       24383
Categories
FreeBSD/Unix

packet filter NAT that uses ng_eiface addreses

In the previous post I described how you can create virtual interfaces which can be used to fetch unique address for each interface using dhclient(8). To use these different addresses assigned to different virtual interfaces in your packet filter NAT you can use following pf.conf. It is important to note that the last filter policy needs to be floating as the packets will flow out from the physical interface and the return packets will arrive on the virtual interface.

#
# Macros
#
INT_IF = "em0"
EXT_IF = "bge0"
VIRT_IF1 = "ngeth0"
VIRT_IF2 = "ngeth1"
VIRT_IF3 = "ngeth2"

table <private_nets> persist { 127/8, 10/8, 172.16/12, 192.168/16, 169.254/16 }

#
# Options and default policy
#
set block-policy drop
set state-policy if-bound

#
# Packet normalization
#
scrub in                          all
scrub out on $EXT_IF all random-id
scrub        on $EXT_IF all reassemble tcp

#
# NAT/redirects
#

# NAT
nat on $EXT_IF from <private_nets> to any -> \
    { ($EXT_IF), ($VIRT_IF1), ($VIRT_IF2), ($VIRT_IF3) }

#
# Filter rules
#
block log all
pass on $INT_IF all
pass out all
pass out on $EXT_IF from { ($VIRT_IF1), ($VIRT_IF2), ($VIRT_IF3) } to \
    any keep state (floating)
Categories
FreeBSD/Unix

Creating virtual ethernet interfaces using ng_eiface and ng_bridge

With generic ethernet interface netgraph node it is possible to create new interfaces that are accessible with ifconfig(8) and with ethernet bridging netgraph node type it is possible to bridge between different ethernet node types. By combining these two types it is possible to create multiple interfaces with different ethernet (MAC) address on a same physical interface. Here is sample script that is modified from the netgraph ether.bridge example script that comes with standard FreeBSD distribution:

#!/bin/sh
#
# Author: Jan Melen-jan(at)melen.org
#
# This script sets up an Ethernet bridging network across multiple
# ng_eiface(4) using the ng_bridge(4) and ng_ether(4) netgraph
# node types.
#

BRIDGE_NAME="bnet0"

VIRTUAL_IFACES="ngeth0 ngeth1 ngeth2"
LOCAL_IFACE="bge0"

# Routine to verify node's existence.
vether_verify() {
	ngctl info ${BRIDGE_NAME}: >/dev/null 2>&1
	if [ $? -ne 0 ]; then
		echo "${BRIDGE_NAME}: bridge network not found"
		exit 1
	fi
}

# Start/restart routine.
vether_start() {

	# Load netgraph KLD's as necessary.
	for KLD in ng_ether ng_bridge; do
		if ! kldstat -v | grep -qw ${KLD}; then
			echo -n "Loading ${KLD}.ko... "
			kldload ${KLD} || exit 1
			echo "done"
		fi
	done

	# Reset all interfaces.
	vether_stop

	# Verify all interfaces exist.
	LINKNUM=2
   ETHERADDR=`ifconfig ${LOCAL_IFACE} ether | grep "ether" | cut -b 8-22`
	for ETHER in ${VIRTUAL_IFACES}; do
		if ! ngctl info ${ETHER}: >/dev/null 2>&1; then
           ngctl mkpeer . eiface hook ether
		fi
       ifconfig ${ETHER} ether ${ETHERADDR}${LINKNUM}
		ifconfig ${ETHER} up || exit 1
		LINKNUM=`expr ${LINKNUM} + 1`
	done

	# Create new ng_bridge(4) node, attached to the first interface.
	ngctl mkpeer ${LOCAL_IFACE}: bridge lower link0 || exit 1
	ngctl name ${LOCAL_IFACE}:lower ${BRIDGE_NAME} || exit 1
   ngctl connect ${LOCAL_IFACE}: ${BRIDGE_NAME}: upper link1 || exit 1

	LINKNUM=2
	# Attach other interfaces as well.
	for ETHER in ${VIRTUAL_IFACES}; do
		if [ ${LINKNUM} != 0 ]; then
			ngctl connect ${ETHER}: ${BRIDGE_NAME}: \
			    ether link${LINKNUM} || exit 1
		fi
		LINKNUM=`expr ${LINKNUM} + 1`
	done

	# Set all interfaces in promiscuous mode and don't overwrite src addr.
	for ETHER in ${LOCAL_IFACE}; do
		ngctl msg ${ETHER}: setpromisc 1
		ngctl msg ${ETHER}: setautosrc 0
	done
}

# Stop routine.
vether_stop() {
	ngctl kill ${BRIDGE_NAME}: >/dev/null 2>&1
	for ETHER in ${VIRTUAL_IFACES} ${LOCAL_IFACE}; do
		ngctl kill ${ETHER}: >/dev/null 2>&1
	done
}

# Main entry point.
case $1 in
	start)
		vether_start
		;;
	stop)
		vether_verify
		vether_stop
		;;
	*)
		echo "usage: $0 [ start | stop ]"
		exit 1
esac