The state policy in reply-to rules must be floating otherwise the state wont match to the reply packet that tries to go out from wrong interface e.g. due to that the host has a default route that is bound to other interface.
pass in reply-to (xl0 10.0.0.1) proto tcp from any to 10.0.0.10 \
port 25 flags S/SA keep state (floating)