Florian's weblog

There are two guides out there that help understand the concepts and syntax: “Cluster from scratch” [1], and “Cluster configuration explained” [2], though, there are certain subtleties I had a difficult time to find and/or understand, that’s why I decided to share my poor experience. IRC Freenode #linux-ha is a good place to ask for help too.

1. About ocf:pacemaker:ping resource, in order to monitor the real score associated with each node by the ping resource, you’d have to use:
   • cibadmin -Q | grep pingd | grep value
   • crm_mon -frontA1 | grep ping

To prevent moving resources on loss of a common ping node, you might want to have
dampen >= 2*ping.op_monitor-interval. Read doc[2] for dampen explanations.

2. Location constraints based on connectivity have to use the ocf:pacemaker:ping resource’s name, not the primitive id. Most of the howtos out there to create a ping resource don’t fill the name parameters but only the primitive’s id (reminder: primitive id class:provider:type params name=foo host_list=...). With an empty name, you have to use the default name for an ocf:pacemaker:ping resource which is pingd.
   

   location IPHA-on-connected-node IPHA \
       rule $id="IPHA-on-connected-node-rule" pingd: defined pingd
   

   This constraint (with a score of pingd: instead of +/-INF:) is explained in a good blog entry that summarize ping scoring behavior, syntax and formula. To understand ping scoring, you must read link[3].

3. If you want to receive SNMP traps whenever a resource changes state, you should create an ocf:heartbeat:ClusterMon resource:
   

   primitive SNMPMonitor ocf:heartbeat:ClusterMon \
       params pidfile="/var/run/crm_mon.pid" extra_options="-S 192.168.1.2 -C public" \
       op monitor on-fail="restart" interval="10s"
   

4. The <op> tag is used to define parameters for operations performed by the cluster such as starting or stopping a resource. Eg, you can tell pacemaker that one of your resources takes a long time to start using <op start timeout="3min" ...> (same goes for stop of course). If you don’t, pacemaker will decide your resource has failed because of the default built-in timeout for the start operation ! (see point number 5 below for a concrete example). Finally, the interval parameter is only used for repetitive operations, the only one right now beeing monitor :
   
   

   primitive firewall lsb:my-complex-firewall-initscript \
       op monitor on-fail="restart" interval="10s" \
       op start interval="0" timeout="3min" \
       op stop interval="0" timeout="1min" \
       meta target-role="Started"
   




5. Prior to CentOS 6.2 (I haven’t been able to find the BZ#id in the release notes…), there is an uneeded and bugged check in the shell code of the ip_stop() function in ocf:heartbeat:IPaddr (/usr/lib/ocf/resource.d/heartbeat/IPaddr).
   When trying to stop such a resource, before deleting the alias, the command if route | grep $IP ; then ... will screw your cluster in two case: your node has a really really big local routing table (BGP ?) or you don’t have any DNS resolver reachable.
   The failure will happen because route will take more than 20 seconds which is the default timeout for a stop action. The resource will have an INFINITY failcount and go unmanaged, if it’s part of a bigger shutdown process, it will break here and other node(s) won’t be able to pick up resources: EPIC FAILURE.
   In the process of fixing this issue, route has first been replaced with route -n which is indeed way faster but can also require more than 20 seconds to be browsed (for example a BGP router can have up to 350K lines), then, it’s been totally removed because it’s totally useless: problem solved. So, you can either: update to 6.2, patch your IPaddr shell script, patch from GitHub, move to IPaddr2. [4]




6. By default, when a failed node comes back online it claims back it’s old resources, meaning they are moved, again. You can avoid this by setting a non-zero resource-stickiness.

[1] http://www.clusterlabs.org/doc/en-US/Pacemaker/1.1/html-single/Clusters_from_Scratch/
[2] http://www.clusterlabs.org/doc/en-US/Pacemaker/1.1/html-single/Pacemaker_Explained/
[3] http://www.woodwose.net/thatremindsme/2011/04/the-pacemaker-ping-resource-agent/
[4] http://floriancrouzat.net/2012/01/pacemaker-moving-from-ipaddr-to-ipaddr2/

There are two possible configurations:

1. the syslog one (more elegant)
2. the file one.

The purpose of the article is to help you create a bind-chroot configuration to separate daemon messages from users queries.

Note that with CentOS 6 bind chrooting is done by mount points, see mount | fgrep named while named is started ; you must not symlink configurations files and you should edit files directly under /etc. Though, you should not put your configuration directly in /etc/named.conf but include your own files under /etc/named/ using the include directive.

With a chrooted bind, since log files are physically placed under /var/named/chroot/var/log/, a lazy admin would want to symlink them in /var/log/:

ln -sf /var/named/chroot/var/log/dns.log /var/log/dns.log
ln -sf /var/named/chroot/var/log/dns_queries.log /var/log/dns_queries.log

1. Logging to syslog

1.1 Configure named for syslog

• /etc/named.conf: Remove the logging{} block and include your own file

[...]
include "/etc/named/named.conf.local";

• /etc/named/named.conf.local:

[...]
// Do not print-time, it's redundant with syslog header.
logging {
        channel log_dns {
                syslog local3;
                print-category yes;
                print-severity yes;
                print-time no;
        };
        channel log_queries {
                syslog local4;
                print-category yes;
                print-severity yes;
                print-time no;
        };
        category default {log_dns;};
        category queries {log_queries;};
        category lame-servers { null;};
        category edns-disabled { null; };
};

1.2 Configure rsyslog

• /etc/rsyslog.conf:

# Don't forget to add the chrooted log socket.
$AddUnixListenSocket /var/named/chroot/dev/log
if $syslogfacility-text == 'local3' then /var/named/chroot/var/log/dns.log
& ~
if $syslogfacility-text == 'local4' then /var/named/chroot/var/log/dns_queries.log
& ~

1.3 Configure logrotate

• /etc/logrotate.d/named:

Rsyslog must be reloaded before named in order to create the new empty log files after rotation.

/var/named/chroot/var/log/dns.log
/var/named/chroot/var/log/dns_queries.log
/var/named/data/named.run {
    missingok
    create 0644 named named
    postrotate
        /bin/kill -HUP `cat /var/run/syslogd.pid 2> /dev/null` 2> /dev/null || true
        /sbin/service named reload  2> /dev/null > /dev/null || true
    endscript
}

You’re all set.

2. Logging to files

2.1 Configure named

• /etc/named.conf: Remove the logging{} block and include your own file

[...]
include "/etc/named/named.conf.local";

• /etc/named/named.conf.local:

[...]
logging {
        channel log_dns {
                file "/var/log/dns.log" versions 3 size 10m;
                print-category yes;
                print-severity yes;
                print-time yes;
        };
        channel log_queries {
                file "/var/log/dns_queries.log" versions 3 size 20m;
                print-category yes;
                print-severity yes;
                print-time yes;
        };
        category default {log_dns;};
        category queries {log_queries;};
        category lame-servers { null;};
        category edns-disabled { null; };
};

You’re all set.

As you should guess, one can find a direct link to video (mp4 files stored at Akamai) directly in Facebook video page source code.
Unfortunatly, it’s encoded in a way that prevent copying and pasting the URL directly in your browser. Here’s a stupid sed one-liner to decode the URL and download (wget) the video.

• Open a video page in facebook ;
• Read the source code and search for either, highqual_src, lowqual_src or video_src ;
• Input the newly found URL in the following one-liner:

wget $(echo "<encoded_mp4_url>" | sed -e 'sx\\\u00253Ax:xg;sx\\\u00252Fx/xg;sx\\\u00253Fx?xg;sx\\\u00253Dx=xg;sx\\\u002526x\&xg')