Build & install notes for BLCR & SLURM on CentOS 7.3

Install build tools & dependencies; EPEL required for munge-devel:

# yum -y groupinstall "Development Tools"
# yum -y install epel-release
# yum -y install gtk2-devel hwloc-devel libibmad-devel libibumad-devel lz4-devel mariadb-devel munge-devel ncurses-devel numactl-devel openssl-devel pam-devel perl-ExtUtils-MakeMaker readline-devel rrdtool-devel

BLCR (optional manual dependency):

Official web site provides release up to 0.8.5. Later versions that works with CentOS 7 (kernel 3.10.0) can be found in beta and snapshot distribution directory.

# yum -y install glibc-devel.i686 kernel-devel-`uname -r` libgcc.i686
# wget
# tar zxvf blcr-0.8.6_b4.tar.gz

acinclude.m4 needs to be patched because ./configure ignores weak symbols in System map and it fails to locate a working `rpmbuild`

You may see what changes are needed in this GitHub commit. (Ah yes… You may actually clone / download ZIP and build from there. One more copy of the diff is backed up at this paste.)
After patching acinclude.m4, prepare build environment by running

# ./
# ./configure
# make rpms

Upon successful build, you’ll have RPMs at rpm/RPMS/x86_64; install all RPMs before we go on to test it:

# modprobe blcr_imports
# modprobe blcr
# /usr/libexec/blcr-testsuite/RUN_ME

You have successfully installed BLCR once all tests pass.
The kernel modules for blcr shall load automatically by its init script.

Build SLURM from tar ball:

Download latest tar ball from official web site:

# wget
# rpmbuild -ta slurm-17.02.3.tar.bz2
# cd rpmbuild/RPMS/x86_64/

Install RPMs:

  • slurm
  • slurm-devel
  • slurm-munge
  • slurm-perlapi
  • slurm-plugins
  • slurm-sjobexit (only prior to version 17.02)
  • slurm-sjstat (only prior to version 17.02)
  • slurm-torque

See RPMS INSTALLED section for more details.


Generate slurm.conf using web configurator.
Note StateSaveLocation defaults to /var/spool; this is not ideal since user slurm needs to write to the directory. I personally use /var/spool/slurmctld
For details:man slurm.conf

And a few manual steps before you have a working Slurm installation:

  1. Populate /etc/munge/munge.key on all nodes, enable and start service; key should be owned by munge:munge, mode 0600
  2. Disable or open necessary ports (6817-18) on firewalld
  3. CgroupAutomount=yes in /etc/slurm/cgroup.conf
  4. Enable and start services for slurmctld & slurmd

Basic slurm.conf example:

NodeName=worker[1-8] Sockets=4 CoresPerSocket=1 ThreadsPerCore=1 State=UNKNOWN 
PartitionName=debug Nodes=worker[1-8] Default=YES MaxTime=INFINITE State=UP

See full example of above config file as generated by configurator.html.

Build SLURM from GitHub repo (advanced / incomplete):

# git clone

Switch to release tag:

# cd slurm
# git checkout slurm-17-02-3-1

Configure for build:

# ./configure --enable-pam

You’ll have to populate some values in slurm.spec before it can be build using rpmbuild.

Intel compiler flags to use when compiling large static matrix with Himeno benchmark

When you compile Himeno benchmark (C + MPI, static allocate version), you can predict the amount of memory required by each MPI process: 56byte x Grid size / #PE
If the memory size is greater than 2GB, you may encounter the following error:

relocation truncated to fit: R_X86_64_32S against `.bss'

Some extra Intel compiler flags are needed to enable static allocation of such large array:

# cat Makefile
OBJS = himenoBMTxps.o

# Use mpiicc instead of mpicc; mpicc wraps around gcc 
CC = mpiicc
RM = /bin/rm
PROG = bmt

# Note the following extra flags:
# -mcmodel large
# -shared-intel
# -fpic
CFLAGS = -O3 -mcmodel large -shared-intel -fpic

all : ${PROG}

${PROG} : ${OBJS} param.h
	${CC} -o $@ ${OBJS} ${LDFLAGS}

.c.o : param.h
	${CC} -c ${CFLAGS} $<

clean :
	${RM} -f ${PROG} ${OBJS} core

Himeno benchmark using PGI Compiler & OpenMPI on CentOS 7

Download PGI Community Edition and install all components, we are using 17.4 in this example.

Download Himeno benchmark, we are using “C + MPI, static allocate version" in this example.

To extract lzh on CentOS, you’ll need to install p7zip & pz7ip-plugins from EPEL

# yum -y install epel-release
# yum -y install p7zip p7zip-plugins

Since it extracts to CWD, probably a good idea to put the lzh archive in a new directory before extraction. Extract using command:

# 7z x cc_himenobmtxp_mpi.lzh

Set your bash variables to use PGI Compilers and accompanying OpenMPI library:

# export MPI=/opt/pgi/linux86-64/17.4/mpi/openmpi
# export MANPATH=$MANPATH:$MPI/man
# export PATH=/opt/pgi/linux86-64/17.4/bin:$MPI/bin:$PATH
# export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:$MPI/lib:/opt/pgi/linux86-64/17.4/lib

Inside your himeno source directory, copy Makefile.sample to Makefile.
Since this is static allocate version, we have to use to configure param.h:

# ./ L 1 2 2

I use this for my 4-core Atom box, adjust for your own case accordingly.

`make` yields “bmt"; launch it using `mpirun`:

# mpirun -n 4 ./bmt

Run again to change test parameters. Be sure to `make clean` before building new binary.

Ansible Playbook conditional on task result

Unlike CentOS or RHEL, sar does not by default keep history on Ubuntu. The following snippet enables it.

- name: enable sysstat data collection
    dest: /etc/default/sysstat
    regexp: '^ENABLED='
    line: 'ENABLED="true"'
    state: present
  register: sadcenabled
- name: restart sysstat service (ondemand)
  service: name=sysstat state=restarted
  when: sadcenabled.changed

Only when we have added ‘ENABLED="true"‘ line shall we restart sysstat service.

Or you may use handler, which I should try soon.

vxlan on Fedora / OpenWrt / OpenBSD

This post contains 3 snippets for setting up vxlan on Fedora 25, OpenWrt 15.05.1 (Chaos Calmer) & OpenBSD 6.0.
Update 2017-04-15: Edited pf.conf for OpenBSD
Update 2017-04-16: A similarly setup Fedora host can learn others MAC when being pinged; while OpenBSD still need to ping other hosts to learn their MAC address. (Edit performed in last edit might not work?) Still requires more investigation. 繼續閱讀