Enabling SR-IOV on ConnectX3 card

Requires mstconfig from package mstflint

# mstconfig -d 04:00.0 q
# mstconfig -d 04:00.0 set SRIOV_EN=1 NUM_OF_VFS=16

Reboot to activate new firmware settings

srun: error: PMK_KVS_Barrier duplicate request from task 0

If you are using Intel MPI within a SLURM session and receive such error on mpirun, make sure I_MPI_PMI_LIBRARY is not set.

Slurm accounting association

Setup Slurm to use slurmdb and setup slurmdbd according to accounting guide.
Enforce accounting association.

# grep -e AccountingStorage slurm.conf
AccountingStorageEnforce=associations
AccountingStorageType=accounting_storage/slurmdbd

Once configured, accounts need to be properly associated in SlurmDBD in order to submit jobs.

Setup account association:

sacctmgr: add cluster Name=shannon
sacctmgr: add account Name=em
sacctmgr: add user Name=elton Cluster=shannon account=em
sacctmgr: list association
   Cluster    Account       User  Partition     Share GrpJobs       GrpTRES 
---------- ---------- ---------- ---------- --------- ------- ------------- 
   shannon       root                               1                       
   shannon       root       root                    1                       
   shannon         em                               1                       
   shannon         em      elton                    1