As a technology preview, there is no additional cost for using the Phi's. All that is required is an active Flux allocation and users can test the Phi's. The only other requirement is all Phi jobs must be less than 24 hours long.
The Phi cards support three modes of operation, Automatic Offload, Compiler Assisted Offload, and Native. The first two are well tested, the last works but is not as well tested. All intel-comp compiler and mkl math library modules on Flux support the Phi.
You can request a phi with PBS with:
qsub -I -l nodes=1:mics=2 -q flux -A account_flux -l qos=flux -V
This will provide two Phi's and one CPU. Flux currently has one node with 8 Phi cards.
PBS will do set two variables:
PBS_MICFILE -> list of hostnames of assigned phi's good for native mode.
OFFLOAD_DEVICES -> csv list of devices for controlling auto offload (MKL) or compiler assisted offload.
$HOME is mounted on the card, as is all the software but /scratch currently is not, it should sometime in the future. This should only affect users running Native Phi code.
For software the Phi requires some environment changes to work. We created a module called xeon-mic. When loaded it will set some sane defaults but will print what it set. Users are encouraged to experiment with many of the settings available.
module load xeon-mic
In future posts we will show examples using the Phi.
