GLCPC Blue Waters Allocations Call Open

The Great Lake Consortium for Petascale Computation (GLCPC) is the easiest way to get access for moderate users of HPC CPU/GPU resources to gain access and use the Blue Waters super computer.

Proposals are due November 3rd, full details are available at the GLCPC website.

What is the difference between GLCPC Blue Waters allocations?
When there is the NSF Petascale Computing Resource Allocation (PRAC) program for Blue Waters?

The bar is higher for the NSF allocations, they expect users to have significant resource need and run at extreme scale.  If your work falls into this area please apply to the NSF program, which are due November 14th.  GLCPC is only granted a small portion of the Blue Waters system ~3.5 Million node hours. GLCPC is also only available to GLCPC members, which Michigan is one.

The availability of time on Blue Waters is broken down as:

• 80% or more NSF Petascale Computing Resource Program (PRAC)
• 7% University of Illinois at Urbana-Champaign
• 2% Great Lake Consortium for Petascale Computation
• 1% Teaching/Education

For most HPC users access via GLCPC is the most appropriate choice, and for a small subset the NSF Petascale program also applies.

How does GLCPC compare to XSEDE?
When should I pursue XSEDE or GLCPC?

GLCPC in the number of hours it can give away is about on par with medium to small XSEDE allocations.  Currently XSEDE also does not have a long lived large scale GPU/CUDA capable machine while Blue Waters is capable of that.

As always contact us at hpc-support@umich.edu if you have questions.

Using FFTW For Optimum Performance

FFTW is probably the most popular DFT library ever.  What many don't realize is that there are many ways to calculate an FFT depending on the number of samples.  FFTW addresses this by creating a plan and using dynamic code generation to try to quickly calculate an optimal method.

What is not realized is that the number of samples should be kept multiples of small primes, 2,3,5 etc.  Calculating an FFT where the number of samples is a single large prime is the worse case.

Eg:

Plan for 999983 samples used: 99522862.000000 add's, 53927851.000000 multiplies, 42864846.000000 fused Multiply-Add

real    0m0.873s

Plan for 999984 samples used: 41417486.000000 add's, 28948213.000000 multiplies, 6618732.000000 fused Multiply-Add

real    0m0.312s

So in this sample code going from 999983, a prime number, to 999984, who's prime factors are 2 x 2 x 2 x 2 x 3 x 83 x 251, resulted in a performance improvement of about 3x.

Performance isn't always simple.

4th Quarter XSEDE Allocations Research Proposal Dates

XSEDE allocation research request (XRAC) proposals are due October 15th. An overview of XSEDE is on the ARC website, in general it is a large set of national resources for computational research. They are at no cost, but do require a proposal for access. These proposals are simple and most get some sort of an award/resource.

Brock Palen at ARC is the local XSEDE Campus Champion and is available for help learning, using, and accessing XSEDE resources.  He can be contacted at hpc-support@umich.edu.

Researchers who want to get started sooner and/or are new to XSEDE resources should use a Startup proposal which are accepted year round and require significantly less documentation.

ARC Fladoop Data Platform (Hadoop)

ARC and CAEN-HPC recently announced our second generation Hadoop cluster. The first generation cluster lasted about a year and was validating the software and usefulness of the platform.  This new generation platform represents a real investment into hardware to match the software to enable true data science at scale.

The Fladoop Cluster

Architecture

One of the core components of Hadoop is a distributed file system known as HDFS. The power of Hadoop is the ability to break work up and move it to chunks of the data, known as a block, on a drive local to the machine doing the work.

Compared to a traditional compute focused HPC cluster such as Flux this feature of Hadoop requires nodes with many local hard drives and a fast TCP network for shuffling the large quantity of data between nodes.

Fladoop the ARC Hadoop cluster for Data Science consists of 7 data nodes. Each of these nodes has 64GB of main memory, 12 cpu cores and 12 local hard drives.  The network tying them together is 40Gbit Ethernet from Chelsio T580-LP-CR all attached to an Arista 7050q switch.  This is 40 times higher bandwidth per host than standard ethernet, and 4times faster than standard high performance 10Gig-e.

HDFS -- Hadoop File System

HDFS has some unique features to both help with data integrity and performance.  HDFS directly controls each hard drive independently.  Because hard drives fail, HDFS by default copies the data 3 times, to at least 2 unique nodes, and tracks how many copies are available at any time.  If a drive or node fails HDFS automatically makes new copies form the remaining copies.

These copies also can lead to additional performance at the cost of total available space. Because Hadoop tries to move work to where the data are local, there are now 3 possibilities to do that before spilling over to accessing the data over the network.

Getting Started

The use of the new cluster is at no cost. People interested in using it should contact hpc-support@umich.edu to get an account, and look over the documentation.  ARC also provides consulting for exploring if the platform will work well for your application.

In future posts we will show how Hadoop and the packages that go with it, Hive, Pig, and Spark, etc. are powerful and easy to use.

ARC/Flux User Meetup

Friday September 26th we will be hosting another ARC/Flux User Meetup. These are open get togethers where the Flux operators and campus HPC support will be in one location. You can drop in ask questions, learn new things, and generally get updated on ways you can further computational work at Michigan.

• September 26th 1:30-5:00
• Duderstadt Center Conference Room 3  (1129)

3D Rendering using Blender and CUDA on HPC Clusters

Blender is a popular Open Source 3D modeling system.  Recently the question was asked can one use the Flux cluster for 3D rendering. We were curious about this, and we wanted to support our students. Clusters like Flux though are normally built with scientific use in mind and we didn't know if what we had would support Blender. Turns out this was all easier than we thought.

blender-batch.sh

GPU Rendering with CYCLES and CUDA

What we found though is we wanted to use the CYCLES render, and not only that we wanted to run it on the FluxG GPU service.  Why GPUs?  Lets us use the current standard CYCLES benchmark file and compare GPU to CPU performance.

Blender Rendering Benchmark (mpan)

Hardware	Tile Settings (XxY)	Time	Speedup
1 CPU E5-2670	16x16	10m:17s	1x
4 CPU E5-2670	16x16	2m:48s	3.7x
16 CPU E5-2670	16x16	46S	13.4x
1 K20X GPU	256x256	40S	15.4x
2 K20X GPU	256x256	24S	25.7
4 K20X GPU	256x256	18S	34.3x

Running Blender Better

So we know GPUs are much faster, but Blender when ran in the normal batch mode above ignores any settings you pass in a python input. We want to be able to control all GPU/CPU settings on the cluster and not open the GUI each time.  The trick, is to read your blend file from the Blender Python API and then change settings.  Look at the tooltips in blender this API is powerful, everything can be controlled from Python.

bmw.py

Lammps 22Jul14

We just installed Lammps 22Jul14 on Flux and Nyx.

LAMMPS is a classical molecular dynamics code, and an acronym for Large-scale Atomic/Molecular Massively Parallel Simulator. 

LAMMPS has potentials for solid-state materials (metals, semiconductors) and soft matter (biomolecules, polymers) and coarse-grained or mesoscopic systems. It can be used to model atoms or, more generically, as a parallel particle simulator at the atomic, meso, or continuum scale. 

What many don't realize is that our lammps builds are very full featured, including GPU support, we could enable other packages but they have never been requested. If you would like them just let us know.

This build has the following packages:

Installed YES: package ASPHERE

Installed YES: package BODY

Installed YES: package CLASS2

Installed YES: package COLLOID

Installed YES: package DIPOLE

Installed YES: package FLD

Installed YES: package GPU

Installed YES: package GRANULAR

Installed  NO: package KIM

Installed YES: package KOKKOS

Installed YES: package KSPACE

Installed YES: package MANYBODY

Installed YES: package MC

Installed YES: package MEAM

Installed YES: package MISC

Installed YES: package MOLECULE

Installed YES: package MPIIO

Installed YES: package OPT

Installed YES: package PERI

Installed YES: package POEMS

Installed YES: package REAX

Installed YES: package REPLICA

Installed YES: package RIGID

Installed YES: package SHOCK

Installed YES: package SRD

Installed  NO: package VORONOI

Installed YES: package XTC

Installed YES: package USER-ATC

Installed YES: package USER-AWPMD

Installed YES: package USER-CG-CMM

Installed YES: package USER-COLVARS

Installed  NO: package USER-CUDA

Installed YES: package USER-EFF

Installed YES: package USER-FEP

Installed YES: package USER-LB

Installed YES: package USER-MISC

Installed YES: package USER-MOLFILE

Installed  NO: package USER-OMP

Installed YES: package USER-PHONON

Installed  NO: package USER-QMMM

Installed YES: package USER-REAXC

Installed YES: package USER-SPH

When we build lammps we provide two builds:

• lmp_linux Doesn't have GPU package
• lmp_gpu  Has GPU package

If you want to start using this new lammps build start with

module rm intel-comp openmpi
module load intel-comp/14.0
module load openmpi/1.6.5/intel/14.0
module load lammps/22Jul14

If you have any questions feel free to contact us.