3D Rendering using Blender and CUDA on HPC Clusters

Blender is a popular Open Source 3D modeling system.  Recently the question was asked can one use the Flux cluster for 3D rendering. We were curious about this, and we wanted to support our students. Clusters like Flux though are normally built with scientific use in mind and we didn't know if what we had would support Blender. Turns out this was all easier than we thought.

blender-batch.sh

GPU Rendering with CYCLES and CUDA

What we found though is we wanted to use the CYCLES render, and not only that we wanted to run it on the FluxG GPU service.  Why GPUs?  Lets us use the current standard CYCLES benchmark file and compare GPU to CPU performance.

Blender Rendering Benchmark (mpan)

Hardware	Tile Settings (XxY)	Time	Speedup
1 CPU E5-2670	16x16	10m:17s	1x
4 CPU E5-2670	16x16	2m:48s	3.7x
16 CPU E5-2670	16x16	46S	13.4x
1 K20X GPU	256x256	40S	15.4x
2 K20X GPU	256x256	24S	25.7
4 K20X GPU	256x256	18S	34.3x

Running Blender Better

So we know GPUs are much faster, but Blender when ran in the normal batch mode above ignores any settings you pass in a python input. We want to be able to control all GPU/CPU settings on the cluster and not open the GUI each time.  The trick, is to read your blend file from the Blender Python API and then change settings.  Look at the tooltips in blender this API is powerful, everything can be controlled from Python.

bmw.py

Using MATLAB with a GPU

While testing the new FluxG GPU service we did some testing with matlab/2013a its support for GPUs in the Parallel Computing Toolbox.  Below are some examples, and how I sped up a code by using a GPU.

In my examples I use a small GPU call to wake up the GPU, the first time you use a GPU the startup time is long, about 12 seconds. All GPU operations after that will be very fast. I will also use hwloc-bind command to control MATLABs built in threading.

The easiest way to use a GPU with MATLAB is to use gpuArray() to move data to the GPU and then call an MATLAB GPU Enabled function on that data.

1 CPU: 0.59s 1 GPU: 0.12s  Speedup: 4.9x

The next step is todo more of your computation on the GPU including data generation. As the computational complexity rises that can stay on the GPU before moving data with gpuArray() and gather()the greater your performance benefit will be.  This example will take a FFT of a 2D function.

1 CPU: 25.66s 1 GPU: 1.38s  Speedup: 18.59x

We used the vector form of the MATLAB operations which should be optimal.  Another benefit of vector form, not only is it faster than using nested loops (uncomment the loop code in the second example if you are curious) MATLAB can also use multiple cores on vector forms for functions working on data that are large enough.  How does this compare to a single GPU?

1 CPU: 25.66s
1 GPU: 1.38s  Speedup: 18.59x
8 CPU: 4.61s  Speedup: 5.56x
16 CPU: 2.65s  Speedup: 9.68x

Based on the Flux rates as of 1/2014, $60/GPU-Month and $6.60/CPU-Month, codes need to have greater than 9x speedup from a single GPU to make up for the cost. Even this is not exactly correct as each Flux GPU comes with 2 host CPU cores.

• GPU Enabled Functions in MATLAB
• MATLAB GPU Page
• MATLAB Vectorization (Not GPU related)
• Establishing MATLAB Arrays on a GPU

Theano for GPU Computing

In the near future Flux will be offering GPGPU services based on the Nvidia K20x GPU.

A user had requested support for Theano for GPU computing, so we installed it:

Theano is a Python library that allows you to define, optimize, and evaluate mathematical expressions involving multi-dimensional arrays efficiently. Theano features:

• tight integration with NumPy – Use numpy.ndarray in Theano-compiled functions.
• transparent use of a GPU – Perform data-intensive calculations up to 140x faster than with CPU.(float32 only)
• efficient symbolic differentiation – Theano does your derivatives for function with one or many inputs.
• speed and stability optimizations – Get the right answer for log(1+x) even when x is really tiny.
• dynamic C code generation – Evaluate expressions faster.
• extensive unit-testing and self-verification – Detect and diagnose many types of mistake.

 To use Theano for GPU's run it as so:
Setting device=gpu rather than gpu# lets our system assign the correct GPU for you.  For testing on CPU set device=cpu. Theano has many configuration options but the above are the most common.

 If you want to use Theano on multiple GPUs in a single job contact us.