Sending large files to people

In a previous post we described the Globus file transfer service.  It is designed to transfer single or multiple large data sets between two sites.  Here we show an alternate approach using the Filesender service offered by Internet2 that is focused on file transfers of any size between individual’s desktop or laptop computers. It is a particularly convenient way to overcome email attachment file size limitations.

The process is real easy, just follow these simple steps:

1. Point your web-browser at: https://filesender.internet2.edu
2. Login using "University of Michigan" as your Organization.
1. Then use your UMich Uniqname and Kerberos password
3. Fill in all the information fields of the form, especially the email address.
4. Upload your files.
5. Then click on the "Send" button. The recipient will be sent an email with a link and instructions on downloading the files. When the download is complete you will receive an email telling you so.

Does someone need to send you large files but is not from an Internet2 member Institution?  No problem!  After you log into the filesender site select "Guest Voucher" at the top of the page.  You can then have an email sent to the other person with a link allowing them to send a file or multiple files back to you or anyone else. Be advised though that this is a one time use for each voucher.

As always, the HPC support staff on campus are available to help, simply send an email to hpc-support@umich.edu

Intel Xeon Phi's Available on Flux

ARC-TS now has available as a technology preview, 8 Intel Xeon Phi (Wikipedia) 5110p cards.  These are known as MIC's or Many Integrated Core.  These are an accelerator card that fits into a slot on a Flux compute node and a code can offload portions or all of the work to the card.

As a technology preview, there is no additional cost for using the Phi's.  All that is required is an active Flux allocation and users can test the Phi's. The only other requirement is all Phi jobs must be less than 24 hours long.

The Phi cards support three modes of operation, Automatic Offload, Compiler Assisted Offload, and Native.  The first two are well tested, the last works but is not as well tested.  All intel-comp compiler and mkl math library modules on Flux support the Phi.

You can request a phi with PBS with:

qsub -I -l nodes=1:mics=2  -q flux -A account_flux -l qos=flux -V

This will provide two Phi's and one CPU. Flux currently has one node with 8 Phi cards.

PBS will do set two variables:

PBS_MICFILE   -> list of hostnames of assigned phi's  good for native mode.
OFFLOAD_DEVICES  -> csv list of devices for controlling auto offload (MKL) or compiler assisted offload.


$HOME is mounted on the card, as is all the software but /scratch currently is not,  it should sometime in the future.  This should only affect users running Native Phi code.

For software the Phi requires some environment changes to work.  We created a module called xeon-mic.  When loaded it will set some sane defaults but will print what it set.  Users are encouraged to experiment with many of the settings available.

module load xeon-mic

In future posts we will show examples using the Phi.

Flux and Value Storage outage March 28th 10:00pm

The ARC cluster Flux and Engineering cluster Nyx will be unavailable for jobs March 28th at 10:00pm. There is an emergency update to the ITS Value Storage systems on that date.
http://status.its.umich.edu/outage.php?id=93178

Flux and Nyx rely on Value Storage and thus will also not be available during that time.  We expect the outage to be finished quickly and any queued jobs will run as expected once the service is completed.

At the start of the outage, login and transfer nodes will be rebooted.  Users will be unable to login until after the service is restored.

Any jobs that request more walltime than remains until the start of the outage will be held and started after the systems return to service.

To find the maximum walltime you can request and have your job start

prior to the outage can be found with our walltime calculator.

module load flux-utilsmaxwalltime

Allocations that are active on that date will be extended by one day

at no cost

If you have any questions feel free to ask us at hpc-support@umich.edu

For immediate updates watch: https://twitter.com/umcoecac

Data will be deleted from /scratch on Flux if unused for 90 days

Over the past several months, a huge amount of data (491 TB) has accumulated in the /scratch directory on the Flux computing cluster. /scratch is meant for data relating to currently running jobs, and the buildup of data is threatening the performance of Flux for all users.
Therefore, ARC will begin deleting data from /scratch that have not been accessed for 90 consecutive days.

Flux account owners with unused data have begun receiving emails warning that their data will be deleted.

Account owners in this situation can move their data to another system such as ITS Value Storage or their own equipment using the dedicated transfer nodes on Flux with high speed network connectsion available for that purpose.

For more information on Value Storage, see the ITS website.
For more information on transfer nodes, see the ARC website.
If you have any questions, please contact hpc-support@umich.edu.

Flux Adds New 20 core Nodes

Flux has been expanded to include 126 new nodes. These are IBM* NeXtScale based systems.  Each Chassis holds 12 nx360 M4 nodes.

Details are:

• 2 x 10 core 2.8 Ghz Intel E5-2680v2
• 90+ GB of DDR 1866Mhz RAM
• FDR** Infiniband

A clever eye may notice that 126 nodes does not evenly divide into chassis that hold 12 nodes. Under FoE we operate some additional nodes totaling 156.

*IBM has sense sold their x86 line to Linovo

** While the servers have FDR adaptors, the fabric it connects to is still QDR based and will perform as such.

Using Infiniband with MATLAB Parallel Computing Toolbox

In High Performance Computing (HPC) there are a number of network types commonly used, among these are: Ethernet, the common network found on all computer equipment. Infiniband, a specialty high performance low latency interconnect common on commodity clusters.  There are also several propriety types and a few other less common types but I will focus on Ethernet and Infiniband.

Ethernet and really its mate protocol, TCP, are the most common supported MPI networks.  Almost all computer platforms support this network type and can be as simple as using your home network switch.  It is ubiquitous and easy to support.  Networks like Infiniband though require special drivers, uncommon hardware but the effort is normally worth it.

The MATLAB Parallel Computing Toolbox provides a collection of functions that allow users of MATLAB to utilize multiple compute nodes to work on larger problems.  Many may not realize that MathWorks chose to use the standard MPI routines to implement this toolbox.  MathWorks also chose for ease of use to ship MATLAB with the Mpich MPI library, and the version they use only support Ethernet for communication between nodes.

As noted Ethernet is about the slowest common network used in parallel applications. The question is how much can this impact performance.

Mmmmm Data:

The data was generated on 12 nodes of Xeon x5650 total 144 cores. The code was the stock MATLAB paralleldemo_backslash_bench(1.25) from MATLAB 2013b.  You can find my M-Code at Gist.

The data show two trends, one is independent of the network type.  That is many parallel algorithms do not scale unless the amount of data for each core to work on is sufficiently large. In this case for Ethernet especially the peak performance is never reached.  What should be really noted though is that without Infiniband at many problem sizes over half of the performance of the nodes is lost. The second trend is that network really matters.

How to have MATLAB use Infiniband?

MathWorks does not ship an MPI library with the parallel computing toolbox that can use infiniband by default. This is reasonable, I would be curious how large the average PCT cluster is, and/or how big the jobs ran on the toolbox are.  Lucky for us MathWorks allows a way for introducing your own MPI library.  Let me be the first to proclaim:

Thank you MathWorks for adding mpiLibConf.m as a feature. -- Brock Palen

In the above test we used Intel MPI for the infiniband test and mpich for the ethernet test.  The choice of MPI is important.  The MPI standard enforces a shared API but not a shared ABI.  Thus the MPI library you substitute needs to match the one MATLAB is compiled against. Lucky for us they used mpich, so any mpich clone should work; mvapich, IntelMPI, etc.

If you are using the MATLAB Parallel Computing Toolbox on more than one node, and if your cluster has a network other than Ethernet/TCP (there are non-tcp ethernet networks that perform very well) I highly encourage that the effort be put in to ensure you use that network.

For Flux users we have this setup, but you have to do some setup for yourself before you see the benefit.  Please visit the ARC MATLAB documentation, or send us a question at hpc-support@umich.edu.

Q1 XSEDE Research Proposal Deadline

The next XSEDE Research proposal deadline is January 15th.  If you are looking to get more work done, or to scale to new levels read more on the ARC site.