Setting up passwordless SSH

1. Run ssh-keygen on a Unix machine you typically use (can be Athena, or a Darwin cluster head node). Accept the default settings and leave the passphrase blank. This creates files in ~/.ssh
2. Append the contents of ~/.ssh/id_rsa.pub to the file ~/.ssh/authorized_keys on each machine you want to be able to connect to. The Darwin cluster nodes share a filesystem, so you only need to do this once on one of the head nodes.
3. Copy the file ~/.ssh/id_rsa (private key) to the same location on each machine you want to connect from. Again, once this file is in ~/.ssh on Darwin you are all set for the cluster.
4. The file id_rsa needs to have strict permissions. If you get errors about this, run chmod 400 id_rsa.

scp file user@machine:path

Darwin specifications

• Darwin compute cluster
  • Head node: beagle.darwinproject.mit.edu
  • Nodes: 128
  • Per-node CPU: 2 x 2-core 3.00GHz Intel(R) Xeon(TM), 4MB L2 cache
  • Per-node memory: 8 GB
• Darwin visualization cluster
  • Head node: evolution.darwinproject.mit.edu
  • Nodes: 60
  • Per-node CPU: 2 x 2-core 2.66GHz Intel(R) Xeon(TM), 4MB L2 cache
  • Per-node memory: 6 GB

Logging In

Compiling MPI and OpenMP programs

Running MPI programs

After compiling MPI programs, use qsub and mpirun to execute:

echo '/opt/openmpi/bin/mpirun -np # /path/to/yourMPIProgram' | qsub -pe mpich_mx #

where # is the number of processors and yourMPIProgram is the MPI executable to be run. qsub is the interface to the Sun Grid Engine batch queue, which ensures that the cluster is shared fairly.

Anything printed to standard out will be available in a file in your home directory called STDIN.oID where ID is the job ID. STDIN refers to the fact that the job description was provided on standard input via a pipe instead of in a shell script file. The standard error stream is in a similar file, named with an e instead of the o.

When using a private cluster, one can specify machines to use manually using a hostfile, as follows:

mpirun -np # -machinefile hostfile ./yourMPIProgram

"hostfile" is a text file listing all the hosts you want to use. It might look like this:

node-1
node-2
node-3
node-4

Timing your runs

MPI example

#include "mpi.h"
#include "unistd.h"
#include "stdio.h"

int main(int argc, char **argv)
{
  char hostname[256];
  int size,rank;

  MPI_Init(&argc, &argv);

  MPI_Comm_size(MPI_COMM_WORLD, &size);
  MPI_Comm_rank(MPI_COMM_WORLD, &rank);

  gethostname(hostname, sizeof(hostname));

  printf("Hi, I am %d of %d and my hostname is %s\n", rank, size, hostname);

  MPI_Finalize();
}

OpenMP example

#include <math.h>
#include <stdio.h>

#define N 16384
#define M 10

double dotproduct(int, double *);


double dotproduct(int i, double *x)
{  
  double temp=0.0, denom;
  int j;

  for (j=0; j<N; j++) {
    // zero based!!
    denom = (i+j)*(i+j+1)/2 + i+1;
    temp = temp + x[j]*(1/denom);  
  }

  return temp;
}


int main()
{
  double *x = new double[N];
  double *y = new double[N];
  double eig = sqrt(N);
  double denom,temp;
  int i,j,k;

  for (i=0; i<N; i++) {
    x[i] = 1/eig;
  }

  for (k=0;k<M;k++) {

    y[i]=0;

    // compute y = Ax
#pragma omp parallel for shared(y)
    for (i=0; i<N; i++) {     
      y[i] = dotproduct(i,x);
    }
    
    // find largest eigenvalue of y
    eig = 0;    
    for (i=0; i<N; i++) {
      eig = eig + y[i]*y[i];
    }    
    eig = sqrt(eig);
   
    printf("The largest eigenvalue after %2d iteration is %16.15e\n",k+1, eig);

    // normalize
    for (i=0; i<N; i++) {
      x[i] = y[i]/eig;
    }
  }
}

Julia example

You can add processors to Julia's pool from the prompt as follows:

julia> @time @parallel (+) for i=1:10000000; randn(); end
elapsed time: 1.0429379940032959 seconds
-842.00907574328687133

julia> @time @parallel (+) for i=1:10000000; randn(); end
elapsed time: 0.58234810829162598 seconds
430.07946679243082144

julia> addprocs_sge(2)
ok

julia> @time @parallel (+) for i=1:10000000; randn(); end
elapsed time: 3.28881287574768066 seconds
2124.42978527200511962

julia> @time @parallel (+) for i=1:10000000; randn(); end
elapsed time: 0.31914687156677246 seconds
283.82680807441192883

Note one minor annoyance: due to Julia's Just-In-Time (JIT) compilation model, things take longer the first time you do them. The extra delay is typically about 1-3 seconds. As the code cache warms up over a session, these delays become shorter and rarer.

Emacs setup

(require 'julia-mode "/home/bezanson/julia/contrib/julia-mode.el")

Julia MPI

mpirun -np # /home/bezanson/.julia/mpi/juliampi sourcefile.jl