Бенчмарк HPL для NVIDIA GPU

Как установить HPL для GPU?

INSTALL HPL 2.0 c OpenBLAS

Подробнее об установке OpenMPI, CUDA, OpenBLAS.

Настроим переменные окружения подключив модули и скачаем HPL 2.0.

$ module add openmpi/v2.1.6
$ module add cuda/v9.2
$ module add openblas/v0.3.6
$ wget https://developer.download.nvidia.com/assets/cuda/secure/AcceleratedLinpack/hpl-2.0_FERMI_v15.tgz
$ tar -xvf hpl-2.0_FERMI_v15.tgz
$ mv hpl-2.0_FERMI_v15.tgz hpl-2.0
$ cd hpl-2.0

Перед сборкой следует отредактировать несколько файлов. Первым будет Make.CUDA в директории hpl-2.0. Скопируем следующий код в Make.CUDA:

$ cat Make.CUDA
  SHELL        = /bin/sh
  CD           = cd
  CP           = cp
  LN_S         = ln -fs
  MKDIR        = mkdir -p
  RM           = /bin/rm -f
  TOUCH        = touch
  ARCH         = CUDA
  
  TOPdir       = /home/user/hpl-2.0
  INCdir       = $(TOPdir)/include
  BINdir       = $(TOPdir)/bin/$(ARCH)
  LIBdir       = $(TOPdir)/lib/$(ARCH)
  HPLlib       = $(LIBdir)/libhpl.a
  
  MPdir        = /nfs/software/openmpi/v2.1.6
  MPinc        = -I$(MPdir)/include
  MPlib        = -L$(MPdir)/lib -lmpi
  
  LAdir        = /nfs/software/openblas/v0.3.6
  LAinc        = -I$(LAdir)/include
  LAlib        = -L$(TOPdir)/src/cuda -ldgemm -L/nfs/software/cuda/v9.2/lib64 -lcuda -lcudart -lcublas -L$(LAdir)/lib -lopenblas
  F2CDEFS      = -DAdd__ -DF77_INTEGER=int -DStringSunStyle
  HPL_INCLUDES = -I$(INCdir) -I$(INCdir)/$(ARCH) $(LAinc) $(MPinc)
  HPL_LIBS     = $(HPLlib) $(LAlib) $(MPlib)
  HPL_OPTS     =  -DCUDA
  HPL_DEFS     = $(F2CDEFS) $(HPL_OPTS) $(HPL_INCLUDES)
  CC           = mpicc
  CCFLAGS      = -fopenmp -lpthread -fomit-frame-pointer -O3 -funroll-loops $(HPL_DEFS)
  CCNOOPT      = $(HPL_DEFS) -O0 -w
  LINKER       = $(CC)
  LINKFLAGS    = $(CCFLAGS)
  ARCHIVER     = ar
  ARFLAGS      = r
  RANLIB       = echo
  MAKE         = make TOPdir=$(TOPdir)

11. Путь до директории hpl-2.0
17. Путь до OpenMPI
21. Путь до OpenBLAS
23. Путь до CUDA lib64

Заменим в файле hpl-2.0/src/cuda/cuda_dgemm.c следующие строки:

$ mcedit src/cuda/cuda_dgemm.c
  …
  // handle2 = dlopen ("libmkl_intel_lp64.so", RTLD_LAZY);
  handle2 = dlopen ("libopenblas.so", RTLD_LAZY);
  …
  // dgemm_mkl = (void(*)())dlsym(handle, "dgemm");
  dgemm_mkl = (void(*)())dlsym(handle, "dgemm_");
  …
  // handle = dlopen ("libmkl_intel_lp64.so", RTLD_LAZY);
  handle = dlopen ("libopenblas.so", RTLD_LAZY);
  …
  // mkl_dtrsm = (void(*)())dlsym(handle2, "dtrsm");
  mkl_dtrsm = (void(*)())dlsym(handle2, "dtrsm_");

Соберем и запустим HPL на 4x GPU:

$ make arch=CUDA
$ cd bin/CUDA
$ export LD_LIBRARY_PATH=/home/user/hpl-2.0/src/cuda/:$LD_LIBRARY_PATH
$ mpirun -np 4 ./xhpl
  ================================================================================
  HPLinpack 2.0  --  High-Performance Linpack benchmark  --   September 10, 2008
  Written by A. Petitet and R. Clint Whaley,  Innovative Computing Laboratory, UTK
  Modified by Piotr Luszczek, Innovative Computing Laboratory, UTK
  Modified by Julien Langou, University of Colorado Denver
  ================================================================================

  An explanation of the input/output parameters follows:
  T/V    : Wall time / encoded variant.
  N      : The order of the coefficient matrix A.
  NB     : The partitioning blocking factor.
  P      : The number of process rows.
  Q      : The number of process columns.
  Time   : Time in seconds to solve the linear system.
  Gflops : Rate of execution for solving the linear system.

  The following parameter values will be used:

  N      :   25000
  NB     :     768
  PMAP   : Row-major process mapping
  P      :       2
  Q      :       2
  PFACT  :    Left
  NBMIN  :       2
  NDIV   :       2
  RFACT  :    Left
  BCAST  :   1ring
  DEPTH  :       1
  SWAP   : Spread-roll (long)
  L1     : no-transposed form
  U      : no-transposed form
  EQUIL  : yes
  ALIGN  : 8 double precision words

  --------------------------------------------------------------------------------

  - The matrix A is randomly generated for each test.
  - The following scaled residual check will be computed:
        ||Ax-b||_oo / ( eps * ( || x ||_oo * || A ||_oo + || b ||_oo ) * N )
  - The relative machine precision (eps) is taken to be               1.110223e-16
  - Computational tests pass if scaled residuals are less than                16.0

  ================================================================================
  T/V                N    NB     P     Q               Time                 Gflops
  --------------------------------------------------------------------------------
  WR10L2L2       25000   768     2     2              16.72              6.232e+02
  --------------------------------------------------------------------------------
  ||Ax-b||_oo/(eps*(||A||_oo*||x||_oo+||b||_oo)*N)=        0.0019019 ...... PASSED
  ================================================================================

  Finished      1 tests with the following results:
                1 tests completed and passed residual checks,
                0 tests completed and failed residual checks,
                0 tests skipped because of illegal input values.
  --------------------------------------------------------------------------------

  End of Tests.
  ================================================================================