Complie HPL 2.2 failed with OpenBLAS Develop

[JohnCenteno]

Dear Xianyi

Do you know why I complie HPL 2.2 failed with OpenBLAS dev, but it worked with GotoBLAS2

/root/OpenBLAS/lib/libopenblas.a(memory.o): In function openblas_fork_handler': memory.c:(.text+0x300): undefined reference topthread_atfork'
collect2: ld returned 1 exit status
make[2]: *** [dexe.grd] Error 1
make[2]: Leaving directory /root/hpl-2.2/testing/ptest/Linux_PII_CBLAS' make[1]: *** [build_tst] Error 2 make[1]: Leaving directory/root/hpl-2.2'
make: *** [build] Error 2
[root@localhost hpl-2.2]#

Thanks.
John Centeno

[brada4]

You need to add -pthread to linker lines to allow this symbol. If you may call BLAS from concurrent threads it must be USE_OPENMP=1 instead. Other option is to build dynamic OpenBLAS so that it imports -lgomp or -lpthread by itself. Please tell how it goes

[JohnCenteno]

Dear Adrew,

Thanks a lot. I am not good at Linux, so I am not sure how to add -pthread to linker lines" or how to imports lgomp or -lpthread

With Best Regards.
John Centos

[JohnCenteno]

Dear Andrews,

I also tried "gmake USE_OPENMP=1" & gmake PREFIX=/root/OpenBLAS
vi .bash_profile
export PATH=/root/mpich/bin:$PATH
export OPENBLAS_NUM_THREADS=1

With Best Regards.
John Centeno

[brada4]

Please catch FULL OUTPUT of build and attach as a text file (DO NOT PASTE IN COMMENT FORM)

[karanchhabra]

Hi,

I'm facing problems in compiling HPL for testing the GPU on my system but while compiling I'm facing the following error. I've tried every possible solution available on goolge but can't resolve it.

mpicc -fopenmp -lpthread -fomit-frame-pointer -O3 -funroll-loops -DAdd__ -DF77_INTEGER=int -DStringSunStyle -DCUDA -I/root/hpl/hpl-2.0_FERMI_v15_latest/include -I/root/hpl/hpl-2.0_FERMI_v15_latest/include/CUDA -I/root/hpl/mpich/include -I/usr/local/cuda-9.1/include -o /root/hpl/hpl-2.0_FERMI_v15_latest/bin/CUDA/xhpl HPL_pddriver.o HPL_pdinfo.o HPL_pdtest.o /root/hpl/hpl-2.0_FERMI_v15_latest/lib/CUDA/libhpl.a -L /root/hpl/hpl-2.0_FERMI_v15_latest/src/cuda -ldgemm -L/usr/local/cuda/lib64 -lcublas -lcuda -lcudart -L/root/hpl/OpenBLAS-0.2.20/libopenblas.so -lpthread /root/hpl/mpich/lib/libmpi.so
/root/hpl/hpl-2.0_FERMI_v15_latest/lib/CUDA/libhpl.a(HPL_idamax.o): In function HPL_idamax': HPL_idamax.c:(.text+0x1a): undefined reference to idamax_'
/root/hpl/hpl-2.0_FERMI_v15_latest/lib/CUDA/libhpl.a(HPL_dgemv.o): In function HPL_dgemv': HPL_dgemv.c:(.text+0xba): undefined reference to dgemv_'
HPL_dgemv.c:(.text+0x136): undefined reference to dgemv_' /root/hpl/hpl-2.0_FERMI_v15_latest/lib/CUDA/libhpl.a(HPL_dcopy.o): In function HPL_dcopy':
HPL_dcopy.c:(.text+0x1e): undefined reference to dcopy_' /root/hpl/hpl-2.0_FERMI_v15_latest/lib/CUDA/libhpl.a(HPL_daxpy.o): In function HPL_daxpy':
HPL_daxpy.c:(.text+0x2f): undefined reference to daxpy_' /root/hpl/hpl-2.0_FERMI_v15_latest/lib/CUDA/libhpl.a(HPL_dscal.o): In function HPL_dscal':
HPL_dscal.c:(.text+0x22): undefined reference to dscal_' /root/hpl/hpl-2.0_FERMI_v15_latest/lib/CUDA/libhpl.a(HPL_dtrsv.o): In function HPL_dtrsv':
HPL_dtrsv.c:(.text+0xc0): undefined reference to dtrsv_' /root/hpl/hpl-2.0_FERMI_v15_latest/lib/CUDA/libhpl.a(HPL_dger.o): In function HPL_dger':
HPL_dger.c:(.text+0x73): undefined reference to dger_' HPL_dger.c:(.text+0xa1): undefined reference to dger_'
collect2: error: ld returned 1 exit status
make[2]: *** [dexe.grd] Error 1
make[2]: Leaving directory /root/hpl/hpl-2.0_FERMI_v15_latest/testing/ptest/CUDA' make[1]: *** [build_tst] Error 2 make[1]: Leaving directory /root/hpl/hpl-2.0_FERMI_v15_latest'
make: *** [build] Error 2

Kindly help.
Thanks,
Karan Chhabra

[martin-frbg]

Looks like you have an error on that long mpicc command line - where you have "-L/root/OpenBLAS-0.2.20/libopenblas.so" it should probably be "-L/root/OpenBLAS-0.2.20 -lopenblas"

[karanchhabra]

libopenblas.so is the library which has to be mentioned in there in order for hpl to compile.

Followed your suggesstion and got following error:
mpicc -fopenmp -lpthread -fomit-frame-pointer -O3 -funroll-loops -DAdd__ -DF77_INTEGER=int -DStringSunStyle -DCUDA -I/root/hpl/hpl-2.0_FERMI_v15_latest/include -I/root/hpl/hpl-2.0_FERMI_v15_latest/include/CUDA -I/root/hpl/mpich/include -I/usr/local/cuda-9.1/include -o /root/hpl/hpl-2.0_FERMI_v15_latest/bin/CUDA/xhpl HPL_pddriver.o HPL_pdinfo.o HPL_pdtest.o /root/hpl/hpl-2.0_FERMI_v15_latest/lib/CUDA/libhpl.a -L /root/hpl/hpl-2.0_FERMI_v15_latest/src/cuda -ldgemm -L/usr/local/cuda/lib64 -lcublas -lcuda -lcudart -L/root/hpl/OpenBLAS-0.2.20/lib -lopenblas -lpthread /root/hpl/mpich/lib/libmpi.so
/bin/ld: cannot find -lopenblas
collect2: error: ld returned 1 exit status
make[2]: *** [dexe.grd] Error 1
make[2]: Leaving directory /root/hpl/hpl-2.0_FERMI_v15_latest/testing/ptest/CUDA' make[1]: *** [build_tst] Error 2 make[1]: Leaving directory /root/hpl/hpl-2.0_FERMI_v15_latest'
make: *** [build] Error 2

[martin-frbg]

Well, are you certain you have it in /root/hpl/OpenBLAS-0.2.20/lib ? Normally this gets built directly under OpenBLAS-0.2.20. The "-L" tells the compiler/linker where to search, the -l tells it what library - and you leave off the "lib" prefix of the name on Unix/Linux.

[karanchhabra]

I'm stuck on this since last 2 weeks and can't find any solution on web.

Versions installed:

1. mpich-3.2.1
2. OpenBLAS-0.2.20
3. hpl-2.0_FERMI_v15_latest

It is running perfectly fine for testing CPUs but in case of benchmarking GPUs i have to use the customized version of the benchmark provided by nvidia.

I did install all the pre requisites for this:

1. nvidia driver.
2. cuda installation toolkit.
   now i'm not able to compile hpl using the version provided by NVIDIA.

My Make.CUDA file is as follows:

---

• shell --------------------------------------------------------------

---

SHELL = /bin/sh

CD = cd
CP = cp
LN_S = ln -fs
MKDIR = mkdir -p
RM = /bin/rm -f
TOUCH = touch

---

• Platform identifier ------------------------------------------------

---

ARCH = CUDA

---

• HPL Directory Structure / HPL library ------------------------------

---

TOPdir =/root/hpl/hpl-2.0_FERMI_v15_latest
endif
INCdir = $(TOPdir)/include
BINdir = $(TOPdir)/bin/$(ARCH)
LIBdir = $(TOPdir)/lib/$(ARCH)

HPLlib = $(LIBdir)/libhpl.a

---

• Message Passing library (MPI) --------------------------------------

---

MPinc tells the C compiler where to find the Message Passing library
header files, MPlib is defined to be the name of the library to be
used. The variable MPdir is only used for defining MPinc and MPlib.

MPdir = /root/hpl/mpich
MPinc = -I$(MPdir)/include
MPlib = $(MPdir)/lib/libmpi.so

---

• Linear Algebra library (BLAS) -----------------------------

---

LAinc tells the C compiler where to find the Linear Algebra library
header files, LAlib is defined to be the name of the library to be
used. The variable LAdir is only used for defining LAinc and LAlib.

LAdir = /root/hpl/OpenBLAS-0.2.20
LAinc =
LAlib = -L $(TOPdir)/src/cuda -ldgemm -L/usr/local/cuda/lib64 -lcublas -lcuda -lcudart -L$(LAdir)/libopenblas.so -lpthread

F2CDEFS = -DAdd__ -DF77_INTEGER=int -DStringSunStyle

---

• HPL includes / libraries / specifics -------------------------------

---

HPL_INCLUDES = -I$(INCdir) -I$(INCdir)/$(ARCH) $(LAinc) $(MPinc) -I/usr/local/cuda-9.1/include
HPL_LIBS = $(HPLlib) $(LAlib) $(MPlib)

• Compile time options -----------------------------------------------

-DHPL_COPY_L force the copy of the panel L before bcast;
-DHPL_CALL_CBLAS call the cblas interface;
-DHPL_DETAILED_TIMING enable detailed timers;
-DASYOUGO enable timing information as you go (nonintrusive)
-DASYOUGO2 slightly intrusive timing information
-DASYOUGO2_DISPLAY display detailed DGEMM information
-DENDEARLY end the problem early
-DFASTSWAP insert to use DLASWP instead of HPL code

HPL_OPTS = -DCUDA

HPL_DEFS = $(F2CDEFS) $(HPL_OPTS) $(HPL_INCLUDES)

---

• Compilers / linkers - Optimization flags ---------------------------

---

CC = mpicc
CCFLAGS = -fopenmp -lpthread -fomit-frame-pointer -O3 -funroll-loops $(HPL_DEFS)
CCNOOPT = $(HPL_DEFS) -O0 -w

LINKER = $(CC)
LINKFLAGS = $(CCFLAGS)

ARCHIVER = ar
ARFLAGS = r
RANLIB = echo

---

MAKE = make TOPdir=$(TOPdir)

[martin-frbg]

So your LAlib line says the libopenblas.so lives in /root/hpl/OpenBLAS-0.2.20/lib (it takes the LAdir and appends /lib to it). This will be correct for an installed OpenBLAS (i.e. you ran "make install" after building it) but not for the original build directory. Can you please check where the libopenblas.so is on your system ?

[karanchhabra]

Ok. I have it in the right location /root/hpl/OpenBLAS-0.2.20/libopenblas.so.

mpicc -fopenmp -lpthread -fomit-frame-pointer -O3 -funroll-loops -DAdd__ -DF77_INTEGER=int -DStringSunStyle -DCUDA -I/root/hpl/hpl-2.0_FERMI_v15_latest/include -I/root/hpl/hpl-2.0_FERMI_v15_latest/include/CUDA -I/usr/local/cuda-9.1/include -o /root/hpl/hpl-2.0_FERMI_v15_latest/bin/CUDA/xhpl HPL_pddriver.o HPL_pdinfo.o HPL_pdtest.o /root/hpl/hpl-2.0_FERMI_v15_latest/lib/CUDA/libhpl.a -L /root/hpl/hpl-2.0_FERMI_v15_latest/src/cuda -ldgemm -L/usr/local/cuda/lib64 -lcublas -lcuda -lcudart -L/root/hpl/OpenBLAS-0.2.20/libopenblas.so -lpthread
/root/hpl/hpl-2.0_FERMI_v15_latest/lib/CUDA/libhpl.a(HPL_idamax.o): In function HPL_idamax': HPL_idamax.c:(.text+0x1a): undefined reference to idamax_'
/root/hpl/hpl-2.0_FERMI_v15_latest/lib/CUDA/libhpl.a(HPL_dgemv.o): In function HPL_dgemv': HPL_dgemv.c:(.text+0xba): undefined reference to dgemv_'
HPL_dgemv.c:(.text+0x136): undefined reference to dgemv_' /root/hpl/hpl-2.0_FERMI_v15_latest/lib/CUDA/libhpl.a(HPL_dcopy.o): In function HPL_dcopy':
HPL_dcopy.c:(.text+0x1e): undefined reference to dcopy_' /root/hpl/hpl-2.0_FERMI_v15_latest/lib/CUDA/libhpl.a(HPL_daxpy.o): In function HPL_daxpy':
HPL_daxpy.c:(.text+0x2f): undefined reference to daxpy_' /root/hpl/hpl-2.0_FERMI_v15_latest/lib/CUDA/libhpl.a(HPL_dscal.o): In function HPL_dscal':
HPL_dscal.c:(.text+0x22): undefined reference to dscal_' /root/hpl/hpl-2.0_FERMI_v15_latest/lib/CUDA/libhpl.a(HPL_dtrsv.o): In function HPL_dtrsv':
HPL_dtrsv.c:(.text+0xc0): undefined reference to dtrsv_' /root/hpl/hpl-2.0_FERMI_v15_latest/lib/CUDA/libhpl.a(HPL_dger.o): In function HPL_dger':
HPL_dger.c:(.text+0x73): undefined reference to dger_' HPL_dger.c:(.text+0xa1): undefined reference to dger_'
collect2: error: ld returned 1 exit status
make[2]: *** [dexe.grd] Error 1
make[2]: Leaving directory /root/hpl/hpl-2.0_FERMI_v15_latest/testing/ptest/CUDA' make[1]: *** [build_tst] Error 2 make[1]: Leaving directory /root/hpl/hpl-2.0_FERMI_v15_latest'
make: *** [build] Error 2

[martin-frbg]

Still not quite right, can you please change the LAlib line in your file to
LAlib = -L $(TOPdir)/src/cuda -ldgemm -L/usr/local/cuda/lib64 -lcublas -lcuda -lcudart -L$(LAdir) -lopenblas -lpthread

[karanchhabra]

Martin,

Thank you very much!

It worked. Such a small issue and i wasn't able to fix it for 2 weeks. Hopefully i can run it successfully and benchmark my system's GPUs.

mpicc -fopenmp -lpthread -fomit-frame-pointer -O3 -funroll-loops -DAdd__ -DF77_INTEGER=int -DStringSunStyle -DCUDA -I/root/hpl/hpl-2.0_FERMI_v15_latest/include -I/root/hpl/hpl-2.0_FERMI_v15_latest/include/CUDA -I/usr/local/cuda-9.1/include -o /root/hpl/hpl-2.0_FERMI_v15_latest/bin/CUDA/xhpl HPL_pddriver.o HPL_pdinfo.o HPL_pdtest.o /root/hpl/hpl-2.0_FERMI_v15_latest/lib/CUDA/libhpl.a -L /root/hpl/hpl-2.0_FERMI_v15_latest/src/cuda -ldgemm -L/usr/local/cuda/lib64 -lcublas -lcuda -lcudart -L/root/hpl/OpenBLAS-0.2.20 -lopenblas -lpthread
make TOPdir=/root/hpl/hpl-2.0_FERMI_v15_latest /root/hpl/hpl-2.0_FERMI_v15_latest/bin/CUDA/HPL.dat
make[3]: Entering directory /root/hpl/hpl-2.0_FERMI_v15_latest/testing/ptest/CUDA' ( cp ../HPL.dat /root/hpl/hpl-2.0_FERMI_v15_latest/bin/CUDA ) make[3]: Leaving directory /root/hpl/hpl-2.0_FERMI_v15_latest/testing/ptest/CUDA'
touch dexe.grd
make[2]: Leaving directory /root/hpl/hpl-2.0_FERMI_v15_latest/testing/ptest/CUDA' make[1]: Leaving directory /root/hpl/hpl-2.0_FERMI_v15_latest'

Thanks,
Karan

[karanchhabra]

Hi,

Compiling HPL was successful but facing problems in running the benchmark.

When I run the code it complaints about missing libmkl_intel_lp64.so but when i do ldd for xhpl, it shows no dependency on libmkl_intel_lp64.so.

Any suggesstions.

What I run:
/root/hpl/mpich/bin/mpirun -np 1 -hostfile nodes ./run_linpack

Error:
libmkl_intel_lp64.so: cannot open shared object file: No such file or directory

Dependency tree:
ldd xhpl
linux-vdso.so.1 => (0x00007ffeb597c000)
libpthread.so.0 => /lib64/libpthread.so.0 (0x00007fd1e10f4000)
libdgemm.so.1 => /root/hpl/hpl-2.0_FERMI_v15_latest/src/cuda/libdgemm.so.1 (0x00007fd1e0eeb000)
libcublas.so.9.1 => /usr/local/cuda/lib64/libcublas.so.9.1 (0x00007fd1dd954000)
libcuda.so.1 => /usr/lib64/nvidia/libcuda.so.1 (0x00007fd1dcdb4000)
libcudart.so.9.1 => /usr/local/cuda/lib64/libcudart.so.9.1 (0x00007fd1dcb45000)
libopenblas.so.0 => /root/hpl/openblas/lib/libopenblas.so.0 (0x00007fd1dbbb6000)
libmpi.so.12 => /root/hpl/mpich/lib/libmpi.so.12 (0x00007fd1db737000)
libgomp.so.1 => /lib64/libgomp.so.1 (0x00007fd1db510000)
libc.so.6 => /lib64/libc.so.6 (0x00007fd1db14d000)
/lib64/ld-linux-x86-64.so.2 (0x00005633fe7d7000)
librt.so.1 => /lib64/librt.so.1 (0x00007fd1daf45000)
libdl.so.2 => /lib64/libdl.so.2 (0x00007fd1dad40000)
libstdc++.so.6 => /lib64/libstdc++.so.6 (0x00007fd1daa38000)
libm.so.6 => /lib64/libm.so.6 (0x00007fd1da736000)
libgcc_s.so.1 => /lib64/libgcc_s.so.1 (0x00007fd1da51f000)
libnvidia-fatbinaryloader.so.390.30 => /usr/lib64/nvidia/libnvidia-fatbinaryloader.so.390.30 (0x00007fd1da2d3000)
libgfortran.so.3 => /lib64/libgfortran.so.3 (0x00007fd1d9fb1000)
libquadmath.so.0 => /lib64/libquadmath.so.0 (0x00007fd1d9d74000)

*Can't see libmkl_intel_lp64.so mentioned anywhere there but still everytime i run HPL it complaints about missing libmkl_intel_lp64.so

Thanks,
Karan

[martin-frbg]

Probably one of those files was created with the Intel compiler and depends on libmkl, try doing the ldd on that libdemm.so.1 for instance.

[karanchhabra]

Not sure but here is the output:

--ldd /root/hpl/hpl-2.0_FERMI_v15_latest_copy/src/cuda/libdgemm.so.1

    linux-vdso.so.1 =>  (0x00007ffc73972000)
    libcudart.so.9.1 => /usr/local/cuda-9.1/targets/x86_64-linux/lib/libcudart.so.9.1 (0x00007fd6f12a9000)
    libcuda.so.1 => /usr/lib64/nvidia/libcuda.so.1 (0x00007fd6f0708000)
    libmpi.so.12 => /root/hpl/mpich/lib/libmpi.so.12 (0x00007fd6f0289000)
    libc.so.6 => /lib64/libc.so.6 (0x00007fd6efec6000)
    libdl.so.2 => /lib64/libdl.so.2 (0x00007fd6efcc1000)
    libpthread.so.0 => /lib64/libpthread.so.0 (0x00007fd6efaa5000)
    librt.so.1 => /lib64/librt.so.1 (0x00007fd6ef89d000)
    libm.so.6 => /lib64/libm.so.6 (0x00007fd6ef59a000)
    libnvidia-fatbinaryloader.so.390.30 => /usr/lib64/nvidia/libnvidia-fatbinaryloader.so.390.30 (0x00007fd6ef34e000)
    libgfortran.so.3 => /lib64/libgfortran.so.3 (0x00007fd6ef02c000)
    libgcc_s.so.1 => /lib64/libgcc_s.so.1 (0x00007fd6eee15000)
    libquadmath.so.0 => /lib64/libquadmath.so.0 (0x00007fd6eebd9000)
    /lib64/ld-linux-x86-64.so.2 (0x000055a1bd532000)

[martin-frbg]

Ok, that leaves libcuda I guess ?

[karanchhabra]

I've the following from:
--ldd /usr/lib64/nvidia/libcuda.so.1

    linux-vdso.so.1 =>  (0x00007ffef41c7000)
    libm.so.6 => /lib64/libm.so.6 (0x00007f8a3ba04000)
    libc.so.6 => /lib64/libc.so.6 (0x00007f8a3b641000)
    libdl.so.2 => /lib64/libdl.so.2 (0x00007f8a3b43c000)
    libpthread.so.0 => /lib64/libpthread.so.0 (0x00007f8a3b220000)
    librt.so.1 => /lib64/librt.so.1 (0x00007f8a3b018000)
    libnvidia-fatbinaryloader.so.390.30 => /usr/lib64/nvidia/libnvidia-fatbinaryloader.so.390.30 (0x00007f8a3adcb000)
    /lib64/ld-linux-x86-64.so.2 (0x00005622f7b92000)

Not sure, I could be totally wrong but is it something to do with the compiler section in Make.CUDA

• Compilers / linkers - Optimization flags ---------------------------

---

next two lines for GNU Compilers:
CC = mpicc
CCFLAGS = -fopenmp -lpthread -fomit-frame-pointer -O3 -funroll-loops $(HPL_DEFS)
CCNOOPT = $(HPL_DEFS) -O0 -w

On some platforms, it is necessary to use the Fortran linker to find
the Fortran internals used in the BLAS library.

LINKER = $(CC)
LINKFLAGS = $(CCFLAGS)

ARCHIVER = ar
ARFLAGS = r
RANLIB = echo

I used mpif77 as a linker while running it for CPUs but i guess this shouldn't be a problem as it is a linker.?

[martin-frbg]

No idea, but I do not think mpif77 has anything to do with it. Are you sure that the xhpl binary you are running the ldd on is actually the one that gets called by your run_linpack script ?

[karanchhabra]

yes it is the one which gets called by my run_linpack script. I did check all dependencies for xhpl and there sub dependencies but can't see any relevance to lib_mkl. The most wiered thing is when i'm not using intel's MKL than why is HPL asking me for the library which is specifically provided by intel with MKL.

run_linpack:
----#!/bin/sh

----export HPL_DIR=/root/hpl/hpl-2.0_FERMI_v15_latest
----CPU_CORES_PER_GPU=128
----export CUDA_DGEMM_SPLIT=0.21
----export CUDA_DTRSM_SPLIT=0.11
----$HPL_DIR/bin/CUDA/xhpl > $HPL_DIR/HPL.out

[matveevma]

Hi all!

This problem with HPL very important for me.
@karanchhabra if you use

$ cd HplSourceDir
$ grep -rn mk

you will see many matches with src/cuda/cuda_dgemm.c. Content this source file include some logic about MKL, ACML and GOTO. I think libdgemm.so compiled espetially for MKL.

I am still try find solution for HPL without Intel implementations. Do you have new results?

[brada4]

Probably if you remove either MKL or OpenBLAS from build machine, HPL goes with one remaining.

[matveevma]

Similar error about MKL: libmkl_intel_lp64.so: cannot open shared object file: No such file or directory. Maybe exist newest libdgemm.so for CUDA?

[brada4]

This is not Intel support forum.
You need to look for "MKL link line advisor" to fix MKL error.

If you want OpenBLAS, do your best to move MKL out of compilers way.

[matveevma]

Ok, i will install ACML or GOTO because dgemm.c included it.

Thanks!

[martin-frbg]

Perhaps someone on the nvidia CUDA ZONE forum will know the answer (though I see now that karanchhabra did not get much help there last year) - maybe you can even get away with copying some other library to the libmkl_intel_lp64.so name (if nothing actually uses it)

[brada4]

@matveevma I actually asked to remove surplus libraries, not to add more and more.

[matveevma]

As i say earlier, at first my system have not BLAS libraries. Then i try compile HPL with ATLAS - return MKL error, then deleted ATLAS. Next try - with Lapack from netlib - return MKL error.

libmkl_intel_lp64.so: cannot open shared object file: No such file or directory

grep -rn mkl show matches with dgemm source and i see hardcode about MKL in file `src/cuda/cuda_dgemm.c". I will asked NVIDIA (again) about newest libdgemm.so.

[martin-frbg]

@matveevma I guess the source for their CUDA dgemm is not available, and you have to rely on source comments in the objects ? (Could be NVIDIA relies on some environment variable being set or unset for their decision to import mkl, but I should really hope they remembered to document that somewhere. I suppose strace is no help either)

[brada4]

There is original HPL without nvidia's customisation:
http://www.netlib.org/benchmark/hpl/
Does it work?

[matveevma]

Yes, running without problem with IntelMPI and MKL. I am checking another CPU cases with OpenMPI and none MKL

[brada4]

Yes, you need to dig through HPL-2.3 configure script
4136: cat <<HPLEOF > hplvars.txt
I see no chance it could detect absent MKL libraries.
Certainly MKL and OpenBLAS and reference BLAS are all equally supported.

[matveevma]

Fixed by link

[user1@ac922 hpl-2.0_FERMI_v15]$ vi ./src/cuda/cuda_dgemm.c
…
// handle2 = dlopen (“libmkl_intel_lp64.so”, RTLD_LAZY);
handle2 = dlopen (“libopenblas.so”, RTLD_LAZY);
…
// dgemm_mkl = (void(*)())dlsym(handle, “dgemm”);
dgemm_mkl = (void(*)())dlsym(handle, “dgemm_”);
…
// handle = dlopen (“libmkl_intel_lp64.so”, RTLD_LAZY);
handle = dlopen (“libopenblas.so”, RTLD_LAZY);
…
// mkl_dtrsm = (void(*)())dlsym(handle2, “dtrsm”);
mkl_dtrsm = (void(*)())dlsym(handle2, “dtrsm_”);
…

Thanks all!

[brada4]

Probably you should report it to cudahpl authors, not here, default hpl picks up cuda or openblas just fine

[matveevma]

default hpl picks up cuda or openblas just fine

What you mean? CUDA support have only HPL 2.0 from NVIDIA. It is very old version for FERMI architecture and build from source code with newerlest libraries not so easy now.

[martin-frbg]

Thx, added to the faq in the wiki (for want of a better place)

[brada4]

I mean absence of newer nvidia distribution means that open-source HPL distribution already works fine with CUDA (it imports only dgemm from supposed BLAS library)

[martin-frbg]

@brada4 I believe you must be mistaken - I see no mention of CUDA whatsoever in either the online HPL documentation at netlib.org nor the INSTALL (etc) document included in the hpl-2.3 release of december 2018. I do not think the modifications in the NVIDIA version of HPL are as simple as just specifying libcuda as its BLAS.

[matveevma]

@martin-frbg, i think @brada4 idea very interesting. I will check integrating dgemm functionality to another blas libraries. I mean building HPL 2.3 from netlib with openblas, cuda and openmpi and i hope it will be work with new HPL version because blas libraries newest too.

The second way to use new HPL with cuda :

$ diff hpl-2.0-directory hpl-2.3-directory >> diff-hpl-versions
$ diff hpl-2.0-cuda-directory hpl-2.3-directory >> diff-cuda

Remove copyrights for easy compare diff results.

This case works fine for me.

[brada4]

@martin-frbg

Yes, you need to dig through HPL-2.3 configure script
4136: cat < hplvars.txt
I see no chance it could detect absent MKL libraries.

[MarpleJane]

Still not quite right, can you please change the LAlib line in your file to LAlib = -L $(TOPdir)/src/cuda -ldgemm -L/usr/local/cuda/lib64 -lcublas -lcuda -lcudart -L$(LAdir) -lopenblas -lpthread

Helps a lot!