Compiling cuda binaries

[amitd]

I've been trying to get SMARTG working on my machine, and been having trouble compiling the binaries. I get this error when trying to create module_from_buffer: cuModuleLoadDataEx failed: a PTX JIT compilation failed - ptxas application ptx input, line 9; fatal : Unsupported .version 6.3; current version is '6.1'

I've tried every combination of the cudatoolkit library, and now I'm wondering if it is a hardware support issue with the binary. Do you mind sharing which versions of CUDA have worked for you or a list of supported hardware? For reference I am using a GTX 1080.

A

[LaurentWandrebeck]

Hello amitd,

We currently use Cuda 10.0.130 coming straight from official Nvidia repository: https://developer.nvidia.com/cuda-downloads with the recommended driver (410.79) provided by the same repository. Our development platform runs CentOS 7.6 with 6x0, 9x0 and 10x0 models.
Could you tell us more about your software programs versions so we can try to find where the problem is ?
We’ll add in a following release versions used for building it and enhance our software versions/hardware models tests.

Thanks for your interest in SMART-G.

[amitd]

Thanks for the help. In the end I had to use CUDA 8.0 as pycuda would only compile with that version. I am on Ubuntu 18.04, which means I also had to point CUDA to a version of gcc and g++ below 5.0.

[clemgoyens]

Hi Amitd,

I got the same error and followed your recommendations (also working on Ubuntu 18.04):
I installed CUDA 8.0 from the toolkit archive and then added the symbolic links to gcc-4.8 and g++-4.8.
However I still have an error when trying to run smartg "cuInit failed: no CUDA-capable device is detected"
Could you maybe share how you managed to succeed?
Thanks a lot!!
Regards,
Clémence

[fsteinmetz]

Dear Clémence,

Concerning pycuda installation we are internally using either pip install pycuda or https://anaconda.org/lukepfister/pycuda.
Also, please make sure to use latest version 0.9.2 which ships with source code instead of compiled binaries : it easens significantly the flexibility of the installation.
Cheers,
François

[clemgoyens]

Dear Francois,

Thanks for your reply!
I have smartg-v0.9.2. with the dependencies installed as mentioned in the readme.md, i.e.,
"conda install -c conda-forge lukepfister::pycuda numpy scipy nose notebook matplotlib netcdf4 progressbar2 pyhdf"

After to test smartg on my computer I tried to run the demo_notebook with jupyter. When importing the modules I got the error:
"ImportError: libcurand.so.8.0: cannot open shared object file: No such file or directory"
So I followed the recommendations of Amitd and installed CUDA 8.0 (as according to the error message I believed the error was related to the wrong CUDA version?? I had CUDA V9.1.85 already installed).
Downloaded and installed the cuda_8.0.61 version from https://developer.nvidia.com/cuda-toolkit-archive
I kept all default installation requirements except:
"Install NVIDIA Accelerated Graphics Driver for Linux-x86_64 375.26?
(y)es/(n)o/(q)uit: n"

After I linked gcc and g++-4.8 to my cuda installation:
ln -s /usr/bin/g++-4.8 /usr/local/cuda-8.0/bin/g++
ln -s /usr/bin/gcc-4.8 /usr/local/cuda-8.0/bin/gcc

and added the following paths in my environment:
PATH=$PATH:/usr/local/cuda-8.0/bin
CUDADIR=/usr/local/cuda-8.0
LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/usr/local/cuda-8.0/lib64

After this I'm able to import all the modules and functions when running demo_notebook but when I want to run smartG I got the error message
"RuntimeError: cuInit failed: no CUDA-capable device is detected"

I'm not an expert in such things so maybe I missed something stupid? I also found a lot of different ways on google to install CUDA on Ubuntu 18.04.2 LTS (including suggestions to reinstall Ubuntu at the end - which I would like to avoid ).

Thanks in advance for your help!!
Regards,
Clémence

[LaurentWandrebeck]

Hello Clémence,

your hardware may be too new for cuda 8. Can you tell me which card you possess ? Have you been able to run any cuda code (cuda samples are an easy try) with cuda 8 ?

Also, I would need the driver version you use. Could you please try installing pycuda with pip install instead of lukepfister conda package ?

Another way to run smartg with a newer cuda version would be to keep lukepfister package, make a symbolic link named libcurand.so.8.0 pointing to libcurand.so.9.0 (or .1 or .2) if you’re willing to run it with cuda 9.0, 9.1 or 9.2.

for example:

Code: Select all

cd /usr/local/cuda-9.2/targets/x86_64-linux/lib/
ln -s libcurand.so.9.2 libcurand.so.8.0

Hope that helps,

[clemgoyens]

Hi Laurent,

Thanks a lot for you help and sorry for my late reply. I had to focus on other things for some time and didn't had the time to do all your tests.

Can you tell me which card you possess ?
VGA compatible controller: Intel Corporation Skylake GT2 [HD Graphics 520] (rev 07)

Have you been able to run any cuda code (cuda samples are an easy try) with cuda 8 ?
No I keep having the same error:
./deviceQuery Starting...
CUDA Device Query (Runtime API) version (CUDART static linking)
cudaGetDeviceCount returned 38
-> no CUDA-capable device is detected
Result = FAIL


At this stage I only have CUDA Version 8.0.61/CUDA Patch Version 8.0.61.2 installed
So if you think the best way is to make a symbolic link to CUDA version 9, I will try to reinstall CUDA version 9!

Thanks again for your help!

[LaurentWandrebeck]

Hello Clémence,

Sorry, our README was probably not clear enough. You need a Nvidia graphic card to be able to run any CUDA code. Without such hardware, you simply can’t run SMART-G. If you need any advice to buy some card, feel free to ask.

Best,