Skip to content

question on interaction of compiler options  #517

Closed
Closed
question on interaction of compiler options #517

Description

@grburgess
grburgess
opened on Jan 20, 2022

Summary:

I'm curious how the build of Cmdstan with a make/local file and the compiling of a model with supplied compiler args interact with one another

Description:

I'm constantly doing a lot of profiling on models where I'm switching at the CXXFlags/ STAN options when building Cmdstan. Since one can specify those flags in the make/local as well as when building a model via cmdstanpy, I was curious how these things interact. E.g., If I build Cmdstan:

make build

and my make/local has

STAN_OPENCL=True

Does that opencl option when building the model model override what is in make/local? So, if I set to false in cmdstanpy, it will be overridden, but if I do not set it, then it will always be tru?

Additional Information:

Provide any additional information here.

Current Version:

1.0.0

Activity

  1. ahartikainen commented on Jan 20, 2022

    @ahartikainen
    ahartikainen
    on Jan 20, 2022
    Contributor

    Good question. ( I don't have an answer to this)

    which one is preferred a cmdline argument or the make/local file.

    (Question for anyone needing to debug this, is there a way to print in somewhere while building is going on)

  2. grburgess commented on Jan 20, 2022

    @grburgess
    grburgess
    on Jan 20, 2022
    ContributorAuthor

    I have noticed that if I turn on OpenCL in the make/local, but say nothing in the cmdline, I get the following output during the build:

    WARNING:cmdstanpy:g++ -funroll-loops -mfpmath=sse -DEIGEN_USE_BLAS -DEIGEN_USE_LAPACKE -mtune=native -march=native -std=c++1y -pthread -D_REENTRANT -Wno-sign-compare -Wno-ignored-attributes   -I stan/lib/stan_math/lib/opencl_3.0.0  -DSTAN_THREADS -I stan/lib/stan_math/lib/tbb_2020.3/include -flto -fuse-linker-plugin -fdevirtualize-at-ltrans -fweb -fivopts -ftree-loop-linear -floop-strip-mine -floop-block -floop-nest-optimize -ftree-vectorize -ftree-loop-distribution -funroll-loops -floop-unroll-and-jam -fsplit-loops -fvisibility=hidden -fvisibility-inlines-hidden -DSTAN_NO_RANGE_CHECKS -O3 -I src -I stan/src -I lib/rapidjson_1.1.0/ -I lib/CLI11-1.9.1/ -I stan/lib/stan_math/ -I stan/lib/stan_math/lib/eigen_3.3.9 -I stan/lib/stan_math/lib/boost_1.75.0 -I stan/lib/stan_math/lib/sundials_6.0.0/include -I stan/lib/stan_math/lib/sundials_6.0.0/src/sundials    -DBOOST_DISABLE_ASSERTS  -DSTAN_OPENCL -DOPENCL_DEVICE_ID=0 -DOPENCL_PLATFORM_ID=0 -DCL_HPP_TARGET_OPENCL_VERSION=120 -DCL_HPP_MINIMUM_OPENCL_VERSION=120 -DCL_HPP_ENABLE_EXCEPTIONS -Wno-ignored-attributes      -DSTAN_NO_RANGE_CHECKS  -c -MT stan/src/stan/model/model_header_threads_opencl_nochecks.hpp.gch -MT stan/src/stan/model/model_header_threads_opencl_nochecks.d -MM -E -MG -MP -MF stan/src/stan/model/model_header_threads_opencl_nochecks.d stan/src/stan/model/model_header.hpp

--- Translating Stan model to C++ code ---
bin/stanc --include_paths=/home/jburgess/sw/whimstan/whimstan/stan_code --o=/home/jburgess/sw/whimstan/whimstan/stan_code/whim.hpp /home/jburgess/sw/whimstan/whimstan/stan_code/whim.stan
--include_paths is deprecated and Will be removed in Stan 2.32.0. Please use --include-paths.
Warning in '/home/jburgess/sw/whimstan/whimstan/stan_code/whim.stan', line 43, column 2: Declaration
    of arrays by placing brackets after a variable name is deprecated and
    will be removed in Stan 2.32.0. Instead use the array keyword before the
    type. This can be changed automatically using the auto-format flag to
    stanc

--- Compiling, linking C++ code ---
g++ -funroll-loops -mfpmath=sse -DEIGEN_USE_BLAS -DEIGEN_USE_LAPACKE -mtune=native -march=native -std=c++1y -pthread -D_REENTRANT -Wno-sign-compare -Wno-ignored-attributes   -I stan/lib/stan_math/lib/opencl_3.0.0  -DSTAN_THREADS -I stan/lib/stan_math/lib/tbb_2020.3/include -flto -fuse-linker-plugin -fdevirtualize-at-ltrans -fweb -fivopts -ftree-loop-linear -floop-strip-mine -floop-block -floop-nest-optimize -ftree-vectorize -ftree-loop-distribution -funroll-loops -floop-unroll-and-jam -fsplit-loops -fvisibility=hidden -fvisibility-inlines-hidden -DSTAN_NO_RANGE_CHECKS -O3 -I src -I stan/src -I lib/rapidjson_1.1.0/ -I lib/CLI11-1.9.1/ -I stan/lib/stan_math/ -I stan/lib/stan_math/lib/eigen_3.3.9 -I stan/lib/stan_math/lib/boost_1.75.0 -I stan/lib/stan_math/lib/sundials_6.0.0/include -I stan/lib/stan_math/lib/sundials_6.0.0/src/sundials    -DBOOST_DISABLE_ASSERTS  -DSTAN_OPENCL -DOPENCL_DEVICE_ID=0 -DOPENCL_PLATFORM_ID=0 -DCL_HPP_TARGET_OPENCL_VERSION=120 -DCL_HPP_MINIMUM_OPENCL_VERSION=120 -DCL_HPP_ENABLE_EXCEPTIONS -Wno-ignored-attributes      -DSTAN_NO_RANGE_CHECKS  -c -Wno-ignored-attributes -fweb -fivopts -ftree-loop-linear -floop-strip-mine -floop-block -floop-nest-optimize -ftree-vectorize -ftree-loop-distribution -funroll-loops -floop-unroll-and-jam -fsplit-loops -fvisibility=hidden -fvisibility-inlines-hidden -flto -fuse-linker-plugin -fdevirtualize-at-ltrans -x c++ -o /home/jburgess/sw/whimstan/whimstan/stan_code/whim.o /home/jburgess/sw/whimstan/whimstan/stan_code/whim.hpp
g++ -funroll-loops -mfpmath=sse -DEIGEN_USE_BLAS -DEIGEN_USE_LAPACKE -mtune=native -march=native -std=c++1y -pthread -D_REENTRANT -Wno-sign-compare -Wno-ignored-attributes   -I stan/lib/stan_math/lib/opencl_3.0.0  -DSTAN_THREADS -I stan/lib/stan_math/lib/tbb_2020.3/include -flto -fuse-linker-plugin -fdevirtualize-at-ltrans -fweb -fivopts -ftree-loop-linear -floop-strip-mine -floop-block -floop-nest-optimize -ftree-vectorize -ftree-loop-distribution -funroll-loops -floop-unroll-and-jam -fsplit-loops -fvisibility=hidden -fvisibility-inlines-hidden -DSTAN_NO_RANGE_CHECKS -O3 -I src -I stan/src -I lib/rapidjson_1.1.0/ -I lib/CLI11-1.9.1/ -I stan/lib/stan_math/ -I stan/lib/stan_math/lib/eigen_3.3.9 -I stan/lib/stan_math/lib/boost_1.75.0 -I stan/lib/stan_math/lib/sundials_6.0.0/include -I stan/lib/stan_math/lib/sundials_6.0.0/src/sundials    -DBOOST_DISABLE_ASSERTS  -DSTAN_OPENCL -DOPENCL_DEVICE_ID=0 -DOPENCL_PLATFORM_ID=0 -DCL_HPP_TARGET_OPENCL_VERSION=120 -DCL_HPP_MINIMUM_OPENCL_VERSION=120 -DCL_HPP_ENABLE_EXCEPTIONS -Wno-ignored-attributes      -DSTAN_NO_RANGE_CHECKS        -Wl,-L,"/home/jburgess/.cmdstan/cmdstan/stan/lib/stan_math/lib/tbb" -Wl,-rpath,"/home/jburgess/.cmdstan/cmdstan/stan/lib/stan_math/lib/tbb"   -flto -fuse-linker-plugin -fdevirtualize-at-ltrans   /home/jburgess/sw/whimstan/whimstan/stan_code/whim.o src/cmdstan/main_threads_opencl_nochecks.o -lblas -llapack -llapacke      -lOpenCL  -Wl,-L,"/home/jburgess/.cmdstan/cmdstan/stan/lib/stan_math/lib/tbb" -Wl,-rpath,"/home/jburgess/.cmdstan/cmdstan/stan/lib/stan_math/lib/tbb"   stan/lib/stan_math/lib/sundials_6.0.0/lib/libsundials_nvecserial.a stan/lib/stan_math/lib/sundials_6.0.0/lib/libsundials_cvodes.a stan/lib/stan_math/lib/sundials_6.0.0/lib/libsundials_idas.a stan/lib/stan_math/lib/sundials_6.0.0/lib/libsundials_kinsol.a  stan/lib/stan_math/lib/tbb/libtbb.so.2 -o /home/jburgess/sw/whimstan/whimstan/stan_code/whim
rm -f /home/jburgess/sw/whimstan/whimstan/stan_code/whim.o

    I do see the OpenCL stuff popping up.

  3. WardBrian commented on Jan 20, 2022

    @WardBrian
    WardBrian
    on Jan 20, 2022
    Member

    CmdStanPy does do something special related to opencl, which is if you pass the use-opencl flag to stanc, it will also turn on STAN_OPENCL in the C++ options.

    There are two things I'm not sure about -

    1. How command line arguments and make/local compose
    2. What STAN_OPENCL does if the model is not compiled with use-opencl in stanc.

    @rok-cesnovar - do you have a better answer for either of those?

  4. mitzimorris commented on Jan 20, 2022

    @mitzimorris
    mitzimorris
    on Jan 20, 2022
    Member

    command line arguments to make override make/local argument.
    https://ftp.gnu.org/old-gnu/Manuals/make-3.79.1/html_chapter/make_9.html#SEC90

    here's what's in the CmdStan docs w/r/t OpenCL - @rok-cesnovar are these correct? https://mc-stan.org/docs/cmdstan-guide/parallelization.html#opencl

  5. rok-cesnovar commented on Jan 20, 2022

    @rok-cesnovar
    rok-cesnovar
    on Jan 20, 2022
    Member

    What STAN_OPENCL does if the model is not compiled with use-opencl in stanc.

    Nothing really as none of the OpenCL-related stuff will be used, so this just adds a bit more code in C++ so a bit longer compile times. You need the use-opencl flag in stanc3 in order to generate the model with the OpenCL-related constructs (matrix_cl class, etc) and setting STAN_OPENCL then enables the OpenCL code in Stan Math so that the generated C++ actually compiles.

    To sum up:

    • no use-opencl stanc3 flag & STAN_OPENCL=False -> normal sequential case
    • no use-opencl stanc3 flag & STAN_OPENCL=True -> same as sequential code, a bit longer compile times excepted
    • use-opencl stanc3 flag & STAN_OPENCL= False -> if any OpenCL code was generated in stanc3, the C++ will not compile
    • use-opencl stanc3 flag & STAN_OPENCL= True -> OpenCL model

    are these correct?

    They are.

    @grburgess if you are ever in doubt about what gets passed to the compiler, I suggest using ˙make print-variable` command. Its precisely used for debugging these kind of stuff.

    I set STAN_OPENCL=false in make/local and run:

    rok@rok-Z97X-UD5H:~/Desktop/cmdstan$ make print-STAN_OPENCL
STAN_OPENCL = false
rok@rok-Z97X-UD5H:~/Desktop/cmdstan$ make STAN_OPENCL=true print-STAN_OPENCL
STAN_OPENCL = true

  6. rok-cesnovar commented on Jan 20, 2022

    @rok-cesnovar
    rok-cesnovar
    on Jan 20, 2022
    Member

    And make is its own beast, I never assume I know anything without triple-checking and even then I don't trust it haha

  7. grburgess commented on Jan 20, 2022

    @grburgess
    grburgess
    on Jan 20, 2022
    ContributorAuthor

    Awesome. I think I understand the workflow now. Thanks!

  8. stan-dev
    locked and limited conversation to collaborators on Jan 21, 2022
  9. WardBrian
    converted this issue into a discussion #519 on Jan 21, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions