Can not compile ConvertUBOToPushConstant.cpp when using external SPIRV-Tools

[uilianries]

Hello! 👋

We are updating DiligentCore in Conan Center to API256019. The Conan recipe does not use the vendored ThirdParty/SPIRV-Tools tree, instead, it uses an installed SPIRV-Tools that only ships the public headers (include/spirv-tools/).

That layout still works for SPIRVTools.cpp (#include "spirv-tools/optimizer.hpp"). It fails on the custom pass added in #737:

Graphics/ShaderTools/src/ConvertUBOToPushConstant.cpp:30:10: fatal error: 'source/opt/pass.h' file not found

You can see my full build log: diligent-core-256019-macos-armv8-clang17-release-static.log

I understand the default build is consistent: add_subdirectory(ThirdParty/SPIRV-Tools) gives source/opt/pass.h (and ir_context.h, type_manager.h, decoration_manager.h). #362 then skips that subdirectory when SPIRV-Tools-static/SPIRV-Tools-opt already exist. ShaderTools CMake also does get_target_property(SOURCE_DIR SPIRV-Tools-opt) and adds that path for those private headers.

That works if the existing target still comes from an vendorized SPIRV-Tools source directory. It does not work for an imported target from find_package. Also, Khronos does not install source/opt/, and SOURCE_DIR is not a checkout.

It would help packagers if Diligent could keep supporting installed SPIRV-Tools as in #362, without requiring those private headers. Regards!

[lolrobbe2]

@uilianries currently also trying to update recipe. and hit the same problem

[lolrobbe2]

Graphics/ShaderTools/src/ConvertUBOToPushConstant.cpp:30:10: fatal error: 'source/opt/pass.h' file not found

note the file is availble. just not found

edit:

try adding the following in the replace_files:

    def _patch_sources(self):
        apply_conandata_patches(self)
        replace_in_file(self, os.path.join(self.source_folder, "CMakeLists.txt"),
                        "project(DiligentCore)",
                        "project(DiligentCore)\n\ninclude(conan_deps.cmake)")
        replace_in_file(
            self,
            os.path.join(self.source_folder, "ThirdParty", "CMakeLists.txt"),
            "if (NOT ${DILIGENT_NO_GLSLANG} AND (NOT TARGET glslang))",
            "if (NOT ${DILIGENT_NO_GLSLANG} AND (NOT TARGET glslang AND NOT TARGET glslang::glslang))"
        )

        # 2. Disable USE_SPIRV_TOOLS in ShaderTools (prevents missing source/opt/pass.h)
        replace_in_file(
            self,
            os.path.join(self.source_folder, "Graphics", "ShaderTools", "CMakeLists.txt"),
            "set(USE_SPIRV_TOOLS TRUE)",
            "set(USE_SPIRV_TOOLS FALSE)"
        )

conandata.yml:

sources:
  "api.256019":
    url: https://github.com/DiligentGraphics/DiligentCore/archive/744f079.tar.gz
    sha256: a227d7c83c92a10c41eec1366d1d303b2150df3c6ad11d4dcec044d37d2a61fb

[TheMostDiligent]

There seems to be no easy fix to that issue right now. Disabling USE_SPIRV_TOOLS will make run-time shader compilation in Vulkan dysfunctional.
The best solution would be to upstream the pass to SPIRV-Tools.

[TheMostDiligent]

@hzqst FYI

[lolrobbe2]

@TheMostDiligent i have gotten past said issue. but still niot getting full build.

conanfile.py:

from conan import ConanFile
from conan.tools.cmake import CMake, CMakeToolchain, cmake_layout, CMakeDeps
from conan.errors import ConanInvalidConfiguration
from conan.tools.build import cross_building, check_min_cppstd
from conan.tools.scm import Version
from conan.tools.files import rm, get, rmdir, rename, collect_libs, export_conandata_patches, copy, apply_conandata_patches, replace_in_file
from conan.tools.microsoft import visual
from conan.tools.apple import is_apple_os
import os
from pathlib import Path
required_conan_version = ">=1.52.0"


class DiligentCoreConan(ConanFile):
    name = "diligent-core"
    url = "https://github.com/conan-io/conan-center-index"
    homepage = "https://github.com/DiligentGraphics/DiligentCore"
    description = "Diligent Core is a modern cross-platfrom low-level graphics API."
    license = "Apache-2.0"
    topics = ("graphics")
    settings = "os", "compiler", "build_type", "arch"
    options = {
        "shared": [True, False],
        "fPIC":   [True, False],
        "with_glslang": [True, False],
    }
    default_options = {
        "shared": False	,
        "fPIC": True,
        "with_glslang": True
    }
    short_paths = True

    @property
    def _minimum_compilers_version(self):
        return {
            "Visual Studio": "16",
            "msvc": "192",
            "gcc": "6",
            "clang": "3.4",
            "apple-clang": "5.1",
        }

    @property
    def _minimum_cpp_standard(self):
        return 14

    def validate(self):
        if self.settings.compiler.get_safe("cppstd"):
            check_min_cppstd(self, self._minimum_cpp_standard)
        min_version = self._minimum_compilers_version.get(str(self.settings.compiler))
        if not min_version:
            self.output.warning("{} recipe lacks information about the {} compiler support.".format(
                self.name, self.settings.compiler))
        else:
            if Version(self.settings.compiler.version) < min_version:
                raise ConanInvalidConfiguration("{} requires C++{} support. The current compiler {} {} does not support it.".format(
                    self.name, self._minimum_cpp_standard, self.settings.compiler, self.settings.compiler.version))
        if visual.is_msvc_static_runtime(self):
            raise ConanInvalidConfiguration("Visual Studio build with MT runtime is not supported")

    def export_sources(self):
        copy(self, "conan_deps.cmake", src=self.recipe_folder, dst=os.path.join(self.export_sources_folder, "src"), keep_path=False)
        export_conandata_patches(self)

    def source(self):
        get(self, **self.conan_data["sources"][self.version], strip_root=True)

    def package_id(self):
        if visual.is_msvc(self.info):
            if visual.is_msvc_static_runtime(self.info):
                self.info.settings.compiler.runtime = "MT/MTd"
            else:
                self.info.settings.compiler.runtime = "MD/MDd"

    def generate(self):
        tc = CMakeToolchain(self)
        tc.variables["DILIGENT_BUILD_SAMPLES"] = False
        tc.variables["DILIGENT_NO_FORMAT_VALIDATION"] = True
        tc.variables["DILIGENT_BUILD_TESTS"] = False
        tc.variables["DILIGENT_NO_DXC"] = True
        tc.variables["DILIGENT_NO_GLSLANG"] = not self.options.with_glslang
        #tc.variables["SPIRV_CROSS_NAMESPACE_OVERRIDE"] = self.dependencies["spirv-cross"].options.namespace
        tc.variables["DILIGENT_CLANG_COMPILE_OPTIONS"] = ""
        tc.variables["DILIGENT_MSVC_COMPILE_OPTIONS"] = ""
        tc.variables["ENABLE_RTTI"] = True
        tc.variables["ENABLE_EXCEPTIONS"] = True
        tc.variables["DILIGENT_BUILD_CORE_THIRD_PARTY"] = False
        spirv_tools_package = Path(self.dependencies["spirv-tools"].package_folder)

        # ...\spirv-tools\<hash>\p
        # -> ...\spirv-tools\<hash>\b\src
        spirv_tools_src = (
            spirv_tools_package.parent / "b" / "src"
        ).as_posix()

        spirv_tools_include = (
            Path(spirv_tools_src) / "include"
        ).as_posix()

        spirv_tools_build = (
            spirv_tools_package.parent / "b" / "build"
        ).as_posix()

        print(f"SPIRV-Tools source directory: {spirv_tools_src}")
        print(f"SPIRV-Tools include directory: {spirv_tools_include}")
        print(f"SPIRV-Tools build directory: {spirv_tools_build}")

        tc.variables["CONAN_SPIRV_TOOLS_SOURCE_DIR"] = spirv_tools_src
        tc.variables["CONAN_SPIRV_TOOLS_INCLUDE_DIR"] = spirv_tools_include
        tc.variables["CONAN_SPIRV_TOOLS_BUILD_DIR"] = spirv_tools_build

        spirv_headers_package = Path(
            self.dependencies["spirv-headers"].package_folder
        )

        spirv_headers_include = (
            spirv_headers_package / "include"
        ).as_posix()
        spirv_headers_unified_include = (
                    spirv_headers_package / "include" / "spirv" / "unified1"
                ).as_posix()
        print(f"SPIRV-Headers include directory: {spirv_headers_include}")
        print(f"SPIRV-Headers unified include directory: {spirv_headers_unified_include}")

        tc.variables["CONAN_SPIRV_HEADERS_INCLUDE_DIR"] = spirv_headers_include
        tc.variables["CONAN_SPIRV_HEADERS_UNIFIED_INCLUDE_DIR"] = spirv_headers_unified_include

        tc.variables[self._diligent_platform()] = True

        if self.settings.os == "Windows":
            tc.variables["VK_USE_PLATFORM_WIN32_KHR"] = True
            tc.variables["DILIGENT_USE_VOLK"] = True
        tc.generate()

        deps = CMakeDeps(self)
        deps.generate()

    def layout(self):
        cmake_layout(self, src_folder="src")

    def configure(self):
        if self.options.shared:
            self.options.rm_safe("fPIC")

    def config_options(self):
        if self.settings.os == "Windows":
            self.options.rm_safe("fPIC")

    def _patch_sources(self):
        apply_conandata_patches(self)
        replace_in_file(self, os.path.join(self.source_folder, "CMakeLists.txt"),
                        "project(DiligentCore)",
                        "project(DiligentCore)\n\ninclude(conan_deps.cmake)")
        replace_in_file(
            self,
            os.path.join(self.source_folder, "ThirdParty", "CMakeLists.txt"),
            "if (NOT ${DILIGENT_NO_GLSLANG} AND (NOT TARGET glslang))",
            "if (NOT ${DILIGENT_NO_GLSLANG} AND (NOT TARGET glslang AND NOT TARGET glslang::glslang))"
        )

        # 2. Disable USE_SPIRV_TOOLS in ShaderTools (prevents missing source/opt/pass.h)


    def build_requirements(self):
        self.tool_requires("cmake/[>=3.24]")

    def requirements(self):
        self.requires("opengl/system")
        if self.settings.os == "Linux":
            self.requires("wayland/1.24.0")

        self.requires("spirv-headers/1.4.350.0")
        self.requires("spirv-cross/1.4.350.0")
        self.requires("spirv-tools/1.4.350.0")
        if self.options.with_glslang:
            self.requires("glslang/1.4.350.0")
        self.requires("vulkan-headers/1.4.350.0")
        self.requires("vulkan-validationlayers/1.4.350.0")
        self.requires("volk/1.4.350.0")
        self.requires("xxhash/0.8.3")
        if self.settings.os in ["Linux", "FreeBSD"]:
            self.requires("xorg/system")
            if not cross_building(self, skip_x64_x86=True):
                self.requires("xkbcommon/1.13.1")

    def _diligent_platform(self):
        if self.settings.os == "Windows":
            return "PLATFORM_WIN32"
        elif self.settings.os == "Macos":
            return "PLATFORM_MACOS"
        elif self.settings.os == "Linux":
            return "PLATFORM_LINUX"
        elif self.settings.os == "Android":
            return "PLATFORM_ANDROID"
        elif self.settings.os == "iOS":
            return "PLATFORM_IOS"
        elif self.settings.os == "Emscripten":
            return "PLATFORM_EMSCRIPTEN"
        elif self.settings.os == "watchOS":
            return "PLATFORM_TVOS"

    def build(self):
        self._patch_sources()
        cmake = CMake(self)
        # By default, Diligent builds static and shared versions of every main library. We select the one we
        # want based on options.shared in package(). To avoid building every intermediate library as SHARED,
        # we have to disable BUILD_SHARED_LIBS.
        # However, BUILD_SHARED_LIBS cannot be disabled normally (in the toolchain in configure()), because
        # Conan outputs that override after the standard line that enables BUILD_SHARED_LIBS. Since the latter
        # is a CACHE variable that cannot be overwritten with another set(), we have to specify it on the
        # command-line, so it takes effect before the toolchain is parsed.
        cmake.configure(variables={"BUILD_SHARED_LIBS": "OFF"})
        cmake.build()

    def package(self):
        cmake = CMake(self)
        cmake.install()
        rmdir(self, os.path.join(self.package_folder, "Licenses"))
        rmdir(self, os.path.join(self.package_folder, "lib"))
        rmdir(self, os.path.join(self.package_folder, "bin"))
        copy(self, "License.txt", dst=os.path.join(self.package_folder, "licenses"), src=os.path.join(self.package_folder, self.source_folder))

        if self.options.shared:
            copy(self, pattern="*.dylib", dst=os.path.join(self.package_folder, "lib"), src=self.build_folder, keep_path=False)
            copy(self, pattern="*.so", dst=os.path.join(self.package_folder, "lib"), src=self.build_folder, keep_path=False)
            copy(self, pattern="*.dll", dst=os.path.join(self.package_folder, "bin"), src=self.build_folder, keep_path=False)
            rm(self, os.path.join(self.package_folder, "lib"), "*.a", recursive=True)
            if self.settings.os != "Windows":
                rm(self, os.path.join(self.package_folder, "lib"), "*.lib", recursive=True)
        else:
            copy(self, pattern="*.a",   dst=os.path.join(self.package_folder, "lib"), src=self.build_folder, keep_path=False)
            copy(self, pattern="*.lib", dst=os.path.join(self.package_folder, "lib"), src=self.build_folder, keep_path=False)
            rm(self, os.path.join(self.package_folder, "lib"), "*.dylib", recursive=True)
            rm(self, os.path.join(self.package_folder, "lib"), "*.so", recursive=True)
            rm(self, os.path.join(self.package_folder, "lib"), "*.dll", recursive=True)

        copy(self, pattern="*.fxh", dst=os.path.join(self.package_folder, "res"), src=self.source_folder, keep_path=False)
        copy(self, "File2String*",  dst=os.path.join(self.package_folder, "bin"), src=self.source_folder, keep_path=False)
        rm(self, "*.pdb", self.package_folder, recursive=True)
        # MinGw creates many invalid files, called objects.a, remove them here:
        rm(self, "objects.a", self.package_folder, recursive=True)

    def package_info(self):
        self.cpp_info.libs = collect_libs(self)
        # included as discussed here https://github.com/conan-io/conan-center-index/pull/10732#issuecomment-1123596308
        self.cpp_info.includedirs.append(os.path.join(self.package_folder, "include"))
        self.cpp_info.includedirs.append(os.path.join(self.package_folder, "include", "Common"))

        self.cpp_info.includedirs.append(os.path.join("include"))
        self.cpp_info.includedirs.append(os.path.join("include", "Common", "interface"))
        self.cpp_info.includedirs.append(os.path.join("include", "Platforms", "interface"))
        self.cpp_info.includedirs.append(os.path.join("include", "Graphics", "GraphicsEngine", "interface"))
        self.cpp_info.includedirs.append(os.path.join("include", "Graphics", "GraphicsEngineVulkan", "interface"))
        self.cpp_info.includedirs.append(os.path.join("include", "Graphics", "GraphicsEngineOpenGL", "interface"))
        self.cpp_info.includedirs.append(os.path.join("include", "Graphics", "GraphicsAccessories", "interface"))
        self.cpp_info.includedirs.append(os.path.join("include", "Graphics", "GraphicsTools", "interface"))
        self.cpp_info.includedirs.append(os.path.join("include", "Graphics", "HLSL2GLSLConverterLib", "interface"))
        archiver_path = os.path.join("include", "Graphics", "Archiver", "interface")
        if os.path.isdir(archiver_path):
            self.cpp_info.includedirs.append(archiver_path)
        self.cpp_info.includedirs.append(os.path.join("src"))
        self.cpp_info.includedirs.append(os.path.join("include", "Primitives", "interface"))
        self.cpp_info.includedirs.append(os.path.join("include", "Platforms", "Basic", "interface"))
        if self.settings.os == "Android":
            self.cpp_info.includedirs.append(os.path.join("include", "Platforms", "Android", "interface"))
        elif is_apple_os(self):
            self.cpp_info.includedirs.append(os.path.join("include", "Platforms", "Apple", "interface"))
        elif self.settings.os == "Emscripten":
            self.cpp_info.includedirs.append(os.path.join("include", "Platforms", "Emscripten", "interface"))
        elif self.settings.os == "Linux":
            self.cpp_info.includedirs.append(os.path.join("include", "Platforms", "Linux", "interface"))
        elif self.settings.os == "Windows":
            self.cpp_info.includedirs.append(os.path.join("include", "Platforms", "Win32", "interface"))
            self.cpp_info.includedirs.append(os.path.join("include", "Graphics", "GraphicsEngineD3D11", "interface"))
            self.cpp_info.includedirs.append(os.path.join("include", "Graphics", "GraphicsEngineD3D12", "interface"))

        #self.cpp_info.defines.append("SPIRV_CROSS_NAMESPACE_OVERRIDE={}".format(self.dependencies["spirv-cross"].options.namespace))
        self.cpp_info.defines.append("{}=1".format(self._diligent_platform()))

        if self.settings.os in ["Macos", "Linux"]:
            self.cpp_info.system_libs = ["dl", "pthread"]
        if self.settings.os == 'Macos':
            self.cpp_info.frameworks = ["CoreFoundation", 'Cocoa', 'AppKit']
        if self.settings.os == 'Windows':
            self.cpp_info.system_libs = ["dxgi", "shlwapi"]

find_package(SPIRV-headers REQUIRED CONFIG)
find_package(SPIRV-Tools REQUIRED CONFIG)
find_package(spirv-cross REQUIRED CONFIG)
find_package(volk REQUIRED CONFIG)
find_package(xxHash REQUIRED CONFIG)

if (NOT DILIGENT_NO_GLSLANG)
    find_package(glslang REQUIRED CONFIG)

    add_library(glslang INTERFACE)

    target_link_libraries(glslang INTERFACE
        glslang::glslang
    )

    target_include_directories(glslang INTERFACE
        ${glslang_INCLUDE_DIR}/glslang
    )

    add_library(SPIRV ALIAS glslang::SPIRV)
endif()

add_library(SPIRV-Headers ALIAS SPIRV-Headers::SPIRV-Headers)
add_library(spirv-tools-core ALIAS spirv-tools::spirv-tools)
# Conan's SPIRV-Tools package contains the source tree under res/.
# Diligent ShaderTools includes private SPIRV-Tools headers such as
# <source/opt/pass.h>.
function(conan_patch_diligent_shadertools)
    if(TARGET Diligent-ShaderTools)
        target_include_directories(Diligent-ShaderTools
            PRIVATE
                ${CONAN_SPIRV_TOOLS_SOURCE_DIR}
                ${CONAN_SPIRV_TOOLS_INCLUDE_DIR}
                ${CONAN_SPIRV_TOOLS_BUILD_DIR}
                ${CONAN_SPIRV_HEADERS_INCLUDE_DIR}
                ${CONAN_SPIRV_HEADERS_UNIFIED_INCLUDE_DIR}
        )

        message(STATUS
            "Conan: SPIRV-Tools source: ${CONAN_SPIRV_TOOLS_SOURCE_DIR}"
        )
        message(STATUS
            "Conan: SPIRV-Tools include: ${CONAN_SPIRV_TOOLS_INCLUDE_DIR}"
        )
    endif()
endfunction()

cmake_language(DEFER CALL conan_patch_diligent_shadertools)

[lolrobbe2]

current problem: it cannot find the vulkan entry point for windows

  Building Custom Rule D:/Conan/b/dilig61bc4324b18d1/b/src/Graphics/GraphicsEngineVulkan/CMakeLists.txt
  DLLMain.cpp
     Creating library D:/Conan/b/dilig61bc4324b18d1/b/build/Graphics/GraphicsEngineVulkan/Release/GraphicsEngineVk_64r.lib and object D:/Conan/b/dilig
  61bc4324b18d1/b/build/Graphics/GraphicsEngineVulkan/Release/GraphicsEngineVk_64r.exp
Diligent-GraphicsEngineVk-static.lib(SwapChainVkImpl.obj) : error LNK2001: unresolved external symbol vkCreateWin32SurfaceKHR [D:\Conan\b\dilig61bc432
4b18d1\b\build\Graphics\GraphicsEngineVulkan\Diligent-GraphicsEngineVk-shared.vcxproj]
D:\Conan\b\dilig61bc4324b18d1\b\build\Graphics\GraphicsEngineVulkan\Release\GraphicsEngineVk_64r.dll : fatal error LNK1120: 1 unresolved externals [D:
\Conan\b\dilig61bc4324b18d1\b\build\Graphics\GraphicsEngineVulkan\Diligent-GraphicsEngineVk-shared.vcxproj]

[TheMostDiligent]

This seems to be unrelated to SPIRV-Tools.
Looks like VK_USE_PLATFORM_WIN32_KHR may be missing when building Volk.

[lolrobbe2]

it turns out when i added the build directories mannualy it build. but conan paths don't really line up well.

[lolrobbe2]

@uilianries, @TheMostDiligent https://github.com/lolrobbe2/conan-center-index/tree/update-dilligent-core is where the fixes have been added, the only problem is being able to tell volk to build for windows. The rest builds.

I am going to bed, 3:30 is a bit late.

[TheMostDiligent]

How was volk built previously? It's always been there

[lolrobbe2]

The vulkan related packages where updated recently for the matest version so there may have been some changes

[lolrobbe2]

@uilianries , @TheMostDiligent i got it building :)