[Crash] Pixel shader error with Ogre D3D11 [solved]

[eaducac]

I've searched around for this issue and couldn't find any references to it, so I thought I'd ask here. I'm completely new to CEGUI, Ogre, DirectX, and 3D graphics in general, so apologies if I'm unfamiliar with some of the concepts.

I'm experiencing a crash when using CEGUI 0.8.2 with Ogre set to use D3D11. After some digging it appears to be something with CEGUI and not Ogre. The "essential section" of my CEGUI.log:

Code: Select all

31/10/2013 00:30:18 (Std)    ---- Version: 0.8.2 (Build: Oct 30 2013 Debug Microsoft Windows MSVC++ 11.0 32 bit) ----
31/10/2013 00:30:18 (Std)    ---- Renderer module is: CEGUI::OgreRenderer - Official OGRE based 2nd generation renderer module. ----
31/10/2013 00:30:18 (Std)    ---- XML Parser module is: CEGUI::ExpatParser - Official expat based parser module for CEGUI ----
31/10/2013 00:30:18 (Std)    ---- Image Codec module is: OgreImageCodec - Integrated ImageCodec using the Ogre engine. ----
31/10/2013 00:30:18 (Std)    ---- Scripting module is: None ----

This is the first time I've ever used Ogre or CEGUI, so I'm trying to create a simple test window with a single button in it using the following code, which works with D3D9 and OpenGL:

Code: Select all

// GUI
      m_renderer = &CEGUI::OgreRenderer::bootstrapSystem();
      CEGUI::ImageManager::setImagesetDefaultResourceGroup("Imagesets");
      CEGUI::Font::setDefaultResourceGroup("Fonts");
      CEGUI::Scheme::setDefaultResourceGroup("Schemes");
      CEGUI::WidgetLookManager::setDefaultResourceGroup("LookNFeel");
      CEGUI::WindowManager::setDefaultResourceGroup("Layouts");

      CEGUI::SchemeManager::getSingleton().createFromFile("TaharezLook.scheme");
      CEGUI::GUIContext& context = CEGUI::System::getSingleton().getDefaultGUIContext();
      context.getMouseCursor().setDefaultImage("TaharezLook/MouseArrow");

      // for some reason this code crashes on DX11 with an assertion failure on mPixelShader in OgreD3D11HLSLProgram.cpp
      // test button
      CEGUI::Window *testWindow = CEGUI::WindowManager::getSingleton().createWindow("DefaultWindow", "RootWindow");
      // setRootWindow() appears to replace tutorial's use of setGUISheet()
      context.setRootWindow(testWindow);
      CEGUI::PushButton *testButton =
         static_cast<CEGUI::PushButton *>(CEGUI::WindowManager::getSingleton().createWindow("TaharezLook/Button", "TestButton"));
      testWindow->addChild(testButton);
      testButton->setPosition(CEGUI::UVector2(CEGUI::UDim(0.5, -50), CEGUI::UDim(0.5, -12)));
      testButton->setSize(CEGUI::USize(CEGUI::UDim(0, 100), CEGUI::UDim(0, 25)));
      testButton->setText("Yeah!");

When I set the Ogre renderer to D3D11, it dies with an assertion failure in Ogre. The Ogre log shows the following:

Code: Select all

00:57:55: OGRE EXCEPTION(3:RenderingAPIException): Cannot assemble D3D11 high-level shader __cegui_internal_ps__ Errors:
F:\Games\REimp\Shader@0x07CDB328(1,105): error X3000: syntax error: unexpected token 'texture'
 in D3D11HLSLProgram::loadFromSource at ..\..\..\..\..\RenderSystems\Direct3D11\src\OgreD3D11HLSLProgram.cpp (line 452)
00:57:55: High-level program __cegui_internal_ps__ encountered an error during loading and is thus not supported.
OGRE EXCEPTION(3:RenderingAPIException): Cannot assemble D3D11 high-level shader __cegui_internal_ps__ Errors:
F:\Games\REimp\Shader@0x07CDB328(1,105): error X3000: syntax error: unexpected token 'texture'
 in D3D11HLSLProgram::loadFromSource at ..\..\..\..\..\RenderSystems\Direct3D11\src\OgreD3D11HLSLProgram.cpp (line 452)
00:57:55: Texture: _cegui_ogre_0: Loading 1 faces(PF_A8B8G8R8,256x256x1) with 0 generated mipmaps from Image. Internal format is PF_A8B8G8R8,256x256x1.
00:57:55: Texture: _cegui_ogre_2: Loading 1 faces(PF_A8B8G8R8,256x256x1) with 0 generated mipmaps from Image. Internal format is PF_A8B8G8R8,256x256x1.


I found the __cegui_internal_ps__ shader in Renderer.cpp in project CEGUIOgreRenderer-0:

Code: Select all

static Ogre::String S_hlsl_ps_source(
    "float4 main(float4 colour : COLOR,"
    "            float2 texCoord : TEXCOORD0,"
    "            uniform sampler2D texture : TEXUNIT0) : COLOR"
    "{"
    "    return tex2D(texture, texCoord) * colour;"
    "}"
);

I don't know HLSL, but I searched MSDN and saw that the tex2D intrinsic is deprecated in Shader Model 4, which is what it looks to me like CEGUI is trying to use:

Code: Select all

// create pixel shader
    d_pimpl->d_pixelShader = Ogre::HighLevelGpuProgramManager::getSingleton().
        createProgram("__cegui_internal_ps__",
               Ogre::ResourceGroupManager::DEFAULT_RESOURCE_GROUP_NAME,
               d_pimpl->d_useGLSL ? "glsl" : "hlsl", Ogre::GPT_FRAGMENT_PROGRAM);

    d_pimpl->d_pixelShader->setParameter("entry_point", "main");

    if (d_pimpl->d_useGLSL)
        d_pimpl->d_pixelShader->setParameter("target", "arbfp1");
    else if (Ogre::GpuProgramManager::getSingleton().isSyntaxSupported("ps_4_0"))
        d_pimpl->d_pixelShader->setParameter("target", "ps_4_0");
    else if (Ogre::GpuProgramManager::getSingleton().isSyntaxSupported("ps_2_0"))
        d_pimpl->d_pixelShader->setParameter("target", "ps_2_0");
    else
    {
        d_pimpl->d_vertexShader.setNull();
        d_pimpl->d_pixelShader.setNull();

        CEGUI_THROW(RendererException(
            "OgreRenderer::initialiseShaders: No supported syntax - "
            "unable to compile '__cegui_internal_ps__'"));
    }

    d_pimpl->d_pixelShader->setSource(d_pimpl->d_useGLSL ? S_glsl_ps_source :
                                                           S_hlsl_ps_source);
    d_pimpl->d_pixelShader->load();

So, after the line where it sets the target to ps_4_0, I tried adding an option I found in the Ogre code:

Code: Select all

d_pimpl->d_pixelShader->setParameter("enable_backwards_compatibility", "yes");

since that MSDN page says this instrinsic is still supported with the legacy compile option, and I realized from MSDN that "texture" is a keyword in HLSL, so I renamed the texture parameter to _texture. With both of those changes in place, the shader appears to compile, but Ogre is now reporting a D3D exception:

Code: Select all

01:19:06: OGRE EXCEPTION(3:RenderingAPIException): D3D11 device cannot draw
Error Description:ID3D11DeviceContext::Draw: Vertex Shader - Pixel Shader linkage error: Signatures between stages are incompatible. Semantic 'COLOR' is defined for mismatched hardware registers between the output stage and input stage.
ID3D11DeviceContext::Draw: Rasterization Unit is enabled (PixelShader is not NULL or Depth/Stencil test is enabled and RasterizedStream is not D3D11_SO_NO_RASTERIZED_STREAM) but position is not provided by the last shader before the Rasterization Unit.
 in D3D11RenderSystem::_render at ..\..\..\..\..\RenderSystems\Direct3D11\src\OgreD3D11RenderSystem.cpp (line 2816)

The last items in the CEGUI.log:

Code: Select all

31/10/2013 01:19:06 (Std)    Creating falagard mapping for type 'TaharezLook/Tooltip' using base type 'CEGUI/Tooltip', window renderer 'Core/Tooltip' Look'N'Feel 'TaharezLook/Tooltip' and RenderEffect ''. (00A1C888)
31/10/2013 01:19:06 (Std)    Creating falagard mapping for type 'TaharezLook/StaticImage' using base type 'DefaultWindow', window renderer 'Core/StaticImage' Look'N'Feel 'TaharezLook/StaticImage' and RenderEffect ''. (00A1C888)
31/10/2013 01:19:06 (Std)    Creating falagard mapping for type 'TaharezLook/StaticText' using base type 'DefaultWindow', window renderer 'Core/StaticText' Look'N'Feel 'TaharezLook/StaticText' and RenderEffect ''. (00A1C888)
31/10/2013 01:19:06 (Std)    Creating falagard mapping for type 'TaharezLook/ItemListbox' using base type 'CEGUI/ItemListbox', window renderer 'Core/ItemListbox' Look'N'Feel 'TaharezLook/ItemListbox' and RenderEffect ''. (00A1C888)
31/10/2013 01:19:06 (Std)    Creating falagard mapping for type 'TaharezLook/ListboxItem' using base type 'CEGUI/ItemEntry', window renderer 'Core/ItemEntry' Look'N'Feel 'TaharezLook/ListboxItem' and RenderEffect ''. (00A1C888)
31/10/2013 01:19:06 (Std)    Creating falagard mapping for type 'TaharezLook/GroupBox' using base type 'DefaultWindow', window renderer 'Core/Default' Look'N'Feel 'TaharezLook/GroupBox' and RenderEffect ''. (00A1C888)
31/10/2013 01:19:06 (Std)    Creating falagard mapping for type 'TaharezLook/Tree' using base type 'CEGUI/Tree', window renderer 'Core/Tree' Look'N'Feel 'TaharezLook/Tree' and RenderEffect ''. (00A1C888)
31/10/2013 01:19:06 (Std)    [OgreRenderer] Created texture: DejaVuSans-12_auto_glyph_images_32

This honestly looks like a bug in the shader to me, but I couldn't find anyone else with the same problem, so I didn't want to report it as such without checking here first. Is there something that I need to differently when using DirectX 11 or is the shader broken in this scenario?

[ranboh]

I had this problem. I think it is due to 'texture' being a keyword in the latest shader model. I changed the declaration to this:

Code: Select all

static Ogre::String S_hlsl_ps_source(
    "float4 main(float4 position : POSITION,"
    "            float2 texCoord : TEXCOORD0,"
   "            float4 colour : COLOR,"
    "            uniform sampler2D texture1 : TEXUNIT0) : COLOR"
    "{"
    "    return tex2D(texture1, texCoord) * colour;"
    "}"
);

Of course I ran into another unrelated problem downstream that I haven't been able to solve.

[eaducac]

Thanks ranboh. With some further Googling, I've managed to get this working by editing the vertex and pixel shaders. This is what I've come up with for the vertex shader:

Code: Select all

static Ogre::String S_hlsl_vs_source(
    "uniform float4x4 worldViewProjMatrix;"
    "struct VS_OUT {"
    "   float4 position : SV_POSITION;" /* this must be an SV_POSITION and not a POSITION or you get the error "position is not provided by the last shader before the Rasterization Unit."*/
    "   float2 uv : TEXCOORD0;"
    "   float4 colour : COLOR;"
    "};"
    "VS_OUT main(float4 position : SV_POSITION," // same
    "            float2 uv : TEXCOORD0,"
    "            float4 colour : COLOR)"
    "{"
    "    VS_OUT o;"
    "    o.uv = uv;"
    "    o.colour = colour;"
    "    o.position = mul(worldViewProjMatrix, position);"
    "    return o;"
    "}"
);

and pixel shader:

Code: Select all

static Ogre::String S_hlsl_ps_source(
    "float4 main(float4 position : SV_POSITION," // the first 3 inputs need to match the vertex shader output
    "   float2 uv : TEXCOORD0,"
    "   float4 colour : COLOR,"
    "   uniform sampler2D _texture : TEXUNIT0) : COLOR" // texture is a keyword so this parameter had to be renamed
    "{"
   "    return tex2D(_texture, uv) * colour;"
    "}"
);

Additionally, in OgreRenderer::initialiseShaders(), after setting the target to ps_4_0, you have to add the line

Code: Select all

d_pimpl->d_pixelShader->setParameter("enable_backwards_compatibility", "yes");

in order to allow the use of the DX9 intrinsic function tex2D.

With these changes in place, my code works with both D3D11 and D3D9. It seems fairly certain to me now that this is a bug in the shader, so I'm going to report this as such.