Shader/Program applied to window?

[zadirion]

Now, I've done some research and from what I understand, RenderEffect is what I need to implement (using CEGUI 0.7.5) so that I can apply a shader to a CEGUI window. The RenderEffect interface documentation is pretty self explanatory, I know how to tie it to the rest of my code. What I do not know is how to do shader stuff inside it?
Thanks in advance for any help!

Footnote: I am basically asking the same that was asked about two years ago here.
I'm creating a new topic because that one is posted in the Advanced section, this is a beginner question, and I am a beginner myself.

[CrazyEddie]

How, and indeed if, you achieve this completely depends on which renderer module you want to use. Basically the situation is as follows:

Should work fine / tested as working

• OpenGL renderer
• Direct3D 9 renderer

Might work, might not - depends on a lot of things. Not tested at all.

• Ogre Renderer
• Irrlicht Renderer

Definitely will not work without modifications to the renderer module code.

• Direct3D 10 renderer
• Direct3D 11 renderer

Yes, the facility is supposed to work in all of them, but it's somewhat a work in progress.

Ok, so basically in performPreRenderFunctions you perform your set up operations, like setting the shader / program to use and initialising any uniform values and such for the given rendering pass of the RenderEffect. These are then cleaned up in performPostRenderFunctions. Depending on your needs this could be all that is required - if you do not need customised geometry for instance. Now the key thing is that the shader set up and what have you is done using the facilities of underlying API - this is why Ogre/irrlicht are 'unknown', since they may make this hard or even impossible, whereas obviously for GL and D3D 9 it's simple enough. D3D10 and D3D11 definitely will not work, this is due to them using a shader to do all rendering already, and this is indiscriminately set in all cases (replacing any shader set by the RenderEffect) - this can be considered a bug

HTH

CE

[zadirion]

Hello CrazyEddie,

Thank you for your comprehensive reply. I already figured out a 'sort of' a solution yesterday (after an entire weekend of head busting). It involved creating a class (FrameWindowWithMaterial) that derives from FrameWindow and implements selfDraw. In it, I call FrameWindow::selfDraw. The window is configured to render to a texture through the AutoRenderingSurface property. I then load and copy an ogre material containing the needed shaders, and simply replace the material's texture with the texture CEGUI rendered my window to. I then create an Ogre::ManualObject through which I draw my texture at an appropriate position corresponding to the CEGUI window. The window is thus drawn using the material's shaders. Tested this with the default programs Ogre comes with, and works great.

Downsides:
1) i have to handle the EventMoved window event, to update the location of the ManualObject on the screen whenever the window position changes.
2) I had to bootstrap(RenderTarget *) CEGUI to render to a texture, so that CEGUI does not itself render directly to the screen when i call FrameWindow::selfDraw (otherwise two windows would show up: mine, and the one rendered by CEGUI). I simply ignore the texture CEGUI renders to. This has the obvious downside that all windows not using my approach described above will not be visible (they render to the texture, and the texture is ignored)

Advantages:
1) I simply set the material with FrameWindowWithMaterial::SetMaterial(string aMaterialName), and the window renders using that material (shaders and all)
2) it uses the Ogre renderer, which means it supports whatever opengl / directx Ogre supports, my code remains independent of that

The reason the solution you gave didn't come to my mind is because the approach feels too much like it breaks encapsulation: you set the active shader somewhere, then exit the function in which you set it, and by the time the actual rendering is done the state of the active shader may or may not have changed in the meantime (from some other unknown location in the code), leading to bugs like the DX10/11 one you were referring to, where the current shader is reset before you have time to actually render the window)

My solution feels like a hack, and I definitely hate it. I'm at work now, but will try your solution when I get back home, and I'll think it all through, evaluating the advantages and downsides of each approach.
What I am trying to avoid from your solution is directly using the GL and/or D3D api. Want to keep it all neat and cross-platform.

In the meantime, is there any way to mitigate Downside 2) ? Perhaps be able to only instruct a particular window not to render to screen somehow? I mean only render to texture, not to screen. Then I can leave CEGUI rendering directly to screen, and only instruct particular windows (like my own) to render to texture only.

Thanks

[CrazyEddie]

I think I should have been more thorough in my initial reply here

The use of RenderEffect::performPreRenderFunctions is absolutely the right place to be doing any shader set up since it's called right before the geometry for the Window is sent for final rendering.

The basic algorithm is:

Code: Select all

for each pass in RenderEffect
   call RenderEffect:: performPreRenderFunctions(pass)
   render geometry buffer content

RenderEffect::performPostRenderFunctions

The issues with D3D10/11 come because the GeometryBuffer itself reset the shader in the 'render geometry buffer content' step.

The key to the rendering algorithm above is that when using the RenderEffect with a Window that has the AutoRenderingSurface property set, the default geometry drawn is two triangles, and the texture used is the texture holding the content of the window. This texture is always set up, and is set up after the performPreRenderFunctions call in each pass - this is one point that could cause issues, not the setting of the texture, but perhaps the setting of some other states related to that. So, ideally, we could 'apply' some ogre material in the performPreRenderFunctions call and for most / many cases we would be done, however, the ogre interface is not friendly towards doing that at the level of Material (somewhat because Material/Technique/Pass is a similar - though much more advanced - abstraction as what the RenderEffect is supposed to be, except one is driven by Ogre at higher levels and the other is driven by CEGUI at higher levels).

So, the conclusion is that to use performPreRenderFunctions you need some way to activate appropriate render system settings from the Material. If this is possible, I think all will be well. If not, I think perhaps your existing solution may be the best that is possible given the current state of things (clearly you could bypass Material, and attack things from a lower level, but most people will not want to do this, and it's somewhat prone to break due to the use of low level / internal ogre calls).

is there any way to mitigate Downside 2) ? Perhaps be able to only instruct a particular window not to render to screen somehow? I mean only render to texture, not to screen.

I think probably the cleanest way is to override RenderEffect::realiseGeometry for the RenderEffect and simply return false. This will mean that no geometry is added for the window, so nothing should get drawn back to the owning cegui rendering surface.

Finally, I will mention that there has been a fair amount of discussion related to creating a new Ogre renderer module that operates at a higher level, and which would allow the use of Ogre Materials much more cleanly. However, this has not even entered a 'work in progress' stage, and whether it will ever happen is a matter of great debate.

CE

[zadirion]

Thanks CE for the clarifications.

I will investigate this when I get home, and will come back with a conclusion.
I hope this thread helps other CEGUI users, the library is definitely well-written and thought-through.

Regards,
Zad

[zadirion]

So, the conclusion is that to use performPreRenderFunctions you need some way to activate appropriate render system settings from the Material. If this is possible, I think all will be well. If not, I think perhaps your existing solution may be the best that is possible given the current state of things (clearly you could bypass Material, and attack things from a lower level, but most people will not want to do this, and it's somewhat prone to break due to the use of low level / internal ogre calls).

Ok change of approach.
The FrameWindowWithMaterial approach was definitely bad because of the amount of hackery involved, and a few other reasons. RenderEffect is definitely the way to go. Implemented it in a class named MaterialEffect and assigned the effect to my window (now a usual FrameWindow like any other, with AutoRenderSurface property enabled, and some transparency). Then all I need to do is create the MaterialEffect on to who'se constructor I pass a material name, and the effect will render my window using that material.
Internally, I do just as you recommended, on performPreRenderFunctions I set up the vertex and fragment programs, along with any parameters needed to be passed to the programs, which I all read from the Ogre::Material passed to the effect. On performPostRenderFunctions, I reset the vertex and fragment programs. However, instead of using the native API (opengl or d3d), I use the Ogre::RenderSystem wrapper to set the programs, parameters, textures, etc.
Below, I am attaching the result:


As you can see, the shader does not blur whatever is behind the window, it only acts on the window. Pretty sure I'm missing something trivial here, so, how did you get this to work in your "CEGUI Glass-like Shader Demo" ?

Other than that, I guess this is pretty much the approach one should take to apply shaders to a window. I really hope this thread helps whoever else is searching for a way to achieve this.

[CrazyEddie]

It's cool that you went that route, and that it works

Your approach to the blurring is slightly askew, as you already gathered. To blur what is beneath, you have to be able to read back what has previously been rendered - and that means rendering to a texture (since you can't usually directly sample the frame buffer). I would say the "usual" approach to achieve that would be to enable AutoRenderingSurface on whatever you're using for a root window - then you can get access to that texture and sample it in the shader to do the blur and to blend with the (non-blurred) pixel from the window being drawn. In my demo I used two passes and had an additional buffer for the intermediate results, thinking about this again now, I'm not certain if that intermediate buffer is really required.

HTH

CE.

[zadirion]

Thanks, CE, the answer was staring me in the face.

Have a nice day!