Composite test using mental ray image based lighting & Final Gather

here is a short workflow / write up on my experience working with image based lighting and final gather, hopefully you might find something useful out of it.

It all started when I created this image.

The cg was rendered with mental ray’s final gather and a dodgy image based lighting setup by mapping a 8bit image onto a sphere all in the one pass. There are obvious problems with the lighting and also there was no alpha matte which captured final gathers ambient shadows. The image was very roughly comped in photoshop and I had to manually mask in the ambient shadows, if you look closely at the hi-res image you will see what I mean when I say it was dodgy.

So it was decided that I should fix this image with a realistic lighting setup and to be able to get an alpha matte straight from maya which included all the shadow information.

Here is the final result with color & level correction, beauty pass rendered with Mental Rays IBL node using a 32 bit HDRI + an occlusion pass using final gather.

In my opinion the result is a much better composite with realistic lighting that matched the background plate. So onto how I did this;

Setting up the IBL beauty pass

Setting up the IBL is fairly straight forward, you can find the steps to do so from this great tutorial

Some caveats you need to know when using IBL are;

Using Mental Ray’s IBL node will increase your render time, the default settings when you setup IBL are not optimised, for this scene I used the following setting.

Note the very low U/V quality – the reason is, currently Maya 2011 there is a current bug with the mentalrayIBL node which causes your render time to sky rocket if you use settings above 7 x 7. There are 2 fixes one is to simply set your quality below 7×7 and bump up your samples in the render settings

The other fix to add this additional attribute to the miDefaultOptions.

You can find Autodesk’s full bug report and solution on this page.

This is the raw render of the scene using IBL only.

Lighting is definitely getting better and you will notice the nice ray trace shadows been generated from the HDRI. Sweet! but than I ran into a bloody big problem, I couldn’t extract the shadows to go onto the backplate. This is what the alpha channel looked like.

Notice the problem? The alpha of the floor is a solid. grrrrrrr. Which prevented me from extracting the shadows casted from the IBL. So I spent another good few days researching and finally came upon the solution.

Extract IBL ray trace shadows using useBackground

The solution is to generate an alpha matte which includes shadow information is to assign Maya’s useBackground shader. You can do this by assigning the shader to the floor plane and using the following settings.

You will also need to change the render stats of the floor geometry as well.

Now if you hit render, the floor will be black ( thats normal as we want the shadow to be black ) but now if you check the alpha channel you will see a alpha matte with the raytrace shadows, woot! 🙂

Separate shadow render pass.

For my composite I didn’t need a separate pass for the shadow, but if you needed a shadow only pass, you can easily create a new render layer and assign a surface shader with full black output opacity to all the geometry except for the floor, which will give you a alpha matte with just the shadows. Another use for this render pass would be to use a solid green color to the surface shader, which you could use for a greenkey in addition to the alpha. Very handy.

Ambient occlusion with mib_fg_occlusion

Part of making the final result realistic I also generated an ambient occlusion pass. Lately I’ve been using final gather for my renders, so I decided to use final gather to generate the occlusion. I know there are some debates on AO and FG-AO, but I found in this case FG AO generated a better result, to set this up I simply created a new render layer, added all the geometry MINUS the IBL sphere to the layer and used the occlusion preset in the renderlayer attributes.

Than I simply replaced the mib_amb_occlusion node with a mib_fg_occlusion node and switch on final gather for the occlusion layer, this is the result.

Putting it all together

So now I have the IBL generated beauty pass + a alpha matte that contains shadow information + an AO pass, its time to put it all together. I used photoshop cs5 for the comp, with all the mattes it was a simple process of multiplying the AO pass over the beauty pass and using the alpha matte to key out the background.

If you get some halo’ing and edge noise after applying the alpha mask, try using photoshop’s layer > matting > remove matte, this should give you a nice clean cut, most compositors will have a similar function.

This is the raw comp without any post work.

Final shot & conclusion

The completed shot included color and level correction as well as adjusting the opacity and masking out some of the AO pass.

So there you have it, the workflow I used to create this image. I hope you find this tutorial useful and please if you have any comments, questions or suggestions to improve this workflow let me know.

If you are curious about the green monster in the shot, he is the lead in a short I’m developing called “Zloki and Ze Pengs” , check out the facebook fanpage for more detail.


  1. Dude…… You’re a rendering beast!
    Fantastic work! It all looks amazing 🙂
    I just gotta learn to do this for my show reel….. 😛


Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s