Friday, November 30, 2012

Using screen-buffer masks in Unity Pro for a fog of war effect.

As I prototyped Convo, it became clear that I would need some sort of fog of war / sight radius visualization. Depth masks weren't very good for this because of the uneven terrain, and the other fog of war solutions I found involved a tile system (nah), or a vertex alpha plane (eww), or some other pretty convoluted thing. Lame. So I did a bunch of research and figured out my own technique... best of all, this visualization uses no script code at all, it's just shaders and geometry.

Also, you don't have to use this for fog of war. You can use it anytime you need to mask-off certain bits of the camera view on a per-object, per-triangle, or per-pixel basis. Like, maybe you'd want some stuff to glow red?...

The gist: (you will need Unity Pro since this uses render textures)
1) A camera's render texture is in RGBA format. You can technically do whatever you want with the alpha channel; most of Unity's built-in shaders use it to mask out alpha textures for various image effects.
2) If we use a shader that writes only to that alpha, we can use it to mask objects or pixels.
3) Then, we edit the image effect shader to modulate an effect with the alpha channel values.

If you need some more details and shader code, read on...

NOTE: Keep in mind that the implementation in this post is purely visual -- it's up to you to do your own visibility checks / instantiate or hide units / manage data / so on.

First, create a mesh. This is the thing that'll be the screen-buffer mask. I made a sphere with inverted normals that I scale to the unit's sight radius. You can texture it with a mask or just apply it to the whole mesh. I did the latter for simplicity. (You could also procedurally generate a mesh for extra fanciness.)

Next, you'll need to create a special masking shader. This is the simple one that I modified from a base shader in this forum thread:
Shader "Screen Buffer Mask" {
    Properties {
        _Color ("Main Color, Alpha", Color) = (1,1,1,1)

    Category {
        Tags {"RenderType" = "Opaque" "Queue"="Transparent"}

        ZWrite On   // for more on Z-buffer stuff and Offset, see here
        ZTest GEqual   // GEqual = mask stuff in front of the mask geo
        Lighting Off
        Color [_Color]   // change alpha in material to tweak mask strength

        SubShader {
            Pass {
                Colormask A   // render only to alpha
                Offset -1, -1
                Cull Back
The most important shader command here is ColorMask A. It tells the shader to render to the screen buffer's alpha channel ONLY. You can also set it to ColorMask RGBA if, say, you wanted to do something with a self-illumination alpha mask or something.

I set ZTest to GEqual (unlike the depth mask technique I mentioned in a previous article, which is LEqual) because I want this particular mask to show through objects that are in front of it, or greater or equal (GEqual) to it -- and since my mesh is an inverted sphere, it ends up masking all the geometry inside the volume of the sphere, kind of. (Play with different ZTest settings.)

Next, create a material with the shader and assign the material to a mesh renderer.

To make sure your mask is working properly, set the rendering mode on your Scene View in Unity to "Alpha" -- because it's invisible in RGB mode -- and if it's not working, double-check your shader code or the color settings on your material.

As you can see above, I've set the Alpha Color on my material to "0", so it masks all the geometry inside the sphere with a heavy black.

The masking is done. Now we need to do something with the mask, like an image effect that takes the render texture (a screenshot of the current camera view, updated every frame) and does something to it. Here, I've modified the stock Grayscale image effect shader to modulate the effect according to the render texture's alpha value:
Shader "Hidden/Grayscale Effect (Screen Buffer Masked)" {
Properties {
 _MainTex ("Base (RGB)", 2D) = "white" {}
 _RampTex ("Base (RGB)", 2D) = "grayscaleRamp" {}

SubShader {
 Pass {
  ZTest Always Cull Off ZWrite Off
  Fog { Mode off }
               #pragma vertex vert_img
               #pragma fragment frag
               #pragma fragmentoption ARB_precision_hint_fastest 
               #include "UnityCG.cginc"

               uniform sampler2D _MainTex;
               uniform sampler2D _RampTex;
               uniform half _RampOffset;

               fixed4 frag (v2f_img i) : COLOR
                fixed4 original = tex2D(_MainTex, i.uv);
                fixed grayscale = Luminance(original.rgb);
                half2 remap = half2 (grayscale + _RampOffset, .5);
                fixed4 output = (original * (1 - original.a)) +
                      (tex2D(_RampTex, remap) * original.a);
                output.a = original.a;
                return output;


Fallback off
The highlighted line does this: first it takes the original screenshot and renders the full-color image according to an inverted mask, then it adds a grayscaled version on top according to regular mask strength. (e.g. black in an alpha mask is 0, and 1 - 0 = 1, so original * 1 + grayscale * 0 = original, and original * 0 + grayscale * 1 = grayscale.)

Don't forget to drag the Grayscale Effect script onto your camera object, and replace the default shader with your own screen-buffery variant.

(TIP: If you want to blur the screen mask before applying an image effect, take a look at how the stock Glow Effect does it in the script. Watch out though, it's a pretty expensive thing to do.)