Radiator Blog: Working with custom ObjectPreviews and SkinnedMeshRenderers in Unity

Tuesday, June 7, 2016

Working with custom ObjectPreviews and SkinnedMeshRenderers in Unity

Unity's blendshape controls -- basically just a list of textboxes -- were going to cause me a lot of pain. After wrestling with broken AnimationClips for my previous attempt at facial expressions in my game Stick Shift, I decided to actually invest a day or two into building better tools for myself, inspired partly by Valve's old Faceposer tool for Source Engine 1.

To do that, I scripted the Unity editor to draw a custom inspector with sliders (based on Chris Wade's BlendShapeController.cs) along with an interactive 3D face preview at the bottom of the inspector.

The workflow I wanted was this:

Save blendshape settings as a "preset" in a separate asset file.
Load the blendshapes and blend freely between them whenever, in-game or in-editor.
Be able to tweak each file really easily, with a built-in 3D preview of the result.

As usual with anything related to editor scripting, the Unity documentation is brief or non-existent, and at worse, straight-up misleading. For much of this, I was relying heavily on Matt Rix's amazing UnityDecompiled repo, where you can see all the internal Unity editor code and reverse-engineer whatever feature you need. (Don't worry, Unity is unofficially OK with it.) Unfortunately, while you might think you can do ___ because some part of the editor does it too, those functions are often locked in sealed internal classes that you can't easily access except through some messy code reflection, so be careful.

I also learned a lot from Tim Aksu's primer to the ObjectPreview system, the editor system that displays any kind of asset preview from 2D to 3D to sounds, usually at the bottom of your inspector. If you want to work with ObjectPreview, definitely read that first.

The two most important steps here are to override HasPreviewGUI( ), which flags to the editor to display a preview pane or not, and then override OnPreviewGUI( ), where you actually do your GUI drawing. (Also, for the sake of completeness, here's a link to the scant Unity docs on ObjectPreview.)

When the editor draws a real-time 3D preview for a Mecanim animator, or displays a static thumbnail for your 3D model, it hooks into this ObjectPreview system via RenderTextures.

Your main goal is to setup this "mini scene" that's rendered just for this little preview window. For most 3D previews, you setup an internal camera, use Graphics.DrawMesh( ) to instantaneously draw a mesh without having to instantiate a GameObject, and then grab the resulting RenderTexture from the camera. To help you do that, Unity has a totally undocumented helper class called PreviewRenderUtility that sets up this internal in-editor scene with its own camera and everything. But that still doesn't change one very important fact -- this is all still taking place in your currently open scene, with all the fog and lighting and shadows etc that entails. If you don't want fog in your ObjectPreview, you might have to manually turn it off.

My big problem, though, was that for facial expressions I needed to draw an animated skinned mesh, not just a plain mesh. It turns out, Unity's internal Mecanim 3D preview AvatarPreview uses internal functions to cleanly manage and instantiate a GameObject, via an inaccessible function called EditorUtility.InstantiateForAnimatorPreview( )

... and that general approach is basically what I did in the end:

I instantiated a separate "preview" prefab, hid it away from the rest of my scene, applied any animations or blendshapes, and then called SkinnedMeshRenderer.BakeMesh( ) to bake it down into a plain Mesh suitable for Graphics.DrawMesh( ). It works surprisingly well.

For your reference, I've included most of my ObjectPreview code below. Much of it is borrowed from the UnityDecompiled ModelInspector.cs, so look there for some extra help. There's also probably a lot of bad broken things in my code, but it works for me and lets me do what I need to do, so I'm pretty much calling it done. Good luck in custom ObjectPreview land!

using UnityEngine; using UnityEditor; using System; using System.Collections; [CustomPreview(typeof(YourCustomScriptableObject))] // THIS IS VERY IMPORTANT, this is so the editor knows what this is supposed to be previewing at all public class SkinnedMeshObjectPreviewExample : ObjectPreview { PreviewRenderUtility m_PreviewUtility; GameObject previewPrefab, previewInstance; SkinnedMeshRenderer skinMeshRender, eyeballRender; Mesh previewMesh, previewMeshEyeball; Material previewMaterial; static Vector2 previewDir = new Vector2(-180f, 0f); static float lerpSpeed = 0.5f; // very important to override this, it tells Unity to render an ObjectPreview at the bottom of the inspector public override bool HasPreviewGUI() { return true; } private void Init() { if (this.m_PreviewUtility == null) { this.m_PreviewUtility = new PreviewRenderUtility(); this.m_PreviewUtility.m_CameraFieldOfView = 27f; RefreshPreviewInstance(); } } // the main ObjectPreview function... it's called constantly, like other IMGUI On*GUI() functions public override void OnPreviewGUI(Rect r, GUIStyle background) { // if this is happening, you have bigger problems if (!ShaderUtil.hardwareSupportsRectRenderTexture) { if (Event.current.type == EventType.Repaint) { EditorGUI.DropShadowLabel(new Rect(r.x, r.y, r.width, 40f), "Mesh preview requires\nrender texture support"); } return; } Init(); previewDir = Drag2D(previewDir, r); if (Event.current.type != EventType.Repaint) { // if we don't need to update yet, then don't return; } m_PreviewUtility.BeginPreview(r, background); // set up the PreviewRenderUtility's mini internal scene DoRenderPreview(); Texture image = m_PreviewUtility.EndPreview(); // grab the RenderTexture resulting from DoRenderPreview() > RenderMeshPreview() > PreviewRenderUtility.m_Camera.Render() GUI.DrawTexture(r, image, ScaleMode.StretchToFill, false); // draw the RenderTexture in the ObjectPreview pane EditorGUI.DropShadowLabel(new Rect(r.x, r.y, r.width, 40f), target.name); } private void DoRenderPreview() { if ( skinMeshRender == null ) { RefreshPreviewInstance(); } // very important: we have to call BakeMesh, to bake that animated SkinnedMesh into a plain static Mesh suitable for Graphics.DrawMesh() previewMesh = new Mesh(); previewMeshEyeball = new Mesh(); skinMeshRender.BakeMesh( previewMesh ); eyeballRender.BakeMesh( previewMeshEyeball ); // now, actually render out the RenderTexture RenderMeshPreview( previewMesh, m_PreviewUtility, previewMaterial, previewDir, -1); } void RefreshPreviewInstance () { // if we already instantiated a PreviewInstance previously but just lost the reference, then use that same instance instead of making a new one var oldInstance = GameObject.Find( "FaceMorphemePreviewInstance" ); if ( oldInstance != null ) { previewInstance = oldInstance; } else { // no previous instance detected, so now let's make a fresh one // very important: this loads the PreviewInstance prefab and temporarily instantiates it into PreviewInstance previewPrefab = (GameObject)AssetDatabase.LoadAssetAtPath("Assets/gayclub/prefabs_npc/facepreview.prefab"); previewInstance = (GameObject)GameObject.Instantiate( previewPrefab, previewPrefab.transform.position, previewPrefab.transform.rotation ); previewInstance.name = "FaceMorphemePreviewInstance"; // HideFlags are special editor-only settings that let you have *secret* GameObjects in a scene, or to tell Unity not to save that temporary GameObject as part of the scene previewInstance.hideFlags = HideFlags.DontSaveInEditor | HideFlags.DontSaveInBuild; // you could also hide it from the hierarchy or inspector, but personally I like knowing everything that's there } var sRenders = previewInstance.GetComponentsInChildren(); // when instantiating stuff in the scene with HideFlags, that stuff often gets broken when loading a new scene... if ( sRenders == null || sRenders.Length == 0 ) { // so I setup a check to detect that case and clean-up Debug.Log("cleaning up leftover PreviewInstance!"); OnDestroy(); RefreshPreviewInstance(); return; } else { // if nothing is wrong, then we proceed normally skinMeshRender = sRenders[0]; eyeballRender = sRenders[1]; } previewMaterial = skinMeshRender.sharedMaterial; // use sharedMaterial or else you will instantiate a new material } void RenderMeshPreview (Mesh mesh, PreviewRenderUtility previewUtility, Material material, Vector2 direction, int meshSubset ) { if (mesh == null || previewUtility == null) { return; } // Measure the mesh's bounds so you know where to put the camera and stuff Bounds bounds = mesh.bounds; float magnitude = bounds.extents.magnitude; float distance = 4f * magnitude; // setup the ObjectPreview's camera previewUtility.m_Camera.backgroundColor = Color.gray; previewUtility.m_Camera.clearFlags = CameraClearFlags.Color; previewUtility.m_Camera.transform.position = new Vector3( 0f, 0.85f, -0.5f); // this used to be "-Vector3.forward * num" but I hardcoded my camera position instead previewUtility.m_Camera.transform.rotation = Quaternion.identity; previewUtility.m_Camera.nearClipPlane = 0.3f; previewUtility.m_Camera.farClipPlane = distance + magnitude * 1.1f; // figure out where to put the model Quaternion quaternion = Quaternion.Euler(0f, direction.x, 0f); // this used to have "Quaternion.Euler(direction.y, 0f, 0f) * " to apply pitch rotation as well, but I didn't need it Vector3 pos = quaternion * -bounds.center; // we are technically rendering everything in the scene, so scene fog might affect it... bool fog = RenderSettings.fog; // ... let's remember the current fog setting... Unsupported.SetRenderSettingsUseFogNoDirty(false); // ... and then temporarily turn it off // submesh support, in case the mesh is made of multiple parts int subMeshCount = mesh.subMeshCount; if (meshSubset < 0 || meshSubset >= subMeshCount) { for (int i = 0; i < subMeshCount; i++) { // PreviewRenderUtility.DrawMesh() actually draws the mesh previewUtility.DrawMesh(mesh, pos, quaternion, material, i); previewUtility.DrawMesh(previewMeshEyeball, pos, quaternion, material, i); // my model's eyeballs are a separate mesh, so I have to draw that too } } else { // no submeshes here so let's just draw it normally previewUtility.DrawMesh(mesh, pos, quaternion, material, meshSubset); previewUtility.DrawMesh(previewMeshEyeball, pos, quaternion, material, meshSubset); } // VERY IMPORTANT: this manually tells the camera to render and produce the render texture previewUtility.m_Camera.Render(); // reset the scene's fog from before Unsupported.SetRenderSettingsUseFogNoDirty(fog); } // this is where you draw any settings or controls above the ObjectPreview public override void OnPreviewSettings() { GUILayout.Label( "LerpSpeed: " ); lerpSpeed = EditorGUILayout.Slider( lerpSpeed, 0.1f, 0.5f, GUILayout.Width( 100f ) ); } // cleanup stuff so we don't leak everywhere public void OnDestroy() { Debug.Log("FaceMorphemePreview OnDestroy()"); if (this.m_PreviewUtility != null) { this.m_PreviewUtility.Cleanup(); this.m_PreviewUtility = null; } if ( previewInstance != null ) { GameObject.DestroyImmediate( previewInstance ); } } // from http://timaksu.com/post/126337219047/spruce-up-your-custom-unity-inspectors-with-a // this is our inlined version of EditorGUIUtility.Draw2D(), which is internal or something public static Vector2 Drag2D(Vector2 scrollPosition, Rect position) { int controlID = GUIUtility.GetControlID("Slider".GetHashCode(), FocusType.Passive); Event current = Event.current; switch (current.GetTypeForControl(controlID)) { case EventType.MouseDown: if (position.Contains(current.mousePosition) && position.width > 50f) { GUIUtility.hotControl = controlID; current.Use(); EditorGUIUtility.SetWantsMouseJumping(1); } break; case EventType.MouseUp: if (GUIUtility.hotControl == controlID) { GUIUtility.hotControl = 0; } EditorGUIUtility.SetWantsMouseJumping(0); break; case EventType.MouseDrag: if (GUIUtility.hotControl == controlID) { scrollPosition -= current.delta * (float)((!current.shift) ? 1 : 3) / Mathf.Min(position.width, position.height) * 140f; scrollPosition.y = Mathf.Clamp(scrollPosition.y, -90f, 90f); current.Use(); GUI.changed = true; } break; } return scrollPosition; } }

One last note: the comments here are unmonitored because I'm busy. You can certainly ask for help, but don't expect a response from me. Sorry.

Tuesday, June 7, 2016

Working with custom ObjectPreviews and SkinnedMeshRenderers in Unity

RADIATOR

DESIGN BLOG

Monthly Archive

Topics