Unity's blendshape controls -- basically just a list of textboxes -- were going to cause me a lot of pain. After wrestling with broken AnimationClips for my previous attempt at facial expressions in my game Stick Shift, I decided to actually invest a day or two into building better tools for myself, inspired partly by Valve's old Faceposer tool for Source Engine 1.
To do that, I scripted the Unity editor to draw a custom inspector with sliders (based on Chris Wade's BlendShapeController.cs) along with an interactive 3D face preview at the bottom of the inspector.
The workflow I wanted was this:
Save blendshape settings as a "preset" in a separate asset file.
Load the blendshapes and blend freely between them whenever, in-game or in-editor.
Be able to tweak each file really easily, with a built-in 3D preview of the result.
As usual with anything related to editor scripting, the Unity documentation is brief or non-existent, and at worse, straight-up misleading. For much of this, I was relying heavily on Matt Rix's amazing UnityDecompiled repo, where you can see all the internal Unity editor code and reverse-engineer whatever feature you need. (Don't worry, Unity is unofficially OK with it.) Unfortunately, while you might think you can do ___ because some part of the editor does it too, those functions are often locked in sealed internal classes that you can't easily access except through some messy code reflection, so be careful.
I also learned a lot from Tim Aksu's primer to the ObjectPreview system, the editor system that displays any kind of asset preview from 2D to 3D to sounds, usually at the bottom of your inspector. If you want to work with ObjectPreview, definitely read that first.
The two most important steps here are to override HasPreviewGUI( ), which flags to the editor to display a preview pane or not, and then override OnPreviewGUI( ), where you actually do your GUI drawing. (Also, for the sake of completeness, here's a link to the scant Unity docs on ObjectPreview.)
When the editor draws a real-time 3D preview for a Mecanim animator, or displays a static thumbnail for your 3D model, it hooks into this ObjectPreview system via RenderTextures.
Your main goal is to setup this "mini scene" that's rendered just for this little preview window. For most 3D previews, you setup an internal camera, use Graphics.DrawMesh( ) to instantaneously draw a mesh without having to instantiate a GameObject, and then grab the resulting RenderTexture from the camera. To help you do that, Unity has a totally undocumented helper class called PreviewRenderUtility that sets up this internal in-editor scene with its own camera and everything. But that still doesn't change one very important fact -- this is all still taking place in your currently open scene, with all the fog and lighting and shadows etc that entails. If you don't want fog in your ObjectPreview, you might have to manually turn it off.
My big problem, though, was that for facial expressions I needed to draw an animated skinned mesh, not just a plain mesh. It turns out, Unity's internal Mecanim 3D preview AvatarPreview uses internal functions to cleanly manage and instantiate a GameObject, via an inaccessible function called EditorUtility.InstantiateForAnimatorPreview( )
... and that general approach is basically what I did in the end:
I instantiated a separate "preview" prefab, hid it away from the rest of my scene, applied any animations or blendshapes, and then called SkinnedMeshRenderer.BakeMesh( ) to bake it down into a plain Mesh suitable for Graphics.DrawMesh( ). It works surprisingly well.
For your reference, I've included most of my ObjectPreview code below. Much of it is borrowed from the UnityDecompiled ModelInspector.cs, so look there for some extra help. There's also probably a lot of bad broken things in my code, but it works for me and lets me do what I need to do, so I'm pretty much calling it done. Good luck in custom ObjectPreview land!
One last note: the comments here are unmonitored because I'm busy. You can certainly ask for help, but don't expect a response from me. Sorry.