First I tried the traditional "use a bunch of tree objects" approach. The nature soft-occlusion shaders, 1500+ poly trees and non-configurable shadowcasting made it way too expensive, even for a modest thicket of 20 trees and substantial billboarding. Our angled top-down perspective also demanded some degree of detail but with a long sightline to give the impression of a forest.
Then I remembered some early forestry visualization experiments by one of the original founders of Day of Defeat, Tim "Waldo" Holt; he used detail sprites in the Source Engine to draw a giant forest at a decent framerate. (Though in the end, he switched to the Torque Engine.)
So what if we just used billboards from the beginning? Enlarge the grass detail sprites to look like trees, and...
Okay, much more lush and performance friendly, but given that characters and buildings in our game use shadow casters, the lack of any shadowing solution for our "detail trees" made them look inconsistent and floating off the ground. We needed shadows for these trees.
My first instinct was to use another detail object as a "canopy shadow" and place it underneath the trees. However, that was too hard to work with because the grass texture shader only supports 1-bit cutout alpha transparency, so the shadows were opaque -- plus, we can't impose a minimum size on detail objects, so sometimes I'd have a 0.1 unit wide canopy shadow that looked more like a small shrub. Worst of all, if you tried to erase detail shadow objects, you'd end up erasing detail trees too.
I was about to give up when I realized I could use my canopy shadows as actual tree objects! Make a "tree" that's a single plane, facing upwards, slightly elevated off the ground with a Transparent \ Diffuse shader. Because it's not a Nature \ Soft Occlusion shader, it breaks lighting / shadows / billboarding on-purpose! Plus, because it's not a detail object, you can erase / paint them on a separate layer without affecting any of the detail trees!
3D sidescrollers, top-down strategy and tower defense games.
FPS games or third-person camera games.
+ Lots of high-fidelity trees for relatively little cost
+ Terrain script still manages placement / LoD
+ Tree shadow solution supports partial transparency
+ Swaying trees!
+ Easy asset creation for new trees (directly paint the billboards)
- Not good if you need anything to collide with trees
- If the camera can rotate at all, or will be very close to the forest, then these billboarded trees might be too obvious
- Hand-placed shadows, even if quickly painted with the terrain tool, can be time-consuming. You'll have to re-paint tree shadows every time you re-paint detail trees, or else they won't match up.