There are several causes that may generate this error. You can get more information selecting the Forest object, and clicking General->Forest Stats.
Usually, it happens because one of more limits has been exceeded. These are the limits used by Forest:
Display->Max Items / Max Faces
Limits the maximum number of items or mesh's faces created by the plugin. The default value is 500,000 objects and 5 million faces. When this limit is reached, the plugin builds an "F" icon, and the Forest Stats window shows a warning message.
Note: The "Max Faces" value is ignored when using the supported engines (as Mental Ray, VRay or Maxwell) and Display->Render->Mode is set to "Automatic" mode, since these renderers are not limited by the number of faces.
If you use one of the engines supported natively, set always Render Mode->Automatic. In this case, the "Max Faces" limit is not used, because Forest uses a custom geometry shader to generate the geometry.
By default the Max Items value is 500K (half million). Usually it is not necessary to increase this limit except in huge scenes. Also we suggest not using more than 500K items in Mental Ray, and 3 Mill. in VRay because the render times would grow exponentially (see Rendering Best Practices for details).
Distribution Map->Max Density
Defines the limit of possible tree positions in the area (by default 10 million). Take care: this includes both white and black pixels in the distribution map, so you may reach the density limit using an empty map and low X/Y values, but not get a single tree in the scene.
To define the position of each item in the scene Forest creates internally a virtual "grid", defined by the Distribution Map and the X/Y Density values. If for example, you use a low density map (as "Very Scattered"), you may be tempted to increase the density lowering X/Y Size. This is not efficient, because most of the positions in the map will be gaps (black pixels). We use this parameter to avoid getting higher densities of the grid (but with few items at all).
To create dense distributions, we suggest first to set a high density map (as "Full"), or lower the "Threshold" value. Then, adjust the X/Y parameters to get the desired result. To prevent the "grid" look of the "Full" map, turn on Transform->Random Translation to 10 or 20%.
Why we add so many limits ? Well, because it is very common than novice users try to create huge distributions, in their early scenes with the plugin. We prefer they get a warning message, instead a crash of Max or the render engine.
- Distribution Map help reference
- Display help reference