Is it OK if Some Simulations Fail during a HEEDS Design Study?

error-designsDuring a design exploration study, HEEDS makes many calls to your simulation model to evaluate potential designs. This means that your model needs to accurately predict design performance values (objectives and constraints) over a wide range of inputs (design variables). Most modern simulation models satisfy this requirement without difficulty.

But in some cases, it is too much to ask that a model be perfect for all combinations of variable values. For example:

  • In a shape optimization problem, some combinations of shape parameter values might produce invalid geometries, making it impossible to generate a CAD model for those designs. Ideally, shape parameters should be defined in a way that ensures all geometries are valid, but that is not always a realistic expectation.
  • Nonlinear or dynamic CAE models occasionally experience problems with convergence or other kinds of numerical errors. Hopefully your models are robust, but it is more difficult to predict the behavior of some designs than others, so numerical errors will occur now and then.

Of course there are many other reasons why a simulation model might terminate prematurely or predict incorrect results. Because many of these cases are unavoidable, HEEDS has been designed to be robust against these model failures. We refer to these as error designs

How does HEEDS handle error designs?

For error designs, it is not possible to calculate accurate values of the objectives and constraints. Sometimes, it is not possible to calculate any output values at all. If there are no output values for a design, then HEEDS cannot judge how well it meets the design goals. Even worse, if these values are artificially good or bad, then the search process could be misled.

Therefore, HEEDS ignores error designs so that they cannot corrupt the search. The overall performance metric of error designs is not even calculated. Nearby or similar designs can still be explored without bias, but error designs will not be considered further. Error designs do, however, count toward the design exploration budget.

How do error designs affect the search?

Because an error design is ignored, it has no influence on the search. Except for the wasted CPU time spent on the failed simulations, there are no negative consequences when the percentage of error designs is relatively small.

However, a productive design space exploration involves the accumulation of information about the design landscape, which does require successful design evaluations. When the percentage of error designs is too high, it is difficult for HEEDS to effectively search the design space. As a rule of thumb, 20% or fewer error designs is usually acceptable.  There is no reason to be alarmed or concerned about a small percentage of error designs.  If the number of error designs is much greater than 20%, it is recommended that more attention be given to making your model robust.

There are some possible exceptions to this rule of thumb. For example, if the error designs result from users-defined run conditions or other expected behaviors, then your search results may still be valid even though the percentage of error designs is large. These cases require proper judgment.

error-design-piechart1      error-design-piechart2
Figure 1. (a) Less than about 20% error designs is usually considered acceptable. (b) When much more than 20% of designs are error designs, it becomes difficult for HEEDS to search the space effectively, so improvements to the analysis model may be warranted.

How does HEEDS know if a design is an error?

Error designs can occur for a variety of reasons and under different execution environments. So HEEDS has multiple ways to check the validity of a result.

When HEEDS executes simulation models on the local computer, it knows that an analysis error has occurred if the analysis process ends but the defined responses are not found in the output files (or the output files don’t exist). This is the simplest case.

For other situations, you may need to define Run Conditions, Success Conditions, or Forced Error Bounds to assist HEEDS in identifying error designs. We will not discuss these techniques in detail here.

How can I detect error solutions before executing a large design study?

When each design evaluation takes hours to complete, it might be expensive to find out near the end of a design study that your CAE analysis model is not as robust as you need it to be. It would be nice to know this before you start exploring the design space.

Each simulation model is unique, so there is no one-size-fits-all approach to testing your models. Generally speaking, though, it is often a good idea to thoroughly exercise parts of your defined analysis process, or simpler versions of your models, prior to starting a large design study. Here are a few examples:

  • For shape optimization problems, perform an initial study using only the CAD model or the [CAD + Meshing] models to check for invalid geometries and/or mesh quality issues. This might be done with a Latin hypercube sampling within a design of experiments (DOE) study. In this way, you can quickly explore many combinations of variables in various parts of the design space. If this results in a large percentage of bad geometries or meshes with poor quality, then you may need to improve the models before starting the optimization study.
  • For nonlinear or dynamic analyses, perform a study using a simplified model first in order to verify expected behavior and trends. For example, run a linear model, a model with reduced load levels (smaller nonlinearities) or a model that considers only the initial portion of the total time event. These studies might be parametric optimization studies or DOE studies, depending on your testing goals.

The time required to perform these inexpensive test runs prior to launching a large-scale optimization study is usually time well spent. You will have more confidence in your models and hopefully avoid the expense of rerunning your studies multiple times due to unknown errors.

We hope this tip helps you to discover better designs, faster.

Leave a Reply

Your email address will not be published. Required fields are marked *