Expression queue tuning

The expression engine might need tuning based on the server resources, expression count, and complexity for each installation. Generally, the process involves tuning and measuring the performance for one parameter at a time repeatedly until the performance cannot be increased further. A track of records must be maintained. A new tuning is required for the change of hardware, usage, and setup. As an administrator, the following application settings must be checked for each installation.

The maximum (not guaranteed) number of hours the expression queue history is retained. Only the run expressions are pruned. This is done by the formula cleanup job.

Tuning tip: Executed queue entries in the formula queue table are kept as execution history. This history is used to check for execution loops and to generate statistics. If statistics are not needed, this number can be lowered in a high-throughput setup to keep the table size down. Never set this value to zero, as proper loop checking requires at least an hour of history.

The desired maximum number of entries in the expression queue. The cleanup job tries to keep the entry count below this number by pruning the history. Only the run expressions are pruned. This overrules the formula.queue.maxage setting.

This number can be decreased to keep the formula queue size down during high load periods, and to save history otherwise. A good approximation of this number is to keep it a little below the average throughput within the formula.queue.maxage period.

The number of expression queue entries that are allocated for one thread at each allocation run. The entries are marked as “started”. The thread runs through them, and complete them before allocating a new batch.

This is an important setting. Background information: Expressions are executed concurrently, out of order on several threads on several nodes at the same time, but always in a dependency-aware chronological order. For example, if two expressions on the queue are unrelated, they might be executed in a different order to the one in which they were added. Consequently, expression entries might not be executed if there are instances of it, or entries whose values it depends on, that are unexecuted before it. This, and the concurrency, requires allocation to avoid collisions, bad order, and redundant work.

As allocation is an expensive operation, several expressions are allocated together with the above requirement, and with the condition that they are executed in a given order. If a lot of expressions are allocated, the allocation overhead is saved, but that "locks up" concurrency, and vice versa. The general rule is to keep this number down if the expressions form long dependency chains, and increase it if they are short, or dependencies are few. You can change this value at runtime and therefore, should tune it until a reasonable optimum is found. Values less than 10 or more than 200 are rarely optimal.

The frequency of cleaning the expression queue. This is where maxage and maxsize are enforced. Normally only queue pruning takes place. The first cleaning takes place when the server starts, at which point special maintenance is done. Also, every two weeks a timer expression integrity check is done (at night).

Most of the time the default is reasonable. If the throughput is high, cleaning takes longer, but should be performed more often, and vice versa. Beware of increasing this time too much, as this might cause execution outages or bad performance if the load suddenly increases significantly (for example, after an import or added attribute).

The frequency of checking the expression queue for new unallocated expressions to run. When a batch is finished, a new check takes place immediately without waiting for the interval. If the queue still contains unevaluated entries when the next interval occurs, a new worker thread is spawned to increase the effort.

In earlier versions of Focal Point, the execution was asynchronous and immediate. When expressions moved to the database, this was no longer possible without the risk of deadlocks. This is the interval at which the expression queue is checked for new expressions to run, that is, the maximum time (along with the execution time) that the user must wait (read: refresh) for an expression to be executed. Polling for unevaluated queue entries is not expensive if there are none, and therefore the time could be short. However, if the interval is too short it consumes resources unnecessarily. This also affects the ramping of concurrency. If the worker thread fired at the previous interval time is not finished within the interval, another thread (till thread.per.nodes) is started. If ramping is too slow, or if lower evaluation latencies are required, you can lower this number. If expressions are rarely used, you can increase it slightly.

The maximum number of concurrent threads running expressions for each node. The actual maximum number dynamically depends on whether there are free threads in the application thread pool.

The concurrency ramping described in background.interval increases the number of expression evaluation threads on the node at each interval if there is more work to do. This requires free threads in the application thread pool, which is automatically tuned with regard to the number of CPU cores (virtual or physical). It is also capped to not consume the last thread in the pool. You can experiment with this number. Better performance is not guaranteed though, as it depends largely on the concurrency ability of the database to keep up. A sign that this is too low is that the queue most often has unevaluated expressions, and that the database load is low.