Qlik Associative Engine Resource Management

Importance of Resource Management in Qlik

The user experience will be bad if your Qlik dashboard takes a long time to display the data. Slowness can come from several factors: undersized infrastructure, use of a lot of data, bad modelling, bad practices in the development of charts and tables…

It should be ensured that each of the elements listed below is optimized. Make sure that the machine on which Qlik Sense is installed has sufficient resources (CPU and RAM) allocated for Qlik Sense. Preferably, install Qlik Sense on a dedicated machine in order to benefit from all the resources.

The slowness of an application can also come from the fact that the application contains a very large amount of data with regard to the capacity of the machine. Qlik Sense works by default with In-Memory technology, so all the data is loaded to RAM.

Memory/RAM

The main memory RAM is the primary storage for all data to be analysed by the Qlik Associative Engine. The engine mainly allocates memory for:

• The unaggregated dataset that is defined by the document data model.
• The aggregated data (cached result sets) and the calculations defined by the user interface.
• The session state for each user of the document.
• Temporary allocations for helper tables used during calculations.

When a user requests a document, the Qlik Associative Engine loads it into memory (if it has not already been loaded). The dataset for a document is only loaded once, regardless of the number of concurrent users.

When a user makes a selection in the document, the Qlik Associative Engine performs real-time calculations for that selection. Newly calculated results are added to the in-memory cache, which is shared with all users. User session states are also stored in memory and most of the session information is shared between sessions in the same state.

Example: Allocation of memory when loading a single document

The following figure shows an example of the memory allocation over time, with the Qlik Associative Engine running on a clean server and users interacting with a document.

When the document is first loaded into memory, it immediately takes up a block of memory. As users begin interacting with the document and making calculation requests, the result sets are cached and stored in RAM. This takes up the largest chunk of memory, but cached result set can be served without any additional calculation, so they are quite useful. A small amount of memory is allocated to keeping track of the state of active user sessions.

Example: Allocation of memory when loading multiple documents

The following figure shows how multiple documents can fit into RAM, even when the total amount of allocated memory touches the Working set Low /Min memory usage limit.

To release memory to load new documents, the Qlik Associative Engine purges cached result sets. The amount of RAM that is used to cache result sets is the floating amount between the Working set Low / Min memory usage setting and the amount that is consumed by the documents and session state information.

How much RAM is consumed per application? 

RAMInitial = SizeOnDisk × FileSizeMultiplier; this is the initial RAM footprint for any application

FileSizeMultiplier: range between 2-10 (this is a compression* ratio depending on the data contained in the app)

*Compression is based upon the data, and how much we can compress depends upon the homogeneity of the data. The more homogeneous, the more compression Qlik can achieve.
More information on one way of optimizing a document can be found on the Qlik Design Blog: Symbol Tables and Bit-Stuffed Pointers

RAMperUser = RAMinitial × userRAMratio; this is the RAM per each incremental user

userRAMratio: range between 1% -10%

Total RAM used per app:

TotalRAM = (RAMperUser × Number of users) + RAMinitial

Example:

SizeOnDisk = 1 GB

File Size Multiplier 6 (range is from 2 to 10)

RAMinitial = 1 * 6 = 6 GB

If we take a RAM Ratio per User of 6% then it is 0.06 (range is from 1 to 10)

RAMperUser = 6 x 0.06 = 0.36GB

Then RAM Required for 30 Users

TotalRAM = (RAMperUser × No. users) + RAMinitial
TotalRAM =  (0.36 * 30) + 6
TotalRAM = 16.8 GB