Transfer Service

The Transfer Service is part of the Value Cache component, which is known internally as mrCache. It transfers data from the Value Cache into the case data database. When a project has AutoUpdateDataModel set to 2, data is transferred immediately from the cache into the case data; when the property is set to 1 (the default), data is written for each submitted page; when this property is set to zero, data is transferred periodically whenever the Transfer Service runs, which, by default, is every five minutes.

The Transfer Service works by performing two passes. The first pass builds a list of files that are ready for transfer and then transfers the files. An increase in the number of cache files will result in an increase in the amount of time required to identify the files that are ready for transfer. The number of engines are not an area of concern as the engines work independently, and normally work on different cache files.

For information about the points for and against immediate writing to the case data, and how to change the default setting, see Using the Value Cache and the Transfer Service.

The Transfer Service uses the project's IsDatabaseUpToDate property to track the status of its case data database. When the interviewer session engine creates a new data file, this property is set to True because there is currently no data to transfer for the interview. If AutoUpdateDataModel is set to 2, so that responses are transferred to the case data as soon as they are given, the database should always be up to date so IsDatabaseUpToDate should always be True. If AutoUpdateDataModel is set to zero, or there are problems writing to the database, IsDatabaseUpToDate becomes False and the Transfer Service will know that it needs to copy the responses for this interview into the case data.

Data files remain in the Value Cache until their interviews complete. Stopped and timed out interviews do not count as completed interviews. When an interview completes, the file's IsReadyForTransfer property is set to True indicating that the data is ready to be written to the case data database. (To transfer files for stopped and timed out interviews, see Flagging files for transfer from the value cache.)

When the Transfer Service runs, it performs the following tasks:

▪Scans the Value Cache for files belonging to those projects where IsReadyForTransfer is True and copies the responses into the project's case data using the connection string defined in the project's properties. Files are copied in the order in which they were last updated, earliest first. If the data is copied successfully, the Transfer Service sets IsDatabaseUpToDate to True. If the transfer fails for any reason, the value of this property remains unchanged and the file is moved into the TransferFailed subfolder to await a retry. By default, up to three retries are normally made (four attempts in all).

▪If IsDatabaseUpToDate to True, deletes or moves transferred files from the cache. If a CompletedFolder is defined (see Value cache settings) the file is moved to this folder, otherwise the file is deleted.

▪Deletes files that have been waiting in the Value Cache for more than 48 hours. Typically, these are files for interviews that have either timed out or been stopped and that have not been restarted, but it may also include files in the TransferFailed folder that have been tried the maximum number of times without success.

▪When determining the optimum scan interval for clusters with high traffic, the time taken to scan and process the files should be less than the scan interval. The scan time will increase as the cache folder is scanned and the transfer queue length increases. The scan time will subsequently decrease as the files are processed. The scan phase is effectively lengthened when more files are added to the cache.

The IsReadyForTransfer=True files are then added into the transfer queue if the engine has the project loaded. If the engine does not have the project loaded, and the internal DateLastUpdated value is more than four times the scan interval (for example, it was written more than two hour ago), then the transfer service will add automatically add files to transfer queue. This ensures that the interviewing engine also performs the file the transfers; if the interviewing engine fails, another engine will eventually begin transferring files.

The transfer queue is ordered by the DateLastUpdated value that is embedded in each file. This ensures that the older files are transferred before the newer files.

After the directory scan is complete the file transfer process begins. When IsDatabaseUpToDate=False, the file data is transferred to the database and the file is then moved to the Completed folder. When IsDatabaseUpToDate=True, the file is immediately moved to the Completed folder without further processing.

When multiple engines have the same project loaded, it is possible that both engines will add the same cache file to their transfer queues. One engine will transfer the file and the other engines will ignore the file if it has already been processed.

▪Each engine has a single transfer service thread. Each thread will start scanning when the engine starts and after the first interview is run on that engine. Each engine has a preference to only transfer cache files for projects that it has loaded.