Synchronization Web Service working environment
The resources that the Synchronization Web Service accesses are located on the UNICOM Intelligence Interviewer. As a result, users must have the proper authority to access the server resources. Authority can be obtained by registering the user/computer combination and passing the client GUID (as well as user ID and password). This step is only necessary when registering a new user. It requires the user to be signed onto the Console. Subsequent synchronizations do not require this step, and as such, users only need to send the computer ID to create a synchronization session prior to synchronizing data.
Detailed synchronization steps
1 Client: The client sends a
sync_start message (see
Begin method) to the server.
Server: The server updates its synchronization status information for this computer and sets up a new session for the synchronization request.
2 Client: The client initiates a
Get request to retrieve updated user information. Updated information includes changed passwords, roles, or permissions.
Server: The server retrieves the user information for users on this system (as recorded by the server) and sends a user record for each user.
Client: The client receives user records and updates its local authentication information.
3 Client and server: For each project, upload cache (chjs), sample (mdd and ddf), and quota (mdd and ddf):
a Client generates a unique package ID for the upload package.
b Client builds manifest of uploaded project data (all cache, sample, and quota).
c Client zips cache, sample, and quota into a single compressed package.
d Server receives uploaded cache (chjs), sample (mdd and ddf), and quota (mdd and ddf) package:
e Check whether the project exists; if not, move package intoInbox\FailedUploadPackages folder and return an upload failed signal to the client. The error log entry
Unable to save package {0} to Inbox: Project {1} does not exist in DPM, it has already been removed. Move packageID {2} to FailedUploadPackages folder
is logged to the MIV log file.
f If the project exists, check whether the package size equals the received package size; if not, move the package into Inbox\FailedUploadPackages, and return an upload failed signal to the client. The error log entry Unable to save package {0} to Inbox: Package size should be {1} bytes, but receive {2} bytes. Move packageID {3} to FailedUploadPackages folder is logged to the MIV log file.
g If the requested package size equals the received package size, save the package (per the packageID that corresponds to the new folder) and save the request .xml, with the ClientInfo\MachineID node to the Inbox folder. The saved package contains one manifest .xml file and one project data .zip file. Return an upload success signal to the client and let the background worker thread process the saved upload package in the Inbox folder.
h If the client receive the success signal, move the cache files into the Archive folder, and do nothing for sample and quota mdd and ddf files. If the upload failed, only add the project to the UploadFailedProjects list to avoid downloading a failed project. Continue uploading data for the next project, even if the previous project upload failed.
i Refer to step 9 for further details on the worker thread process.
4 Client: The client initiates a Post request to retrieve shared content. The client saves shared content to a local folder.
5 Client and server: For each project – download project files:
a Client builds manifest of the current project files and uploads to the server (for comparison changes purposes).
b Server generates a unique download package ID.
c Server compares the client-supplied manifest with the current project files and hashes and builds a change manifest (GET and DELETE)
d Server zips GET files into a download package; the manifest and package ID are returned to the client.
e Client parses the XML to extract the download package ID. Now ready to download the project files.
f Client reads the download stream and saves to a local file.
g Client expands download package and copies new or changed files to the project. Obsolete files are also removed at this time.
The client initiates a
Get request to retrieve the sample, case (only case data with sample is downloaded), and quota data (if any).
h When the project includes sample, the sample management name is appended to the Get request, and each client’s sample record information (SampleId, ReturnTime and CheckSum string), which is used to determine whether or not to download the sample, is appended into a Get request if the client has an existing sample.
i When the project includes quota, the quota project name is appended to the Get request.
The server receives a
Get request to retrieve the sample, case (only case data with sample is downloaded), and quota data (if any).
j When the project includes sample, attempt to retrieve the sample and sample history for the request interviewer:
Rule 1: If the client’s sample is the same as the server’s sample, or the client’s sample is newer than the server’s sample, do not download the server’s sample to the client. Otherwise, download the sample to the client. CheckSum is used to check if the client and server sample are the same and is created by appending all column string values (except the Serial column, since Serial is always different between the server and client).
Rule 2: If the client does not have the sample on the server (for example, it is a newly assigned sample record), download the sample to the client.
Rule 3: If the client has a sample that does not exist on the server, or is reassigned to another interviewer, the client removes the sample.
Rule 4: When downloading the sample, also download the sample history data to the client.
k When the project includes sample, attempt to download associated case data that is based on the download sample.
l When the project includes quota, always download quota data.
m The server zips the sample, case, and quota data into a download package.
The client is now ready to update the local sample, case, and quota data (if any):
n The client parses the XML to extract the download package ID. The client is now ready to update local sample data, case, and quota data (if any).
o The client reads the download stream and saves the information to a local temp folder.
p When the project includes sample, write the download sample to the local Participants mdd/ddf file.
If the sample already exists on the client, overwrite the full sample record and update the internal LocalChanged field to false to avoid uploading unchanged sample to the server (unless continues a survey using the sample; when the survey starts, sets the LocalChanged flag true).
If the sample is newly assigned, insert the sample into the local Participants mdd/ddf.
Delete samples that belong to the current interviewer but are now reassigned to another interviewer, or samples that are deleted from the server. The server download stream contains information regarding which samples are reassigned to other interviewers or deleted from the server.
q When the project includes sample, verify if the local case data needs to be written or updated.
If the case data does not exist on the client, write the case data to a local project mdd/ddf.
If the case data already exists on the client, overwrite the client's case data with the server's case data.
r When the project includes quota, update the client's quota data with the server data based on the following rule:
Rule 1: When the server's TimeLastUpdated value is newer than the client’s, update CompleteCount based on the server’s CompleteCount value. Otherwise keep the client’s CompleteCount value. Always update the PreviousCompleteCount value to equal the CompleteCount value.
All other .ddf quota columns are overwritten based on the server's data.
6 Client and server: For each log file package, extract the upload package zip file into a <subfolder> on the server (all client log files are synchronized and stored centrally on the server).
a Upload log files.
b Client generates a unique package ID for the upload package.
c Client builds a manifest of uploaded log files.
d Client zips the log files into a single compressed package.
e Server receives uploaded log zip file, saves it to the Inbox folder, returns an upload success signal to the client. The background worker thread process saves the upload package to the Inbox folder.
Refer to step 9 for further details on the worker thread process.
7 Client and server: The client uses a
Get request to retrieve updated configuration options from the server.
a The server retrieves the configuration file for the remote client, from DPM Site > Properties > RemoteDevices > Configuration, and sends it to the client.
b The client receives updated configuration information and saves it to the SiteInfo.xml file. The client then rereads the updated configuration.
8 Client: The client sends a
sync_end message (see
End method) to the server.
9 Server: Background worker thread process
The worker thread scans all of the Inbox sub-folders every second (other than the InvalidCacheFiles, ProcessedUploadPackages and FailedUploadPackages sub-folders) for the creation date and time.
Every sub-folder represents a package that needs to be processed.
For each project data (cache, sample, quota) package:
a Extract the package zip file to a temporary folder.
b Process the sample (if any) and retrieve the client's changed samples. The internal LocalChanged field is used to identify whether the sample needs to be updated to the server). When LocalChanged is true, update the sample to the server (if it is not a rejected sample).
c The Synchronization service will reject the sample based on following rules. All reject rules are based on sample data, and all rejected sample, sample history, related cache file, and related quota are written to the SyncReject folder (which is set as a DPM property):
Rule 1: When the sample’s Queue status is Complete in the server database.
Rule 2: When the sample’s Queue status is Active in the server database.
Rule 3: When the sample’s ReturnTime in the server database is newer than client’s.
The sample is rejected when any of these rules are met, and all rejected sample, sample history, related cache file, and related quota (if any) are written to the SyncReject folder (which is set in the DPM property).
d When the sample is not rejected, the full sample record, and related sample history, is updated to the server.
e A serial number is generated for each cache file. If the case data's Respondent.ID exists in the database, Respondend.Serial is returned; if not, a new serial number is generated for Respondent.ID.
If the case data's Respondent.ID belongs to the rejected sample list, move the cache file into the SyncReject folder.
If case data exists on the server, and the server case data's DataCollection.FinishTime is newer than the client’s, delete the client cache file since it is older than the server cache file.
Otherwise, the IsReadyForTransfer property is set to True, and the IsDatabaseUpToDate property is set to False. The cache file is then copied to the Cache directory. The Cache directory is configured in DPM via the Site > Properties > InterviewCacheUNC property, enabling it to be handled by the TransferService when it runs.
The TransferService transfers all
.chj files in the Cache directory to the database. This includes
.chj files that were created via web interviewing and personal
.chj files that have been copied to the directory. Completed and partially completed cache files can be synchronized. If errors are encountered when processing a cache file, the file is moved to the Inbox\InvalidCacheFiles folder, while the next cache file is processed. See
Transfer Service for more information.
f For project that include sample, the sample table‘s Serial column is updated to equal to server case data’s Respondend.Serial column.
g For projects that include quota, check the Respondend.Serial for each quota record against the list of rejected serials. If the serial is in the rejected list, write the quota CompleteHistory to the SyncReject folder.
h For non-rejected quota data:
Update the QUOTA_PersonalCompleted table based on the client's CompleteCount.
Update the QUOTA_QuotaCount table's CompleteCount field. The client's quota mdd/ddf has an internal PreviousCompleteCount field that is used to record the increase count from the last synchronization. The server CompleteCount is incremented by the difference between the client's CompleteCount and the client's PreviousCompleteCount.
A new row is then inserted into the QUOTA_Serial server table for every Respondend.Serial in the client.
i When errors are encountered during package processing, the failed package is moved to the Inbox\FailedUploadPackages folder.
See also