Cleaning data using a DMS file
The DMS file provides a powerful scripting solution to cleaning data:
▪You can clean the data using SQL syntax, mrScriptBasic, or a combination of both SQL syntax and mrScriptBasic. However, unless you are an expert in SQL syntax, mrScriptBasic is generally easier to read and understand.
▪By default, the clean data is written to a different data source, thus keeping the original “dirty” data intact. An advantage of keeping the original data is that it is available for comparison if subsequently there are any queries about the validity of the changes made to the data during the cleaning process. However, the original data can be overwritten if required.
▪You can clean the data and export it to more than one data format (such as IBM SPSS Statistics .sav file and Quantum .dat file) in one operation.
▪You can use the same cleaning algorithms on data that has been collected and entered using a combination of methods; such as a combination of UNICOM Intelligence Interviewer - Server, UNICOM Intelligence Interviewer - Paper/Scan Add-on, and manual data entry of UNICOM Intelligence Interviewer - Paper questionnaires.
▪You can first write a report on the errors and use this to design the cleaning algorithms, which you then run.
▪You can clean the data in stages. For example, at the end of each day or week of a long-term project.
▪You can use the three data cleaning system variables (DataCleaning.Status, DataCleaning.ReviewStatus, and DataCleaning.Note) to mark a case as requiring review and write notes about the problems found.
▪You can remove variables that are not required for analysis. You do this by setting up a metadata filter. For more information, see
Filtering data in a DMS file.
See