EVS2017: What has been done to achieve good data quality?
EVS2017 main goals:
- Transparency for all partners involved in the survey (EVS planning groups, countries, users) and good documentation of all internal processes of data processing and documentation so that the same standards and recommendations apply not only to the first countries in the field but also to the last,
- Implementation of review rounds to make sure that only data and documents approved by the EVS teams and the EVS countries are published,
- Provision of comprehensive documentation of the published data (guidelines, master/field questionnaires, method/variable/ harmonization reports, country-specific notes), which not only helps the users to explore the data but also shows how the recommendations and standards have been implemented to allow an assessment of the data quality.
- Provision of the data as soon as possible via pre-releases. Even though the goal is to provide high data quality and good documentation already for the pre-release, this allows us to incorporate the researchers as reviewers and to improve the data step by step for the full release.
The full documentation, concerning the methodological guidelines followed, data processing, as well as country specific information, is here accessible. We pursued these goals in each phase of the life cycle of the EVS wave 2017:
- At the beginning of the survey project
- The guidelines and standards were designed taking into account the experience of EVS and external experts as well as the results and experiences of the EVS wave 2008.
- All guidelines and workflows were discussed and agreed upon with the EVS countries.
- During the survey project
- Project management: A project management portal (myEVS) supported the collaboration between the members of the EVS central team and the countries during the consultation phase and data processing; the portal guaranteed secure access to the whole communication and the transferred files for all those involved.
- Consultation phase: In the consultation phase, sign-off processes were carried out (related to sampling design, field questionnaire, and classifications) to ensure that the EVS recommendations were appropriately implemented in the countries.
- Consultation during Fieldwork and Data processing: The questions of the countries during the fieldwork and data preparation were answered by the EVS and external GESIS experts; furthermore, a ‘Fieldwork Reporting and Monitoring’ tool was implemented, which supported the Methodology group in monitoring FW progress of the countries.
- Data processing: For data preparation, a workflow was implemented with several steps for checking/cleaning and standardization/harmonization of the national data as well as detailed process documentation was agreed. The internal process documentation was used as the basis for the documentation of the data to be published. In general, all data modifications were agreed upon with the countries. EVS and external experts especially supported the generation of further classifications for pre-classified variables such as education, and the construction of the weight variables. All constructed variables are documented. For harmonized variables, we additionally provide information on the standard applied (source and date), the mapping of national categories to the categories of the harmonized variable in a harmonization report, and the source variable containing the original data. This allows users not only to track the recording but also to replicate the mapping easily.
- The data anonymization comprised three steps: (1) general anonymization measures implemented by the data archive, (2) review of the measures by the country and implementation of further country-specific measures; (3) review of the proposals of the countries to see if they are useful for other EVS countries.
- The national data and documents to be published were finally checked by the respective countries; the integrated data and documentation were made available for review by all members of the EVS project before publication.
- And finally the publication of the results
- Pre-releases were agreed upon in order to be able to provide the data quickly but with good quality and documentation. The pre-releases differ from the full release by (1) light version of documentation; (2) not yet all countries or variables are included in the integrated datasets (3) not yet all datasets are published.
- Light version does not mean that necessary documentation is missing, it means updated and more detailed documentation:
For the full release, we plan an update of existing documentation (additionally):
- All reports: we use official publication formats and covers, add names of authors
- Variable Report: add frequency tables broken down by country for all variables and add a section ‘Changes from previous releases’
- Method Report: add country reports with more detailed information of the Methodological Questionnaire
For the not yet publish data files, we prepare the following documentation:
- Add on sensitive Data: Variable Report
- Add on country-specific variables: Variable overview including original language and English question texts/answer categories
- Individual country datasets: Slim versions of the variable report and method report; all harmonized variables are documented in the harmonization reports of the integrated dataset (see here)