Skill Lab: Science Detective has three goals. The first goal is to build a community of the ScienceAtHome games’ players by providing them with a fun gaming platform while giving them feedback in the shape of a Cognitive map, which can serve as performance assessment in other games (i.e. commercial off-the-shelf games) or self assessment (i.e. what are your cognitive strengths and weaknesses). The second goal is to create a base of player profiles linked to the other games from ScienceAtHome, allowing better game optimization for future Citizen Science projects. The third goal is to create a database of different cognitive indicators for people with different age, gender, and cultural background, thus allowing drawing perspectives on cognitive normality, cognitive differentiation (e.g. exceptional abilities), and cognitive decline (e.g. dementia. See for example ‘Sea hero quest’ [1]).

Skill Lab: Science Detective aims to explore cognitive skills at different levels, from finely grained skills (such as memory and reaction times[2][3]) to systemic organized high-order cognitive skills (such as Executive functions and Visuospatial reasoning [3][4]). We use the principles of task analysis in human problem-solving [5] for systematically and procedurally evaluate the use of different cognitive skills in problem-solving, how these skills interact, and how they are presented in each of the tasks.

It is not the first time this type of studies are made, for example, the game ‘Sea Hero Quest’ aims to explore and understand dementia based on the analysis of big data of players regarding their spatial navigation skills. Other games such as ‘Crayon Physics Deluxe’ [6] [7] [8] aims for the creation of learning assessment tools for Newtonian physics, seeking not only to assess learning by means of video games but to understand how players and learners generate creative solutions to problems that cannot be tackled by using brute force. Additionally, other authors [9] have explored the use of video games as means of cognitive evaluation batteries for understanding cognitive processes such as spatial and logic reasoning under a processual perspective.

To achieve these goals, Skill Lab: Science Detective is building under the principles of Evidence-centred Design and Stealth Assessment. Evidence-centred Design is a way to collect information which allows us generating inferences about what people know, believe, and do when they are solving a problem in a specific environment[7][8]. Stealth Assessment is a way to invisibly include Evidence-centred Design under the layer of games, so players do not feel evaluated while playing, thus exerting their skills as they will normally do in their everyday life [8].

In addition to this, the motivation that is inherent to the games, as well as some gamification elements, make Skill Lab an engaging tool that can serve multiple purposes in science, from the creation of player profiles for citizen science projects to high-scale assessment of cognitive skills.

Additionally, Skill Lab: Science Detective is built under the principles of neuropsychological [10] and new trends in educational assessment [8], allowing outlining cognitive skills at different levels, including the way different skills interact when doing contextualized tasks.

Sources:

1. Glitchers. (2016). Sea hero quest. Retrieved from www.seaheroquest.com/en/
2. Eysenck, M., & Keane, M. (2003). Cognitive Psychology A Student’s Handbook (4th ed.). USA: Psychology Press.
3. Garnham, A., & Oakhill, J. (1994). Thinking and Reasoning. USA: Basil Blackwell Inc.
4. Holyoak, K., & Morrison, M. (2005). The Cambridge Handbook of Thinking and Reasoning. USA: Cambridge University Press.
5. Newell, A., & Simon, H. (1972). Human Problem Solving. USA: Prentice-Hall.
6. Petri Purho. 2009. Crayon Physics Deluxe.
7. Shute, V., Moore, G., & Wang, L. (2015). Measuring Problem Solving Skills in Plants vs. Zombies 2. In Proceedings of the 8th International Conference on Educational Data Mining (pp. 428–431). Spain: ERIC.
8. Shute, V., & Ventura, M. (2013). Stealth Assessment. Measuring and Supporting Learning in Video Games. England: MIT Press.
9. Castaño, C. & Chavés, L. (2010). ‘Estudio piloto de razonamiento probabilístico, razonamiento silogístico y toma de decisiones por medio de una batería de evaluación procesual (software) en niños entre cinco y nueve años de edad de la ciudad de Medellín’. Thesis to achieve the title of Psychologist. University of Antioquia, Medellín, Colombia.
10. Luria, A., & Tsvetkova, L. (1990). The Neuropsychological Analysis of Problem Solving. (R. Sbordone, Ed., A. Mikheyev, Trans.). USA: Paul M. Deutsch Press, Inc.

When you play Skill Lab: Science Detective you can contribute to our research in understanding cognitive skills. Another benefit is the chance to get an overview of your own skills when solving problems in the game. We call this visualization the Cognitive map. The map is unlocked when you complete your ScienceAtHome User Profile, the questionnaire, and have played all of the minigames.

The Cognitive map shows the five major skills that are assessed in the game. These are reasoning, memory, language, perception and executive function. Under these, are smaller subdivisions such as working memory and prospective memory accompanied by a brief in-game explanation.

You can view your strengths in a bar chart. Here, it’s important to understand how we make these estimations. We use the data gathered from the game to optimize an estimate of your skills. It does not mean that it’s a measure that is comparable to another player’s estimates.

The data generating the bar chart visualization is dependent on data from your other skills. If your memory skills appear to be stronger than your language skills, it does not mean that your language skills are necessarily bad, just weaker than your memory skills. The cognitive map is therefore not relevant for making comparisons between different players.

Some skill estimates are not accessible in the bar graph yet, as we still need more data to estimate them. These skills can be viewed on the map but without any data.

What are the validation tasks in Skill Lab: Science Detective used for?

We need to use cognitive tasks alongside the games as a means of calibration, that is to say, as a way to ensure that the skills tapped by the games are the same tapped by psychological experiments. This is important because we need to be sure the games are providing reliable data for understanding cognitive skills. In addition, the data from the validation tasks are also used to improve the model we use to estimate the cognitive skills.

We know that the tasks are time-consuming and less entertaining than the game itself. However, the more people play these tasks and games, the more accurate our estimations will be in the future.

As a reward for your dedication we will also give you a diploma of participation as proof of your contribution to science and to reveal the type of detective you are!

We hope that the potential of this research will encourage you to complete the validation tasks in Skill Lab: Science Detective.

Thanks again for supporting our work. Your help is invaluable to our Citizen science project!

Skill Lab: Science Detective explores cognitive skills at different levels. In the game, we use the principles of task analysis in human problem-solving to systematically and procedurally evaluate the use of different cognitive skills in problem-solving, how these skills interact, and how they are presented in each of the tasks. To answer some of the questions you may have about the mechanics of the game, how and what data we collect and how we use it we have created an FAQ to help you navigate the research taking place in the game.

Is a citizen science approach reliable in social and cognitive science?
Researchers need people to participate in their experiments. In the social sciences, researchers often work with small pools of participants with low variability, sometimes relying only on laboratory studies. This has resulted in researchers having a very good understanding of how a particular group of people behave under a particular set of circumstances but fails to provide information from the broader population in everyday contexts. At ScienceAtHome, we want to democratize science and draw upon a wider pool of participants to improve the quality and generalizability of our research. Such an approach is also an opportunity to communicate science to a wider audience. We are aware that introducing social science studies to the “wild” reduces the control of variables and noise. However, with the citizen science mission, we believe that a sufficient amount of people from all demographics can be mobilized to overcome these issues and deliver more general insights into human nature. Given all the methodological challenges we will face, we see our Skill Lab: Science Detective game as the first, modest step into this new direction for conducting social science on a massive scale.

Is this AI research?
Due to the attention in the media, AI research has become a very broad envelope of research topics. Within the field, there is a growing awareness that humans have unique ways of processing large amounts of data very efficiently in situations where quick, intuitive responses are required. This has led to an increasing focus on understanding human decisions, actions, and interactions. One dream within the AI community is to discover “the secret of human data processing” and put this in the form of algorithms for the next generation of AI. However, the more closely we examine the human mind, the more complexity appears, so this “dream” may not be very feasible. Instead, many researchers (including us) believe that we should focus on learning more about the intricacies of humans, not in order to replace them, but in order to better understand how we can generate digital interfaces between human and algorithms that make optimal use of both. We call this hybrid intelligence. We believe that the challenge for the future lies in knowing our human strengths as accurately as possible in order to help shape the best possible future for all.

Who is ScienceAtHome?
ScienceAtHome is a diverse team of scientists, game developers, designers and visual artists based at Aarhus University, Denmark. We create fun scientific games, with the aim of revolutionizing scientific research and teaching by game-play.

What is my data used for?
We are using your data to learn how different people solve problems and make decisions. This data enables us to create Cognitive maps which can help in understanding individual behavior, group interaction and address societal questions in general. We also hope to use this data to develop hybrid computation algorithms to solve problems more efficiently. This could allow us to solve problems currently too complicated for humans or computers to solve alone.

Is my data going to be used for advertising or commercial purposes?
Not at all. Your data will only be used only for research purposes.

Who has access to my data?
Only the scientific team at ScienceAtHome who are directly related to the data will have access to it.

Does anyone else have access to my data
ScienceAtHome may collaborate with other faculties, professors, or students in the future. In that case, our collaborators will be allowed only to a pseudonymized version of the data, and only to the data required to answer their research questions.

What rights do I have regarding my data?
Under the European General Data Protection Regulation, you have the rights to withdraw from the research whenever you want and ask for your personally identifiable information to be deleted. For a full description of your rights, please read our Privacy Policy and Terms of services in the Legal section of scienceathome.org

What is your ultimate goal in gathering this data?
We have two goals in mind when creating Skill Lab: Science Detective. The first goal is to understand how people make decisions and solve problems, which can help us understanding cognitive skills, how they develop, and how they interact, as well as how people interact in groups, and which group dynamics are more efficient.
Also, we want to optimize algorithms used to solve complex problems such as cooling down atoms, understanding turbulence and finding novel ways of solving scheduling tasks in complex networks (often from the technical, natural sciences), and our citizen science research has shown that humans are surprisingly good at this. We want to dive into these citizen's Cognitive maps, in order to understand what people are doing and how this info can be used to optimize computational algorithms.

What exactly is Cognitive mapping?
Cognitive mapping is a technique where a ‘map’ or a ‘blueprint’ is created of how you use different cognitive skills based on the different ways you solve problems. It works in a similar way to some psychological tests, but this technique relies on your actions while solving problems rather than your answers to standardized questions.

How can the Cognitive map be used?
At ScienceAtHome we will use Cognitive mapping to help build a model for understanding how people solve problems and make decisions. This can help us to generate a better understanding of human mind and how cognitive skills interact with each other, as well as optimize the way artificial intelligence solves difficult problems.

There are multiple other uses for Cognitive mapping. For example, it has been used in game research to improve the user experience in games. Some psychologists use Cognitive mapping for career assessment. In certain cases, it is even usable for target marketing. At ScienceAtHome we will only use your data for solving scientific questions about the mind, the society, Artificial Intelligence, and to improve the player experience in our own games.

When can I view my personal Cognitive map?
Your Cognitive map and player persona (the type of detective you are) will be automatically unlocked once you have completed your ScienceAtHome User profile, the questionnaire, and have played all of the minigames.

Can I compare my Cognitive map with my friends’ map?
The Cognitive map supplied to you in Skill Lab: Science Detective is specially tailored for you, which means that it is based on your own strengths and weaknesses without comparing it to the general population. You can compare it with your friends or other people, but it will carry little value; apart from you being able to see the strengths and weakness of your friend and for them to know yours: It will not say if you are ‘better’ than someone else, or vice versa.

Are there any risks to Cognitive mapping?
Cognitive mapping is not risky in itself, but it can be misused. For example, if a person not familiar with the theory behind the map uses the data provided by Skill Lab: Science Detective to make a psychological assessment, it might cause harm. Nevertheless, at ScienceAtHome we are not using the tool for these purposes, and our legal statements forbid the use of Cognitive mapping for any type of assessment outside our scientific team.

What is the fastest way to get my Cognitive map?
The fastest way is to enter the game and play the games located in each of the labs, and completing your ScienceAtHome User profile and the questionnaire. You can keep track of your progress via the progress bar in the bottom of the screen. 

Am I allowed to use the game for assessment?
You are not allowed to use the game for group-assessment or to assess other people. The feedback provided by the games are indicators (values interpreted within a research context), that work only for yourself to have an idea of which are your personal strengths and weaknesses. They are not values directly referring to whom you or other people are unless correctly interpreted within a context.

What are the scientific bases for what you are doing?
We work under the principles of cognitive psychology and cognitive neuropsychology. Particularly for Skill Lab: Science Detective we are using Citizen Science as a participation technique, and the principles of ‘stealth assessment’ and ‘evidence-based design’ to create Cognitive maps. Similar projects have adopted these techniques in games such as ‘Crayon Physics Deluxe’ and 'Sea Hero Quest'.

What is the relation to the British brain training experiment organized by BBC in 2009?
There is no relation to the brain training experiment or any other brain training apps in the market. Different types of research have proven that brain training video games are not effective to train general skills, but the proficiency in solving the specific games. Because we know there’s no relationship between playing the games and general skill training (this relationship is also called transfer), we cannot claim that our games will help your general skills in any way.

When will I get my Player persona?
Your player persona (the type of detective you are) will be automatically unlocked once you have played all of the games located in each lab.

Where/when can I see my High Scores?
We will make high scores available in the ScienceAtHome Hub in-game in the near future.

When will I get my participation diploma?
You will be eligible for a diploma when you have played every game located in the labs, and completed the Validation tasks, the Questionnaire, and your ScienceAtHome User profile

What information will my participation diploma contain?
The diploma that ScienceAtHome will award you for your participation in helping to build Skill Lab: Science Detective will contain your Player persona (the type of detective you are), and an acknowledgment of how you have helped the science behind the game.

Are you measuring intelligence?
No. Intelligence or IQ is a clinical measure. We are gathering data on how developed some of your cognitive skills are and how they interact when you play games and solve problems for research (non-clinical) purposes.

How much should I trust the Player persona and Cognitive mapping?
We are in the process of creating and validating our models. Although we have made some early validations, the model is always developing and improving. So it might be that for starters you don’t feel that the persona or the Cognitive map say much about you, or feel that they are incomplete. However, the more you and other people play, the more we can improve it. Also, in the future, we will be adding more games to expand the map of cognitive abilities we are generating.