Frequently Asked Questions

Q: Who can take part in the project?

A: We aim to get the research community involved in this project. Therefore, we require that at least one member of each research team has an active affiliation with a research institution or has already obtained a doctorate degree.

Q: What is a research team?

A: A research team consists of one or two researchers who want to contribute an algorithm to the project.

Q: What are my duties as a research team?

A: Your task is to develop a total of four algorithms (they can be identical) which aggregate predictions in four different domains and yield an aggregated prediction value as a result. You will also need to review a few other teams’ submissions and answer some questionnaires.

Q: Do I need to have programming skills to participate?

A: Yes, each team submits fully programmed algorithms in R or Python. See the section on algorithm development.

Q: How do you determine the winner of the competition?

A: We determine the most accurate prediction algorithm in each domain as follows: We apply the algorithm to each question of each run of each wave of data collection. For each one, we determine the absolute deviation from the respective true values. Then, we take the mean of the 128 (4 questions X 8 runs X 4 waves) absolute deviations. In each domain, the team that submitted the algorithm with the lowest mean absolute deviation wins.

Q: Did an ethics review board evaluate this project?

A: Yes, we had the review board of the University of Innsbruck review and approve the project (ref: 78/2023).

Q: Whom can I contact if I have concerns, questions, or comments?

A: Please email us at email@woccap.com or contact the project coordinators directly.

Q: How do you collect the predictions for the different domains?

A: We will run monthly prediction surveys on Prolific.

Q: Who are the participants?

A: US-based participants will be invited from a selection of the Prolific database which is defined as follows: having a past approval rate of at least 95% and having completed at least 100 studies.

Q: How many predictions will I be able to use in my algorithms?

A: We aim to collect 640 predictions every month from a panel of participants. Monthly data will be split into non-overlapping datasets of 80 predictions each. Algorithms will aggregate the 80 predictions of these datasets and we will run each algorithm against each one of the 8 subsets to generate multiple independent observations.

Q: Will participants be screened?

A: We will implement a CAPTCHA task to all studies on a general welcome screen and exclude those that answer this incorrectly before randomizing participants to individual studies.

Q: Can I include additional screening questions in the prediction survey?

A: You can propose additional measures to be included in the prediction survey and endorse those proposed by other research teams. The project coordinators will make the final selection.

Q: Can I exclude any observations in my algorithms?

A: Observations can be excluded via the proposed algorithm (i.e., assigning a weight of ‘0’ to responses that meet certain criteria).

Q: What does it mean to develop an aggregation algorithm?

A: Your team needs to come up with an idea about how to aggregate the individual predictions made by the participants of the prediction surveys. You will have access to the codebook and demo datasets to develop your algorithm.

Q: Do I need to implement the algorithm that I have come up with in code?

A: Yes! You will need to write the code that implements your aggregation algorithm in either R or Python. You will have access to a codebook specifying exactly which variables are available and how information is coded. You will also get a demo dataset with random data for testing.

Q: Can I test my implementation somehow?

A: Yes, you can upload your algorithm to our sandboxed testing environment. We will execute your code against a demo dataset and report all outputs and errors.

Q: Can I use additional packages in my code?

A: Yes, if you write the code in Python, simply submit a requirements.txt file for use with pip alongside your algorithms. If using R, make sure that your script installs the necessary packages.