Exam Reliability Study
Structured, anonymized exam outcome & rubric data · 2024–2026

Methodology & Transparency

How this study works

Full transparency about what data we collect, why we collect it, how it is anonymized, and how we will analyze and publish findings.

Independence declaration

This study is independent, non-commercial research. It is not affiliated with, sponsored by, funded by, or endorsed by PSIA (Professional Ski Instructors of America), AASI (American Association of Snowboard Instructors), or any of their regional divisions, affiliates, or chapter organizations. No compensation is received from any ski resort, employer, certification body, or industry organization.

The study has no institutional affiliation. It is operated independently by researchers with a professional interest in measurement reliability and fair assessment standards in ski and snowboard instructor certification.


Research design

This is an observational, self-report study. Participants voluntarily submit structured data about their own exam experience. Data collection covers the 2024–25 and 2025–26 PSIA/AASI exam seasons.

We are not conducting an experiment and are not manipulating any variables. Our goal is to describe the distribution of exam outcomes and rubric scores across disciplines, levels, divisions, and seasons — and to identify patterns that may indicate systematic inconsistency in how exam standards are applied.

Primary research questions

Limitations


What data is collected

Data fieldPurposePublished?
Discipline & credential levelStratify outcomes by exam typeAggregated
Exam season (2024–25 / 2025–26)Identify temporal trendsAggregated
Exam outcome (pass / fail / partial)Core dependent variableAggregated
Division / regionIdentify geographic variabilityAggregated
Attempt numberTrack repeat attempt outcomesAggregated
Rubric scores per itemAssess applied scoring consistencyAggregated means only
Exam format & conditionsControl variablesAggregated
Group size & examiner countStructural exam variablesAggregated
Demo quality & feedback deliveryProcess quality indicatorsAggregated
Teaching experience rangeControl for candidate backgroundAggregated
Home resortIdentify patterns by mountainAggregated (min. n≥15)
Employer / ski schoolIdentify patterns by employerAggregated (min. n≥15)
Candidate notes (free text)Qualitative contextNever published
Email address (optional)Study updates onlyNever published
Uploaded assessment sheetVerify self-reported scoresNever published

What data is NOT collected


Anonymization process

Before any analysis or publication, all submissions are reviewed to remove or generalize any information that could identify an individual candidate or examiner. Free-text fields (candidate notes, examiner feedback summaries) are reviewed manually and are never published verbatim. Uploaded assessment sheets are used solely for internal score verification and are stored in an access-controlled location outside the public web root.

Published findings will contain only aggregated statistics (means, distributions, counts, percentages). No individual submission record will ever be published.


Why we are not publishing interim data

Publishing preliminary pass/fail rates or score distributions before data collection closes would introduce response bias: candidates who see early results may be more or less likely to submit based on whether the results match their expectations, and may anchor their self-reported scores to published distributions. To protect the integrity of the dataset, outcome data will remain private until the collection period ends.

We will publish participation counts (once a meaningful threshold is reached) but no outcome statistics until the study is complete.


Study timeline

Sign up for notification when results are published.

Contribute your exam results → About this study