Assessment Framework & Performance Management Policy
ASF Surveyor Training Programme — Assessment Design, Standard-Setting, Moderation, and Quality Assurance
Aligned with ISQua EEA STP 4th Edition (2021) Criteria 5.1–5.4
ISQua EEA STP Criterion References
5.1 “The training programme has a defined and transparent assessment framework that is shared with participants prior to or at the start of the programme.” MET
5.2 “Assessments are valid, reliable and fair and use multiple assessment methods.” MET
5.3 “There is a defined and transparent standard-setting process for determining pass marks.” MET
5.4 “There is a defined process for the moderation and quality assurance of assessments.” MET
5.1 — Assessment Framework
ASF’s Assessment Framework specifies the full set of assessments used in the Surveyor Training Programme, their purpose, format, timing, and pass standards. The Framework is published in the Trainee Handbook and provided to all trainees at enrolment (minimum 14 days before the programme commences), fulfilling the transparency requirement of Criterion 5.1.
Assessment Overview
| Assessment | Criterion Addressed | Format | Weight | Pass Mark | Timing |
|---|---|---|---|---|---|
| Written Knowledge Test | 5.2, 5.3 | 80 MCQ (2 hrs) | Pass/Fail | 70% (56/80) | End of didactic phase (Day 3) |
| OSCE (Objective Structured Clinical Examination) | 5.2, 5.3 | 6 stations × 10 min | Pass/Fail | 70% aggregate; no station <50% | End of didactic phase (Day 4) |
| Supervised Survey Log | 5.2 | Portfolio (3 observations) | Pass/Fail | Satisfactory on all 3 | Mentorship phase (months 1–6) |
| Final Competency Assessment | 5.2, 5.3 | 1-day supervised survey + structured debrief | Pass/Fail | 70% on structured observation form | End of mentorship phase |
| Diversity & Equity Reflection | 4.4, 5.2 | 500-word reflective case vignette | Pass/Fail | Satisfactory (rubric-scored) | After Diversity module |
5.2 — Validity, Reliability, Fairness & Multiple Methods
Validity
Assessment validity is established through a systematic blueprint mapping each assessment item to the ASF Surveyor Competency Framework (derived from the JCI Surveyor Competency Framework, 2020) and to the ISQua EEA STP competency domains. The Written Knowledge Test and OSCE blueprints are reviewed annually by the Assessment Lead and an independent content expert to ensure content validity.
Construct validity for the OSCE is maintained by using standardised scenarios reviewed against current accreditation practice. Face validity is confirmed through pilot testing with a panel of experienced surveyors before each cohort.
Reliability
Interrater reliability for the OSCE and Final Competency Assessment is assessed using Cohen’s kappa (κ). The minimum acceptable threshold is κ ≥ 0.70, consistent with published standards for high-stakes health professional assessments (Landis & Koch, 1977). Where κ falls below 0.70 on any station, the Assessment Lead convenes a calibration session before results are finalised.
Internal consistency of the Written Knowledge Test is monitored annually using Kuder-Richardson Formula 20 (KR-20). A KR-20 value < 0.70 triggers item analysis and test revision.
Fairness & Reasonable Adjustments
ASF’s assessment framework complies with the obligation of non-discrimination under Loi n°2008-496 du 27 mai 2008 and the UN Convention on the Rights of Persons with Disabilities (CRPD, Article 24). Trainees with documented disabilities or specific learning needs may apply for reasonable adjustments (e.g., extended time, alternative formats). All reasonable adjustment requests are reviewed by the Programme Director within 10 working days of receipt. Adjustments do not compromise the validity of the assessment or the standards required for qualification.
Multiple Assessment Methods
ASF uses five distinct assessment formats (Written Knowledge Test, OSCE, Supervised Survey Log, Final Competency Assessment, Reflective case vignette), drawing on three assessment paradigms: knowledge testing, performance observation, and reflective practice. This multi-method approach is consistent with best practice in health professional education (Miller, 1990; Norcini & McKinley, 2007) and mitigates the measurement error inherent in any single method.
5.3 — Standard-Setting Process
Pass marks for all summative assessments are set using defensible, evidence-based standard-setting methods. The standard-setting process is documented and transparent, satisfying Criterion 5.3.
Written Knowledge Test — Angoff Method
The pass mark for the Written Knowledge Test is determined using the Modified Angoff method (Angoff, 1971; Kane, 2001). A panel of 3–5 subject-matter experts, each with ≥5 years of active accreditation surveying experience, independently estimates the probability that a “minimally competent” candidate would answer each item correctly. The mean probability across experts and items establishes the criterion-referenced pass mark. The Angoff-derived pass mark is reviewed against the empirical distribution of candidate scores; if the pass mark falls below the 10th percentile of historical performance, it is re-examined.
OSCE & Final Competency Assessment — Borderline Group Method
Pass marks for the OSCE and Final Competency Assessment are set using the Borderline Group method (Boursicot et al., 2007). Assessors classify each candidate as Clear Fail / Borderline Fail / Borderline Pass / Clear Pass at each station. The mean score of the Borderline Pass group defines the pass mark. This method anchors the standard to actual candidate performance and is widely used in OSCE-format health professional assessments.
Pass Mark Review Cycle
All pass marks are reviewed at the end of each cohort against: (a) pass rates; (b) post-hoc item analysis; (c) trainee feedback; and (d) any changes to the ISQua EEA STP or ASF Surveyor Competency Framework. Revised pass marks take effect from the subsequent cohort.
5.4 — Moderation & Quality Assurance
Pre-Assessment QA
- All assessment instruments (MCQ bank, OSCE station scripts, observation forms) are reviewed by the Assessment Lead and one independent reviewer before each cohort for accuracy, currency, and bias
- OSCE scenarios are piloted with a small group (minimum 3) of experienced surveyors not involved in delivery; timing and difficulty are adjusted based on pilot feedback
- MCQ items are classified by cognitive level (Bloom’s taxonomy levels 1–3) and domain; a minimum of 40% of items must be at application or analysis level (Bloom level 2–3)
In-Assessment Moderation
- OSCE assessors attend a mandatory calibration briefing (minimum 2 hours) before each assessment day, including review of anchor cases and rating scale anchors
- Interrater reliability data (Cohen’s κ) are computed in real time for paired-assessor OSCE stations; stations with κ < 0.70 are reviewed before results are confirmed
- Written Knowledge Tests are scored by automated marking; a 10% sample of answer sheets is manually verified against machine scores
Post-Assessment Review
- Item analysis is conducted after every Written Knowledge Test: items with facility value <0.20 or >0.95, or point-biserial correlation <0.15, are flagged for review or removal
- Assessment outcomes (pass rates, mean scores, standard deviation) are reported to the Programme Director and Quality Committee within 10 working days of each assessment event
- A full psychometric review report is produced annually and retained as documentary evidence for accreditation review
External Moderation
An external examiner (ISQua fellow or equivalent; not employed by ASF) reviews a sample of OSCE recordings and Written Knowledge Test results annually. The external examiner’s report is presented to the Programme Committee and any recommended changes are logged and actioned within one review cycle (12 months). External examiner reports are retained for 7 years.
References & Regulatory Basis
Angoff, W.H. (1971) — “Scales, norms, and equivalent scores.” In R.L. Thorndike (Ed.), Educational Measurement (2nd ed., pp. 508–600). American Council on Education, Washington DC. — Original description of the Angoff standard-setting method.
Kane, M.T. (2001) — “So much remains the same: Conception and status of validation in setting cutscores.” In G.J. Cizek (Ed.), Setting Performance Standards (pp. 53–88). Lawrence Erlbaum. — Contemporary framework for validity in standard-setting.
Boursicot, K., Roberts, T. & Pell, G. (2007) — “Using borderline methods to compare passing standards for OSCEs across different medical schools.” Medical Education, 41(11), 1024–1031. — Borderline Group method for OSCE standard-setting.
Landis, J.R. & Koch, G.G. (1977) — “The measurement of observer agreement for categorical data.” Biometrics, 33(1), 159–174. — Benchmark for interpreting Cohen’s κ in assessor agreement.
Miller, G.E. (1990) — “The assessment of clinical skills/competence/performance.” Academic Medicine, 65(9 Suppl), S63–67. — The Miller pyramid framework for health professional assessment.
Norcini, J. & McKinley, D.W. (2007) — “Assessment methods in medical education.” Teaching and Teacher Education, 23(3), 239–250. — Best-practice evidence for multi-method assessment.
Loi n°2008-496 du 27 mai 2008 — Non-discrimination in vocational training. JORF, 28 mai 2008.
UN Convention on the Rights of Persons with Disabilities (CRPD, 2006) — Article 24: Education; reasonable accommodation obligations.