Programme Effectiveness Evaluation
ASF Surveyor Training Programme — Monitoring, Evaluation & Continuous Improvement Framework
Aligned with ISQua EEA STP 4th Edition (2021) Criteria 2.7, 2.8, 4.9, 5.5
ISQua EEA STP Criteria Addressed
| Criterion | Exact ISQua Text (4th Ed., 2021) | Status |
|---|---|---|
| 2.7 | “The organisation monitors the effectiveness of the training programme.” | MET |
| 2.8 | “The organisation uses the results of monitoring to continuously improve the training programme.” | MET |
| 4.9 | “The assessment process is regularly reviewed and updated to reflect current evidence and best practice.” | MET |
| 5.5 | “The organisation demonstrates continuous improvement in the quality of the surveyor training programme over time.” | MET |
Evaluation Framework (Criteria 2.7 & 2.8)
ASF’s programme evaluation model is structured on the Kirkpatrick Four-Level Model of Training Evaluation (Kirkpatrick & Kirkpatrick, 2006), the most widely cited and validated framework in professional training evaluation literature. All four levels are evaluated for each training cohort.
| Kirkpatrick Level | What is Evaluated | ASF Method | Frequency | Threshold |
|---|---|---|---|---|
| Level 1 — Reaction | Trainee satisfaction and perceived relevance of training | Post-module satisfaction survey (10-item, 5-point Likert scale); end-of-programme survey | After each module and at programme end | Mean ≥4.0/5.0 on overall satisfaction |
| Level 2 — Learning | Knowledge and skill acquisition against stated learning outcomes | Written MCQ examination (minimum 70% pass mark); OSCE (minimum 70%); portfolio review | At programme end; OSCEs at supervised practice stage | ≥70% of trainees achieve pass mark; mean score ≥75% |
| Level 3 — Behaviour | Application of learning in the survey workplace (transfer of training) | Post-qualification supervisor survey (6 months after qualifying); surveyor self-assessment report | 6 months post-qualification; annually for active surveyors | ≥80% of respondents confirm competence demonstrated in practice |
| Level 4 — Results | Impact on accreditation quality and surveyed facility outcomes | Analysis of ASF survey quality indicators (interrater reliability, complaint rates, appeal rates); correlation with facility re-accreditation outcomes | Annual analysis (calendar year) | Surveyor interrater reliability Cohen’s κ ≥0.70; complaint rate <3% of surveys |
Assessment Review (Criterion 4.9)
The assessment tools used in the Surveyor Training Programme are reviewed annually to ensure they remain current, valid, and aligned with best practice in professional assessment. The review process covers:
Annual Assessment Review Elements
- Content currency: MCQ item bank reviewed against the latest ISQua EEA STP edition and any updates to the accreditation standards used in ASF surveys; outdated items withdrawn and new items developed
- Psychometric analysis: Item difficulty indices (p-values), item discrimination indices (D-values), and Cronbach’s alpha reliability coefficient calculated for each examination administration. Items with p < 0.20 or p > 0.90 or D < 0.20 are reviewed for revision or removal
- Standard setting: Angoff standard-setting procedure (Angoff, 1971; Kane, 2001) conducted before each new cohort by a panel of minimum 5 qualified surveyors to establish the pass mark; documented in the standard-setting report retained in the QMS
- OSCE station review: Each OSCE station reviewed by the independent Assessment Panel annually; marking criteria updated where relevant; inter-rater reliability (Cohen’s κ) calculated for each OSCE administration
- External benchmarking: Assessment methodology benchmarked against equivalent ISQua-accredited programmes annually; external psychometric consultant review every 3 years
Continuous Improvement Demonstration (Criterion 5.5)
ASF is committed to demonstrating continuous improvement in programme quality over time. Evidence of improvement is documented and reported as part of the ISQua accreditation self-assessment process.
Key Quality Indicators (KQIs) — Trended Annually
| KQI | Measure | Target | Improvement Action if Below Target |
|---|---|---|---|
| KQI-1 | Trainee first-attempt pass rate (written examination) | ≥80% | Module content review; assessment item analysis; additional remediation support offered |
| KQI-2 | Trainee satisfaction (overall programme, Level 1 Kirkpatrick) | Mean ≥4.0/5.0 | Focus group with trainees; module-level root cause analysis; trainer performance review |
| KQI-3 | Surveyor interrater reliability (Cohen’s κ) | ≥0.70 (substantial agreement) | Calibration workshop for all active surveyors; targeted standardisation training |
| KQI-4 | Complaint and appeal rate (% of surveys resulting in a formal complaint or appeal) | <3% | Root cause analysis of each complaint; systemic issues addressed in next cohort; trainer briefing updated |
| KQI-5 | CPD compliance rate among active surveyors (% meeting 15-hour annual minimum) | ≥95% | Reminder communications at 9 and 11 months; CPD bursary scheme for financial barriers |
| KQI-6 | Diversity of trainee cohort (gender balance; geographic distribution across France) | No single group >60% of cohort; representation from ≥5 regions | Targeted recruitment outreach; bursary scheme; regional partnership development |
KQI data for the preceding 3 years are published in the ASF Annual Quality Report, accessible to all trainees via the LMS portal and to external stakeholders on request. Trends are presented to the Board annually and to ISQua at each accreditation review.
References & Regulatory Basis
Kirkpatrick, D.L. & Kirkpatrick, J.D. (2006) — Evaluating Training Programs: The Four Levels (3rd ed.). San Francisco: Berrett-Koehler Publishers.
Angoff, W.H. (1971) — “Scales, norms and equivalent scores.” In R.L. Thorndike (Ed.), Educational Measurement (2nd ed., pp. 508–600). Washington DC: American Council on Education.
Kane, M.T. (2001) — “Current concerns in validity theory.” Journal of Educational Measurement, 38(4), 319–342. doi.org/10.1111/j.1745-3984.2001.tb01130.x
Landis, J.R. & Koch, G.G. (1977) — “The measurement of observer agreement for categorical data.” Biometrics, 33(1), 159–174. — Provides the benchmark for interpreting Cohen’s κ: κ ≥0.70 = substantial agreement.
HAS (2012) — Évaluation des formations: guide méthodologique. Haute Autorité de Santé, Saint-Denis La Plaine. has-sante.fr