Validity and Reliability of 'Objective Structured Assessment of Technical Skills' (OSATS) for Vaginal Delivery

OSATS for Normal delivery

Authors

Jaweria Faisal Akhtar Saeed Medical college & Farooq Hospital Rawalpindi

DOI:

https://doi.org/10.51273/esc24.25132024

Keywords:

Cronbach's alpha, Validity, OSATS, Reliability, Workplace-based assessment.

Abstract

Objective: To assess validity and reliability metrics of objective structured assessment of technical skills
(OSATS) for assessing the competence of technical skills of vaginal delivery
Materials and Methods: Co-relational reliability study, the validation study was conducted at the Department
of Obstetrics and Gynaecology of Akhtar Saeed Medical College and Farooq Hospital Rawalpindi from April
to December 2023. The study comprised nine candidates at different training levels (LHV, staff, postgraduate
trainees & senior registrars), who were evaluated on OSATS by 8 evaluators over 54 NVD observations. Cronbach's
alpha and intraclass correlation coefficient (ICC) were used to estimate reliability metrics of OSATS scores.
The construct validity; that OSATS can measure and reveal the underlying construct, was determined by three
shreds of evidence; evidence of content, response process and internal structure, and evidence of improvement in
performance scores as the procedure-specific experience accrues with OSATS using linear mixed model and
evidence of the discriminatory power of the tool between various levels of the training program, with one-way ANOVA.
Results: The OSATS tool demonstrated good reliability metrics with Cronbach's alpha of 0.897 and good inter-
rater reliability calculated by intraclass correlation coefficient as 0.801. OSATS was validated as examinee
scores improved showing gradual progression on performance curves with succeeding OSATS encounters.
Validation also yielded that the tool can differentiate among different training levels with a p-value of 0.019
Conclusion: The OSATS tool was found to be valid and reliable for assessing the competence of technical skills of vaginal delivery.

References

Strandbygaard J, Scheele F, Sørensen JL. Twelve tips for assessing surgical performance and use of technical assessment scales. Med Teach 2017 Jan 2; 39(1):32-7

Khaliq T. Reliability of Results Produced Through Objectively Structured Assessment of Technical Skills (OSATS) for Endotracheal Intubation (ETI). JCPSP 2013; 23(1): 51-5.

Norcini JJ. Work based assessment ABC of learning and teaching in medicine. BMJ 2003; 326:753-5.

Zoller A, Holle T, Wepler M, Radermacher P, Nussbaum BL. Development of a novel global rating scale for objective structured assessment of technical skills in an emergency medical simulation training. BMC Medical Education 2021 Dec; 21:1-8.

Martin JA, Regehr G, Reznick R, Macrae H, Murnaghan J, Hutchison C, Brow M. Objective structured assessment of technical skill (OSATS) for surgical residents. British Journal of Surgery 1997 Feb; 84(2):273-8.

Schmidt MW, Haney CM, Kowalewski KF, Bintintan VV, Abu Hilal M, Arezzo A, et al. Development and validity evidence of an objective structured assessment of technical skills score for minimally invasive linear-stapled, hand-sewn intestinal anastomoses: the A-OSATS score. Surgical endoscopy 2022 Jun; 1:1-3.

Asif H, McInnis C, Dang F, Ajzenberg H, Wang PL, Mosa A, et al. Objective Structured Assessment of technical skill (OSATS) in the Surgical Skills and Technology Elective Program (SSTEP): comparison of peer and expert raters. The American Journal of Surgery 2022 Feb 1; 223(2):276-9.

Hatala R, Cook DA, Brydges R, Hawkins R. Constructing a validity argument for the Objective Structured Assessment of Technical Skills (OSATS): a systematic review of validity evidence. Adv in Health Sci Educ 2015; 20:1149–75.

Antomarchi J, Delotte J, Jordan A, Tran A, Bongai A. Development and validation of an objective structured assessment of technical skill tool for the practice of vertex presentation delivery. Arch Gynecol Obstet 2014; 290:243–7.

Treil L, Neumann N, Chanes N, Lejay A, Bourcier T, Bismuth J, et al. Objective Evaluation of Clock Face Suture Using the Objective Structured Assessment of Technical Skill (OSATS) Checklist. In EJVES Vascular Forum 2022 Jan 1; 57: 5-11.

Kyriazos TA, Stalikas A. Applied Psychometrics: The Steps of Scale Development and Standardization Process. Psychology 2018; 9: 2531-60.

Steger D, Jankowsky K, Schroeders U, Wilhelm O. The road to hell is paved with good intentions: How common practices in scale construction hurt validity. Assessment 2023 Sep; 30(6):1811-24.

Amirrudin M, Nasution K, Supahar S. Effect of variability on Cronbach alpha reliability in research practice. Jurnal Matematika, Statistika dan Komputasi 2021; 17(2):223-30.

Carsuzaa F, Payen C, Gallet P, Favier V. French translation and validation of the OSATS tool for the assessment of surgical skill. Journal of Visceral Surgery 2023 Dec 1; 160(6):402-6.

Fouillen KJ, Duncan HF, de Vries P, Chevalier V. Development and preliminary validation of an Objective Structured Assessment of Technical Skills (OSATS) for a partial pulpotomy procedure. International Endodontic Journal 2023 May 25; 56(8):1011-21.

Sürücü L, Maslakci A. Validity and reliability in quantitative research. Business & Management Studies: An International Journal. 2020 Sep 25; 8(3):2694-726.

Xue C, Yuan J, Lo GG, Chang AT, Poon DM, Wong OL, et al. Radiomics feature reliability assessed by intraclass correlation coefficient: a systematic review. Quantitative Imaging in Medicine and Surgery. 2021 Oct; 11(10):4431.

Maybodi FR, Keshmiri F, Kazemipoor M, Owlia F. Assessment of suturing and scaling skills of periodontology and oral medicine residents by OSATS method: a pilot study. BMC Medical Education 2023 Nov 21; 23(1):889.

Ramazani F, Wright ED, Randall DR, Lin JR, Jeffery CC. Developing an Objective Structured Assessment of Technical Skills (OSATS) for Microlaryngoscopy. The Laryngoscope 2023 Feb 23.

Jakubowski L, Leader BA, Ishman SL, Chun R, Tarima SS, Parikh S, Skinner M, Thorne M, Weatherly R, Wiet G, Brown DJ. Validation of an objective assessment tool for tonsillectomy in otolaryngology resident training. The Laryngoscope. 2021 Feb; 131(2):E359-66.

Vaidya A, Aydin A, Ridgley J, Raison N, Dasgupta P, Ahmed K. Current status of technical skills assessment tools in surgery: a systematic review. Journal of Surgical Research. 2020 Feb 1; 246:342-78.

Navrazhina K, Alam M, Tung R, Cressey BD, Decker A, Surprenant D, et al. A blinded, multirater and multi-institutional study evaluating the Objective Structured Assessment of Technical Skills (OSATS) tool in dermatology education. JAAD 2021 Nov 1; 85(5):1346-8.

Chavescampos ME, de Oliveira MM, de Assis LB, Reis AB, Gonçalves FB. Validation of the objective structured assessment of technical skill in brasil. Revista da Associação Médica Brasileira 2020; 66(3):328-33.

Benmansour M, Malti A, Jannin P. Deep neural network architecture for automated soft surgical skills evaluation using objective structured assessment of technical skills criteria. International Journal of Computer Assisted Radiology and Surgery. 2023 Jan 25; 18:1-9.

Downloads

Published

2024-07-18

How to Cite

Faisal J. Validity and Reliability of ’Objective Structured Assessment of Technical Skills’ (OSATS) for Vaginal Delivery: OSATS for Normal delivery. Esculapio - JSIMS [Internet]. 2024 Jul. 18 [cited 2024 Nov. 21];20(2). Available from: https://esculapio.pk/journal/index.php/journal-files/article/view/1074