WHEN THE SUN RISES WE WORK HARD TO DELIVER Understanding the use of

WHEN THE SUN RISES WE WORK HARD TO DELIVER Understanding the use of pre- and post-tests in the grade 8 and 9 Natural Sciences teachers in Mpumalanga province Presenters: Maria Kekana & Sipho Dlamini O.R Tambo MST Academy (Mpumalanga) MSTA 1

PRESENTATION OUTLINE 1. Background of MSTA 2. Introduction 3. Aim and research question 4. Rationale 5. Methodology 6. Literature review 7. Results 8. Recommendations 9. Conclusion 2

Background of O.R. Tambo MSTA MSTA was established by the Mpumalanga Department of Education for: Increasing the number of learners taking Mathematics, Sciences and technology Improving quality passes (50% and above) in MST subjects 3 MSTA

Background of MSTA cont STRUCTURE OF THE ACADEMY 23 MSTA linked schools 26 MSTA linked schools Sub-hub Acorns-Oaks Bohlabela Sub-hub Khumbula Sec Ehlanzeni MSTA Main-hub Emalahleni 24 MSTA linked schools Sub-hub Izimbali Comb. Gert Sibande Sub-hub Kwandebele SS Nkangala 24 MSTA linked schools 4

Introduction A huge budget is invested annually in teacher trainings impact of these trainings is important Teachers attend trainings to upgrade their subject content –so immediate impact need to be assessed. There is an interest to explore if pre- and post-testing are effective in measuring the impact of teacher trainings. A long term evaluation of all the MSTA teacher trainings is still to be conducted. In the beginning of each training session, a pre-test is administered and on the last day of the training a post test is administered. 5 MSTA

Aim and research question Research aim To investigate the effectiveness of using pre- and post-tests to measure the impact of teacher trainings. Research question: To what extent a pre- and post-testing measure the impact of teacher trainings? Research sub-questions: Is there any difference in scores between the pre-test and the post-test during the MSTA teacher trainings? (quantitatively.) H0: The median of the difference score is equal to zero (H0:Me 0) (both 2016 and 2017 pre- and post-tests) To what extent did the teachers benefit from the Natural Sciences teacher trainings? (qualitatively) 6

Rationale Pre-test and post-test were scarcely administered in the past during teacher trainings by the Department of Basic Education in South Africa (S.A). However, recently this practice of pre- and post-testing was introduced and implemented to assess the immediate impact of teacher trainings. The view of using pre- and post-test in teacher trainings is growing at all levels in the Department of Basic Education ranging from cluster to provincial teacher trainings. This study is one among the few to focus on exploring the extent at which pre and post-testing measure the impact of teacher trainings in South Africa. 7

Literature review Pre-tests reveal the learners’ abilities before entry and provide the baseline in the beginning of the programme training. Post testing illustrate the individual teacher’s gain from the training. Challenges of taking long to mark the tests and also accuracy in marking can be minimized by online testing. Literature also revealed that online testing went smoothly as there were no logistical challenges; grading was done automatically online and the instructor workload reduced since marking was conducted online too. The US Department of Education argue that pre- and post-tests have some benefits and according to them the benefits include the following: Indicates student progress from a particular programme to keep them informed of their progress. Administrators can use the tests to improve programmes. Measure effectiveness of programmes/trainings. For reporting purposes. To see where training needs to be improved in terms of the presenter or probably an approach needs to be changed. 8

Literature review cont Lessons learnt from literature reviewed were that there are factors that may affect the pre- and post-test writing and these factors that may impact pre-post-tests results are: Practice effects where teachers take the same tests/questions repeatedly, this could be mitigated by having a pool of questions/question banks to choose from as pre- and post-tests are developed. Fatigue effects which could be due to hunger or getting tired as you write the test. It is suggested that this factor can be mitigated by having short pre- and post-tests. Motivational effects which could be due to teachers’ willingness to excel and therefore do a lot of self-study as opposed to being taught; the opposite is also true when a teacher is demotivated, this may be mitigated by encouraging all teachers to succeed. The manner in which the test is administered is another factor and can be mitigated by avoiding a noisy, venue where there are some distractions when writing tests. Length of pre- and post-tests might also be an important factor, too short might miss the purpose and produce invalid scores and too long might create boredom. Mitigate by having reasonable length of pre- and post-tests. To see where training needs to be improved in terms of the presenter or probably an approach needs to be changed. 9

Methodology The mixed method approach was used to achieve the aim of this research. The paradigm used in this research is based on pragmatism which integrates the positivistic and interpretive research paradigms. This research was conducted in Mpumalanga Province of South Africa in the O.R. Tambo MST Academy teacher trainings Data was collected from the grade 8 & 9 Natural Sciences teacher trainings in 2016 (5 days) and 2017 (3 days). All the pre- and post-tests of teachers who attended the workshops that occurred over the winter school holidays were collected. These were then recorded to see if there were any differences between the pre-test and post-tests scores. The teachers’ scores from the pre- and post-tests were analysed quantitatively using Wilcoxon signed-rank test, as well as qualitatively using qualitative content analysis of the semi-structured interviews and pre and post-tests records. Ethical considerations: teachers used codes on their pre and post-tests scripts/ teachers’ informed consent 10

Methodology cont There are several statistical techniques that were considered to analyse the quantitative data in this study. Among others are t-test, ANOVA and Wilcoxon signed-rank test. The t-test can be used for two independent group as well for one group with two variables (Maree, 2017). ANOVA is well suited for analyzing the data where there are two independent groups involved. Wilcoxon signed-rank test is similar to a t-test where two variables are compared in a single group (Maree, 2017) Certain assumptions like random sampling, normality, equal variances and others does not matter much in the Wilcoxon signed-rank test unlike it matters with t-test. It is against this background that Wilcoxon signed-rank test was used in this study. 11

Results 2016 NS workshop scores TEACHERS CODE PRE TEST % POST TEST % VARIANCE 1.TC 2706 43 36 -7 2.AB773 45 52 7 3.WXYZ 33 66 33 4.FF01 27 38 11 5.2580 48 75 27 6.PM102 45 31 -14 7.HDB808 40 94 54 8.NEC 527 59 10 -49 9.CZWO 69 64 -5 10.SIS 245 57 26 -31 11.MSN 217 50 86 36 12.SKN 40 83 43 13.VVV 123 59 72 13 14.AMG 14 33 56 23 15.LLB 101 26 19 -7 16.MLK 1 48 69 21 17.HKZ 07 46 42 -4 12

Results cont. 2017 NS workshop scores TEACHERS CODE PRE TEST % POST TEST % VARIANCE 1.MASH 1975 75 82 7 2.0766619295 31 94 63 3.HOT2E 38 82 44 4.AKA 44 58 14 5.CD 25 75 94 19 6.DAM 705 69 71 2 7.FAM 59 44 76 32 8.FRW 950 31 100 69 9.ENS 35 75 94 19 10.91 25 82 57 11.DNC 75 88 13 12.0823100408 38 76 38 13.MN63 68 94 26 14.KWA 81 88 7 15.0828120086 13 94 81 16.BMDS 56 88 32 17.3664 A 63 71 8 18.1440 44 71 27 19.MP 805.H 38 72 34 13

Results cont. 2016: The p-value is 0.00298 which is less than 0.005. This suggest that the null hypothesis of zero median difference is rejected the median is significantly different from zero. The data provides sufficient evidence that the teacher training had a positive influence on the natural sciences teachers. 2017 The p-value is 0.00 which is less than 0.005. This suggest that the null hypothesis of zero median difference is rejected the median is significantly different from zero. The data provides sufficient evidence that the teacher training had a positive influence on the Natural Sciences teachers. 2016 2017 No. attended 66 41 Wrote both pre- and post 49% 56% Wrote only pre- 27% 12% Wrote only post- 15% 32% Did not write both 9% 0% 14

Results cont. 2016: 7 Teachers whose scores dropped. TEACHERS CODE PRE TEST % POST TEST % 1) 45 31 2) NEC 527 59 10 3) CZWO 69 64 4) SIS 57 26 5) HKZ 46 42 6) TPM51 50 49 7) BNM 43 17 PM 102 15

Qualitative results The pre- and post-tests scores were recorded and they were received. The total number of teachers who attended the NS workshop in 2016 was 66. Out of the 66 only 32 wrote both pre and post tests and this translated to 49% of teachers who wrote both tests. Out of the 66 teachers, 18 wrote only pre- test which translated to 27%. Furthermore, 10 out of 66 teachers wrote post-test only (15%) and 6 did not write both the pre and posttests which is about 9%. The total number of teachers who attended the NS workshop in 2017 was 41. Out of the 41 only 23 wrote both pre and post-tests which translated to 56%. Five of the 41 teachers wrote only pre- test which is 12%. Thirteen of the 41 wrote post-test (32%) only. There were no teachers who did not write both the pre and post-tests in 2017 NS teacher trainings. 16

Qualitative results Reasons for not writing pre or post: -searching for training venues, -underestimated travelling time, -start from work, -received the communique very late -requested to leave early -unknown about the remainder of the teachers Teachers welcomed the pre and post lest: they able to measure themselves There were no minimum standards in place for setting a pre and post test The pre and post test did not adhere to any cognitive levels of difficulty (e.g. Bloom’s Taxonomy) 17

Discussion and conclusion It has been statistically proven in this research that pre and post-testing during teacher training do measure the impact of the teacher trainings. Therefore, pre and post testing can be effective tools to be utilized in teacher trainings. The are no standards in place to set pre and post tests While pre and post test have positive intentions , absence of set standards could lead to compliance Teachers have embraced the pre and post tests Pre- and post-testing do measure the impact of short term teacher trainings to a large extent Mayer and Lloyd, 2011 disagree that short term training are effective and have an impact on teacher knowledge Some literature also advice is that: Make sure the test is designed to be used as a pre-post-test: check if the test is aligned to the participants ; there should be different forms of assessment in the tests. Ensure that the test is age, skill levels and background appropriate: the test should be adaptive i.e. one can adapt it to different participants; should be reviewed. It should measure what it supposed to measure: indicate outcomes e.g. recall, understanding etc. Consider using same assessment as peers: take tests from other countries or districts or circuits provided contexts are similar. Conditions for taking the pre-post tests should be similar: choose place conducive for writing the pre-post; environment for pre should be similar to the post one; time taken be the same too. 18

Recommendations teacher trainings should be prolonged for them to have impact on teacher content knowledge – post-testing can be administered months after the training was concluded pre and post-testing alone may not be sufficient to measure the impact of teacher trainings, learner performance on the content that the teachers were trained can be used as a measure to assess the impact Setting standards and etc. (e.g. US Department of Education) to be considered when setting pre- and post tests It is believed that what the teachers have learnt would not immediately be translated into knowledge. Guskey (2002) argued that change takes time and effort to be realized. 19

THANK YOU 20