As we work in an international organization, we often get a question such as: “What is the right assessment for our country, an international assessment, a regional assessment, or our own national assessment?” This is probably due to the tremendous growth in the number of assessments.
Since the International Association for Evaluation of the Educational Achievement (IEA) implemented the first international mathematics study in the late 1950s, many large-scale assessments have appeared in the field, and much more countries have been participating in these assessment studies. For example, when OECD first implemented the Programme for International Student Achievement (PISA) in 2000, there were 28 countries who participated, while in 2012, this number increased to some 65. At the same time, there have been a number of regional assessments that have collected assessment data since the 1990s. These include Southern and East Consortium for Monitoring Educational Quality (SACMEQ), Programme d’Analyse des Systèmes Educatifs des Pays de la CONFEMEN (PASEC), Laboratorio Latinoamericano de Evaluación de la Calidad de la Educación (LLECE), etc. Many countries have also established national assessments, it has been reported that most of the countries have conducted at least one national assessment since 2000, the year of the Dakar EFA conference.
There is no straight answer to the question, but certain issues must be considered in order to decide which assessment might be most appropriate for your country.
1. Subject matter – What subjects are considered important in your country? They could be basic cognitive skills such as mathematics, reading, and science. How about life skills, such as health knowledge, global citizenship, sustainable development knowledge, entrepreneurship? Most of the international and regional assessments deal with the basic cognitive skills. This was the case for the initial SACMEQ study; however, during the SACMEQ Assembly of Ministers in 2005, Ministers wished to have information on the level of HIV and AIDS knowledge of the Grade 6 pupils. A new initiative, South East Asia Primary Learning Metrics (SEA-PLM), which is organized by the UNICEF and South East Asia Ministries of Education Organization (SEAMEO) is planning to measure global citizenship, based on the regionally common values.
2. Test framework – Do you want to see if the students have acquired the knowledge taught at school, or see if the schools prepared students to be ready for the world of work? Most of the assessments, such as IEA’s TIMSS, PIRLS, as well as regional assessments such as SACMEQ, PASEC, and LLECE are based on the school curriculum. For example, in the 1990s when Zimbabwe launched its first national assessment, the concern was to see if the intended school curriculum was actually implemented by the teachers and achieved by the pupils, and therefore, this became the model for SACMEQ. On the other hand, PISA and PISA for Development are exceptions, where the test framework is rather forward looking and assesses the students’ competencies to be demonstrated in a totally novel situation.
3. Target population – Which grade level or age level needs to be assessed? Are you concerned about the early grade pupils who need to be diagnosed? Then what language should be used for the tests? If the local mother tongue is used as the language of instruction, then standardization of such variety of test forms could be a big challenge. On the other hand, you may be challenged if you administer the tests in a unique language at this early level. This was the reasons in which SACMEQ countries opted for Grade 6. While PASEC also test at Grade 6, they also test Grade 2, and therefore a careful test construction and test operation would be required. IEA’s TIMSS administers at Grade 4, as well as Grade 8 to see the progress between these two grades. Even if you are concerned about the comprehension of school subjects at the end of the cycle, the question may not be about which grade level, but which age level. The grade-based target population is the norm when the test framework is based on the school curriculum, prescribed for different grade levels. However, PISA and PISA for Development use the age-level target population, 15 years of age. The challenge for many developing countries using the age-based target population is the large proportion of the students who may belong to different grade levels or different cycles due to repetitions or late entries to schools, or already dropped out of the system.
4. What to do with the assessment data – What is the purpose of measuring the learning achievement? If it is for a certification of a cycle or selection into a higher cycle, then public examinations would be sufficient. If the countries would like to connect the achievement data with other background information such as teacher characteristics, teacher performance, school resources, home background, etc., then background questionnaires are required. You may be also interested in evaluating the competency of teachers. In this case, a teacher test must be considered, as was the case for SACMEQ studies since 2000. This allowed SACMEQ policy makers to realize not only the teachers’ pedagogical skills but also the subject knowledge was critical for learning improvement. Does the international and/or regional assessment that you are about to join include the kind of information that are the priority area of the Ministry?
5. Research capacity needs – The Ministries of Education will have to ensure that they have the technical capacity to carry out the assessment. This includes the sampling of schools and students, test construction and item analyses, data processing, data analyses, and report writing. The advantage of belonging to regional or international assessments would be that the same methodology is shared and a common capacity building is usually provided by the technical teams for the participating members. In certain cases, participating member countries could collaborate and help each other, as experienced by some SACMEQ countries.
6. Ownership – Capacity building should lead to the countries’ ownership of the assessment study. This includes ownership of the data, instruments, and all the methodologies. This would allow countries to do more autonomous secondary data analyses using the data collected, after the international and/or regional reporting, so that the assessment results and policy recommendations could be also linked back to the original policy concerns. Both SACMEQ and PASEC use this strategy where country teams are responsible for producing more focused analyses.
The Ministry may be also interested in joining all of the existing international and regional assessment networks, as well as conducting its own national assessment. But we need to remember that to improve quality of education, it’s not the level of participation in assessment studies that matters, it’s the way assessment results are used. After all, the assessment is merely one small part within the policy reform.