Flawed data released under MySchool

Today, the Australian Curriculum, Reporting and Assessment Authority (ACARA) released flawed NAPLAN data comparisons for the ACT on the MySchool website.

The data reported for the ACT do not present a reliable indicator of NAPLAN performance in the ACT. This is because of flaws in a new method for comparing the actual performance of a school with a statistical prediction of how the school should have performed in NAPLAN.

ACARA, other states and territories, and education data experts agree that there are anomalies in the ACT data.

With the support of all states and territories and the federal government, ACARA will seek expert advice to review the approach adopted for NAPLAN comparisons this year. Until this review is completed and the flaws corrected the comparisons cannot be relied on as a measure of the ACT’s NAPLAN performance.

ACARA’s new method for benchmarking school NAPLAN results predicts the score that a student should have achieved in NAPLAN based on four factors. These factors are their parents’ job title and highest level of education, as well as the remoteness of their school and the percentage of Indigenous students at the school.

The average of predicted student scores for a school indicates the “expected” school performance, which is then compared with the actual average school score.

The method for predicting student scores does not work in the ACT.

The ACT has a predominately public service workforce where many people hold the job title ‘manager’ across a range of employment classifications. 65 per cent of ACT students have a parent with a bachelor’s degree or above, compared to the national average of 35 per cent.

These two factors together artificially increase predicted NAPLAN scores. Comparing the predicted and actual scores results in a large difference that falsely suggests underperformance of ACT schools, both government and non-government.

There is also a problem with establishing meaningful statistical prediction for the ACT because of the relatively small number of schools and students.

The change to how ACARA compares results between schools has meant that the data comparisons released today for ACT schools are even more unreliable and invalid than in previous years.

As I have consistently raised, NAPLAN is misused when schools are compared as though it is a competition. And there is more to measuring school performance than NAPLAN.

ACT schools, students and teachers continue to perform well. This is reinforced by the ACT bucking the national trend in the most recent OECD’s Programme for International School Assessment (PISA) test.

PISA tests 15-year-old students across the world conducted in 2018 and the results show that ACT students perform better on average than students across Australia, and better than most of the participating countries when it comes to mathematics, scientific literacy and reading literacy.

Specifically, the ACT ranked 3rd in reading (Australia ranked 17th), 4th in Science (Australia ranked 18th), and 11th in maths (Australia ranked 30th).

Additionally, the most recent ACT Post School Destinations and Pathways report, released last year, shows 93 per cent of young people who graduated in 2017 were employed or undertaking further study in 2018 — 6 per cent higher than the national average. Further, 97 per cent of 2015 year 12 graduates reported that they were employed or studying in 2018.

The life outcomes of a student is the whole point of school and the education system.

NAPLAN is a narrow point in time assessment that provides limited information about only a few aspects among all the learning happening in schools.

The aggregate result of the 2019 NAPLAN tests show that the ACT continues to perform well achieving top or equal top by mean score in 18 of 20 areas tested in NAPLAN 2019.

The ACT Government aims to give every child and young person a great education and, in turn, the positive life experiences that flow from that. Our school leaders, teachers and support staff work every day to equip our students for success.

/Public Release. View in full here.