Robert Sabia, CSC Docket No. 2019-2186

BackgroundRobert Sabia appeals the scoring of the promotional examination for Fire Captain (PM1051V), Paterson. It is noted that the appellant passed the subject examination with a final score of 83.470 and his name appears as the 41st ranked eligible on the subject list.

It is noted for the record that this two-part examination consisted of a written multiple-choice portion and an oral portion. Candidates were required to pass the written portion of the examination, and then were ranked on their performance on both portions of the examination. The test was worth 80 percent of the final score and seniority was worth the remaining 20 percent. Of the test weights, 31.35% of the score was the written multiple-choice portion, 22.49% was the technical score for the evolving exercise, 7.53% was the supervision score for the evolving exercise, 4.28% was the oral communication score for the evolving exercise, 19.23% was the technical score for the arriving exercise, 7.53% was the supervision score for the arriving exercise, and 7.59% was the oral communication score for the arriving exercise.

Every candidate received a Candidate Feedback Report at examination review which explained standardization in layman’s terms and provided all the calculations relative to the individual candidate. Z-scores were used as the type of standardized score. Standardizing scores places scores of different scales on the same scale. This is necessary as the written multiple-choice exam, the six oral exam scores, and seniority were on different scales. On appeal, the appellant calculates his own score without using standardization, but merely multiplying the weights of each portion of the test to the scores he received and adding them, and then multiplying this score by 80% and adding a weighted seniority score.

CONCLUSION

N.J.S.A. 11A:4-1(b) gives the Civil Service Commission (Commission) the statutory authority to rate examinations, and Commission staff rated this examination in a proper manner. See N.J.A.C. 4A:4-2.15(a). The appellant’s method of scoring is not valid, as it does not account for the different scales used in scoring. Standardization preserves the relative weighting of each of the components of the examination. Under this process, a standardized z-score represents both the relative position of an individual score in a distribution as compared to the mean and the variation of scores in the distribution. Z-scores will form a distribution identical to the distribution of raw scores; the mean of z-scores will equal zero and the variance of a z-distribution will always be one, as will the standard deviation. This places all scores on the same scale, that is, it provides a score that is directly comparable within and between different types of scores. A negative z-score indicates the score is below the distribution mean, while a positive z-score indicates the score is above the distribution mean. These scores are then “normalized.” Standardization allows for the comparison of scores that are from different normal distributions. When an examination has multiple disparate scores, weighting each of them and adding the weights together results in a nonsensical final average, since each score has a different normal distribution. The appellant’s calculations do not include standardized scoring and are simply incorrect.

The appellant is arguing that a statistically incorrect method of scoring should be used. The argument raised by the appellant reflects a basic misunderstanding of scoring. As evidenced above, there is no statistical basis for adding three diverse scores to arrive at a final average without standardization. The testing process involves different tests, multiple-choice and oral, and seniority, so it is critical that scores be standardized before they are combined, which puts all scores onto the same playing field. Without standardization, the final score has no meaning, as all of the variables are not in proportion to one another. Put another way, the scores for the multiple-choice exam, the oral exam, and seniority are not on the same scale until they are standardized, regardless of weighting. Candidates do not have the authority to determine proper scoring methods.

The appellant states that z-scores are designed to keep scores closer together when tests are very difficult to more fairly determine scores. This statement is simply untrue. Z-scores measure the number of standard deviations a score is from the mean, and are measured in standard deviation units. Thus, they transform scales in order to equalize them. The appellant questions whether the z-score was based on the entire candidate population or only on the scores of passing candidates. He surmises that the scores were formulated incorrectly, but does not assert which way he believes is the most appropriate method. The most appropriate method, which was used, was to include only scores of passing candidates on the written exam. The scores of candidates who failed the written portion are eliminated from the process and do not contribute towards the calculations of the written mean and standard deviation. This scoring method was developed according to psychometrically approved standards and approved by the U.S. Department of Justice. There is no evidence of a scoring error.

A thorough review of the appellant’s submissions and the test materials indicates that the decision below is amply supported by the record, and the appellant has failed to meet his burden of proof in this matter.

ORDER

Therefore, it is ordered that this appeal be denied.

ClosingThis is the final administrative determination in this matter. Any further review should be pursued in a judicial forum.