David Bialas, CSC Docket No. 2023-1995

BackgroundDavid Bialas appeals his score on the promotional examination for Battalion Fire Chief (PM3380C), Elizabeth. It is noted that the appellant passed the examination with a final average of 87.310 and ranks sixth on the eligible list.

The subject promotional examination was held on May 19, 2022, and 16 candidates passed. This two-part examination consisted of an integrated system of simulations designed to generate behavior similar to that required for success on the job. The first part consisted of multiple-choice items that measured specific work components identified and weighted by the job analysis. The second part consisted of three oral scenarios: Supervision, Administration and Incident Command. The examination was based on a comprehensive job analysis conducted by the Civil Service Commission, which identified the critical areas of the job. The weighting of the test components was derived from the job analysis data.

Each candidate in a given jurisdiction was scored by a team of three different Subject Matter Experts (SMEs), who were trained in current technical scoring procedures. Each of these SMEs were current or retired fire officers who held the title of Battalion Fire Chief (or Fire Officer 2) or higher. Candidates were also assessed by three New Jersey Civil Service Commission (Commission) employees trained in oral communication assessment. As part of the scoring process, an SME observed and noted the responses of a candidate relative to the knowledge, skills and abilities (KSAs) that each exercise was designed to measure. An assessor also noted any weaknesses that detracted from the candidate’s overall oral communication ability. Each assessor then rated the candidate’s performance according to the rating standards and assigned the candidate a technical or oral communication score on that exercise.

In order to preserve the relative weighting of each of the components of the examination, the ratings for each portion were adjusted by a well-recognized statistical process known as “standardization.” Under this process, the ratings are standardized by converting the raw scores to z-scores, an expression of the deviation of the score from the mean score of the group in relation to the standard deviation of scores for the group. Each portion of the examination had a relative weight in its relation to the whole examination. Thus, the z-score for the multiple-choice portion was multiplied by a test weight of 36.53%, the oral technical scores were multiplied by a test weight of 53.91% and the oral communication scores were multiplied by a test weight of 9.56%. The weighted z-scores were summed and this became the overall final test score. This was weighted and added to the weighted seniority score. The result was standardized, then normalized, and rounded up to the third decimal place to arrive at a final average.

For the Supervision scenario, the appellant scored a 5 on the technical component and a 5 on the oral communication component. On the Administration scenario, the appellant scored a 5 on the technical component and a 3 on the oral communication component. Finally, on the Incident Command scenario, the appellant scored a 5 on the technical component and a 4 on the oral communication component.

Appellant's argumentThe appellant challenges his scores for the oral communication components of the Administration and Incident Command scenarios. On the Administration scenario, the assessor indicated that the appellant displayed a major weakness in word usage and grammar, as evidenced by the use of filler words “um” and “uh” more than 90 times. The assessor also stated that the appellant repeated words and phrases within a sentence and used incorrect grammar. Based on these findings, the aseessor awarded the appellant a score of 3. On appeal, the appellant, citing Michael Erard, Um. . . : Slips, Stumbles and Verbal Blunders, and What They Mean (2007), presents that speech without ums did not emerge as a cultural standard until the 20th century. He argues that some research-based theories have indicated use of filler words is correlated with the speaker having a large vocabulary. He adds that in the case of first responders, because the National Incident Management System (NIMS) instructs first responders to use common and easily understood terms, speakers often search for the simplest term. In addition, he asserts that the scoring of filler words can lead a grader to perceive other words like “the” or “a” to be filler words and he avers that the assessor even added a filler word after the use of “the” in one of the examples on his scoring sheet, when in actuality, he paused and repeated part of what he was saying to establish and clarify an idea he was detailing. As to incorrect grammar, the appellant avers that the assessor incorrectly quoted him and erroneously lowered his score.

On the Incident Command scenario, the assessor indicated that the appellant displayed a minor weakness in word usage, using filler words like “um” and “uh” in excess of 55 times during his response. Accordingly, the assessor awarded the appellant a score of 4 for the oral communication component of this scenario. On appeal, the appellant asserts that he only used filler words 41 times. He presents that while he understands the need for a concise message absent of ums, he believes that an overcount of his use of filler words produced a lower score than he should have received.

CONCLUSION

Initially, the Commission observes that scoring for the subject examination is not strictly pass or fail. Rather, candidates are given a final average based on performance. The use of filler words undoubtedly undermines the quality and clarity of a presentation, as it is easier to quickly understand and process information that is not obscured by utterances like “uh” or “um.” Thus, it is more than appropriate to rate a candidate’s presentation as “optimal” or “more than acceptable” if they use few or no filler words, while giving “acceptable” or lower ratings to candidates who use filler words at a greater rate. The Commission also emphasizes that the use of a flexible, holistic approach to assess whether and to what extent the use of filler words impacts the effectiveness of a candidate’s oral communication performance does not render the use of this metric invalid or arbitrary. In this regard, the Commission notes, for example, that two candidates may utter 20 “uhs” during their full presentations, but one candidate’s presentation might reasonably be considered more ineffective if they utter “uh” 20 times in the span of a minute than another who utters the same 20 “uhs” over a 10-minute period.

As to the appellant’s Administration scenario presentation, a review of his performance confirms that his use of filler words, including “um” and “uh,” dozens of times was a major weakness. Beyond that, there were numerous instances where the appellant’s fast rate of speech caused him to stumble and repeat words and phrases within sentences, further bolstering the assessor’s finding that the appellant displayed a major weakness in word usage/grammar. Accordingly, the appellant’s score of 3 for the oral communication component of the Administration scenario is sustained.

Further, a review of the appellant’s presentation for the Incident Command scenario confirms that he uttered filler words like “um” and “uh” more than 55 times and that the appellant’s use of filler words clearly constituted a minor weakness in word usage. As such, the record supports the assessor’s award of 4 for the oral communication component of his Incident Command score. Such a score would be justified in this case, even if, as the appellant claims, he used filler words 41 times for this scenario.

Accordingly, a thorough review of the appellant’s submissions and the test materials indicates that the decision below is amply supported by the record, and the appellant has failed to meet his burden of proof in this matter.

ORDER

Therefore, it is ordered that this appeal be denied.

ClosingThis is the final administrative determination in this matter. Any further review should be pursued in a judicial forum.