Keith Kaufman, CSC Docket No. 2023-1957

BackgroundKeith Kaufman appeals his score on the promotional examination for Fire Officer 2 (PM4200C), Jersey City. It is noted that the appellant passed the examination with a final average of 86.410 and ranks 21st on the eligible list.

The subject promotional examination was held on May 19, 2022, and 39 candidates passed. This two-part examination consisted of an integrated system of simulations designed to generate behavior similar to that required for success on the job. The first part consisted of multiple-choice items that measured specific work components identified and weighted by the job analysis. The second part consisted of three oral scenarios: Supervision, Administration and Incident Command. The examination was based on a comprehensive job analysis conducted by the Civil Service Commission (Commission), which identified the critical areas of the job. The weighting of the test components was derived from the job analysis data. It is noted that candidates were told the following prior to beginning their presentations for each scenario: “In responding to the questions, be as specific as possible. Do not assume or take for granted that general actions will contribute to your score.”

Each candidate in a given jurisdiction was scored by a team of three different Subject Matter Experts (SMEs), who were trained in current technical scoring procedures. Each of these SMEs were current or retired fire officers who held the title of Battalion Fire Chief (or Fire Officer 2) or higher. Candidates were also assessed by three Commission employees trained in oral communication assessment. As part of the scoring process, an SME observed and noted the responses of a candidate relative to the knowledge, skills and abilities (KSAs) that each exercise was designed to measure. An assessor also noted any weaknesses that detracted from the candidate’s overall oral communication ability. Each assessor then rated the candidate’s performance according to the rating standards and assigned the candidate a technical or oral communication score on that exercise.

In order to preserve the relative weighting of each of the components of the examination, the ratings for each portion were adjusted by a well-recognized statistical process known as “standardization.” Under this process, the ratings are standardized by converting the raw scores to z-scores, an expression of the deviation of the score from the mean score of the group in relation to the standard deviation of scores for the group. Each portion of the examination had a relative weight in its relation to the whole examination. Thus, the z-score for the multiple-choice portion was multiplied by a test weight of 36.53%, the oral technical scores were multiplied by a test weight of 53.91% and the oral communication scores were multiplied by a test weight of 9.56%. The weighted z-scores were summed and this became the overall final test score. This was weighted and added to the weighted seniority score. The result was standardized, then normalized, and rounded up to the third decimal place to arrive at a final average.

Each oral examination question, and overall oral communication, was rated on a five-point scale, with 5 as the optimal response, 4 as a more than acceptable passing response, 3 as a minimally acceptable passing response, 2 as a less than acceptable response, and 1 as a much less than acceptable response.

On the Supervision scenario, the appellant scored a 5 on the technical component and a 4 on the oral communication component. On the Administration scenario, the appellant scored a 5 on the technical component and a 4 on the oral communication component. Finally, on the Incident Command scenario, the appellant scored a 5 on the technical component and a 4 on the oral communication component.

Appellant's argumentThe appellant challenges his scores on the oral communication components of the Supervision and Incident Command scenarios, and his seniority score.

On the Supervision scenario, the assessor found that the appellant displayed a minor weakness in word usage/grammar through the continued use of “ahs” and “ums” throughout his presentation and, as a result, awarded the appellant an oral communication score of 4. On appeal, the appellant acknowledges that he did utter some “ahs” and “ums” during his presentation. However, he asserts, in relevant part, that because he “received a perfect score of 5 on the technical part of the supervision question . . . everything [he] needed to communicate was stated and fully understood.” Accordingly, he contends that his score should be changed from a 4 to a 5.

On to the Incident Command scenario, the assessor awarded the appellant an oral communication score of 4 based on a finding that he displayed a minor weakness in rate of speech by speaking quickly throughout his presentation, making some of his remarks difficult to understand and causing him to occasionally stumble over his words. The assessor cited two specific points in the recording of the appellant’s presentation as examples of the appellant’s stumbles. On appeal, the appellant acknowledges that he stumbled at the two points cited by the assessor, but maintains, in relevant part, that he corrected himself and received credit for saying the right answer. He avers that his rate of speech was consistent with his normal talking speed. He adds that since he “received a perfect score of 5 on the technical part of the incident command question, [he] believe[s] all information was fully understood and [he] said everything he needed to say.”

Finally, concerning seniority the appellant states that he received a notice from this agency which erroneously indicated that his seniority score was 8.2192 and he asks that his seniority score be reviewed.

CONCLUSION

In the instant matter, the appellant relies heavily on his technical component ratings of 5 on the Supervision and Incident Command scenarios to argue that his oral communication scores for these same components should have been 5. The Commission emphasizes that it is this agency’s longstanding policy that technical and oral communication component scores are independent ratings on the examination and that an exemplary or poor technical rating does not have a bearing on oral communication scoring and vice versa. Furthermore, oral communication performance can clearly distinguish candidates, including those delivering presentations with comparable technical details. To wit, it would be disingenuous to argue that the presentation of one candidate who spoke at a low rate of volume, had their speech punctuated by the frequent use of filler words like “ah” and “um,” rarely made eye contact with their audience and routinely made distracting hand gestures would be as understandable, effective and well-received as the presentation of another candidate who gave a response with a comparable level of detail, but without these same oral communication issues. For these reasons, mere arguments that because a candidate received an optimal technical score on a scenario, they should have received a corresponding optimal oral communication score are invalid. Given these considerations, the Commission finds that the appellant’s appeal of his oral communication scores is without merit and must be denied.

Nevertheless, in the interest of completeness, the Commission adds the following observations. Initially, it bears noting that the appellant’s rating of 4 on each of the oral communication scores at issue corresponds to a finding that his responses were “more than acceptable.” Scores of 5 correspond to "optimal” responses.

The appellant does not dispute that he uttered “ums” and “ahs” during his Supervision scenario presentation and that he stumbled at multiple points during his Incident Command presentation. It was thus reasonable for the assessors to find that the appellant displayed minor weaknesses in oral communication for the subject scenarios and to rate his presentations as “more than acceptable” (i.e., award him scores of 4), rather than “optimal” (i.e., scores of 5). Further, a review of the appellant’s Supervision and Incident Command scenarios corroborates the assessors’ oral communication scoring. Accordingly, the appellant has failed to sustain his burden of proof with respect to these scores.

As to the appellant’s seniority score, although the appellant submits a notice which stated that his seniority score was 8.2192, a review of agency records indicates that his final average of 86.410 was based, in part, on a seniority score of 82.192. Specifically, examination seniority is based on the time from the regular appointment date (to the eligible title) to the closing date of the announcement, minus the time spent on layoffs, certain leaves of absence without pay, or suspensions. See N.J.A.C. 4A:4-2.15 (Rating of examinations). The appellant received a permanent appointment to the title of Fire Officer 1, effective July 23, 2019, and the closing date was September 30, 2021. His seniority score of 82.192 reflects a base score of 70, plus 10 points for record of service, plus 2.192 for the 2 years, 2 months and 8 days he was a Fire Officer 1. Time spent in a provisional position or as an “acting” Fire Officer 1 is not added to seniority for any candidate. Accordingly, the record demonstrates that the appellant’s seniority score of 82.192 and his final average of 86.410 are correct.

ORDER

Therefore, it is ordered that this appeal be denied.

ClosingThis is the final administrative determination in this matter. Any further review should be pursued in a judicial forum.