Scott McCarthy, CSC Docket No. 2023-2086

BackgroundScott McCarthy appeals his score on the promotional examination for Battalion Fire Chief (PM3395C), Rahway. It is noted that the appellant passed the examination with a final average of 84.330 and ranks second on the eligible list.

The subject promotional examination was held on May 25, 2022, and four candidates passed. This two-part examination consisted of an integrated system of simulations designed to generate behavior similar to that required for success on the job. The first part consisted of multiple-choice items that measured specific work components identified and weighted by the job analysis. The second part consisted of three oral scenarios: Supervision, Administration and Incident Command. The examination was based on a comprehensive job analysis conducted by the Civil Service Commission (Commission), which identified the critical areas of the job. The weighting of the test components was derived from the job analysis data. It is noted that candidates were told the following prior to beginning their presentations for each scenario: “In responding to the questions, be as specific as possible. Do not assume or take for granted that general actions will contribute to your score.”

Each candidate in a given jurisdiction was scored by a team of three different Subject Matter Experts (SMEs), who were trained in current technical scoring procedures. Each of these SMEs were current or retired fire officers who held the title of Battalion Fire Chief (or Fire Officer 2) or higher. Candidates were also assessed by three Commission employees trained in oral communication assessment. As part of the scoring process, an SME observed and noted the responses of a candidate relative to the knowledge, skills and abilities (KSAs) that each exercise was designed to measure. An assessor also noted any weaknesses that detracted from the candidate’s overall oral communication ability. Each assessor then rated the candidate’s performance according to the rating standards and assigned the candidate a technical or oral communication score on that exercise.

In order to preserve the relative weighting of each of the components of the examination, the ratings for each portion were adjusted by a well-recognized statistical process known as “standardization.” Under this process, the ratings are standardized by converting the raw scores to z-scores, an expression of the deviation of the score from the mean score of the group in relation to the standard deviation of scores for the group. Each portion of the examination had a relative weight in its relation to the whole examination. Thus, the z-score for the multiple-choice portion was multiplied by a test weight of 36.53%, the oral technical scores were multiplied by a test weight of 53.91% and the oral communication scores were multiplied by a test weight of 9.56%. The weighted z-scores were summed and this became the overall final test score. This was weighted and added to the weighted seniority score. The result was standardized, then normalized, and rounded up to the third decimal place to arrive at a final average.

Each oral examination question, and overall oral communication, was rated on a five-point scale, with 5 as the optimal response, 4 as a more than acceptable passing response, 3 as a minimally acceptable passing response, 2 as a less than acceptable response, and 1 as a much less than acceptable response.

On the Supervision scenario, the appellant scored a 4 on the technical component and a 5 on the oral communication component. On the Administration scenario, the appellant scored a 5 on the technical component and a 3 on the oral communication component. Finally, on the Incident Command scenario, the appellant scored a 3 on the technical component and a 4 on the oral communication component.

Appellant's argumentOn appeal, the appellant challenges his scores for the oral communication component of the Administration scenario and the technical component of the Supervision scenario. As a result, the appellant’s test material and a listing of possible courses of action (PCAs) for the scenarios were reviewed.

On the oral communication component of the Administration scenario, the assessor indicated that the appellant displayed a minor weakness in inflection/rate/volume, as evidenced by the use of a monotone inflection. In addition, the assessor stated that the appellant displayed a minor weakness in nonverbal communication by failing to maintain sufficient eye contact, including reading from the booklet during two portions of his presentation. Based upon the foregoing, the assessor awarded the appellant a score of 3 on the oral communication component of the Administration scenario. On appeal, the appellant argues that the reduction in his score for monotone speech was subjective and not a valid criterion for assessing his performance. He emphasizes that he scored a 5 on the technical component and complains that it “feels unfair and unjustified” to rate him at a score of 3 on oral communication because of “intangible and subjective criter[ia] like monotone” speech. Accordingly, the appellant requests reconsideration of his oral communication score on the subject scenario.

In reply, the Commission emphasizes that it is this agency’s longstanding policy that technical and oral communication component scores are independent ratings on the examination and that an exemplary or poor technical rating does not have a bearing on oral communication scoring and vice versa. Furthermore, oral communication performance can clearly distinguish candidates, including those delivering presentations with comparable technical details. To wit, it would be disingenuous to argue that the presentation of one candidate who spoke at a low rate of volume, had their speech punctuated by the frequent use of filler words like “ah” and “um,” rarely made eye contact with their audience and routinely made distracting hand gestures would be as understandable, effective and well-received as the presentation of another candidate who gave a speech with a comparable level of detail, but without these same oral communication issues. For these reasons, mere arguments that because a candidate received a certain technical score on a scenario, they should have received a corresponding oral communication rating for that same scenario are invalid. Beyond this, a review of the appellant’s presentation confirms that he displayed minor weaknesses in inflection/rate/volume and nonverbal communication, as noted by the assessor. Accordingly, the appellant’s score of 3 on the oral communication component of the Supervision scenario is sustained.

The Supervision scenario presents that the candidate is a newly-appointed Battalion Fire Chief and that the department has recently undergone changes in upper management, including the appointment of a new Fire Chief in charge of the department. It further states that the new Fire Chief wants to implement a new and well-defined disciplinary policy. After this progressive discipline policy is implemented, the Fire Chief receives reports that a subordinate Fire Captain under the candidate’s supervision is not enforcing the new policy. Question 1 asks what specific actions the candidate would take to investigate the situation with the Fire Captain. The prompt for Question 2 states that the candidate has learned that the Fire Captain has not been implementing the new policy because she is not sure how to enforce it in certain situations, particularly those she has not dealt with previously. Question 2 then asks what specific topics/actions the candidate should discuss/take in an interview with the Fire Captain based upon the new information.

The SME awarded the appellant a score of 4 on the scenario based upon a finding that the candidate missed a number of PCAs, including, in part, meeting with the Fire Chief at the start of the investigation to get specific information from him in response to Question 1. On appeal, the appellant argues that because there was a well-established chain of command and effective communication channels, it was not necessary to meet with the Fire Chief. He also proffers that because he consulted the legal department regarding the interpretation of the policy and informed the Fire Chief about the disciplinary meetings, he demonstrated his capacity as a leader to make independent and well-informed decisions, and accountability for any actions that follow. Based upon the foregoing, the appellant argues that his score for this component should have been higher.

In reply, at the outset, it is noted that keeping the Fire Chief informed of all findings and actions was a distinct PCA from meeting with the Fire Chief at the start of the investigation and that the appellant did receive credit for keeping the Fire Chief informed of all findings and investigations. Meeting with the Fire Chief to get specific information from him at the start of the investigation is a valid and important step for several critical reasons. The prompt states that the candidate was recently appointed to the title Battalion Fire Chief and that there have been other recent changes in upper management, including the appointment of a new Fire Chief in charge of the department. It further provides that the new Fire Chief wants to implement a new progressive discipline policy. The newness of the candidate to the position, the recency of the Fire Chief’s appointment and the stated intention of the Fire Chief to implement a new policy are all factors that make it logical and imperative to ensure that the candidate and the Fire Chief are on the same page with the expectations of the investigation presented here. Failing to meet prior to beginning the investigation could result in an investigation that is inefficient and/or fails to accomplish the new Fire Chief’s objectives. Beyond this, a review of the appellant’s presentation fails to demonstrate that he otherwise addressed the PCA at issue. For these reasons, the subject PCA was valid and the appellant’s score of 4 on the technical component of the Supervision scenario is sustained.

CONCLUSION

A thorough review of the appellant’s submissions and the test materials indicates that the decision below is amply supported by the record, and the appellant has failed to meet his burden of proof in this matter.

ORDER

Therefore, it is ordered that this appeal be denied.

ClosingThis is the final administrative determination in this matter. Any further review should be pursued in a judicial forum.