Albert Salina, CSC Docket No. 2023-2051

BackgroundAlbert Salina, represented by Joseph A. Burke, Esq., appeals his score on the promotional examination for Battalion Fire Chief (PM3390C), Paterson. It is noted that the appellant passed the examination with a final average of 85.170 and ranks 15th on the eligible list.

The subject promotional examination was held on May 23, 2022, and 45 candidates passed. This two-part examination consisted of an integrated system of simulations designed to generate behavior similar to that required for success on the job. The first part consisted of multiple-choice items that measured specific work components identified and weighted by the job analysis. The second part consisted of three oral scenarios: Supervision, Administration and Incident Command. The examination was based on a comprehensive job analysis conducted by the Civil Service Commission (Commission), which identified the critical areas of the job. The weighting of the test components was derived from the job analysis data. It is noted that candidates were told the following prior to beginning their presentations for each scenario: “In responding to the questions, be as specific as possible. Do not assume or take for granted that general actions will contribute to your score.”

Each candidate in a given jurisdiction was scored by a team of three different Subject Matter Experts (SMEs), who were trained in current technical scoring procedures. Each of these SMEs were current or retired fire officers who held the title of Battalion Fire Chief (or Fire Officer 2) or higher. Candidates were also assessed by three New Jersey Civil Service Commission employees trained in oral communication assessment. As part of the scoring process, an SME observed and noted the responses of a candidate relative to the knowledge, skills and abilities (KSAs) that each exercise was designed to measure. An assessor also noted any weaknesses that detracted from the candidate’s overall oral communication ability. Each assessor then rated the candidate’s performance according to the rating standards and assigned the candidate a technical or oral communication score on that exercise.

In order to preserve the relative weighting of each of the components of the examination, the ratings for each portion were adjusted by a well-recognized statistical process known as “standardization.” Under this process, the ratings are standardized by converting the raw scores to z-scores, an expression of the deviation of the score from the mean score of the group in relation to the standard deviation of scores for the group. Each portion of the examination had a relative weight in its relation to the whole examination. Thus, the z-score for the multiple-choice portion was multiplied by a test weight of 36.53%, the oral technical scores were multiplied by a test weight of 53.91% and the oral communication scores were multiplied by a test weight of 9.56%. The weighted z-scores were summed and this became the overall final test score. This was weighted and added to the weighted seniority score. The result was standardized, then normalized, and rounded up to the third decimal place to arrive at a final average.

For the Supervision scenario, the appellant scored a 5 on the technical component and a 4 on the oral communication component. On the Administration scenario, the appellant scored a 5 on the technical component and a 3 on the oral communication component. Finally, on the Incident Command scenario, the appellant scored a 3 on the technical component and a 5 on the oral communication component.

Appellant's argumentThe appellant challenges his scores for the oral communication component of the Supervision scenario and the technical component of the Incident Command scenario. As a result, the appellant’s test material and a listing of possible courses of action (PCAs) for the scenarios were reviewed.

For the oral communication component of the Supervision scenario, the assessor found that the appellant displayed a minor weakness in organization, as evidenced by two pauses during his presentation. Accordingly, the assessor awarded the appellant a score of 4 for the oral communication component of this scenario. On appeal, the appellant argues that based upon statement in the 2022 2nd Level Fire Supervision Orientation Guide that “[p]ausing occasionally to review notes is expected and will not be penalized,” he should not have been penalized by the assessor1.

In response, a review of the appellant’s presentation supports the assessor’s conclusion that the appellant displayed a minor weakness in organization, as evidenced by two pauses during his presentation. The Commission further notes that these were not the only lengthy pauses. Specifically, approximately 5 minutes and 25 seconds into his presentation for the Supervision scenario, the appellant paused for 24 seconds to review his notes. In the two minutes that followed this initial lengthy pause, the appellant had two additional lengthy pauses of 25 seconds and 23 seconds, respectively. It is noted that while the 2022 2nd Level Fire Supervisor Orientation Guide states that “[p]ausing occasionally to review notes is expected and will not be penalized,” that same passage also tells candidates to “eliminate long pauses,” as reviewers can consider such a deficiency in a presentation. Thus, the appellant had reasonable notice that he could be penalized for such lengthy pauses and his arguments that his score should not have been lowered based upon these pauses is without merit. Accordingly, the appellant’s score of 4 for the oral communication component of the Supervision scenario is sustained.

The Incident Command scenario involves a response to a fire at a local auto parts store and auto repair shop. Question 1 asks what specific actions the candidate would take upon arriving at the scene. The prompt for Question 2 indicates that while crews are involved in extinguishment operations, an explosion occurs on Side C, emergency radio traffic has been transmitted by a fire fighter and structural damage is now visible on Side C. Question 2 asks what specific actions the candidate should now take based upon this new information.

For the technical component of the Incident Command scenario, the assessor awarded the appellant a score of 3 using the “flex rule.” The appellant’s score was based upon his failure to identify the mandatory response of ordering an emergency evacuation in addressing Question 2. On appeal, the appellant argues that he adequately covered the subject PCA at a specific point in his presentation and provides a list of the actions he took, including activating emergency evacuation tones and air horns. In support, the appellant cites New Jersey Division of Fire Safety and Kean University, New Jersey Fire Fighter Skills Addendum (4th ed. 2021); New Jersey Division of Fire Safety, Model Fire Department Incident Management Standard Operating Guides - Booklet 9 (2011); John Norman, Fire Officer’s Handbook of Tactics (5th ed. 2019); and Vincent Dunn, Collapse of Burning Buildings: A Guide to Fireground Safety (2nd ed. 2010). The appellant also cites In the Matter of Daniel Dornacker, Jr. (CSC, decided October 19, 2016) and In the Matter of Collin Caesar (CSC, decided October 19, 2016) to support his claim that he should have been credited with ordering an evacuation based upon his statements that he would sound evacuation tones.

Background1 It is noted that the only recording of the appellant on file is an audio recording. The appellant alleges that a video would have demonstrated that he was looking at his notes during the portions of his presentation where he paused. The Commission accepts the appellant’s contention that he was reviewing his notes at these points.

In reply, generally, candidates must identify all mandatory responses to receive, at minimum, a score of 3. However, a score of 3 may also be achieved via the “flex rule,” where a candidate provides many additional responses, but does not give a mandatory response. However, a score higher than a 3 cannot be provided utilizing the flex rule. Here, a review of the appellant’s presentation demonstrates that he, in relevant part, called for an “orderly withdrawal, everybody out” and that he sounded evacuation tones. As discussed in In the Matter of Daniel Dornacker, Jr., supra, “[o]rdering an ‘orderly withdrawal’ does not have the same sense of urgency of an evacuation, and is not the same.” As such, his reference to an “orderly withdrawal” was insufficient to cover the subject mandatory response. As to sounding evacuation tones, N.J.A.C. 5:75-2.7(d) provides, in pertinent part, that an evacuation order shall only be conducted upon the order of the incident commander or his or her designee. As to the appellant’s arguments that sounding evacuation tones was sufficient to cover the subject mandatory response based on In the Matter of Collin Caesar, supra, the Commission notes that “order evacuation of building/sound evacuation tones” was a singular PCA on the PM1194S examination at issue in that appeal. Ibid. (“the assessors were saying that the candidate failed to address the mandatory PCA of ordering the evacuation of the building or sound evacuation tones in question 2.”) Further, an evacuation signal is only a portion of the evacuation protocol. N.J.A.C. 5:75-2.7(c) states that “[w]henever an emergency evacuation signal is being sounded, there shall also be a radio message transmitted either from the incident scene or from the designated fire department dispatch center announcing the evacuation order. To the extent possible, the radio message should be coordinated with the sounding of the evacuation signal to insure the radio messages are heard.” Thus, because the incident commander or designee ordering an evacuation is a prerequisite to sounding an evacuation signal and because an evacuation signal is only one part of the protocol for ordering an evacuation, the appellant’s statement that he would activate emergency tones and air horns was insufficient to award him credit for the mandatory response of ordering an evacuation. Accordingly, these statements and the other actions cited by the appellant in response to Question 2 were insufficient to award him credit for the mandatory response of ordering an evacuation. Therefore, the appellant’s score of 3, pursuant to the flex rule, for the technical component of the Incident Command scenario is appropriatr.

CONCLUSION

A thorough review of the appellant’s submissions and the test materials indicates that the decision below is amply supported by the record, and the appellant has failed to meet his burden of proof in this matter.

ORDER

Therefore, it is ordered that this appeal be denied.

ClosingThis is the final administrative determination in this matter. Any further review should be pursued in a judicial forum.