Steven Ali, CSC Docket No. 2023-1955

BackgroundSteven Ali appeals his score on the promotional examination for Battalion Fire Chief (PM3390C), Paterson. It is noted that the appellant failed the subject examination.

The subject promotional examination was held on May 23, 2022, and 45 candidates passed. This two-part examination consisted of an integrated system of simulations designed to generate behavior similar to that required for success on the job. The first part consisted of multiple-choice items that measured specific work components identified and weighted by the job analysis. The second part consisted of three oral scenarios: Supervision, Administration and Incident Command. The examination was based on a comprehensive job analysis conducted by the Civil Service Commission (Commission), which identified the critical areas of the job. The weighting of the test components was derived from the job analysis data. It is noted that candidates were told the following prior to beginning their presentations for each scenario: “In responding to the questions, be as specific as possible. Do not assume or take for granted that general actions will contribute to your score.”

Each candidate in a given jurisdiction was scored by a team of three different Subject Matter Experts (SMEs), who were trained in current technical scoring procedures. Each of these SMEs were current or retired fire officers who held the title of Battalion Fire Chief (or Fire Officer 2) or higher. Candidates were also assessed by three New Jersey Civil Service Commission employees trained in oral communication assessment. As part of the scoring process, an SME observed and noted the responses of a candidate relative to the knowledge, skills and abilities (KSAs) that each exercise was designed to measure. An assessor also noted any weaknesses that detracted from the candidate’s overall oral communication ability. Each assessor then rated the candidate’s performance according to the rating standards and assigned the candidate a technical or oral communication score on that exercise.

In order to preserve the relative weighting of each of the components of the examination, the ratings for each portion were adjusted by a well-recognized statistical process known as “standardization.” Under this process, the ratings are standardized by converting the raw scores to z-scores, an expression of the deviation of the score from the mean score of the group in relation to the standard deviation of scores for the group. Each portion of the examination had a relative weight in its relation to the whole examination. Thus, the z-score for the multiple-choice portion was multiplied by a test weight of 36.53%, the oral technical scores were multiplied by a test weight of 53.91% and the oral communication scores were multiplied by a test weight of 9.56%. The weighted z-scores were summed and this became the overall final test score. This was weighted and added to the weighted seniority score. The result was standardized, then normalized, and rounded up to the third decimal place to arrive at a final average.

For the Supervision scenario, the appellant scored a 4 on the technical component and a 4 on the oral communication component. On the Administration scenario, the appellant scored a 2 on the technical component and a 3 on the oral communication component. Finally, on the Incident Command scenario, the appellant scored a 2 on the technical component and a 3 on the oral communication component.

Appellant's argumentThe appellant challenges his scores for the technical component of the Supervision, Administration and Incident Command scenarios1. As a result, the appellant’s test material and a listing of possible courses of action (PCAs) for the scenarios were reviewed.

The Supervision scenario involves the candidate’s subordinate, a Fire Captain, violating the leave policy by being absent without leave approval from a supervisor. Question 1 asks what specific actions the candidate, as the Battalion Fire Chief supervising the said Fire Captain, should take. Question 2 indicates that while investigating, the candidate discovers that the same Battalion Fire Chief was disciplined for violating the leave policy in the same manner nine months earlier and asks, based on this information, what specific topics should be discussed in a meeting with the Fire Captain and what specific actions should be taken.

1 The appellant also questions why his seniority score was marked as “fail” on the candidate feedback sheet he received at the review session which followed the scoring of the subject examination instead of listing a numerical calculation. As noted in the 2022 2nd Level Fire Supervisor Orientation Guide, only candidates with passing examination scores will have their overall test score weighted 70% and seniority score weighted 30% in determining their final score.

For the Supervision scenario, the SME indicated that the appellant, in response to Question 1, missed a number of PCAs, including the opportunity to get a written report from the outgoing Battalion Fire Chief. Based upon the foregoing, the SME awarded the appellant a score of 4 for the technical component of this scenario. On appeal, the appellant argues that it is not common practice for the incoming Battalion Fire Chief to request a written report from the outgoing Battalion Fire Chief, as it relates to the incoming Battalion Fire Chief’s personnel, but that he covered the necessary action by stating during his presentation that the outgoing Battalion Fire Chief had updated him on the Fire Captain calling out. The appellant argues that because it was up to him to find out what happened, why it happened, how it happened, prevent it from happening again and to report it up the chain of command, he should have been awarded credit for the subject PCA.

In reply, a review of the recording of the appellant’s presentation supports the assessor’s award of a score of 4 for the technical component of the Supervision scenario based upon the appellant’s failure to identify the subject PCA and several other PCAs. Here, the prompt provides that the outgoing Battalion Fire Chief had informed the candidate about the Fire Captain calling out. As such, the appellant saying during his presentation that the outgoing Battalion Fire Chief had updated him on the Fire Captain calling out did not add new information. Further, since candidates were required to investigate an incident that could potentially result in disciplinary action, the Commission finds it reasonable to require candidates to identify the need to get a written report from the outgoing Battalion Fire Chief as a material witness. Regardless, even if the appellant had received credit for the subject PCA, it would not have increased his score for the technical component of the subject scenario. Accordingly, the appellant’s score of 4 for this component is sustained.

The Administration scenario involves the candidate investigating an incident between Fire Fighter Hernandez and a Police Officer which culminated in the arrest of Fire Fighter Hernandez at the scene of a car accident where the candidate was serving as the incident commander.

For the Administration scenario, the SME indicated that the appellant missed a significant number of PCAs, including the opportunities to review the National Fire Incident Response System (NFIRS), to offer the Employee Assistance Program (EAP) and to interview Fire Fighter Hernandez separately. Based upon the foregoing, the SME awarded the appellant a score of 2. On appeal, the appellant concedes that he did not identify any of the foregoing PCAs, but he asserts that his analysis, response and implementation of preventative measures were far more important and critical to the scenario. In this regard, he points to the other actions he identified and argues that his response merits a higher grade than he received.

In reply, because of a review of the recording of the appellant’s Administration scenario supports the assessor’s award of a score of 2 for the technical component and because the appellant has conceded that he did not identify the PCAs at issue, the appellant has not met his burden of proof with respect to this scenario.

The Incident Command scenario involves a response to a fire at a local auto parts store and auto repair shop. Question 1 asks what specific actions the candidate would take upon arriving at the scene. The prompt for Question 2 indicates that while crews are involved in extinguishment operation, an explosion occurs on Side C, emergency radio traffic has been transmitted by a fire fighter and that structural damage is now visible on Side C. Question 2 asks what specific actions the candidate should now take based upon this new information.

For the technical component of the Incident Command scenario, the assessor awarded the appellant a score of 2 based upon his failure to perform a number of mandatory actions, including conducting a primary search and ordering an emergency evacuation, plus his failure to identify a number of additional PCAs, such as the opportunity to consider foam operations. On appeal, the appellant points to a specific portion of his presentation where he maintains that he stated that he would be conducting a search, where the ladder and engine crew would be working in coordination with a 2.5 inch hose line, doing a search for victims who, if found, would be saved and handed off to EMS. He avers that the fact he announced the search at the beginning of the fire implied that the search was a primary search. The appellant further maintains that he addressed the mandatory response of ordering an emergency evacuation in response to Question 2 and called for the apparatus to sound the evacuation air horns. Finally, while the appellant concedes that he did not identify the additional PCA of considering foam operations, he argues that there are multiple ways to handle a fire and that the use of foam in the subject scenario was not critical. He expresses concern that his failure to consider foam operations was a substantial reason for the lowering of his score. He contends that because he successfully controlled the situation “perfectly void of any injuries” he should have received a higher score.

In reply, as noted above, candidates were told the following prior to beginning their presentations for each scenario: “In responding to the questions, be as specific as possible. Do not assume or take for granted that general actions will contribute to your score.” The appellant’s statements were too general to award him credit for the mandatory responses of conducting a primary search and ordering an evacuation. In terms of a search, the appellant indicated that he would, in relevant part, “ventilate from the tower ladder, help to . . . work in coordination with the engine company, looking for victims, we’re going to do some [sic] search. They’re going to be mindful of the secondary means of egress. Be mindful of the flow path, rescuing victims that they come upon.” Since the appellant did not specify that he was searching the structure, he cannot be said to have conveyed that he was conducting a primary search. As to the mandatory response of ordering an emergency evacuation, the Commission observes that the appellant stated, in relevant part, “I’m going to do the evacuation tones. I need everybody to come out. I’m gonna do a PAR on there . . . on here.” N.J.A.C. 5:75-2.7(d) states, in relevant part, that an evacuation order shall only be conducted upon the order of the incident commander or his or her designee. N.J.A.C. 5:75-2.7(e) provides, in relevant part, that “[u]pon being alerted to an emergency evacuation signal, all personnel shall evacuate the hazardous area to an area of obvious safety. As all personnel evacuate, they shall warn others in the event they have not been alerted by the signal.” Based upon the foregoing, it was essential that candidates specifically state that they were ordering an evacuation and not just that they were sounding an evacuation signal. Thus, since the appellant only specified that he was sounding an evacuation signal, he was properly denied credit for the separate, mandatory response of ordering an evacuation in response to Question 2. Finally, it is noted that the PCA of considering foam operations was an additional, rather than mandatory, response to the Incident Command scenario. Even if he had been awarded credit for that additional response, he score would remain unchanged because he failed to identify the mandatory responses noted above. Therefore, the Commission finds that the appellant has failed to sustain his burden of proof.

CONCLUSION

A thorough review of the appellant’s submissions and the test materials indicates that the decision below is amply supported by the record, and the appellant has failed to meet his burden of proof in this matter.

ORDER

Therefore, it is ordered that this appeal be denied.

ClosingThis is the final administrative determination in this matter. Any further review should be pursued in a judicial forum.