Michael Guarino, CSC Docket No. 2023-1951

BackgroundMichael Guarino appeals his score on the promotional examination for Battalion Fire Chief (PM3380C), Elizabeth. It is noted that the appellant passed the examination with a final average of 76.000 and ranks 16th on the eligible list.

The subject promotional examination was held on May 19, 2022, and 16 candidates passed. This two-part examination consisted of an integrated system of simulations designed to generate behavior similar to that required for success on the job. The first part consisted of multiple-choice items that measured specific work components identified and weighted by the job analysis. The second part consisted of three oral scenarios: Supervision, Administration and Incident Command. The examination was based on a comprehensive job analysis conducted by the Civil Service Commission, which identified the critical areas of the job. The weighting of the test components was derived from the job analysis data. It is noted that candidates were told the following prior to beginning their presentations for each scenario: “In responding to the questions, be as specific as possible. Do not assume or take for granted that general actions will contribute to your score.”

Each candidate in a given jurisdiction was scored by a team of three different Subject Matter Experts (SMEs), who were trained in current technical scoring procedures. Each of these SMEs were current or retired fire officers who held the title of Battalion Fire Chief (or Fire Officer 2) or higher. Candidates were also assessed by three New Jersey Civil Service Commission employees trained in oral communication assessment. As part of the scoring process, an SME observed and noted the responses of a candidate relative to the knowledge, skills and abilities (KSAs) that each exercise was designed to measure. An assessor also noted any weaknesses that detracted from the candidate’s overall oral communication ability. Each assessor then rated the candidate’s performance according to the rating standards and assigned the candidate a technical or oral communication score on that exercise. Scoring decisions were based on SME-approved possible courses of action (PCAs) including those actions that must be taken to resolve the situation as presented. For a performance to be acceptable in the technical component for some scenarios, a candidate needed to present the mandatory courses of action for that scenario. Only those oral responses that depicted relevant behaviors that were observable and could be quantified were assessed in the scoring process.

In order to preserve the relative weighting of each of the components of the examination, the ratings for each portion were adjusted by a well-recognized statistical process known as “standardization.” Under this process, the ratings are standardized by converting the raw scores to z-scores, an expression of the deviation of the score from the mean score of the group in relation to the standard deviation of scores for the group. Each portion of the examination had a relative weight in its relation to the whole examination. Thus, the z-score for the multiple-choice portion was multiplied by a test weight of 36.53%, the oral technical scores were multiplied by a test weight of 53.91% and the oral communication scores were multiplied by a test weight of 9.56%. The weighted z-scores were summed and this became the overall final test score. This was weighted and added to the weighted seniority score. The result was standardized, then normalized, and rounded up to the third decimal place to arrive at a final average.

For the Supervision scenario, the appellant scored a 4 on the technical component and a 4 on the oral communication component. On the Administration scenario, the appellant scored a 4 on the technical component and a 3 on the oral communication component. Finally, on the Incident Command scenario, the appellant scored a 2 on the technical component and a 4 on the oral communication component.

Appellant's argumentThe appellant challenges his scores for the technical components of the Administration and Incident Command scenarios.

CONCLUSION

The Administration scenario involves the candidate being tasked with investigating an incident and revising the department’s current pre-incident action plan procedures following a call where a pre-action plan failed to reflect a building’s conversion and the addition of partition walls inside of the structure. Question 1 asks what specific steps the candidate would take to investigate the incident and the lack of updated pre-incident action plan procedures. Question 2 asks what should be included in an updated pre-incident action plan standard operating guidelines/procedures (SOGs/SOPs).

On the Administration scenario, the assessor stated that the appellant missed the opportunity to note the frequency of inspections under the new pre-incident planning program. Accordingly, the assessor awarded the appellant a score of 4. On appeal, the appellant argues that he covered this action at several points in his response by stating he would follow-up, return to the building and revisit the location.

In response, as noted above, candidates were told the following prior to beginning their presentations for each scenario: “In responding to the questions, be as specific as possible. Do not assume or take for granted that general actions will contribute to your score.” The actions cited by the appellant are too general to have conveyed that he would have, in response to Question 2, included the frequency of inspections in pre-incident action plan SOGs/SOPs.

The Incident Command scenario involves the response to a report of smoke at a local food mart that is part of a strip mall. Question 1 asks candidates what specific actions they would take upon arriving at the scene. Question 2 states that during firefighting operations, part of the roof over the fire collapses, trapping an interior crew, and the crew transmits a MAYDAY. Question 2 then asks what specific actions the candidate would take based on this new information.

On the Incident Command scenario, the assessor awarded the appellant a score of 2 based upon his failure to identify the mandatory response of acknowledging the MAYDAY in response to Question 2 and the appellant missing a number of additional opportunities in response to Questions 1 and 2. On appeal, the appellant argues that he covered this action by stating that he would ask for LUNAR and subsequently acknowledging the MAYDAY via LUNAR.

In reply, at the outset, it is noted that the assessor credited the appellant with the additional response of requesting LUNAR, but not the mandatory action of acknowledging the MAYDAY. Acknowledging the MAYDAY is a critical step because firefighters are instructed to listen for an acknowledgment from incident command before giving further information after a MAYDAY, such as the information covered by a LUNAR. A review of the recording does not indicate that the appellant separately acknowledged the MAYDAY in response to Question 2 on the subject scenario. As such, the assessor’s award of a score of 2 for the technical component of this scenario was correct based upon the appellant’s failure to identify the subject mandatory response and a number of additional PCAs.

Accordingly, a thorough review of the appellant’s submissions and the test materials indicates that the decision below is amply supported by the record, and the appellant has failed to meet his burden of proof in this matter.

ORDER

Therefore, it is ordered that this appeal be denied.

ClosingThis is the final administrative determination in this matter. Any further review should be pursued in a judicial forum.