Michael SanPhilip, CSC Docket No. 2023-2312

BackgroundMichael SanPhilip appeals his score on the oral portion of the promotional examination for Fire Lieutenant (PM2358C), Belleville. It is noted that the appellant passed the examination with a final average of 82.280 and ranks seventh on the eligible list.

This two-part examination consisted of a written multiple-choice portion and an oral portion. Candidates were required to pass the written portion of the examination, and then were ranked on their performance on both portions of the examination. The test was worth 80 percent of the final score and seniority was worth the remaining 20 percent. Of the test weights, 35.90% of the score was the written multiple-choice portion, 22.04% was the technical score for the evolving exercise, 7.45% was the supervision score for the evolving exercise, 5.71% was the oral communication score for the evolving exercise, 23.20% was the technical score for the arriving exercise, 5.71% was the oral communication score for the arriving exercise.

The oral portion of the Fire Lieutenant examination consisted of two scenarios: a fire scene simulation with questions designed to measure the knowledge of safe rescue tactics and procedures to safeguard citizens, supervision of fire fighters and the ability to assess fire conditions and hazards in an evolving incident on the fireground (Evolving Scenario); and a fire scene simulation designed to measure the knowledge of safe rescue tactics and procedures to safeguard citizens, supervision of firefighters and the ability to plan strategies and tactics based upon a building’s structure and condition (Arriving Scenario). Knowledge of supervision was measured by a question in the Evolving Scenario, and was scored for that scenario. For the Evolving Scenario, candidates were provided with a 15-minute preparation period, and candidates had 10 minutes to respond. For the Arriving Scenario, a five-minute preparation period was given, and candidates had 10 minutes to respond.

The candidates’ responses were scored on technical knowledge and oral communication ability. Prior to the administration of the exam, a panel of Subject Matter Experts (SMEs) determined the scoring criteria, using generally approved fire command practices, firefighting practices, and reference materials. Scoring decisions were based on SME-approved possible courses of action (PCAs) including those actions that must be taken to resolve the situation as presented. Only those oral responses that depicted relevant behaviors that were observable and could be quantified were assessed in the scoring process. It is noted that candidates were told the following prior to beginning their presentations for each scenario: “In responding to the questions, be as specific as possible. Do not assume or take for granted that general actions will contribute to your score.”

Candidates were rated on a five-point scale, with 5 as the optimal response, 4 as a more than acceptable passing response, 3 as a minimally acceptable passing response, 2 as a less than acceptable response, and 1 as a much less than acceptable response. For each of the scenes, and for oral communication, the requirements for each score were defined.

On the Evolving Scenario, the appellant scored a 2 for the technical component, a 5 for the supervision component, and a 3 for the oral communication component. On the Arriving Scenario, the appellant scored a 2 for the technical component and a 4 for the oral communication component.

Appellant's argumentThe appellant challenges his score for the technical component of the Evolving Scenario. As a result, the appellant’s test material, video, and a listing of PCAs for the scenario were reviewed.

The Evolving Scenario involves a fire at a music store. Upon arrival of the candidate’s crew, the incident commander reports that the fire was knocked down and orders the candidate’s crew to begin salvage and overhaul in the music store as other crews conduct secondary searches. Question 1 then asks what the candidate’s initial actions will be and to describe, in detail, how they and their crew will conduct salvage and overhaul operations at this incident. Question 1 further directs candidates to include descriptions of techniques, firefighter safety concerns, and any coordination with other fire personnel. The prompt for Question 2 states that when conducting overhaul operations in the music store, the candidate and their crew discover a severely compromised structural member with the potential for collapse. Question 2 then asks what actions the candidate should take now.

The SME awarded a candidate a score of 2 on the basis that he failed to identify the mandatory response of notifying the incident commander/supervisor/safety officer through an urgent message in response to Question 2 and missed a number of additional PCAs, including, in part, providing additional ventilation and checking carbon monoxide levels.

On appeal, the appellant argues that he should have been credited with the additional PCA of checking carbon monoxide levels based upon his statement at a specified point that he would “monitor the air quality.” In support, the appellant cites several textbook sources.

In reply, upon review of the appellant’s appeal, the Division of Test Development, Analytics and Administration (TDAA) advises that based upon the statement cited by the appellant and the statements he made surrounding it during his presentation, the appellant should have been credited with the PCA at issue. However, TDAA states that even with the foregoing additional credit, his score of 2 would remain unchanged. The Commission agrees with TDAA’s assessment.

CONCLUSION

A thorough review of the appellant’s submissions and the test materials indicates that the decision below is amply supported by the record and, except as noted above, the appellant has failed to meet his burden of proof in this matter.

ORDER

Therefore, it is ordered that this appeal be granted in part and that appropriate agency records be revised to reflect the above-noted credit changes for the technical component of the Evolving Scenario, but that the appellant’s overall score for this component remain unchanged at 2.

ClosingThis is the final administrative determination in this matter. Any further review should be pursued in a judicial forum.