Matthew Ferrell, CSC Docket No. 2024-1077

BackgroundMatthew Ferrell appeals his score on the oral portion of the promotional examination for Fire Captain (PM5040D), Willingboro. It is noted that the appellant failed the subject examination.

This two-part examination consisted of a written multiple-choice portion and an oral portion. Candidates were required to pass the written portion of the examination, and then were ranked on their performance on both portions of the examination. The test was worth 70 percent of the final score and seniority was worth the remaining 30 percent. Of the test weights, 35.26% of the score was the written multiple-choice portion, 22.04% was the technical score for the evolving exercise, 2.79% was the oral communication score for the evolving exercise, 13.56% was the technical score for the administration exercise, 2.79% was the oral communication score for the administration exercise, 22.04% was the technical score for the arriving exercise, 2.79% was the oral communication score for the arriving exercise.

The oral portion of the second-level Fire Captain examination consisted of three scenarios: a fire scenario simulation with questions designed to measure knowledge and abilities in assessing risk (Evolving); a simulation designed to measure technical knowledge and abilities in administrative duties (Administration); and a fire scenario simulation designed to measure technical knowledge and abilities in strategy and attack plan and hazmat (Arriving). For the Evolving and Administration scenarios, candidates were provided with a 25-minute preparation period for both, and candidates had 10 minutes to respond to each. For the Arriving scenario, a five-minute preparation period was given and candidates had 10 minutes to respond.

The candidates’ responses were scored on technical knowledge and oral communication ability. Prior to the administration of the exam, a panel of Subject Matter Experts (SMEs) determined the scoring criteria, using generally approved fire command practices, firefighting practices, and reference materials. Scoring decisions were based on SME-approved possible courses of action (PCAs) including those actions that must be taken to resolve the situation as presented. For a performance to be acceptable, other than for oral communication, a candidate needed to present the mandatory courses of action for that scenario. Only those oral responses that depicted relevant behaviors that were observable and could be quantified were assessed in the scoring process. Scores were then converted to standardized scores.

Candidates were rated on a five-point scale, with 5 as the optimal response, 4 as a more than acceptable passing response, 3 as a minimally acceptable passing response, 2 as a less than acceptable response, and 1 as a much less than acceptable response. For each of the scenes, and for oral communication, the requirements for each score were defined. It is noted that candidates were told the following prior to beginning their presentations for each scenario: “In responding to the questions, be as specific as possible. Do not assume or take for granted that general actions will contribute to your score.”

On the Evolving Scenario, the appellant scored a 1 on the technical component and a 5 on the oral communication component. On the Administration Scenario, the appellant scored a 2 on the technical component and a 5 on the oral communication component. On the Arriving Scenario, the appellant scored a 2 on the technical component and a 4 on the oral communication component.

The appellant specifically challenges his scores for the technical components of the Evolving and Administration Scenarios. Further, the appellant contends that he conveyed other important pieces of information during each of the three scenarios, such as monitoring the radio, the placement of apparatus and requests for resources. The appellant also questions how he could have earned failing technical component scores when his oral communication ratings were 5 on the Evolving and Administration Scenarios and 4 on the Arriving Scenario. The appellant also alleges that the use of the z-score dilutes the final average and made it increasingly difficult to pass each scenario.

Initially, it is this agency’s longstanding policy that technical and oral communication component scores are independent ratings on the examination and that an exemplary or poor technical rating does not have a bearing on oral communication scoring and vice versa. Furthermore, oral communication performance can clearly distinguish candidates, including those delivering presentations with comparable technical details. To wit, it would be disingenuous to argue that the presentation of one candidate who spoke at a low rate of volume, had their speech punctuated by the frequent use of filler words like “ah” and “um,” rarely made eye contact with their audience and routinely made distracting hand gestures would be as understandable, effective and well-received as the presentation of another candidate who gave a speech with a comparable level of detail, but without these same oral communication issues. For these reasons, mere arguments that because a candidate received a certain technical score on a scenario, they should have received a corresponding oral communication rating for that same scenario are invalid. For each of the three scenarios, one assessor rated the appellant’s identification of relevant PCAs (i.e., determined his technical component rating) while a second assessor focused strictly on scoring the oral communication dimensions based on specific scoring dimensions and without regard to how many PCAs were successfully identified. Accordingly, the appellant’s oral communication scores do not provide a basis to adjust any of his technical component scores. As to the appellant’s score specifically, the Division of Test Development, Analytics and Administration states that the appellant’s failure was not attributable to its use of z- scores. Rather, the appellant failed the subject examination because he failed each of the three oral scenario technical components and had a technical average of 1.667 which fell below the 2.5 average passing threshold. Based upon the foregoing, the Civil Service Commission (Commission) finds that the appellant has failed to sustain his burden of proof with respect to his challenge to his failing technical component scores based upon his oral communication scoring performance and TDAA’s use of z- scores.

AnalysisThe Commission notes that the appellant has also set forth challenges to specific technical items on the Evolving and Administration Scenarios. The Commission will address these remaining arguments on a scenario-by-scenario basis.

The Evolving Scenario involves the response to a fire reported at a home improvement store where the candidate, a second-level supervisor, will be the incident commander and will establish command on scene. Upon arrival, the candidate is greeted by the store manager, who reports that the fire is on the roof and indicates concern that the solar panels on the roof may be the cause of the fire. Question 1 then asks what the candidate’s actions, orders and requests are to fully address the incident. Question 2 advises that multiple crews are reporting that the roof is failing. It then asks the candidate what actions they should take to handle this new information.

The SMEs awarded the appellant a score of 1 on the technical component of the Evolving Scenario, based upon a finding that the appellant missed a significant number of mandatory and additional responses, including monitoring trusses and ordering crews to use master streams to attack the fire on the roof in response to

Question 1, and ordering all crews to evacuate and conducting a Personnel Accountability Report (PAR) in response to Question 2. On appeal, the appellant avers that he covered monitoring trusses in response to Question 1 by stating at a specified point that he would place a 2.5-inch hoseline to address the issue of the open web truss being exposed to fire. He also cites a statement about the need to evacuate the building that he maintains covered the evacuation order PCA in response to Question 2.

In reply, a review of the appellant’s Evolving Scenario presentation confirms the accuracy of the SME’s scoring determinations. As noted above, candidates were told the following prior to beginning their presentations for each scenario: “In responding to the questions, be as specific as possible. Do not assume or take for granted that general actions will contribute to your score.” In this regard, for the mandatory response of monitoring the trusses, the relevant statement from the appellant during his presentation was that "Engine 4 will stretch a 2.5-inch line toward the rear of the structure, underneath the open web steel trusses for the building." This spoke directly to the distinct PCA of stretching hoseline(s) into the building as a precaution but failed to affirmatively indicate that the appellant would monitor or ensure the monitoring of the trusses. As a result, he was properly denied credit for the subject Question 1 mandatory response. Concerning the Question 2 mandatory response of ordering all crews to evacuate, the Commission notes that the appellant stated that he would “issue an urgent mayday message that the roof is failing, ensure that the evacuation tone is given out over the radio and advise dispatch of the same, that the roof is failing.” The SME credited the appellant with the distinct PCAs of sounding evacuation tones and “other plausible (mayday, urgent message).” However, as noted by Vincent Dunn, Safety and Survival on the Fireground 310 (2nd ed. 2015), “a ‘Mayday’ announcement does not effectively cause [a] firefighter to leave a building. Whether this is because it has become too commonly used or because it has other meanings, it should not be used as an emergency evacuation signal.” Since the appellant issued a mayday message, rather than specifically providing a message calling for an evacuation, he was properly denied credit for the distinct PCA of ordering all crews to evacuate. Accordingly, the appellant has failed to sustain his burden of proof and his Evolving Scenario technical component score of 1 is affirmed.

The Administration Scenario presents that the candidate is a newly-appointed second-level fire supervisor (Captain) of Engine 1, shift B and that the shift’s supervisory crew is rounded out by Lieutenant Jones, who is responsible for the pre- shift equipment and apparatus checks. The scenario further indicates that early in the candidate’s shift, the station is dispatched to a structure fire. Upon arrival, the incident commander (IC) orders Ladder 1 to the roof for vertical ventilation. However, as the crew prepares to carry out the IC’s orders, a member of Ladder 1’s crew advises that the chainsaw is missing from one of the compartments. The crew is then forced to obtain a chainsaw from another truck before the proceeding to the roof. Question

1 then asks what actions the candidate should take to fully address this situation. Question 2 states that after the situation has been addressed, the candidate sees Ladder 1’s missing chainsaw in the rear of one of their crew member’s trucks. The truck with the chainsaw in it belongs to Firefighter Lambing. Question 2 then asks what additional actions the candidate should now take.

The SME awarded the appellant a score of 2 based upon his failure to identify a significant number of PCAs, including the opportunity to request a written statement from Firefighter Lambing, completing a written statement regarding the appellant’s actions while at the incident and reviewing all department logs. On appeal, the appellant presents that he mentioned that the chief should be “kept in the loop” regarding the events presented in the scenario and that a remedy needed to be sought, such as the use of the employee assistance program (EAP). Regarding the review of departmental logs, he proffers that he indicated that he would review any prior incidents involving Firefighter Lambing. The appellant presents that “[a]ccording to the agency he represent[s], [he] feel[s] as though” he complied with agency requirements with his response to this scenario.

In reply, as noted above, candidates were told the following prior to beginning their presentations for each scenario: “In responding to the questions, be as specific as possible. Do not assume or take for granted that general actions will contribute to your score.” It is noted that the appellant was credited with the relevant PCAs covered by his specific statements, including advising his supervisor of the situation, reviewing the personnel/training files of both Lieutenant Jones and Firefighter Lambing and offering EAP. Contrary to the appellant’s assertion, a review of his presentation on appeal does not demonstrate that he covered reviewing other departmental records or identified a number of the other PCAs that TDAA, in consultation with the SMEs, determined were actions that should be taken in response to that action. Thus, while the appellant may “feel” that he did everything he needed to with his response to his scenario, it is evident that he was appropriately given a rating of 2 on the technical component of the Administration Scenario.

CONCLUSION

A thorough review of the appellant’s submissions and the test materials indicates that the decision below is amply supported by the record and the appellant has failed to meet his burden of proof in this matter.

ORDER

Therefore, it is ordered that this appeal be denied.

ClosingThis is the final administrative determination in this matter. Any further review should be pursued in a judicial forum.