Matthew Weston, CSC Docket No. 2023-2270

BackgroundMatthew Weston appeals his score on the oral portion of the promotional examination for Fire Lieutenant (PM2374C), Mount Laurel. It is noted that the appellant failed the subject examination.

This two-part examination consisted of a written multiple-choice portion and an oral portion. Candidates were required to pass the written portion of the examination, and then were ranked on their performance on both portions of the examination. The test was worth 80 percent of the final score and seniority was worth the remaining 20 percent. Of the test weights, 35.90% of the score was the written multiple-choice portion, 22.04% was the technical score for the evolving exercise, 7.45% was the supervision score for the evolving exercise, 5.71% was the oral communication score for the evolving exercise, 23.20% was the technical score for the arriving exercise, 5.71% was the oral communication score for the arriving exercise.

The oral portion of the Fire Lieutenant examination consisted of two scenarios: a fire scene simulation with questions designed to measure the knowledge of safe rescue tactics and procedures to safeguard citizens, supervision of fire fighters and the ability to assess fire conditions and hazards in an evolving incident on the fireground (Evolving Scenario); and a fire scene simulation designed to measure the knowledge of safe rescue tactics and procedures to safeguard citizens, supervision of firefighters and the ability to plan strategies and tactics based upon a building’s structure and condition (Arriving Scenario). Knowledge of supervision was measured by a question in the Evolving Scenario, and was scored for that scenario. For the Evolving Scenario, candidates were provided with a 15-minute preparation period, and candidates had 10 minutes to respond. For the Arriving Scenario, a five-minute preparation period was given, and candidates had 10 minutes to respond.

The candidates’ responses were scored on technical knowledge and oral communication ability. Prior to the administration of the exam, a panel of Subject Matter Experts (SMEs) determined the scoring criteria, using generally approved fire command practices, firefighting practices, and reference materials. Scoring decisions were based on SME-approved possible courses of action (PCAs) including those actions that must be taken to resolve the situation as presented. Only those oral responses that depicted relevant behaviors that were observable and could be quantified were assessed in the scoring process.

Candidates were rated on a five-point scale, with 5 as the optimal response, 4 as a more than acceptable passing response, 3 as a minimally acceptable passing response, 2 as a less than acceptable response, and 1 as a much less than acceptable response. For each of the scenes, and for oral communication, the requirements for each score were defined.

For the Evolving Scenario, the appellant scored a 2 for the technical component, a 1 for the supervision component, and a 3 for the oral communication component. For the Arriving Scenario, the appellant scored a 3 for the technical component and a 5 for the oral communication component.

Appellant's argumentThe appellant challenges his score for the technical and supervision components of the Evolving Scenario and for the technical component of the Arriving Scenario.1 As a result, the appellant’s test material, video, and a listing of PCAs for the scenarios were reviewed.

The technical component of the Evolving Scenario provides that the candidate is the First-Level Fire Supervisor of the first responding ladder company dispatched to a report of a two-car motor vehicle accident. Question 1 asks the candidate what specific actions they will take to address this incident. The prompt for Question 2 states that a small pickup truck comes flying down the shoulder on one side of the street where the accident occurred and slams into the back of the parked Engine 3. It further indicates that the front end of the pickup truck is smashed and that Engine 3 skids forward a few inches. Question 2 asks what actions the candidate would take to address this development.

1 The appellant alleges that there was a “radical” amount of human error in the scoring of his examination, such that it does not “meet the State of New Jersey’s standards.” As a result, he requests “corrective actions,” namely a revision of his examination score. As detailed below, the Commission finds these claims are without merit.

On the technical component of the Evolving Scenario, the SME awarded the appellant a score of 2 based on findings that the appellant failed to identify the mandatory response of performing a 360-degree size-up in response to Question 1 and missed a number of additional opportunities in response to both questions, including the opportunity to appoint a safety officer and the opportunity to de-energize both vehicles. On appeal, the appellant argues that because he conveyed the necessary “color of the scene” in his arrival report, he should have been awarded a passing score for the subject scenario, even though he did not specifically perform a 360-degree size- up. The appellant maintains that he only missed two additional responses and that because of this, the SME should have awarded him a passing score of 3 pursuant to the “flex rule.”2

In response, the appellant’s argument that he only missed two additional responses and should have been awarded a passing score of 3 pursuant to the flex rule is misplaced. In order to protect the security of the examination, candidates are not necessarily given an exhaustive list of every PCA they missed in their responses. Here, the two additional PCAs noted by the appellant represent only two examples of the additional numerous PCAs the appellant missed when giving his response to the Evolving Scenario. Although a review of the appellant’s presentation on appeal demonstrates that he should have received credit for the additional PCA of calling for an additional alarm in response to Question 2, the foregoing additional credit does not alter the appellant’s score of 2 on the technical component of the Evolving Scenario.

The supervision component of the Evolving Scenario involves a Fire Fighter beginning to argue with the pickup truck driver who crashed into Engine 3 and the two subsequently beginning to push and shove one another. It then asks what actions the candidate should take to handle the argument and the Fire Fighter both on scene and back at the station. The assessor found that the appellant failed to identify a significant number of PCAs, including, in relevant part, opportunities to speak to the driver and ensure he was alright; review the Fire Fighter’s files; ensure the Fire Fighter receives a reprimand; and documenting all findings and actions. Based upon the foregoing, the SME awarded the appellant a score of 1. On appeal, the appellant argues that he covered each of the PCAs at issue at specified points during his presentation.

In reply, upon review of the appellant’s appeal and his presentation, the Division of Test Development, Analytics and Administration (TDAA) has determined that the appellant should have been credited with the PCAs of ordering the Fire Fighter to walk away/separate the Fire Fighter and the driver; documenting all

2 Generally, candidates must identify all mandatory responses to receive, at minimum, a score of 3.

However, a score of 3 may also be achieved via the “flex rule,” where a candidate provides many additional responses, but does not give a mandatory response. However, a score higher than a 3 cannot be provided utilizing the flex rule.

findings and actions; and informing the Fire Fighter of his right to union representation. Based upon the foregoing, TDAA presents that the appellant’s score for the supervision component of the Evolving Scenario should be raised from 1 to 3. The Commission agrees with TDAA’s findings regarding the technical component of the Evolving Scenario.

The Arriving Scenario involves the response to a fire at a building on a farm which has steel truss construction with corrugated steel walls and a roof with steel I- beams. The building houses farm equipment, bales of hay and straw, diesel fuel, solvents, oils, and lubricants. Question 1 directs candidates to perform their initial reports to the camera as they would upon arrival at this incident. Question 2 asks, after the candidate’s initial report, what specific actions they should take to fully address the incident. The SME indicted that the appellant missed a number of opportunities, including, in part, opportunities to establish a rapid intervention crew (RIC) and to request a rehabilitation unit. Based upon the foregoing, the SME awarded the appellant a score of 3. On appeal, the appellant argues that he performed the aforementioned actions at specified points. In particular, he presents that by requesting four alarms, he went beyond requesting a RIC and effectively requested four of them to make sure that all operating companies were safe.

In reply, upon review of the appellant’s appeal and his presentation, TDAA has determined that the appellant should have been credited with the PCAs of establishing a rapid intervention crew (RIC), requesting hazmat, requesting a utility company and requesting a rehab unit. However, TDAA has also determined that the appellant should not have been credited with the PCA of reporting the steel truss structure to dispatch in response to Question 1 because he only made a general reference to Type II construction, rather than steel trusses specifically. Based upon the foregoing, TDAA maintains that the appellant’s Arriving Scenario technical component score of 3 should remain unchanged. The Commission agrees with TDAA’s assessment.

Finally, TDAA advises that with the scoring change on the supervision component of the Evolving Scenario, the appellant would achieve a passing score on the subject examination. The Commission agrees with TDAA’s assessment regarding this scoring change.

ORDER

Therefore, it is ordered that this appeal be granted in part and that the appellant’s score for the supervision component of the Evolving Scenario be raised from 1 to 3. Additionally, it is ordered that appropriate agency records be revised to reflect the above-noted technical component credit changes for both the Evolving Scenario and the Arriving Scenario, but that the appellant’s overall scores for these components remain unchanged at 2 and 3, respectively. It is further ordered that, since the appellant passed the subject examination based upon the Evolving Scenario supervision component scoring change, that the appellant’s name be added to the Fire Lieutenant (PM2374C), Mount Laurel eligible list with retroactive effect.

ClosingThis is the final administrative determination in this matter. Any further review should be pursued in a judicial forum.