Patrick Calandrillo, CSC Docket No. 2023-2399

BackgroundPatrick Calandrillo appeals his score on the oral portion of the promotional examination for Fire Officer 1 (PM2390C), North Hudson Fire and Rescue. It is noted that the appellant passed the subject examination with a score of 84.010 and ranks 48th on the subject eligible list.

This two-part examination consisted of a written multiple-choice portion and an oral portion. Candidates were required to pass the written portion of the examination, and then were ranked on their performance on both portions of the examination. The test was worth 80 percent of the final score and seniority was worth the remaining 20 percent. Of the test weights, 35.90% of the score was the written multiple-choice portion, 22.04% was the technical score for the evolving exercise, 7.45% was the supervision score for the evolving exercise, 5.71% was the oral communication score for the evolving exercise, 23.20% was the technical score for the arriving exercise, 5.71% was the oral communication score for the arriving exercise.

The oral portion of the Fire Officer 1 examination consisted of two scenarios: a fire scene simulation with questions designed to measure the knowledge of safe rescue tactics and procedures to safeguard citizens, supervision of fire fighters and the ability to assess fire conditions and hazards in an evolving incident on the fireground (Evolving Scenario); and a fire scene simulation designed to measure the knowledge of safe rescue tactics and procedures to safeguard citizens, supervision of firefighters and the ability to plan strategies and tactics based upon a building’s structure and condition (Arriving Scenario). Knowledge of supervision was measured by a question in the Evolving Scenario, and was scored for that scenario. For the Evolving Scenario, candidates were provided with a 15-minute preparation period, and candidates had 10 minutes to respond. For the Arriving Scenario, a five-minute preparation period was given, and candidates had 10 minutes to respond.

The candidates’ responses were scored on technical knowledge and oral communication ability. Prior to the administration of the exam, a panel of Subject Matter Experts (SMEs) determined the scoring criteria, using generally approved fire command practices, firefighting practices, and reference materials. Scoring decisions were based on SME-approved possible courses of action (PCAs) including those actions that must be taken to resolve the situation as presented. Only those oral responses that depicted relevant behaviors that were observable and could be quantified were assessed in the scoring process. It is noted that candidates were told the following prior to beginning their presentations for each scenario: “In responding to the questions, be as specific as possible. Do not assume or take for granted that general actions will contribute to your score.”

Candidates were rated on a five-point scale, with 5 as the optimal response, 4 as a more than acceptable passing response, 3 as a minimally acceptable passing response, 2 as a less than acceptable response, and 1 as a much less than acceptable response. For each of the scenes, and for oral communication, the requirements for each score were defined.

On the Evolving Scenario, the appellant scored a 3 for the technical component, a 4 for the supervision component, and a 3 for the oral communication component. On the Arriving Scenario, the appellant scored a 5 for the technical component and a 5 for the oral communication component.

Appellant's argumentThe appellant challenges his scores for the oral communication and supervision components of the Evolving Scenario. As a result, the appellant’s test material, video, and a listing of PCAs for the scenario were reviewed.

On the oral communication component of the Evolving Scenario, the assessor found that the appellant displayed a major weakness in organization, as evidenced by 10 pauses during his response. Based upon the foregoing, the assessor awarded the appellant a score of 3 for the oral communication component of the Evolving Scenario. On appeal, the appellant argues that most of the pauses in question were “extremely brief” to quickly glance at his notes. He contends that because he gave a thorough, detailed answer and avoided filler words, it was unreasonable and unwarranted to deduct points from his score because of his brief pauses to review notes.

In reply, a review of the appellant’s Evolving Scenario presentation confirms that the assessor properly determined that the appellant displayed a major weakness in organization. As noted by the assessor, the appellant repeatedly paused mid- sentence, creating a stilted delivery that noticeably detracted from the effectiveness of his oral communication. Further, contrary to the appellant’s claims, a number of his pauses to review his notes were lengthy and undercut the flow of his presentation. Therefore, it was appropriate to give him a score of 3, corresponding to a minimally acceptable passing response, rather than a higher rating. Accordingly, the appellant has failed to meet his burden of proof and his score for the oral communication component of the Evolving Scenario is affirmed.

The prompt for the supervision component of the Evolving Scenario states that upon arrival, the candidate discovers that their supply hose was improperly packed after a prior incident earlier that shift, causing a delay in current operations. It then asks the candidate what specific actions they should take on scene and back at the firehouse.

The assessor awarded the appellant a score of 4 based upon a finding that the appellant failed to identify multiple PCAs, including, in part, the opportunity to inspect the rest of the hoses. On appeal, the appellant avers that it was understood that by discussing the disheveled nature of the hoseline and the way it was packed with the chauffer, having it repacked for functional operation purposes and providing retraining on properly packing hoselines, he made it clear that all hoselines on the apparatus would need to be inspected. The appellant further contends that the concise and detailed explanation of each step he would take showed that he had a thorough understanding of how to conduct himself as a first-level fire supervisor.

In reply, as noted above, candidates were told the following prior to beginning their presentations for each scenario: “In responding to the questions, be as specific as possible. Do not assume or take for granted that general actions will contribute to your score.” Here, the appellant’s argument amounts to a request to accept that his stated actions implied that he would inspect the other hoselines. Since such a request would run counter to the clear examination instructions regarding the importance of specificity, it must necessarily fail. Additionally, upon review of the appellant’s presentation on appeal, the Division of Test Development, Analytics and Administration (TDAA) has found that the appellant was erroneously awarded credit for the PCAs of interviewing the firefighters, checking the firefighters’ records, and monitoring the firefighters’ progress. Specifically, because the appellant indicated that he would only interview, check the records of and monitor the progress of the chauffeur, rather than his entire crew, as required under the scoring standard, TDAA submits that his credit for these items should be stricken. However, TDAA also presents that the appellant should have received credit for the PCA of providing any necessary re-training. Based upon these changes, TDAA advises that the appellant’s supervision component score should be lowered from 4 to 3. The Commission agrees with TDAA’s assessment.

ORDER

Therefore, it is ordered that this appeal be denied and that the appellant’s score on the supervision component of the Evolving Scenario for the subject examination be lowered from 4 to 3 with retroactive effect.

ClosingThis is the final administrative determination in this matter. Any further review should be pursued in a judicial forum.