Stefan Vassallo, CSC Docket No. 2023-2053

BackgroundStefan Vassallo appeals his score on the promotional examination for Fire Officer 2 (PM4201C), North Hudson Fire and Rescue. It is noted that the appellant passed the examination with a final average of 78.710 and ranks 20th on the eligible list.

The subject promotional examination was held on May 25, 2022, and 20 candidates passed. This two-part examination consisted of an integrated system of simulations designed to generate behavior similar to that required for success on the job. The first part consisted of multiple-choice items that measured specific work components identified and weighted by the job analysis. The second part consisted of three oral scenarios: Supervision, Administration and Incident Command. The examination was based on a comprehensive job analysis conducted by the Civil Service Commission (Commission), which identified the critical areas of the job. The weighting of the test components was derived from the job analysis data. It is noted that candidates were told the following prior to beginning their presentations for each scenario: “In responding to the questions, be as specific as possible. Do not assume or take for granted that general actions will contribute to your score.”

Each candidate in a given jurisdiction was scored by a team of three different Subject Matter Experts (SMEs), who were trained in current technical scoring procedures. Each of these SMEs were current or retired fire officers who held the title of Battalion Fire Chief (or Fire Officer 2) or higher. Candidates were also assessed by three Commission employees trained in oral communication assessment. As part of the scoring process, an SME observed and noted the responses of a candidate relative to the knowledge, skills and abilities (KSAs) that each exercise was designed to measure. An assessor also noted any weaknesses that detracted from the candidate’s overall oral communication ability. Each assessor then rated the candidate’s performance according to the rating standards and assigned the candidate a technical or oral communication score on that exercise.

In order to preserve the relative weighting of each of the components of the examination, the ratings for each portion were adjusted by a well-recognized statistical process known as “standardization.” Under this process, the ratings are standardized by converting the raw scores to z-scores, an expression of the deviation of the score from the mean score of the group in relation to the standard deviation of scores for the group. Each portion of the examination had a relative weight in its relation to the whole examination. Thus, the z-score for the multiple-choice portion was multiplied by a test weight of 36.53%, the oral technical scores were multiplied by a test weight of 53.91% and the oral communication scores were multiplied by a test weight of 9.56%. The weighted z-scores were summed and this became the overall final test score. This was weighted and added to the weighted seniority score. The result was standardized, then normalized, and rounded up to the third decimal place to arrive at a final average.

Each oral examination question, and overall oral communication, was rated on a five-point scale, with 5 as the optimal response, 4 as a more than acceptable passing response, 3 as a minimally acceptable passing response, 2 as a less than acceptable response, and 1 as a much less than acceptable response.

On the Supervision scenario, the appellant scored a 4 on the technical component and a 3 on the oral communication component. On the Administration scenario, the appellant scored a 5 on the technical component and a 4 on the oral communication component. Finally, on the Incident Command scenario, the appellant scored a 2 on the technical component and a 5 on the oral communication component.

Appellant's argumentOn appeal, the appellant challenges his scores for the oral communication component of the Administration scenario and the technical component of the Incident Command scenario. As a result, the appellant’s test materials and a listing of possible courses of action (PCAs) for the scenarios were reviewed.

On the Administration scenario, the assessor found that the appellant displayed a minor weakness in organization by pausing for extended periods of time, including a seven-second pause. Based upon the foregoing, the assessor awarded the appellant a score of 4 for the oral communication component of the Administration scenario. On appeal, the appellant states that his seven-second pause was to review his notes. Beyond this, he proffers that “[p]ausing can have several positive effects,” including increased clarity and focus, reduced stress, improved decision-making, and promoting greater mindfulness and intent in decision-making.

In reply, although the 2022 2nd Level Fire Supervisor Orientation Guide states that “[p]ausing occasionally to review notes is expected and will not be penalized,” that same passage also tells candidates to “eliminate long pauses,” as reviewers can consider such a deficiency in a presentation. Thus, the appellant had reasonable notice that he could be penalized for lengthy pauses, such as the one noted above, and his argument that his score should not have been lowered because of it is without merit. Even assuming, arguendo, that the appellant’s seven-second pause to review his notes was insufficient to support a finding that he displayed a weakness in organization, a review of his presentation demonstrates that the appellant paused at multiple other points in his presentation and that some of these were unrelated to the appellant’s review of his notes. Finally, although the appellant maintains that “[p]ausing can have several positive effects,” the pauses during the appellant’s presentation were weaknesses, not strengths. Accordingly, the appellant’s score of 4 for the oral communication component of the Administration scenario is sustained.

The Incident Command scenario involves a response to a report of fire at a local movie theater, part of which is in the process of a renovation. The prompt indicates that the first arriving engine company has established command and requested a second alarm. Question 1 asks what specific actions the candidate would take upon arriving at the scene. The prompt for Question 2 states that the party wall separating two theaters collapses during firefighting operations, trapping two firefighters. Question 2 then asks what specific actions the candidate should now take based on this new information.

The SME found that the appellant failed to perform the mandatory action of ensuring proper transfer of command in response to Question 1 and ensuring the removal of trapped firefighters in response to Question 2. The SME further indicated that the appellant missed a number of additional PCAs, including the opportunity to activate the rapid intervention crew (RIC). Based upon the foregoing, the SME awarded the appellant a score of 2 for the technical component of the Incident Command scenario. On appeal, the appellant argues that he covered the PCAs at issue by stating that he would establish command, mentioning that he would rescue and remove all firefighters and civilians, referencing the RIC and discussing transferring command at specified points during his presentation.

In reply, a review of the appellant’s presentation fails to demonstrate that the appellant should have been credited with any of the PCAs at issue. In this regard, it is noted that the first arriving engine company has established command, meaning that the appellant would not be establishing command when he arrived on the scene, but rather would be assuming command from the first arriving engine company’s incident commander. Since the appellant was accepting the transfer of command, rather than establishing command, it was necessary to include relevant details, such as indicating that he would have a face-to-face meeting with the incident commander he was relieving or that he would ensure a transfer of information via the radio as part of that process. Since the appellant failed to do so, he was appropriately denied credit for the mandatory response of ensuring proper transfer of command. As to the mandatory response in Question 2 of ensuring the removal of the trapped firefighters, the point of the presentation the appellant cites was during his response to Question 1 and conveyed that he would perform the PCA of conducting a primary search in response to that question. Critically, because the removal of the trapped firefighters was a distinct mandatory response to Question 2, it needed to be specifically identified during the portion of his response covering Question 2. However, a review of the appellant’s response to Question 2 fails to demonstrate that he covered this mandatory response. Accordingly, because the appellant failed to identify the subject mandatory PCAs, his Incident Command technical score of 2 is sustained1.

CONCLUSION

A thorough review of the appellant’s submissions and the test materials indicates that the decision below is amply supported by the record, and the appellant has failed to meet his burden of proof in this matter.

ORDER

Therefore, it is ordered that this appeal be denied.

ClosingThis is the final administrative determination in this matter. Any further review should be pursued in a judicial forum.