Keith Bierwisch, CSC Docket No. 2023-425

BackgroundKeith Bierwisch appeals his score for the oral portion of the promotional examination for second-level Fire Captain (PM2331C), Irvington. It is noted that the appellant failed the subject examination.

It is noted for the record that this two-part examination consisted of a written multiple-choice portion and an oral portion. The test was worth 70 percent of the final score and seniority was worth the remaining 30 percent. The various portions of the test were weighted as follows: written multiple choice portion, 35.26%; technical score for the Evolving Scenario, 20.77%; oral communication score for the Evolving Scenario, 2.79%; technical score for the Administration Scenario, 13.56%; oral communication score for the Administration Scenario, 2.79%; technical score for the Arriving Scenario, 22.04%; and oral communication score for the Arriving Scenario, 2.79%.

The oral portion of the second-level Fire Captain examination consisted of three scenarios: a fire scenario simulation with questions designed to measure knowledge and abilities in assessing risk (Evolving); a simulation designed to measure technical knowledge and abilities in administrative duties (Administration); and a fire scenario simulation designed to measure technical knowledge and abilities in strategy and attack plan and hazmat (Arriving). For the Evolving and Administration Scenarios, candidates were provided with a 25-minute preparation period for both, and candidates had 10 minutes to respond to each. For the Arriving

Scenario, a five-minute preparation period was given and candidates had 10 minutes to respond.

The candidates’ responses were scored on technical knowledge and oral communication ability. Prior to the administration of the examination, a panel of Subject Matter Experts (SMEs) determined the scoring criteria, using generally approved fire command practices, firefighting practices, and reference materials. Scoring decisions were based on SME-approved possible courses of action (PCAs) including those actions that must be taken to resolve the situation as presented. For a performance to be acceptable, other than for oral communication, a candidate needed to present the mandatory courses of action for that scenario. Only those oral responses that depicted relevant behaviors that were observable and could be quantified were assessed in the scoring process. Scores were then converted to standardized scores.

Candidates were rated on a five-point scale, with 5 as the optimal response, 4 as a more than acceptable passing response, 3 as a minimally acceptable passing response, 2 as a less than acceptable response, and 1 as a much less than acceptable response. For each of the scenes, and for oral communication, the requirements for each score were defined.

For the Evolving Scenario, the appellant scored a 3 for the technical component and a 3 for the oral communication component. For the Administration Scenario, the appellant scored a 2 for the technical component and a 5 for the oral communication component. For the Arriving Scenario, the appellant scored a 1 for the technical component and a 4 for the oral communication component. The appellant challenges his scores for the technical components of the Administration and Arriving Scenarios, and for the oral communication components of the Evolving and Arriving Scenarios. As a result, the appellant’s test material, video, and a listing of PCAs for the scenarios were reviewed.

However, first, the appellant appeals his seniority score. In this regard, N.J.A.C. 4A:4-2.15(b) requires that candidates who do not receive a passing score on one part of an examination shall be deemed to have failed the entire examination and those candidates shall not receive credit for seniority. Thus, since the appellant did not pass the examination, his seniority was not credited to his final score. Additionally, the appellant argues that he was not given the calculations to determine his score. However, the appellant’s scores averaged 2 for technical knowledge, and 4 for oral communicative ability. The Candidate Feedback Report, which contained the scoring calculations, stated, “In order to pass the oral assessment exercise, your average raw score on both technical knowledge and oral communicative ability must be 2.5 or higher. In addition, you must receive a passing technical score (3 or higher) on 2 out of the 3 oral assessment scenarios. Final and seniority scores can now be calculated for those who achieved a passing score.” As such, calculations for standardization of the appellant’s individual scores were not completed as the appellant did not receive a score of 2.5 or higher on technical knowledge, and he received below a score of 3 on two technical components.

As to the oral communication components, the appellant does not outline how his presentations do not contain the weaknesses noted by the assessors. Rather, he states that the reviewers “might not be correct,” and he has no ability to validate the scores. Therefore, he maintains that “this should be stricken.”

In reply, for the Evolving Scenario, the assessor noted weaknesses in nonverbal communication, as evidenced by his lack of eye contact; in rate, which was evidenced by speaking in a monotone voice throughout the presentation; and in organization, by demonstrating multiple pauses and going on a tangent about building structure and risks. For the Arriving Scenario, the assessor noted a weakness in grammar, by using filler words such as “um” and “ah,” and speaking in a stilted manner.

It is noted that the orientation guide available to each candidate indicated that oral communication, the ability to communicate clearly and concisely, was a component of this portion of the examination. One factor in oral communication is inflection/moderation/rate/volume. This factor is defined as speaking at an appropriate rate, maintaining appropriate pitch and volume, and properly using pitch to convey meaning or emphasis. Another factor of oral communication is nonverbal communication, which includes using gestures effectively without causing confusion or distractions and making eye contact when speaking. Candidates were permitted to use their notes, and test conditions were standardized in their application to all candidates, i.e., nonverbal communication (including eye contact) was assessed for all candidates. Another factor was organization, defined as presenting ideas in a logical fashion, stating a topic, and providing supporting arguments as well as a conclusion or summary. Lastly, usage/grammar is a factor defined as using appropriate words and using sentences that are grammatically correct. As to word usage/grammar, the assessor notes referred to the appellant’s use of distracting verbal mannerisms such as “um” and “ah.” It was not acceptable to present many distracting verbal mannerisms, such as “um.” This was an examination setting where candidates were given scenarios, and a question or questions for each scenario, and were required to provide direct answers to those questions and, in this setting, candidates were required to maintain the flow of information. They were required to state what they meant, without excessive pausing and at an appropriate rate, and with no distractions.

A review of the Evolving Scenario demonstrates that the appellant’s lack of eye contact was a weakness. The appellant suggests that “[t]he reviewer may not be correct.” He further asserts that it “doesn’t yield to ability to have proper validity.” However, it is observed that while candidates were permitted to use their notes, they were told to make their presentation to the camera. Further, it is noted that test conditions were standardized in their application to all candidates, i.e., nonverbal communication (including eye contact) was assessed for all candidates. Prior to commencing the examination, the room monitor read the same information to every candidate. When giving instructions, the monitor told the appellant to direct his responses to the video camera. Further, making eye contact when speaking is generally recognized as a factor in effective nonverbal communication and oral communication. The appellant did not look towards the camera for approximately 90 seconds at the beginning of his Evolving Scenario presentation. In addition, the appellant looked down to read for roughly one minute and five seconds out of a one minute and nine second interval later in his presentation. Therefore, the record supports the appellant’s score of 3 for the oral communication component of the Evolving Scenario.

Similarly, a review of the oral communication component of the Arriving Scenario demonstrates that the appellant’s use of filler words was a minor weakness. For example, approximately four minutes and 30 seconds into his Arriving Scenario presentation, the appellant stated:

Have one of the company officers as a staging officer. One not only has a primary water supply to the initial attack engine but a secondary, um, and all of this would be transmitted, uh, to establish command of 18 Didi

Drive with, um, my size-up that I would conduct of I’m establishing a command at 18 Didi Drive, um. This is a residential property a wood frame. . .

This was not the only instance where the appellant used such filler words in this scenario. As noted by the assessor, filler words like “um” and “ah” appeared more than 20 times during his 10-minute answer. This minor weakness detracted from the appellant’s performance and clearly supports the assessor’s score of 4.

Appellant's argumentRegarding the technical components, the appellant argues that two PCAs on the Administration Scenario should be stricken and that the assessors failed to properly credit him with identifying two PCAs on the Arriving Scenario.

The Administration Scenario involves a Fire Fighter in a residence speaking to the elderly residents with his mask down, which is a violation of department policy. Question 1 asked for actions to take to fully address the incident. Question 2 indicated that the elderly wife is confirmed to have COVID-19, and the husband has called to complain and says that he is considering legal action. This question asked for additional actions that should now be taken.

The assessor noted that the appellant missed the opportunities to check the Fire Fighter’s personnel file and to monitor the situation going forward, which were responses to question 1, and to offer the Employee Assistance Program (EAP)/Employee Advisory Service (EAS), a response to question 2. On appeal, the appellant states that he has no right to review the Fire Fighter’s personnel file. He argues that he did not need to offer EAP/EAS as the scenario did not indicate any personal issues with this Fire Fighter.

In reply, the appellant’s presentation has been reviewed, and the review does not find that the appellant stated he would check the Fire Fighter’s personnel file or offer EAP/EAS. Rather, the appellant argues that these actions should not be included in scoring. The Commission notes that the PCAs were developed by SMEs who determined that the candidate should check the Fire Fighter’s personnel file and offer EAP/EAS in response to this incident. The Division of Test Development, Analytics and Administration (TDAA) states that the SMEs considered the PCA of checking the Fire Fighter’s personnel file as critical because it would speak to whether this was his first performance issue or repeated performance issues of a similar nature and, in turn, inform how the progressive disciplinary process may unfold. TDAA maintains that even if the appellant’s jurisdiction would not allow him to personally view a subordinate’s personnel file, it does not change the fact that the appellant, as the supervisor in charge of this matter, would have to ensure that the information is reviewed following this incident, so as to ensure that appropriate action is taken in response. As to offering EAP/EAS, TDAA presents that off-duty problems can contribute to performance issues and Health Insurance Portability and Accountability Act of 1996 (HIPAA) guidelines make it inappropriate for someone within the chain of command to directly inquire about such issues. Conversely, EAP/EAS is a resource that is specifically designed to evaluate and respond to outside issues. For these reasons, the SMEs considered offering EAP/EAS to be an appropriate PCA. The Commission finds that the record supports the rationales for both of these responses and it does not find that the appellant’s arguments that he did not have to take these actions to be persuasive. Accordingly, since the appellant missed these PCAs, and other actions as well, his score of 2 for this scenario is correct.

The Arriving Scenario involves a fire in a two-story, multi-family, wood-framed residential property built in 1995. Upon arriving, it is noticed that grey smoke is seeping out from the closed garage door and the second-floor windows, and there is an orange glow of fire seen through the windows. The question asked for initial concerns and specific actions to take to fully address the incident.

The assessor noted that the appellant failed to order a primary search of 18 Didi Drive, to order crews to check for extension, and to establish a Rapid Intervention Crew (RIC), each of which is a mandatory response. The assessor also indicated that the appellant missed the opportunity to request Red Cross assistance for any displaced residents. On appeal, the appellant states that he established a RIC, allocated resources for the VES method (vent, enter, search), and took other actions.

As noted above, the appellant received a score of 1 for the technical component. In response, a review of the appellant’s presentation demonstrates that he did establish a RIC. However, the assessor properly found that he should not be credited with the mandatory response of ordering a primary search of 18 Didi Drive. In this regard, the appellant stated during his presentation that he “would have a truck company, if we were unaccounted for at 18 Didi from the resident, possibly conduct a vent, enter, search from Side C, um, of the residence so that they can isolate themselves with the bedroom door, ah, and then, uh, return back out.” The appellant’s response falls short because he merely suggests use of the VES method as a possibility, rather than definitively ordering it, even though the prompt establishes that the tenant of 14 Didi Drive advised the candidate that he had not seen any of his neighbors this morning or among the crowd present outside of the property.

BackgroundAs set forth above, the appellant was determined to have missed two additional mandatory responses, as well as several additional responses. However, a review of the appellant’s responses indicate that he established a RIC, i.e., provided one missing mandatory response. Accordingly, the appellant’s score for this component should be changed from 1 to 2. Nonetheless, even with this change, the appellant did not achieve a passing score (i.e., 3 or higher) on two out of the three oral assessment scenarios and, therefore, still failed the subject examination.

CONCLUSION

A thorough review of the appellant’s submissions and the test materials indicates that, except for the technical component of the Arriving Scenario, the decision below is amply supported by the record, and the appellant has failed to meet his burden of proof in this matter.

ORDER

Therefore, it is ordered that the appellant’s score for the technical component of the Arriving Scenario be raised from 1 to 2 and that the remainder of his appeal be denied.

ClosingThis is the final administrative determination in this matter. Any further review should be pursued in a judicial forum.