Richard Gorman, CSC Docket No. 2023-1965

BackgroundRichard Gorman appeals his score on the promotional examination for Fire Officer 3 (PM5162C), Jersey City. It is noted that the appellant passed the examination with a final average of 84.550 and ranks 12th on the eligible list.

The subject promotional examination was held on April 23, 2022, and 13 candidates passed. This was an oral examination designed to generate behaviors similar to those required for success in a job. The examination consisted of four scenario-based oral exercises. Each exercise was developed to simulate tasks and assess the knowledge, skills and abilities (KSAs) important to job performance. These exercises covered four topic areas: 1) Incident Command: Non-Fire Incident, 2) Supervision, 3) Administration, and 4) Incident Command: Fire Incident. The test was worth 70 percent of the final score and seniority was worth the remaining 30 percent. The candidates’ responses were scored on technical knowledge and oral communication ability. The various portions of the test were weighted as follows: technical score for the Incident Command: Non-Fire scenario, 24.42%; oral communication score for the Incident Command: Non-Fire scenario, 3.155%; technical score for the Supervision scenario, 14.17%; oral communication score for the Supervision scenario, 3.155%; technical score for the Administration scenario, 11.81%; oral communication score for the Administration scenario, 3.155%; technical score for the Incident Command: Fire scenario, 36.98%; and oral communication score for the Incident Command: Fire scenario, 3.155%.

Prior to the administration of the exam, a panel of Subject Matter Experts (SMEs) determined the scoring criteria, using generally approved fire command practices, fire fighting practices, and reference materials. Scoring decisions were based on SME-approved possible courses of action (PCAs) including those actions that must be taken to resolve the situation as presented. For a performance to be acceptable in the technical component for some scenarios, a candidate needed to present the mandatory courses of action for that scenario. Only those oral responses that depicted relevant behaviors that were observable and could be quantified were assessed in the scoring process.

This examination was given using the chain oral testing process, and candidates were given 10 minutes to respond to each question. Candidate responses to each question were rated on a five-point scale (1 to 5) from nil response through optimum according to determinations made by the SMEs. Oral communication for each question was also rated on the five-point scale. This five-point scale includes 5 as the optimal response, 4 as a more than acceptable passing response, 3 as a minimally acceptable passing response, 2 as a less than acceptable response, and 1 as a much less than acceptable response. It is noted that candidates were told the following prior to beginning their presentations for each scenario: “In responding to the questions, be as specific as possible. Do not assume or take for granted that general actions will contribute to your score.”

For the Incident Command: Non-Fire Incident scenario, the appellant scored a 5 on the technical component and a 4 on the oral communication component. For the Supervision scenario, the appellant scored a 5 on the technical component and a 3 on the oral communication component. On the Administration scenario, the appellant scored a 2 on the technical component and a 4 on the oral communication component. Finally, for the Incident Command: Fire Incident scenario, the appellant scored a 2 on the technical component and a 3 on the oral communication component.

Appellant's argumentThe appellant challenges his scores on the oral communication components of the Supervision and Administration scenarios and the technical component of the Administration scenario. The appellant also requests that the Civil Service Commission (Commission) review and explain how his seniority score was calculated and factored into his final average for the subject examination. As a result, the appellant’s test material, video recording and a list of PCAs for the scenarios were reviewed.

For the oral communication component of the Supervision scenario, the assessor stated that the appellant displayed a major weakness in word usage/grammar and a minor weakness in inflection/rate/volume. Specifically, with regard to word usage/grammar, the assessor noted that the appellant used filler words 40 times during his presentation. As to inflection/volume/rate, the assessor found that the appellant spoke very quickly during his presentation. Concerning the oral communication component of the Incident Command: Fire Incident, the assessor indicated that appellant displayed a major weakness in word usage/grammar, uttering “uh” or “um” approximately 50 times during his presentation.

On appeal, the appellant argues that the use of filler words like “ah” and “um” is commonly regarded as normal and the average person uses them at a rate of one every 12 seconds (five per minute). He contends that “under the duress and stress of the testing process, the total elimination of filler words is extremely difficult” and he observes that using “ahs” and “ums” at a “low average” between two and four times per minute in a presentation would translate to between 20 and 40 utterances in a 10-minute presentation. As such, the appellant asserts that his rate of speech and filler words fell within “normal speech patterns.” He argues that because the quality of his answers to all four scenarios was consistent, he should be awarded a score of 4 for the oral communication component of the Supervision and Incident Command: Fire Incident scenarios. He also calls this criterion into question because the Division of Test Development, Analytics and Administration (TDAA) does not utilize a specific number or rate of utterances of filler words to calculate a candidate’s score.

AnalysisThe Commission finds that the appellant has failed to meet his burden of proof with regard to his oral communication scores for the subject components. At the outset, the appellant has not cited any source to support his claims about “normal speech patterns.” Even assuming, arguendo, that the figures he presents for normal speech patterns are accurate, it does not render this agency’s assessment of filler words as a component of candidates’ oral communication scores invalid. Scoring for the subject examination is not strictly on a pass/fail basis. Rather, eligibles are given a final average and ranked based on performance. The use of filler words undoubtedly undermines the quality and clarity of a presentation, as it is easier to quickly understand and process information that is not obscured by utterances like “ah” or “um.” Thus, it is more than appropriate to rate a candidate’s presentation as “optimal” or “more than acceptable” if they use few or no filler words, while giving “acceptable” or lower ratings to candidates who use filler words at a greater rate. The Commission also emphasizes that the use of a flexible, holistic approach to assess whether and to what extent the use of filler words impacts the effectiveness of a candidate’s oral communication performance does not render the use of this metric invalid or arbitrary. In this regard, the Commission notes, for example, that two candidates may utter 20 “uhs” during their full presentations, but one candidate’s presentation might reasonably be considered more ineffective if they utter “uh” 20 times in the span of a minute than another who utters the same 20 “uhs” over a 10- minute period. As to the appellant’s presentations specifically, a review of his performance confirms that he uttered “um” or “uh” more than 50 times during his presentations for the Supervision and Incident Command: Fire Incident scenarios and that it was appropriately considered a major weakness. Accordingly, the appellant’s score of 3 for the oral communication components of both the Supervision and Incident Command: Fire Incident scenarios is sustained.

Regarding the appellant’s challenge to his score on the technical component of the Administration scenario, the Administration scenario presents that the Fire Chief has tasked the candidate with preparing for a parade on Halloween evening in coordination with the police department, including ensuring safety and code compliance efforts. Question 1 asks what course of action the candidate should take to complete their task. Question 2 asks how the candidate would handle requests from some department members to be included in the parade and leave requests from others requesting off for the Halloween holiday.

For the technical component of the Administration scenario, the assessor found that the appellant failed to identify a significant number of PCAs, including the opportunities to: review applicable code compliance/apply for permits/determine what criteria must be met during festivities; determine which fire department company would participate in the parade; and seek union input. On appeal, the appellant points to statements he made at the 30:15 and 33:35 marks in his video recording as evidence that he should have been credited with the PCA of reviewing applicable code compliance/apply for permits to determine what criteria need to be met during the event. He also argues that pursuant to the New Jersey Uniform Fire Code (NJUFC) and the International Fire Code (IFC), New Jersey Edition, the individual holding the title of Fire Official would have all authority and responsibility for code compliance and permitting and would act as the liaison for other stakeholders and ensure that all needs were met. He maintains that he addressed the foregoing by indicating during his presentation that the municipal partners and their liaisons would present the facts, code compliance and completed permits to be entered into the incident action plan (IAP). As to the PCAs of determining which fire department company would participate in the parade and seeking union input, the appellant references specific points in the recording of his presentation that he maintains address these actions.

The Commission finds that the appellant has failed to meet his burden of proof with respect to his challenge to the scoring of the technical component of the Administration scenario. The appellant stated at the 30:15 mark in his video recording that the Committee “will ensure that our legal aspects are taken care of and that we will identify all of our areas” and at 33:21 “I will ensure that our SOPs and our plan, our Incident Action Plan that we built on this event, is sustainable. I will research past events and other parties that we had. I will investigate with the Police Department and other areas if there’s new risks involved and I will train and emphasize with all of my Fire Fighters and fire officers that this is a great opportunity to show the public who pays us, who trusts us, that they can continue to trust us.” As noted above, candidates were told when “responding to the questions, be as specific as possible. Do not assume or take for granted that general actions will contribute to your score.” These statements were too general to cover this PCA, as they did not specifically speak to code compliance or permitting and the appellant’s references to the NJUFC and IFC on appeal are insufficient to find he should have been credited with the subject PCA. As to the PCA of determining the fire department company that will participate in the parade, the Commission observes that at 34:45, the appellant stated “I will uh select two fire trucks and locate them into the area and I will rotate ladder trucks and engine [sic] maybe that can accommodate the men to come in and be with their men.” As the appellant did not identify which fire company would be participating in the parade or speak to how he would determine which trucks/fire company would participate in the parade, he was properly denied credit for this PCA.

Finally, with regard to the appellant’s seniority score, examination seniority is based on the time from the regular appointment date (to the eligible title) to the closing date of the announcement, minus the time spent on layoffs, certain leaves of absence without pay, or suspensions. See N.J.A.C. 4A:4-2.15 (Rating of examinations). The appellant received a promotion to Fire Officer 2 on July 12, 2016, and the closing date was February 28, 2022. His seniority score is 85.636. This reflects a base score of 70, plus 10 points for record of service, plus 5.636 for the 5 years, 7 months and 16 days he was a Fire Officer 2. Time spent in a provisional position or as an “acting” Fire Officer 2 is not added to seniority for any candidate. Accordingly, the record demonstrates that the appellant’s seniority score of 85.636 is correct. Further, a review of the appellant’s overall score calculation demonstrates that his final average of 84.550 was correct.

CONCLUSION

A thorough review of the appellant’s submissions and the test materials indicates that the decision below is amply supported by the record, and the appellant has failed to meet his burden of proof in this matter.

ORDER

Therefore, it is ordered that this appeal be denied.

ClosingThis is the final administrative determination in this matter. Any further review should be pursued in a judicial forum.