Vincent Palamara, CSC Docket No. 2023-1971

BackgroundVincent Palamara, Jr. appeals his score on the promotional examination for Fire Officer 2 (PM4200C), Jersey City. It is noted that the appellant passed the examination with a final average of 84.300 and ranks 30th on the eligible list.

The subject promotional examination was held on May 19, 2022, and 39 candidates passed. This two-part examination consisted of an integrated system of simulations designed to generate behavior similar to that required for success on the job. The first part consisted of multiple-choice items that measured specific work components identified and weighted by the job analysis. The second part consisted of three oral scenarios: Supervision, Administration and Incident Command. The examination was based on a comprehensive job analysis conducted by the Civil Service Commission (Commission), which identified the critical areas of the job. The weighting of the test components was derived from the job analysis data. It is noted that candidates were told the following prior to beginning their presentations for each scenario: “In responding to the questions, be as specific as possible. Do not assume or take for granted that general actions will contribute to your score.”

Each candidate in a given jurisdiction was scored by a team of three different Subject Matter Experts (SMEs), who were trained in current technical scoring procedures. Each of these SMEs were current or retired fire officers who held the title of Battalion Fire Chief (or Fire Officer 2) or higher. Candidates were also assessed by three Commission employees trained in oral communication assessment. As part of the scoring process, an SME observed and noted the responses of a candidate relative to the knowledge, skills and abilities (KSAs) that each exercise was designed to measure. An assessor also noted any weaknesses that detracted from the candidate’s overall oral communication ability. Each assessor then rated the candidate’s performance according to the rating standards and assigned the candidate a technical or oral communication score on that exercise.

In order to preserve the relative weighting of each of the components of the examination, the ratings for each portion were adjusted by a well-recognized statistical process known as “standardization.” Under this process, the ratings are standardized by converting the raw scores to z-scores, an expression of the deviation of the score from the mean score of the group in relation to the standard deviation of scores for the group. Each portion of the examination had a relative weight in its relation to the whole examination. Thus, the z-score for the multiple-choice portion was multiplied by a test weight of 36.53%, the oral technical scores were multiplied by a test weight of 53.91% and the oral communication scores were multiplied by a test weight of 9.56%. The weighted z-scores were summed and this became the overall final test score. This was weighted and added to the weighted seniority score. The result was standardized, then normalized, and rounded up to the third decimal place to arrive at a final average.

On the Supervision scenario, the appellant scored a 5 on the technical component and a 4 on the oral communication component. On the Administration scenario, the appellant scored a 5 on the technical component and a 3 on the oral communication component. Finally, on the Incident Command scenario, the appellant scored a 5 on the technical component and a 4 on the oral communication component.

Appellant's argumentThe appellant challenges his scores for the oral communication components of all three scenarios. As a result, the appellant’s test material was reviewed.

On the Supervision scenario, the assessor found that the appellant displayed a minor weakness in word usage/grammar through the continued use of “ahs” and “ums” throughout his presentation and, as a result, awarded the appellant an oral communication score of 4.

On the Administration scenario, the assessor credited the appellant with an oral communication score of 3 based upon finding that the appellant displayed minor weaknesses in word usage, rate and organization. Specifically, the assessor indicated that the appellant’s use of filler words like “ah” and “um” in excess of 50 times constituted a minor weakness in word usage. As to rate, the assessor stated that the appellant displayed a minor weakness by speaking so quickly at times throughout his presentation that he stumbled over his words.

Finally, on the Incident Command scenario, the assessor awarded the appellant an oral communication score of 4 based on a finding that he displayed a minor weakness in rate of speech by speaking quickly throughout his presentation, such that he occasionally stumbled over his words and spoke in bullet points rather than full sentences. The assessor indicated that this pattern was especially pronounced at the beginning of the appellant’s presentation.

On appeal, the appellant denies that he displayed weaknesses in rate and organization, and he argues that because he received a rating of 5 on the technical components for each scenario, the deductions from his oral communication scores were contradictory. The appellant asserts that any answers he repeated reflected being thorough, rather than being disorganized. He avers that his only weakness in oral communication was pausing to ensure that he did not miss anything.

CONCLUSION

In the instant matter, a review of the appellant’s oral presentations confirms the validity of the assessors’ findings. At the outset, qualities like excessive usage of filler words and a fast rate of speech undoubtedly undermine the clarity and effectiveness of a presentation, as it is easier for a listener to fully understand and process information when it is not obscured by utterances like “ah” or “um” and/or delivered at a very brisk pace. Thus, regardless of a candidate’s technical component performance, it is more than appropriate to rate the oral communication component of a candidate’s presentation as “optimal” or “more than acceptable” if they use few or no filler words, while giving “acceptable” or lower ratings to candidates who use filler words at a greater rate. Similarly, for these reasons, it is more than appropriate to factor a candidate’s rate of speech into their oral communication scores even if they identify a substantial number of technical possible courses of action.

As to the appellant’s presentations specifically, a review of his performance confirms that his use of filler words, including “um” and “ah,” was appropriately classified as a minor weakness in word usage for both the Supervision and Administration scenarios. Additionally, the findings that the appellant displayed a minor weakness in rate of speech in both the Incident Command and Administration scenarios, such that it caused him to stumble over his words on several occasions, are substantiated by the record. Finally, a review of the recording of the appellant’s Administration scenario confirms that he displayed a minor weakness in organization by repeating a number of actions in response to Question 2 that he had previously identified in Question 1, including reviewing old and new standard operating procedures (SOPs); looking at building layouts from contractors; and using social media, magazines, flyers and local broadcasting. Critically, it cannot be said that these actions were necessary to address Question 2, which asked “[w]hat should be included in a pre-incident action plan standard operating procedures/guidelines (SOGs/SOPs).” Thus, the appellant’s repetition of these items reflected disorganization rather than thoroughness. Accordingly, based upon the totality of the record, the appellant has failed to meet his burden of proof and his scores on the subject examination are sustained.

ORDER

Therefore, it is ordered that this appeal be denied.

ClosingThis is the final administrative determination in this matter. Any further review should be pursued in a judicial forum.