Dexter Smith, CSC Docket No. 2023-2026

BackgroundDexter Smith, Sr. appeals his score on the promotional examination for Fire Officer 2 (PM4200C), Jersey City. It is noted that the appellant passed the examination with a final average of 80.490 and ranks 39th on the eligible list.

The subject promotional examination was held on May 19, 2022, and 39 candidates passed. This two-part examination consisted of an integrated system of simulations designed to generate behavior similar to that required for success on the job. The first part consisted of multiple-choice items that measured specific work components identified and weighted by the job analysis. The second part consisted of three oral scenarios: Supervision, Administration and Incident Command. The examination was based on a comprehensive job analysis conducted by the Civil Service Commission (Commission), which identified the critical areas of the job. The weighting of the test components was derived from the job analysis data. It is noted that candidates were told the following prior to beginning their presentations for each scenario: “In responding to the questions, be as specific as possible. Do not assume or take for granted that general actions will contribute to your score.”

Each candidate in a given jurisdiction was scored by a team of three different Subject Matter Experts (SMEs), who were trained in current technical scoring procedures. Each of these SMEs were current or retired fire officers who held the title of Battalion Fire Chief (or Fire Officer 2) or higher. Candidates were also assessed by three Commission employees trained in oral communication assessment. As part of the scoring process, an SME observed and noted the responses of a candidate relative to the knowledge, skills and abilities (KSAs) that each exercise was designed to measure. An assessor also noted any weaknesses that detracted from the candidate’s overall oral communication ability. Each assessor then rated the candidate’s performance according to the rating standards and assigned the candidate a technical or oral communication score on that exercise.

In order to preserve the relative weighting of each of the components of the examination, the ratings for each portion were adjusted by a well-recognized statistical process known as “standardization.” Under this process, the ratings are standardized by converting the raw scores to z-scores, an expression of the deviation of the score from the mean score of the group in relation to the standard deviation of scores for the group. Each portion of the examination had a relative weight in its relation to the whole examination. Thus, the z-score for the multiple-choice portion was multiplied by a test weight of 36.53%, the oral technical scores were multiplied by a test weight of 53.91% and the oral communication scores were multiplied by a test weight of 9.56%. The weighted z-scores were summed and this became the overall final test score. This was weighted and added to the weighted seniority score. The result was standardized, then normalized, and rounded up to the third decimal place to arrive at a final average.

Each oral examination question, and overall oral communication, was rated on a five-point scale, with 5 as the optimal response, 4 as a more than acceptable passing response, 3 as a minimally acceptable passing response, 2 as a less than acceptable response, and 1 as a much less than acceptable response.

On the Supervision scenario, the appellant scored a 5 on the technical component and a 4 on the oral communication component. On the Administration scenario, the appellant scored a 2 on the technical component and a 3 on the oral communication component. Finally, on the Incident Command scenario, the appellant scored a 5 on the technical component and a 4 on the oral communication component.

Appellant's argumentThe appellant challenges his scores for the oral communication1 and technical components of the Administration scenario. As a result, the appellant’s test material and a listing of possible courses of action (PCAs) for the scenario were reviewed.

The Administration scenario involves the candidate being tasked with investigating an incident and revising the department’s current pre-incident action plan procedures following a call where a pre-action plan failed to reflect a building’s conversion and the addition of partition walls inside of the structure. Question 1 asks what specific steps the candidate would take to investigate the incident and the lack of updated pre-incident action plan procedures. Question 2 asks what should be included in updated pre-incident action plan standard operating guidelines/procedures (SOGs/SOPs).

1 The exact scope of the appellant’s appeal is unclear. The Commission interprets it as a challenge to the Administration scenario based upon his reference to possible courses of action (PCA) from the technical component for that scenario in his appeal and his scores on that scenario.

On the oral communication component of the Administration scenario, the assessor found that the appellant displayed a major weakness in word usage by uttering “ah/um” over 60 times and “ya know” 15 times. The assessor also found that the appellant displayed a minor weakness in organization by taking five distinct pauses ranging from 8 seconds to 45 seconds in length. Based upon the foregoing, the assessor awarded the appellant an oral communication score of 3.

On the technical component, the SME found that the appellant missed a significant number of PCAs, including the opportunity to interview the incident commander who was present on scene in response to Question 1 and opportunities to gather information about the building and identify the strategy and tactics that should be used in response to Question 2.

On appeal, the appellant argues that his oral communication score should not have been reduced because of his use of filler words like “ah,” “um” and “you know.” In this regard, he maintains that the Harvard Business Review states that it is acceptable to use filler words, as long as they aren’t used excessively2. The appellant alleges that the assessor became bias[ed] about his use of “ah” and “um” and erroneously indicated that he used the phrase “ya know” when rating his presentation. He proffers that the phrase he used was “you know,” which he notes “is a filler phrase.”

In addition, the appellant argues that his technical score for the Administration scenario should have been higher because he stated that he would conduct “fact finding” about the building in question, which “allow[ed] [him] to cover a lot and utilize time more efficiently.” In this regard, he proffers that his reference to “fact finding” established that all necessary people, including the chief, would be interviewed for “their knowledge of the building, what, when, why and how.”

CONCLUSION

In the instant matter, the appellant has failed to sustain his burden of proof. It is the longstanding policy of this agency to evaluate candidates’ use of filler words and phrases as part of the oral communication scoring of their oral presentations. In this regard, the excessive usage of filler words undoubtedly undermines the clarity and effectiveness of a presentation, as it is easier for a listener to fully understand and process information when it is not obscured by frequent utterances of fillers like “ah” and “um.” Thus, regardless of a candidate’s technical component performance, it is more than appropriate to rate the oral communication component of a candidate’s presentation as “optimal” or “more than acceptable” if they use few or no filler words, while giving “acceptable” or lower ratings to candidates who use filler words at a greater rate. Further, Candidates were apprised of this in the 2022 2nd Level Fire Supervisor Orientation Guide, which recommended that candidates practice their oral communication skills and seek to “limit excessive filler words (e.g., “um,” “ah,” “y’know).” See State of New Jersey, Civil Service Commission, 2022 2nd Level Fire Supervision Orientation Guide 16-17, https://www.nj.gov/csc/about/news/safety/pdf/FINAL%202022%202nd%20Level%20 Fire%20Supervisor%20Orientation%20Guide.pdf. Here, the appellant concedes that he used fillers like “you know” during his presentation. Based upon the foregoing and a review of the appellant’s presentation, it is abundantly clear that the assessor’s conclusion that the appellant displayed a major weakness in word usage on the Administration scenario based upon his excessive usage of filler words and phrases was objectively reasonable and consistent with the scoring standards for the subject examination, rather than the product of any “bias.” Similarly, the record confirms that the appellant’s multiple long pauses during his Administration scenario presentation were appropriately characterized as a minor weakness. Accordingly, the appellant’s oral communication score of 3 on the Administration scenario is sustained.

2 The appellant has not furnished a copy of or a citation for the article that purportedly supports this claim. Since pursuant to N.J.A.C. 4A:2-1.4, the appellant bears the burden of proof in this matter and he has failed to provide adequate support for this claim, it will not be considered further.

Similarly, the Commission finds that the appellant’s arguments regarding his Administration scenario technical component score are without merit. As noted above, candidates were told the following prior to beginning their presentations for each scenario: “In responding to the questions, be as specific as possible. Do not assume or take for granted that general actions will contribute to your score.” The Commission observes that the appellant did receive credit for a number of PCAs that he specifically identified in response to Question 1, but that his reference to “fact finding” was a general action that was insufficient to cover any of the PCAs for which he did not receive credit. The appellant’s response failed to cover any of the specific PCAs for Question 2, which asked what should be included in updated pre-incident action plan SOGs/SOPs. Critically, the appellant spoke only in general terms about the need to update the SOGs/SOPs and the need to perform background research, and he stated that he would leave it up to the Fire Chief to decide how to update the SOGs/SOPs, even though Question 2 clearly required him to state what should be included in updated SOGs/SOPs. Therefore, his technical component score of 2 on the Administration scenario is also affirmed.

ORDER

Therefore, it is ordered that this appeal be denied.

ClosingThis is the final administrative determination in this matter. Any further review should be pursued in a judicial forum.