Christopher Edwards, CSC Docket No. 2023-1960

BackgroundChristopher Edwards appeals his score on the promotional examination for Battalion Fire Chief (PM3377C), Camden. It is noted that the appellant passed the examination with a final average of 79.420 and ranks 12th on the on the eligible list.

The subject promotional examination was held on May 23, 2022, and 12 candidates passed. This two-part examination consisted of an integrated system of simulations designed to generate behavior similar to that required for success on the job. The first part consisted of multiple-choice items that measured specific work components identified and weighted by the job analysis. The second part consisted of three oral scenarios: Supervision, Administration and Incident Command. The examination was based on a comprehensive job analysis conducted by the Civil Service Commission (Commission), which identified the critical areas of the job. The weighting of the test components was derived from the job analysis data. It is noted that candidates were told the following prior to beginning their presentations for each scenario: “In responding to the questions, be as specific as possible. Do not assume or take for granted that general actions will contribute to your score.”

Each candidate in a given jurisdiction was scored by a team of three different Subject Matter Experts (SMEs), who were trained in current technical and oral communication scoring procedures. Each SME is a current or retired fire officer who held the title of Battalion Fire Chief (or Fire Officer 2) or higher. Candidates were also assessed by three Commission employees trained in oral communication assessment. As part of the scoring process, an SME observed and noted the responses of a candidate relative to the knowledge, skills and abilities (KSAs) that each exercise was designed to measure. An assessor also noted any weaknesses that detracted from the candidates overall oral communication ability. Each assessor then rated the candidate’s performance according to the rating standards and assigned the candidate a technical or oral communication score on that exercise.

In order to preserve the relative weighting of each of the components of the examination, the ratings for each portion were adjusted by a well-recognized statistical process known as “standardization.” Under this process, the ratings are standardized by converting the raw scores to z-scores, an expression of the deviation of the score from the mean score of the group in relation to the standard deviation of scores for the group. Each portion of the examination had a relative weight in its relation to the whole examination. Thus, the z-score for the multiple-choice portion was multiplied by a test weight of 36.53%, the oral technical scores were multiplied by a test weight of 53.91% and the oral communication scores were multiplied by a test weight of 9.56%. The weighted z-scores were summed and this became the overall final test score. This was weighted and added to the weighted seniority score. The result was standardized, then normalized, and rounded up to the third decimal place to arrive at a final average.

For the Supervision scenario, the appellant scored a 4 on the technical component and a 4 on the oral communication component. On the Administration scenario, the appellant scored a 3 on the technical component and a 3 on the oral communication component. Finally, on the Incident Command scenario, the appellant scored a 3 on the technical component and a 3 on the oral communication component.

Appellant's argumentThe appellant challenges his scores for the oral communication component of the Administration, Supervision and Incident Command scenarios, and the technical portions of the Administration and Incident Command scenarios.

CONCLUSION

For the oral communication component of the Supervision scenario, the assessor indicated that the appellant displayed a minor weakness in word usage/grammar, as evidence by the candidate’s use of “um” and “uh” 17 times and “you know” four times during his presentation. Accordingly, the assessor awarded the appellant a score of 4 for the oral communication component of the Supervision scenario. On appeal, the appellant argues that his use of these filler words did not make his statements incorrect and he maintains that he should not have received a lower score on that basis.

In reply, a review of the appellant’s presentation supports the assessor’s determination that the appellant’s use of filler words like “um,” “uh” and “you know” was a minor weakness that detracted from his performance. Accordingly, the appellant’s score of 4 on the oral component of the Supervision scenario is correct.

On the Administration scenario, the assessor stated that the appellant displayed a major weakness in word usage/grammar, as evidenced by frequently using grammatically incorrect sentences and repeating words/phrases. The assessor also indicated that the appellant used filler words like “uh,” “um” and “you know” approximately 24 times. Based upon the foregoing, the assessor awarded the appellant a score of 3 for the oral communication component of the Administration scenario. On appeal, the appellant argues that his use of these filler words did not make his statements incorrect and he maintains that he should not have received a lower score on that basis.

In reply, a review of the appellant’s presentation supports the assessor’s determination that the appellant displayed a major weakness in word usage/grammar. As to the appellant’s claims that he should not have received a lower score on the basis of his use of filler words, it is noted that in the 2022 2nd Level Fire Supervisor Orientation Guide recommended that candidates practice their oral communication skills and seek to “limit excessive filler words (e.g., “um,” “ah,” “y’know).” See State of New Jersey, Civil Service Commission, 2022 2nd Level Fire Supervision Orientation Guide at 16-17, https://www.nj.gov/csc/about/news/safety/pdf/FINAL%202022%202nd%20Level%20 Fire%20Supervisor%20Orientation%20Guide.pdf. Based upon the foregoing, the appellant’s score of 3 for the oral communication component of the Administration scenario is sustained.

As to the Incident Command scenario, the assessor found that the appellant displayed a major weakness in inflection/rate/volume, as evidenced by rapid speech that inhibited the appellant's enunciation throughout his presentation. As such, the assessor awarded the appellant a score of 3 for oral communication on the subject scenario. On appeal, the appellant argues that his speech was clear and that his rate of speech was appropriate when considering the allotted time for the subject scenario.

In reply, a review of the recording of the appellant’s response to the Incident Command scenario supports the assessor’s determination that the appellant displayed a major weakness in inflection/rate/volume. The appellant’s speech throughout his presentation was less clear overall than in the other scenarios. Importantly, it is observed that his rate of speech increased noticeably during his response to Question 2 and that his enunciation became noticeably less clear as a result. According, the appellant’s score of 3 for the oral communication component of the Incident Command scenario is correct.

Regarding the technical components of the examination, the Administration scenario involves the candidate investigating an incident between a Fire Fighter and a Police Officer which culminated in the arrest of the Fire Fighter at the scene of a car accident where the candidate was serving as the incident commander.

For the technical component of the Administration scenario, the assessor awarded the appellant a score of 3 based upon the number of PCAs he missed, including the opportunity to review the National Fire Incident Reporting System (NFIRS) and the opportunity to review current directives of defining agency roles according to the type of incident (SOPs/SOGs, general orders). On appeal, the appellant argues that he covered the foregoing PCA by stating that he would check the standard operating procedures (SOP) and standard operating guidelines (SOG). In this regard, he maintains that the NFIRS is part of recordkeeping, so the SOPs and SOGs would be the procedures and guidelines for solving issues.

In reply, the Commission observes that while the appellant claims he covered the PCA of reviewing NFIRS by stating that he would check the SOPs and SOGs, the assessor properly denied him credit for both PCAs. As noted above, candidates were told the following prior to beginning their presentations for each scenario: “In responding to the questions, be as specific as possible. Do not assume or take for granted that general actions will contribute to your score.” The appellant’s references to standard procedures and standard guidelines were too general to award him credit for the PCA of reviewing current directives of defining agency roles according to the type of incident, as he failed to make sufficiently clear that he was reviewing SOPs and SOGs as they related to defining agency roles based on the type of incident. Similarly, the reference to SOPs and SOGs was too general to credit the appellant for reviewing NFIRS. Accordingly, the appellant’s score of 3 for the technical component of the Administration scenario is sustained.

The Incident Command scenario involves a response to a fire at a local auto parts store and auto repair shop. Question 1 asks what specific actions the candidate would take upon arriving at the scene. The prompt for Question 2 indicates that while crews are involved in extinguishment operation an explosion occurs on Side C, emergency radio traffic has been transmitted by a fire fighter and that structural damage is now visible on Side C. Question 2 asks what specific actions the candidate should now take based upon this new information.

On the technical component of the Incident Command scenario, the assessor awarded the appellant a score of 3 using the “flex rule,” where a maximum score may be awarded where a candidate fails to identify a mandatory PCA, but identifies additional acceptable actions. A score above 3 cannot be awarded under the flex rule. The appellant’s score was based upon his failure to identify the mandatory response of ordering an emergency evacuation in response to Question 2 and a number of additional opportunities in response to both questions. On appeal, the appellant acknowledges that he did not state that he would perform an emergency evacuation. However, he contends that he covered this action by removing all members during the explosion, calling for a personal accountability report and updating information. He avers that removing the personnel at that location was the same thing as ordering an emergency evacuation.

In reply, the Commission observes that the appellant was credited with the PCA of evacuating the affected area where smoke was present based upon his statements in response to Question 1. However, the appellant did not indicate that he would perform the mandatory action of ordering an emergency evacuation following the explosion referenced in the prompt for Question 2. Accordingly, the Commission finds that the appellant was properly awarded a score of 3 pursuant to the flex rule.

ORDER

Therefore, it is ordered that this appeal be denied.

ClosingThis is the final administrative determination in this matter. Any further review should be pursued in a judicial forum.