Gennaro Basso, CSC Docket No. 2023-1915

BackgroundGennaro Basso appeals his score on the promotional examination for Battalion Fire Chief (PM4444C), Atlantic City. It is noted that the appellant passed the examination with a final average of 83.830 and ranks 16th on the eligible list.

The subject promotional examination was held on May 25, 2022, and 27 candidates passed. This two-part examination consisted of an integrated system of simulations designed to generate behavior similar to that required for success on the job. The first part consisted of multiple-choice items that measured specific work components identified and weighted by the job analysis. The second part consisted of three oral scenarios: Supervision, Administration and Incident Command. The examination was based on a comprehensive job analysis conducted by the Civil Service Commission (Commission), which identified the critical areas of the job. The weighting of the test components was derived from the job analysis data. It is noted that candidates were told the following prior to beginning their presentations for each scenario: “In responding to the questions, be as specific as possible. Do not assume or take for granted that general actions will contribute to your score.”

Each candidate in a given jurisdiction was scored by a team of three different Subject Matter Experts (SMEs), who were trained in current technical scoring procedures. Each of these SMEs were current or retired fire officers who held the title of Battalion Fire Chief (or Fire Officer 2) or higher. Candidates were also assessed by three Commission employees trained in oral communication assessment. As part of the scoring process, an SME observed and noted the responses of a candidate relative to the knowledge, skills and abilities (KSAs) that each exercise was designed to measure. An assessor also noted any weaknesses that detracted from the candidate’s overall oral communication ability. Each assessor then rated the candidate’s performance according to the rating standards and assigned the candidate a technical or oral communication score on that exercise.

In order to preserve the relative weighting of each of the components of the examination, the ratings for each portion were adjusted by a well-recognized statistical process known as “standardization.” Under this process, the ratings are standardized by converting the raw scores to z-scores, an expression of the deviation of the score from the mean score of the group in relation to the standard deviation of scores for the group. Each portion of the examination had a relative weight in its relation to the whole examination. Thus, the z-score for the multiple-choice portion was multiplied by a test weight of 36.53%, the oral technical scores were multiplied by a test weight of 53.91% and the oral communication scores were multiplied by a test weight of 9.56%. The weighted z-scores were summed and this became the overall final test score. This was weighted and added to the weighted seniority score. The result was standardized, then normalized, and rounded up to the third decimal place to arrive at a final average.

Each oral examination question, and overall oral communication, was rated on a five-point scale, with 5 as the optimal response, 4 as a more than acceptable passing response, 3 as a minimally acceptable passing response, 2 as a less than acceptable response, and 1 as a much less than acceptable response.

On the Supervision scenario, the appellant scored a 4 on the technical component and a 2 on the oral communication component. On the Administration scenario, the appellant scored a 5 on the technical component and a 5 on the oral communication component. Finally, on the Incident Command scenario, the appellant scored a 4 on the technical component and a 5 on the oral communication component.

Appellant's argumentThe appellant challenges his scores for the oral communication component of the Supervision scenario and the technical component of the Incident Command scenario. As a result, the appellant’s test material and a listing of possible courses of action (PCAs) for the scenarios were reviewed.

On the oral communication component of the Supervision scenario, the assessor awarded the appellant a score of 2 based on findings that he displayed major weaknesses in word usage/grammar and in specificity/brevity. Specifically, with regard to word usage/grammar, the assessor cited the appellant’s repetition of words and phrases within sentences. The assessor also noted that while concluding his response, the appellant mentioned initially that he responded to Questions 1 and 2 together, but broke them up afterwards and explained them. As to the major weakness in specificity/brevity, the assessor found that the appellant’s response was too brief to effectively demonstrate his oral communication ability. On appeal, the appellant maintains that should not have been marked down for repetition, as he only did so where necessary to clarify which question he was answering. As to brevity, he maintains that his score of 2 was inconsistent with his technical component score of 4. In this regard, he proffers that since his response missed a single PCA and just barely missed a score of 5, his answer could not have been too brief.

In reply, although the appellant relies on his technical component rating of 4 on the Supervision scenario to argue that his oral communication scores for the same component should have been higher than 2, the Commission emphasizes that it is this agency’s longstanding policy that technical and oral communication component scores are independent ratings on the examination and that an exemplary or poor technical rating does not have a bearing on oral communication scoring and vice versa. Furthermore, oral communication performance can clearly distinguish candidates, including those delivering presentations with comparable technical details. To wit, it would be disingenuous to argue that the presentation of one candidate who spoke at a low rate of volume, had their speech punctuated by the frequent use of filler words like “ah” and “um,” rarely made eye contact with their audience and routinely made distracting hand gestures would be as understandable, effective and well-received as the presentation of another candidate who gave a speech with a comparable level of detail, but without these same oral communication issues. For these reasons, mere arguments that because a candidate received an certain technical score on a scenario, they should have received a corresponding oral communication rating for that same scenario are invalid. Given these considerations, the Commission finds that this aspect of the appellant’s arguments regarding his oral communication score is without merit and must be denied. As to repetition and brevity in particular, the Commission finds that the record substantiates the assessor’s findings. Specifically, the appellant repeated a number of actions including, in part, speaking with current crew members and allowing the Fire Captain to provide her side of the story. Even assuming, arguendo, that it was inaccurate to characterize this as an issue of word usage/grammar, the appellant’s acknowledged recapping of his actions in order to clarify which actions were meant to respond to which question would still constitute a minor weakness in organization. Beyond this, the appellant’s response was noticeably brief, lasting less than five minutes, one minute of which involved rehashing items he discussed earlier in his presentation. As such, the assessor’s finding of brevity is also sustained. Based upon the foregoing, the appellant’s oral communication score of 2 on the technical component of the Supervision scenario is affirmed.

The Incident Command scenario involves a response to a report of fire at a local movie theater, part of which is in the process of a renovation. Question 1 asks what specific actions the candidate would take upon arriving at the scene. The prompt for

Question 2 states that the party wall separating two theaters collapses during firefighting operations, trapping two firefighters. Question 2 then asks what specific actions the candidate should now take based on this new information. The SME awarded the appellant a score of 4 for the technical component of the Incident Command scenario, finding that the appellant missed a number of additional opportunities, including, in part, the opportunity to establish a water supply. On appeal, the appellant argues that he covered this PCA at a specified point during his presentation.

In reply, a review of the appellant’s presentation demonstrates that he was properly denied credit for the PCA of establishing a water supply. The appellant, while describing how he would expand the Incident Command System stated, in part, that “[e]ven though we have four hydrants, I would call for a Water Supply Officer to ensure adequate water supply throughout the duration of the incident.” This statement conveyed that he would appoint a Water Supply Officer and he received credit for the associated action of expanding the Incident Command System. However, establishing a water supply is a distinct action which the appellant failed to separately identify. As such, he was properly denied credit for the foregoing PCA and his score of 4 is sustained.

CONCLUSION

A thorough review of the appellant’s submissions and the test materials indicates that the decision below is amply supported by the record and that the appellant has failed to meet his burden of proof in this matter.

ORDER

Therefore, it is ordered that this appeal be denied.

ClosingThis is the final administrative determination in this matter. Any further review should be pursued in a judicial forum.