Javier Matos, CSC Docket No. 2023-1984

BackgroundJavier Matos, represented by Michael L. Prigoff, Esq., appeals his score on the promotional examination for Battalion Fire Chief (PM3377C), Camden. It is noted that the appellant failed the subject examination.

The subject promotional examination was held on May 23, 2022, and 12 candidates passed. This two-part examination consisted of an integrated system of simulations designed to generate behavior similar to that required for success on the job. The first part consisted of multiple-choice items that measured specific work components identified and weighted by the job analysis. The second part consisted of three oral scenarios: Supervision, Administration and Incident Command. The examination was based on a comprehensive job analysis conducted by the Civil Service Commission (Commission), which identified the critical areas of the job. The weighting of the test components was derived from the job analysis data.

Each candidate in a given jurisdiction was scored by a team of three different Subject Matter Experts (SMEs), who were trained in current technical scoring procedures. Each of these SMEs were current or retired fire officers who held the title of Battalion Fire Chief (or Fire Officer 2) or higher. Candidates were also assessed by three Commission employees trained in oral communication assessment. As part of the scoring process, an SME observed and noted the responses of a candidate relative to the knowledge, skills and abilities (KSAs) that each exercise was designed to measure. An assessor also noted any weaknesses that detracted from the candidate’s overall oral communication ability. Each assessor then rated the candidate’s performance according to the rating standards and assigned the candidate a technical or oral communication score on that exercise.

In order to preserve the relative weighting of each of the components of the examination, the ratings for each portion were adjusted by a well-recognized statistical process known as “standardization.” Under this process, the ratings are standardized by converting the raw scores to z-scores, an expression of the deviation of the score from the mean score of the group in relation to the standard deviation of scores for the group. Each portion of the examination had a relative weight in its relation to the whole examination. Thus, the z-score for the multiple-choice portion was multiplied by a test weight of 36.53%, the oral technical scores were multiplied by a test weight of 53.91% and the oral communication scores were multiplied by a test weight of 9.56%. The weighted z-scores were summed and this became the overall final test score. This was weighted and added to the weighted seniority score. The result was standardized, then normalized, and rounded up to the third decimal place to arrive at a final average.

For the Supervision scenario, the appellant scored a 5 on the technical component and a 3 on the oral communication component. On the Administration scenario, the appellant scored a 3 on the technical component and a 2 on the oral communication component. Finally, on the Incident Command scenario, the appellant scored a 5 on the technical component and a 2 on the oral communication component.

Appellant's argumentThe appellant challenges his scores for the oral communication components of the Administration and Incident Command scenarios. As a result, the appellant’s test materials for the scenarios were reviewed.

CONCLUSION

On the Administration scenario, the assessor indicated that the appellant displayed major weaknesses in word usage/grammar and organization. With regard to word usage/grammar, the assessor cited the appellant’s use of “uh” and “um” approximately 34 times and “you know” approximately 31 times. As to organization, the assessor cited examples of the appellant losing his train of thought during the foregoing scenario. On the Incident Command scenario, the assessor indicated that the appellant displayed major weaknesses in word usage/grammar and nonverbal communication. Specifically, in finding a major weakness in word usage/grammar, the assessor cited the appellant’s use of “uh” and “um” approximately 34 times, use of “you know” three times and use of “ok” five times. As to nonverbal communication, the assessor indicated that the appellant failed to maintain sufficient eye contact during his presentation, as he was looking up and down at his notes, continually shifting in his chair and looking at his timer, all of which proved to be distracting. On appeal, the appellant avers that his use of filler words was a minor distraction in these scenarios. In this regard, he notes that he used such filler words 34 times in the Incident Command scenario but received a lower score of 2 than on the Supervision scenario where he uttered them 56 times and received a score of 3. Similarly he presents that his use of filler words in the Administration scenario resulted in a score of 2, even though it was only marginally greater than in the Supervision scenario. Based upon the foregoing, he argues that his oral communication scores for both the Administration and Incident Command scenarios should be raised to 3.

Initially, the Commission observes that the assessor found that the appellant displayed a major weakness in word usage and grammar in all three scenarios. The appellant’s oral communication score on the Supervision scenario is attributable to the assessor finding that the appellant did not display any other weaknesses. Conversely, the appellant’s scores on the Administration and Incident Command scenarios were lower because he displayed a second major weakness in each. Specifically, the second major weaknesses were organization in the Administration scenario and nonverbal communication in the Incident Command scenario. The appellant essentially argues that his use of filler words was comparable in each of the three scenarios. That is correct, but only in the sense that his video presentations conclusively demonstrate that his use of filler words was a major weakness, based upon the litany of “uhs”, “ums”, “you knows” and “oks” he uttered in each presentation. While the appellant does not appear to dispute the finding of other major weaknesses in his oral communication scores for the Administration and Incident Command scenarios, the Commission notes, in the interest of completeness, that a review of the recording of the appellant’s presentation supports the assessor finding a major weakness in organization in the Administration scenario and in nonverbal communication in the Incident Command scenario.

ORDER

Therefore, it is ordered that this appeal be denied.

ClosingThis is the final administrative determination in this matter. Any further review should be pursued in a judicial forum.