Raymond Clark, CSC Docket No. 2024-1906

BackgroundRaymond Clark appeals his score on the oral portion of the promotional examination for Battalion Fire Chief (PM2480E), Ocean City. It is noted that the appellant passed the examination with a score of 81.580 and ranks fifth on the subject eligible list.

This two-part examination consisted of an integrated system of simulations designed to generate behavior similar to that required for success on the job. The first part consisted of multiple-choice items that measured specific work components identified and weighted by the job analysis. The second part consisted of three oral scenarios: Supervision, Administration and Incident Command. The examination was based on a comprehensive job analysis conducted by the Civil Service Commission (Commission), which identified the critical areas of the job. The weighting of the test components was derived from the job analysis data. It is noted that candidates were told the following prior to beginning their presentations for each scenario: “In responding to the questions, be as specific as possible. Do not assume or take for granted that general actions will contribute to your score.”

Each candidate in a given jurisdiction was scored by a team of three different Subject Matter Experts (SMEs), who were trained in current technical scoring procedures. Each of these SMEs were current or retired fire officers who held the title of Battalion Fire Chief (or Fire Officer 2) or higher. Candidates were also assessed by three Commission employees trained in oral communication assessment. As part of the scoring process, an SME observed and noted the responses of a candidate relative to the knowledge, skills and abilities (KSAs) that each exercise was designed to measure. An assessor also noted any weaknesses that detracted from the candidate’s overall oral communication ability. Each assessor then rated the candidate’s performance according to the rating standards and assigned the candidate a technical or oral communication score on that exercise.

In order to preserve the relative weighting of each of the components of the examination, the ratings for each portion were adjusted by a well-recognized statistical process known as “standardization.” Under this process, the ratings are standardized by converting the raw scores to z-scores, an expression of the deviation of the score from the mean score of the group in relation to the standard deviation of scores for the group. Each portion of the examination had a relative weight in its relation to the whole examination. Thus, the z-score for the multiple-choice portion was multiplied by a test weight of 36.53%, the oral technical scores were multiplied by a test weight of 53.91% and the oral communication scores were multiplied by a test weight of 9.56%. The weighted z-scores were summed and this became the overall final test score. This was weighted and added to the weighted seniority score. The result was standardized, then normalized, and rounded up to the third decimal place to arrive at a final average. For each of the scenes, and for oral communication, the requirements for each score were defined. It is noted that candidates were told the following prior to beginning their presentations for each scenario: “In responding to the questions, be as specific as possible. Do not assume or take for granted that general actions will contribute to your score.”

On the Incident Command Scenario, the appellant scored a 2 on the technical component and a 5 on the oral communication component. On the Supervision Scenario, the appellant scored a 3 on the technical component and a 5 on the oral communication component. On the Administration Scenario, the appellant scored a 5 on the technical component and a 5 on the oral communication component.

Appellant's argumentThe appellant challenges his score for the technical component of the Supervision Scenario. As a result, the appellant’s test material, video recording and a list of possible courses of action (PCAs) for the scenario were reviewed.

The Supervision Scenario presents that a female firefighter in the department comes to the candidate, who is the Battalion Fire Chief, and advises that her Captain is assigning her all of the domestic station tasks to complete by herself, while other male members of the company do nothing or handle the more technical tasks. She further presents that when she expressed her concerns to the Captain, he stated that he did so because she was the “new kid on the block.” The prompt then states that after doing research, the candidate has discovered that the Captain has not assigned duties equally and that the female firefighter is receiving all of the station assignments. Question 1 asks how the candidate should ensure that the firefighter’s complaint is addressed. The prompt for Question 2 states that a few days later, the female firefighter informs the candidate that while lifting weights in the fitness room, she received a “joking” comment from one of her male peers that “she needs to head to the kitchen where females belong.” It then asks, based on this new information, what actions the candidate should take.

The SME awarded the appellant a score of 3 on the Supervision Scenario, based upon a finding that he failed to identify a number of PCAs, including, in part, having the female firefighter fill out a written report following the incident described in Question 2 and having the accused firefighter in Question 2 fill out a written report. On appeal, the appellant cites statements he gave during his response to Question 1 in support.

In reply, the appellant’s arguments appear to be based upon a misunderstanding regarding the PCAs for which he did and did not receive credit. Both Questions 1 and 2 had PCAs related to requesting written statements and documenting each incident. However, the scoring summaries the appellant was given during his review session failed to clearly communicate to him that he received credit for the relevant PCAs related to documentation on Question 1, but failed to identify them during his response to Question 2. Upon review of the appellant’s response on appeal, the appellant was properly credited with the PCAs related to documentation of the incident and investigation on Question 1 and was correctly found not to have indicated that he would have the female firefighter or accused firefighter complete written reports regarding the Question 2 incident. However, this review has also demonstrated to both the Division of Test Development, Analytics and Administration and the Civil Service Commission that the appellant was erroneously credited with the Question 2 PCA of monitoring the situation for any future occurrences with the firefighter. Accordingly, credit for this PCA shall be stricken. Nevertheless, even with this reversal of credit, the appellant’s overall score on the technical component of the Supervision Scenario remains 3.

CONCLUSION

A thorough review of the appellant’s submissions and the test materials indicates that the decision below is amply supported by the record and the appellant has failed to meet his burden of proof in this matter.

ORDER

Therefore, it is ordered that this appeal be denied and that appropriate agency records be revised to reflect the above-noted adjustment to the appellant’s scoring records for the technical component of the Supervision Scenario, but that the appellant’s overall score for this component remain unchanged at 3.

ClosingThis is the final administrative determination in this matter. Any further review should be pursued in a judicial forum.