David Guadiane, CSC Docket No. 2025-1589

BackgroundDavid Guadiane appeals his score on the oral portion of the promotional examination for Battalion Fire Chief (PM2227F), Roselle. It is noted that the appellant achieved a passing score of 82.580 on the subject examination and is tied at rank one on the subject eligible list.

This two-part examination consisted of an integrated system of simulations designed to generate behavior similar to that required for success on the job. The first part consisted of multiple-choice items that measured specific work components identified and weighted by the job analysis. The second part consisted of three oral scenarios: Supervision, Administration and Incident Command. The examination was based on a comprehensive job analysis conducted by the Civil Service Commission (Commission), which identified the critical areas of the job. The weighting of the test components was derived from the job analysis data. It is noted that candidates were told the following prior to beginning their presentations for each scenario: “In responding to the questions, be as specific as possible. Do not assume or take for granted that general actions will contribute to your score.”

Each candidate in a given jurisdiction was scored by a team of three different Subject Matter Experts (SMEs), who were trained in current technical scoring procedures. Each of these SMEs were current or retired fire officers who held the title of Battalion Fire Chief (or Fire Officer 2) or higher. Candidates were also assessed by three Commission employees trained in oral communication assessment. As part of the scoring process, an SME observed and noted the responses of a candidate relative to the knowledge, skills and abilities (KSAs) that each exercise was designed to measure. An assessor also noted any weaknesses that detracted from the candidate’s overall oral communication ability. Each assessor then rated the candidate’s performance according to the rating standards and assigned the candidate a technical or oral communication score on that exercise.

In order to preserve the relative weighting of each of the components of the examination, the ratings for each portion were adjusted by a well-recognized statistical process known as “standardization.” Under this process, the ratings are standardized by converting the raw scores to z-scores, an expression of the deviation of the score from the mean score of the group in relation to the standard deviation of scores for the group. Each portion of the examination had a relative weight in its relation to the whole examination. Thus, the z-score for the multiple-choice portion was multiplied by a test weight of 36.53%, the oral technical scores were multiplied by a test weight of 53.91% and the oral communication scores were multiplied by a test weight of 9.56%. The weighted z-scores were summed and this became the overall final test score. This was weighted and added to the weighted seniority score. The result was standardized, then normalized, and rounded up to the third decimal place to arrive at a final average.

On the Supervision Scenario, the appellant scored a 3 on the technical component and a 3 on the oral communication component. On the Administration Scenario, the appellant scored a 4 on the technical component and a 5 on the oral communication component. Finally, on the Incident Command scenario, the appellant scored a 5 on the technical component and a 5 on the oral communication component.

Appellant's argumentThe appellant challenges his score on the oral communication component of the Supervision scenario. As a result, the appellant’s test material and a listing of possible courses of action (PCAs) for the scenario were reviewed.

On the oral communication component of the Supervision scenario, the assessor awarded the appellant a score of 3, based upon a determination that he displayed minor weaknesses in word usage/grammar and in nonverbal communication. Specifically, the assessor cited the appellant’s use of filler words like “um” and “uh” more than 20 times during his presentation as the basis for finding a minor weakness in word usage/grammar. Concerning nonverbal communication, the assessor stated that the candidate displayed a minor weakness by focusing on his notes more than the assessors, especially during the start of his presentation. On appeal, the appellant avers that for nonverbal communication, the 2024 Battalion Fire Chief Orientation Guide did not indicate that the scenario would not be read in the room prior to the presentation or that the assessors would have follow-up questions about the material presented and would ask candidates if they wanted to add additional points before concluding the presentation. He contends that this was different from prior lower-level promotional examinations and distracting to candidates. He argues that these considerations and the fact that between 30 and 40 minutes elapsed from the time he reviewed the scenario to the time he began his presentation created “an unfair disadvantage” and negatively impacted the fair and accurate assessment of his oral communication rating on the subject scenario. He adds that his ratings of 5 for oral communication on the other two scenarios show how knowing that the scenario prompts would not be re-read in the presentation rooms and that the assessors would be asking follow-up questions made a difference in his performance. Finally, the appellant argues that he should have been awarded a higher score because his breaks in speech were minimal, did not significantly disrupt the flow of his presentation and were not long pauses.

In reply, any differences between the administration of the subject examination and the administration of prior examinations, particularly for lower- level titles, are immaterial to the question of whether a given examination was fairly administered. In this regard, the Commission notes that the composition of the examinations for higher-level fire officer titles differs from that for lower-level titles. Further, changes in the content of an examination, venue, personnel, resources and/or other considerations like public health, may necessitate changes to how the examination for any given title may be conducted in subsequent examination cycles. As such, the relevant inquiry to whether this examination was fairly administered is whether the appellant was subjected to the same general conditions as the other candidates who took the subject examination. Here, a review of the presentations of the other three candidates who took the subject examination demonstrates that the Supervision scenario was administered in the same manner for each candidate, where the proctor did not re-read the full scenario prompt and instead simply asked the related scenario question of “[w]hat steps should you take when addressing this problem?” Since the Supervision scenario was administered in the same manner for all of the candidates, the fact the Supervision scenario prompt was not re-read to the appellant immediately prior to his presentation does not provide a basis to adjust his oral communication score for the Supervision scenario.

As to the appellant’s substantive challenge to his Supervision component oral communication score, a review of his presentation confirms that the appellant used filler words and phrases like “um,” “uh” and “you know,” in excess of 20 times during his presentation and that this was a minor weakness in word usage/grammar that detracted from his presentation. In addition, the recording further confirms that the appellant’s eye contact was largely focused on his notes, rather than the assessors, particularly during the earlier portion of his presentation. Accordingly, the appellant has failed to sustain his burden of proof and his Supervision scenario oral communication component score of 3 is affirmed.

CONCLUSION

A thorough review of the appellant’s submissions and the test materials indicates that the decision below is amply supported by the record and the appellant has failed to meet his burden of proof in this matter.

ORDER

Therefore, it is ordered that this appeal be denied.

ClosingThis is the final administrative determination in this matter. Any further review should be pursued in a judicial forum.