Stanley Kolbe, CSC Docket No. 2023-1954

BackgroundStanley Kolbe, Jr. appeals his score on the promotional examination for Battalion Fire Chief (PM3394C), Pleasantville. It is noted that the appellant failed the subject examination.

The subject promotional examination was held on May 25, 2022, and five candidates passed. This two-part examination consisted of an integrated system of simulations designed to generate behavior similar to that required for success on the job. The first part consisted of multiple-choice items that measured specific work components identified and weighted by the job analysis. The second part consisted of three oral scenarios: Supervision, Administration and Incident Command. The examination was based on a comprehensive job analysis conducted by the Civil Service Commission (Commission), which identified the critical areas of the job. The weighting of the test components was derived from the job analysis data. It is noted that candidates were told the following prior to beginning their presentations for each scenario: “In responding to the questions, be as specific as possible. Do not assume or take for granted that general actions will contribute to your score.”

Each candidate in a given jurisdiction was scored by a team of three different Subject Matter Experts (SMEs), who were trained in current technical scoring procedures. Each of these SMEs were current or retired fire officers who held the title of Battalion Fire Chief (or Fire Officer 2) or higher. Candidates were also assessed by three Commission employees trained in oral communication assessment. As part of the scoring process, an SME observed and noted the responses of a candidate relative to the knowledge, skills and abilities (KSAs) that each exercise was designed to measure. An assessor also noted any weaknesses that detracted from the candidate’s overall oral communication ability. Each assessor then rated the candidate’s performance according to the rating standards and assigned the candidate a technical or oral communication score on that exercise.

In order to preserve the relative weighting of each of the components of the examination, the ratings for each portion were adjusted by a well-recognized statistical process known as “standardization.” Under this process, the ratings are standardized by converting the raw scores to z-scores, an expression of the deviation of the score from the mean score of the group in relation to the standard deviation of scores for the group. Each portion of the examination had a relative weight in its relation to the whole examination. Thus, the z-score for the multiple-choice portion was multiplied by a test weight of 36.53%, the oral technical scores were multiplied by a test weight of 53.91% and the oral communication scores were multiplied by a test weight of 9.56%. The weighted z-scores were summed and this became the overall final test score. This was weighted and added to the weighted seniority score. The result was standardized, then normalized, and rounded up to the third decimal place to arrive at a final average.

Each oral examination question, and overall oral communication, was rated on a five-point scale, with 5 as the optimal response, 4 as a more than acceptable passing response, 3 as a minimally acceptable passing response, 2 as a less than acceptable response, and 1 as a much less than acceptable response.

On the Supervision scenario, the appellant scored a 4 on the technical component and a 2 on the oral communication component. On the Administration scenario, the appellant scored a 2 on the technical component and a 3 on the oral communication component. Finally, on the Incident Command scenario, the appellant scored a 2 on the technical component and a 4 on the oral communication component.

Appellant's argumentOn appeal, the appellant challenges his scores for the oral communication components of the Supervision and Incident Command scenarios, the technical components of the Administration and Incident Command scenarios, and seniority. As a result, the appellant’s test material and a listing of possible courses of action (PCAs) for the scenarios were reviewed.

On the Supervision scenario, the assessor found that the appellant displayed major weaknesses in word usage/grammar and in organization, plus a minor weakness in specificity/brevity. Specifically, the appellant’s repeating of words within sentences and his use of “um” excessively while responding was cited as evidence of his major weakness in word usage/grammar. As to organization, the assessor observed that the appellant repeatedly paused throughout his presentation, with one pause exceeding 10 seconds in length. In regard to specificity/brevity, the assessor stated that the appellant’s response was too brief to effectively demonstrate his oral communicative ability. Based upon the foregoing, the assessor awarded the appellant a score of 2 for oral communication on the Supervision scenario. On appeal, the appellant argues that his use of “um” was not excessive and he proffers that his 10- second pause was to review his notes, which he believed was permitted.

In reply, the appellant’s appeal of his oral communication score for the Supervision scenario is without merit. Even assuming, arguendo, that the appellant’s utterances of “um” as a filler word were insufficient alone to find he displayed a minor weakness in word usage/grammar, it is noted that the assessor also cited the appellant’s repeating of words to support their determination on this scoring dimension. A review of the appellant’s presentation confirms that he repeated words in several instances, including stating ". . . she may, she may [sic] be unaware of the whole thing" approximately one minute into his presentation and that "after all, after all [sic] this is done and complete and we found this out, again, we'll talk with talk with [sic] the captain, make sure she understands all the all the [sic] new policies . . ." in the final minute of his presentation. Such repetition, particularly in concert with the appellant’s use of filler words, clearly supported a determination that the appellant displayed a minor weakness in word usage/grammar. As to the appellant’s arguments regarding pauses, it is noted that the appellant’s presentation included a pause of approximately 27 seconds in the final minute of his presentation. Although the 2022 2nd Level Fire Supervisor Orientation Guide states that “[p]ausing occasionally to review notes is expected and will not be penalized,” that same passage also tells candidates to “eliminate long pauses,” as reviewers can consider such a deficiency in a presentation. Thus, the appellant had reasonable notice that he could be penalized for lengthy pauses, such as the one noted above, and his argument that his score should not have been lowered because of it is without merit. Accordingly, the appellant’s score of 2 for the oral communication component of the Supervision scenario is sustained.

On the Incident Command scenario, the assessor indicated that the appellant displayed a minor weakness in word usage/grammar by repeating words more than a dozen times during his responses. Based upon the foregoing, the assessor awarded the appellant a score of 4 for the scenario’s oral presentation component. On appeal, the appellant presents that a person walked into the room during presentation and that this impacted his train of thought.

In reply, it is noted that no fewer than eight of the occasions where the appellant repeated words mid-sentence occurred before the point in the presentation the appellant is referring to. Additionally, the person walking into the room at the point in question was the monitor advising the appellant that he had two minutes remaining to finish his presentation. The Commission notes that this is standard protocol for oral examination presentations. Critically, the video shows that the appellant was informed by the monitor prior to beginning his oral presentations that she would come into the room eight minutes into each presentation to let him know that he had two minutes remaining. For all of these reasons, the appellant losing his train of thought at the point of his presentation in question is immaterial and does not provide a basis to adjust his score. Accordingly, the appellant has failed to sustain his burden of proof and his oral communication score of 4 on the Incident Command scenario is sustained.

BackgroundWith regard to technical component scoring, the appellant appeals his scores on the Administration and Incident Command scenarios.

The Administration scenario provides that the candidate is a newly-promoted Battalion Fire Chief who has been assigned to review and update the current mutual aid contracts. Question 1 asks what steps the candidate should take to update the mutual aid contracts.

The assessor awarded the appellant a score of 2 on the technical component because the appellant failed to identify a significant number of PCAs, including the opportunity to update contracts, as needed. On appeal, the appellant argues that he covered the PCA of updating the contracts as needed at a specified point during his presentation.

In reply, a review of the appellant’s presentation indicates that he spoke only in general terms about having to “reevaluate everything” and “mak[ing] a change” if there are any problems at incidents after a “new mutual aid policy” is implemented and that he did not specify that he was updating mutual aid contracts, as required. Accordingly, the appellant was properly denied credit for this PCA and his score of 2 for the technical component of this scenario is sustained.

The Incident Command scenario involves a response to a report of fire at a local movie theater, part of which is in the process of a renovation. Question 1 asks what specific actions the candidate would take upon arriving at the scene. The prompt for Question 2 states that the party wall separating two theaters collapses during firefighting operations, trapping two firefighters. Question 2 then asks what specific actions the candidate should now take based on this new information.

The assessor found that the appellant failed to identify the mandatory responses of proper transfer of command in response to Question 1 and conducting a Personnel Accountability Report (PAR) in response to Question 2, and that he failed to identify a number of additional responses, including missing the opportunity to ensure the monitoring of the air. Based upon the foregoing, the assessor awarded the appellant a score of 2. On appeal, the appellant argues that he referenced conducting a PAR at a specified point in his presentation.

In reply, a review of the appellant’s response reveals that he discussed the use of accountability tags at the point of his presentation he cited, rather than conducting a PAR. Moreover, the appellant failed to otherwise call for a PAR during his response to Question 2. Accordingly, the appellant has failed to sustain his burden of proof and his score of 2 on the technical component of the Incident Command scenario is affirmed.

Appellant's argumentFinally, the appellant notes that during his review session, his scoring sheet indicated “fail” for his seniority score. He questions this and states that he is the Fire Captain with the most seniority in his department. As noted in the 2022 2nd Level Fire Supervisor Orientation Guide, only candidates with passing examination scores will have their overall test score weighted 70% and seniority score weighted 30% in determining their final score. Since the appellant failed the subject examination, his seniority score was not evaluated.

CONCLUSION

A thorough review of the appellant’s submissions and the test materials indicates that the decision below is amply supported by the record, and the appellant has failed to meet his burden of proof in this matter.

ORDER

Therefore, it is ordered that this appeal be denied.

ClosingThis is the final administrative determination in this matter. Any further review should be pursued in a judicial forum.