William Martin, CSC Docket No. 2024-1880

BackgroundWilliam Martin appeals his score on the oral portion of the promotional examination for Battalion Fire Chief (PM2480E), Ocean City. It is noted that the appellant achieved a passing score of 82.260 on the subject examination and ranks fourth on the subject eligible list.

This two-part examination consisted of an integrated system of simulations designed to generate behavior similar to that required for success on the job. The first part consisted of multiple-choice items that measured specific work components identified and weighted by the job analysis. The second part consisted of three oral scenarios: Supervision, Administration and Incident Command. The examination was based on a comprehensive job analysis conducted by the Civil Service Commission (Commission), which identified the critical areas of the job. The weighting of the test components was derived from the job analysis data. It is noted that candidates were told the following prior to beginning their presentations for each scenario: “In responding to the questions, be as specific as possible. Do not assume or take for granted that general actions will contribute to your score.”

Each candidate in a given jurisdiction was scored by a team of three different Subject Matter Experts (SMEs), who were trained in current technical scoring procedures. Each of these SMEs were current or retired fire officers who held the title of Battalion Fire Chief (or Fire Officer 2) or higher. Candidates were also assessed by three Commission employees trained in oral communication assessment. As part of the scoring process, an SME observed and noted the responses of a candidate relative to the knowledge, skills and abilities (KSAs) that each exercise was designed to measure. An assessor also noted any weaknesses that detracted from the candidate’s overall oral communication ability. Each assessor then rated the candidate’s performance according to the rating standards and assigned the candidate a technical or oral communication score on that exercise.

In order to preserve the relative weighting of each of the components of the examination, the ratings for each portion were adjusted by a well-recognized statistical process known as “standardization.” Under this process, the ratings are standardized by converting the raw scores to z-scores, an expression of the deviation of the score from the mean score of the group in relation to the standard deviation of scores for the group. Each portion of the examination had a relative weight in its relation to the whole examination. Thus, the z-score for the multiple-choice portion was multiplied by a test weight of 36.53%, the oral technical scores were multiplied by a test weight of 53.91% and the oral communication scores were multiplied by a test weight of 9.56%. The weighted z-scores were summed and this became the overall final test score. This was weighted and added to the weighted seniority score. The result was standardized, then normalized, and rounded up to the third decimal place to arrive at a final average.

On the Supervision Scenario, the appellant scored a 4 on the technical component and a 5 on the oral communication component. On the Administration Scenario, the appellant scored a 4 on the technical component and a 5 on the oral communication component. Finally, on the Incident Command Scenario, the appellant scored a 3, pursuant to the flex rule, on the technical component and a 4 on the oral communication component.

Appellant's argumentThe appellant challenges his scores for the technical and oral communication components of the Incident Command Scenario. As a result, the appellant’s test material and a listing of possible courses of action (PCAs) for the scenario were reviewed.

On the oral communication component of the Incident Command Scenario, the assessor awarded the appellant a score of 4, based upon a finding that the appellant displayed a minor weakness in word usage/grammar. In particular, the assessor cited the appellant's frequent use of the phrases “I’m gonna . . .” and “We’re gonna . . .” during his response. On appeal, the appellant argues that while the 2023 Battalion Fire Chief Orientation Guide listed certain distractions to avoid, including “limit excessive filler words (e.g., “um,” “ah,” “y’know”),” it did not say that he would be penalized for using the above-noted phrases, particularly when they were used to introduce full sets of actions. He questions how he could be expected to run an evolving fire ground “without explaining what actions need to be done and by whom.” He further presents that he was careful to avoid the distractions listed in the orientation guide and that if his use of “I’m gonna” was “worthy of a point deduction[,] it should [have] be[en] listed in the elements to consider” in the orientation guide.

In reply, the 2023 Battalion Fire Chief Orientation Guide presented a non- exhaustive list of general deficiencies, but, as noted by the appellant, also advised to “limit excessive filler words.” It also stated “do not ramble and/or repeat the same information over and over.” Thus, candidates had reasonable notice of the need to avoid the repetitive use of words and phrases. Further, it is observed that “I’m gonna” or “we’re gonna” are far from the only phrases one can use to state what actions they and/or their group will perform. Here, a review of the appellant’s presentation confirms that he uttered “I’m gonna” or “we’re gonna” in excess of 30 times. Thus, there is ample support for the assessor’s rating of 4 for the appellant’s Incident Command oral presentation component score and the appellant has failed to sustain his burden of proof with respect to this component.

Regarding the technical component of the Incident Command Scenario, the prompt presents that upon the candidate's arrival back to their firehouse after being dispatched to a call, they notice an abundance of black smoke and flames spewing from the second-floor windows where the bunks are located, with the fire located on Side B of the building. Question 1 asks for the candidate’s specific actions upon arrival at the scene. Question 2 indicates that during overhaul, a firefighter transmits a mayday after seeing two firefighters overcome by smoke in the hallway. It then queries what specific actions should now take, given this new information.

On the technical component, the assessor awarded the appellant a score of 3, pursuant to the flex rule, based on a finding that he missed a number of PCAs, including, in part, the mandatory response of monitoring and protecting the truss roof, and an opportunity to identify command mode. On appeal, the appellant argues that he identified his command mode with the following statements and actions: taking command, naming it, and giving a location; ordering Engine 1 to stretch a 1 ¾ inch line for maneuverability; ordering Engine 2 to report to Side B; ordering the truck company to locate, confine and extinguish the fire; ordering a water supply officer to take up post in case the fire got bigger; stating “if we have to go defensive, we will flank the building, ground monitors, aerials switch to Firefighter mode”; and giving a direction to surround and drown.

In reply, it is noted that the appellant does not argue that he addressed the mandatory response of monitoring and protecting the truss roof and a review of the recording of his presentation confirms that he failed to identify this mandatory PCA. Generally, candidates must identify all mandatory responses to receive, at minimum, a score of 3. However, a score of 3 may also be achieved via the “flex rule,” where a candidate provides many additional responses, but does not give a mandatory response. However, a score higher than a 3 cannot be provided utilizing the flex rule. As such, even if the appellant was correct that he should have been credited with the additional PCA of identifying command mode, because he still missed the mandatory response of monitoring and protecting the truss roof, pursuant to the flex rule, he still could not receive a rating higher than 3 on the Incident Command Scenario. Accordingly, the appellant’s appeal of his Incident Command Scenario technical component score is moot and his rating of 3 is affirmed.

CONCLUSION

A thorough review of the appellant’s submissions and the test materials indicates that the decision below is amply supported by the record and the appellant has failed to meet his burden of proof in this matter.

ORDER

Therefore, it is ordered that this appeal be denied.

ClosingThis is the final administrative determination in this matter. Any further review should be pursued in a judicial forum.