Thomas Higgins, CSC Docket No. 2020-1259

BackgroundThomas Higgins appeals his score on the promotional examination for Battalion Fire Chief (PM2164W), Pleasantville. It is noted that the appellant failed the examination.

This two-part examination consisted of an integrated system of simulations designed to generate behavior similar to that required for success on the job. The first part consisted of multiple-choice items that measured specific work components identified and weighted by the job analysis. The second part consisted of three oral scenarios; a Supervision, Administration and Incident Command scenario. The examination was based on a comprehensive job analysis conducted by the Civil Service Commission, which identified the critical areas of the job. The weighting of the test components was derived from the job analysis data.

For the oral portion, candidates had 60 minutes to prepare for all three scenarios and had 10 minutes per scenario to present their response. For all three oral exercises, the candidate was to assume the role of a Battalion Fire Chief. Candidates were scored based on the content of their response (technical) and the how well they presented their response (oral communication). These components were scored on a scale of 1 to 5 with 1 being the lowest rating and 5 being the highest rating.

Each candidate in a given jurisdiction was scored by a team of three different Subject Matter Experts (SMEs), who were trained in current technical and oral communication scoring procedures. Each SME is a current or retired fire officer who held the title of Battalion Fire Chief (or Fire Officer 2) or higher. As part of the scoring process, an SME observed and noted the responses of a candidate relative to the knowledge, skills and abilities (KSAs) that each exercise was designed to measure. An SME also noted any weaknesses that detracted from the candidates overall oral communication ability. The SME then rated the candidate’s performance according to the rating standards and assigned the candidate a technical or oral communication score on that exercise.

In order to preserve the relative weighting of each of the components of the examination, the ratings for each portion were adjusted by a well-recognized statistical process known as “standardization.” Under this process, the ratings are standardized by converting the raw scores to z-scores, an expression of the deviation of the score from the mean score of the group in relation to the standard deviation of scores for the group. Each portion of the examination had a relative weight in its relation to the whole examination. Thus, the z-score for the multiple-choice portion was multiplied by a test weight of 36.53%, the oral technical scores were multiplied by a test weight of 53.91% and the oral communication scores were multiplied by a test weight of 9.56%. The weighted z-scores were summed and this became the overall final test score. This was weighted and added to the weighted seniority score. The result was standardized, then normalized, and rounded up to the third decimal place to arrive at a final average.

For the technical and oral communication components of the Supervision, Administration and Incident Command scenarios, the appellant received scores of 4, 2 and 1, and 4, 3 and 4, respectively.

Appellant's argumentThe appellant challenges his score for the technical component for the Incident Command scenario. As a result, the appellant’s test material and a listing of possible courses of action (PCAs) for the scenario were reviewed.

CONCLUSION

The Incident Command scenario involved a fire at a powder metallurgy facility. Question 1 asked for specific actions upon arrival at the scene. Question 2 indicated that during fireground operations, there is an explosion inside the facility and a Mayday is being broadcast. It asked for specific actions to be taken based on this new information.

For this scenario, the SME noted that the appellant failed to set up cold, warm and hot zones, failed to set up a collapse zone, failed to ensure monitoring of the air, and failed to protect the storage area (hoseline, removal of drums) all of which were mandatory responses to question 1. On appeal, the appellant states that he assigned a Safety Officer, had firefighters in full PPE and SCBA and maintained constant communication with them, had a Hazmat decontamination area along with EMS, and had a 2.5 inch hoseline with another backup line of the same size to locate, confine and extinguish the fire.

In this instance, one SME indicated that the appellant did not provide mandatory responses. The instructions in the scenario tell candidates to be as specific as possible and not to assume or take for granted that general actions will contribute to a score. Each of the responses that the appellant gave on appeal are different than the actions noted by the assessor, and the appellant requests that the Commission assume that he gave those actions. This is not how scoring works. Credit is not given for information that is implied or assumed. Assigning a Safety Officer is not setting up a collapse zone. If it were, then a collapse zone would be set up each time a Safety Officer was assigned. Having firefighters in full PPE and SCBA and maintaining constant communication with them is not the same as monitoring the air, and contaminants in the air would affect more than just the firefighters. The appellant missed the point of this exercise if he believes that only the firefighters would be affected by air contaminants. Having a Hazmat decontamination area along with EMS is not the same as setting up cold, warm and hot zones. Bringing a 2.5 inch hoseline with another backup line to the seat of the fire in the powder metallurgy facility does not have a line to the storage area. The appellant’s score will not be assigned based on broad assumptions rather than what he actually said. The appellant missed at least four mandatory responses, and his score of 1 for this component is correct.

A thorough review of appellant’s submissions and the test materials indicates that the decision below is amply supported by the record, and the appellant has failed to meet his burden of proof in this matter.

ORDER

Therefore, it is ordered that this appeal be denied.

ClosingThis is the final administrative determination in this matter. Any further review should be pursued in a judicial forum.