Julio Medina, CSC Docket No. 2020-1295

BackgroundJulio Medina appeals his score on the promotional examination for Battalion Fire Chief (PM2157W), Newark. It is noted that the appellant passed the examination with a final average of 78.670 and ranked tenth on the eligible list.

This two-part examination consisted of an integrated system of simulations designed to generate behavior similar to that required for success on the job. The first part consisted of multiple-choice items that measured specific work components identified and weighted by the job analysis. The second part consisted of three oral scenarios; a Supervision, Administration and Incident Command scenario. The examination was based on a comprehensive job analysis conducted by the Civil Service Commission, which identified the critical areas of the job. The weighting of the test components was derived from the job analysis data.

For the oral portion, candidates had 60 minutes to prepare for all three scenarios and had 10 minutes per scenario to present their response. For all three oral exercises, the candidate was to assume the role of a Battalion Fire Chief. Candidates were scored based on the content of their response (technical) and the how well they presented their response (oral communication). These components were scored on a scale of 1 to 5 with 1 being the lowest rating and 5 being the highest rating.

Each candidate in a given jurisdiction was scored by a team of three different Subject Matter Experts (SMEs), who were trained in current technical and oral communication scoring procedures. Each SME is a current or retired fire officer who held the title of Battalion Fire Chief (or Fire Officer 2) or higher. As part of the scoring process, an SME observed and noted the responses of a candidate relative to the knowledge, skills and abilities (KSAs) that each exercise was designed to measure. An SME also noted any weaknesses that detracted from the candidates overall oral communication ability. The SME then rated the candidate’s performance according to the rating standards and assigned the candidate a technical or oral communication score on that exercise.

In order to preserve the relative weighting of each of the components of the examination, the ratings for each portion were adjusted by a well-recognized statistical process known as “standardization.” Under this process, the ratings are standardized by converting the raw scores to z-scores, an expression of the deviation of the score from the mean score of the group in relation to the standard deviation of scores for the group. Each portion of the examination had a relative weight in its relation to the whole examination. Thus, the z-score for the multiple-choice portion was multiplied by a test weight of 36.53%, the oral technical scores were multiplied by a test weight of 53.91% and the oral communication scores were multiplied by a test weight of 9.56%. The weighted z-scores were summed and this became the overall final test score. This was weighted and added to the weighted seniority score. The result was standardized, then normalized, and rounded up to the third decimal place to arrive at a final average.

For the technical and oral communication components of the Supervision, Administration and Incident Command scenarios, the appellant received scores of 5, 3, 3 and 4, 5, 4, respectively.

Appellant's argumentThe appellant challenges his scores for the technical components for the Administration and Incident Command scenarios. As a result, the appellant’s test material and a listing of possible courses of action (PCAs) for the scenarios were reviewed.

CONCLUSION

In the administration scenario, the Deputy Fire Chief assigns the candidate to develop and be ready to implement a preplan for emergencies in places of worship in the first due response area, with an emphasis on risk management. Question 1 asked for actions to be taken to bring the first due response area incident action plans for places of worship in line with the Deputy Fire Chief’s assignment. Question 2 asked for specific information to be included in the pre-plans to effectively cover the arsonist threat.

For this scenario, the SME indicated that the appellant missed the opportunities to outline the building construction and to identify exposures. There are actions to be taken in response to 2. On appeal, the appellant states that he identified target hazards, formed committees and gave a plan of action, and had fire preplanning, inspections and mock drills.

In reply, the instructions in the scenario tell candidates to be as specific as possible and not to assume or take for granted that general actions will contribute to a score. As such, it was made clear that credit cannot be given for responses that are implied or assumed. The appellant argues for credit for assumed responses. That is, he believes that he included outlining building construction as specific information in the pre-plan as he formed committees for various purposes. These are distinct and different actions, and the appellant must verbalize responses if he is to receive credit for them. In this case, a review of the appellant’s presentation indicates that he spent a significant amount of time answering question 1, but did not respond to question 2. As a result, he missed a significant amount of actions which would have enhanced his score. He did not outline the building construction, identify exposures or include any specific information in his pre-plan. His score of 3 for this component is correct.

As a result of the review of this scenario, it was found that the presentation has a weakness in inflection/rate/volume which is defined as failing to speak at an appropriate rate (long pauses/too fast/stumbling) failing to maintain appropriate pitch and volume, and improperly using pitch to convey meaning or emphasis. In this case, the appellant was found to speak too rapidly, such that he could not be understood without re-reviewing the presentation. Throughout the presentation, the appellant poorly enunciated and pronounced words and sentences, such that the presentation was difficult to follow and unclear at times. The appellant’s sentences had pauses after words and phrases, rather than at the end of sentences only, in a halting manner. Viewed holistically, the presentation contained a weakness in inflection/rate/volume and the score for this component should be changed from 5 to 4.

The Incident Command scenario involved a fire at a medical facility which is a chemotherapy center. Question 1 asked for specific actions upon arrival at the scene. Question 2 indicated that during fireground operations, one of the firefighters falls into an intravenous (IV) stand with IV bags containing doxorubicin, a potent chemotherapy drug, soaking himself with the fluid, and he sustained a head injury. It asked for specific actions to be taken based on this new information.

For this scenario, the SME noted that the appellant failed to monitor or protect the truss roof, which was a mandatory response to question 1. It was also indicated that he missed the opportunity to contact a railroad authority to shut down tracks due to smoke, which were additional responses to question 1. He used the “flex” rule to assign a score of 3. On appeal, the appellant states that he stated the building was wood frame construction, utilized a drone, vented the roof from an aerial ladder, reduced his span of control, assigned group leaders, requested progress reports, fought the fire with handlines, and used Thermal Imaging Cameras.

Regarding the flex rule, mandatory responses are responses that are requirements for a performance to be acceptable (a score of 3). Sometimes, a candidate states many additional responses but does not give a mandatory response. The flex rule was designed to allow the SMEs to assign a score of 3 to candidates who fail to give a mandatory response but who provide many additional responses. However, the SMEs cannot provide a score higher than a 3 in those cases. All mandatory responses must be given in order for a performance to be acceptable, whether there is one mandatory response or five of them. It is not assumed that candidates receive a score of 5 which is then lowered for lack of responses. Performances that include mandatory responses get a score of 3, and those without mandatory responses get a score of 1 or 2. Additional responses only increase a score from 3 to 4 or from 3 to 5.

In reply, none of the actions listed by the appellant on appeal indicates that he stated he would monitor or protect the truss roof, a mandatory response. This was a responsibility of the Incident Commander and the candidate is remiss if he did not do so. Again, credit is not given for information that is implied or assumed. The appellant missed the actions listed by the assessor, including a mandatory response, and his score for the technical component is correct.

A thorough review of appellant’s submissions and the test materials indicates that the decision below is amply supported by the record, and the appellant has failed to meet his burden of proof in this matter.

ORDER

Therefore, it is ordered that this appeal be denied and the appellant’s score for the oral communication component of the Administration scenario be reduced from 5 to 4.

ClosingThis is the final administrative determination in this matter. Any further review should be pursued in a judicial forum.