Raylor Adames, CSC Docket No. 2019-2235

BackgroundRaylor Adames appeals his score for the oral portion of the promotional examination for Fire Captain (PM1046V), Newark. It is noted that the appellant passed the subject examination with a final score of 80.690 and his name appears as the 51st ranked eligible on the subject list.

It is noted for the record that this two-part examination consisted of a written multiple-choice portion and an oral portion. Candidates were required to pass the written portion of the examination, and then were ranked on their performance on both portions of the examination. The test was worth 80 percent of the final score and seniority was worth the remaining 20 percent. Of the test weights, 31.35% of the score was the written multiple-choice portion, 22.49% was the technical score for the evolving exercise, 7.53% was the supervision score for the evolving exercise, 4.28% was the oral communication score for the evolving exercise, 19.23% was the technical score for the arriving exercise, 7.53% was the supervision score for the arriving exercise, and 7.59% was the oral communication score for the arriving exercise.

The oral portion of the Fire Captain examination consisted of two scenarios: a fire scene simulation with questions designed to measure the knowledge of safe rescue tactics and procedures to safeguard citizens, supervision of fire fighters and the ability to assess fire conditions and hazards in an evolving incident on the fireground (evolving); and a fire scene simulation designed to measure the knowledge of safe rescue tactics and procedures to safeguard citizens, supervision of firefighters and the ability to plan strategies and tactics based upon a building’s structure and condition (arriving). Knowledge of supervision was measured by questions in both scenarios, and was scored for each. For the evolving scenario, candidates were provided with a 15-minute preparation period, and candidates had 10 minutes to respond. For the arriving scenario, a five minute preparation period was given and candidates had 10 minutes to respond.

The candidates’ responses were scored on technical knowledge and oral communication ability. Prior to the administration of the exam, a panel of Subject Matter Experts (SMEs) determined the scoring criteria, using generally approved fire command practices, firefighting practices, and reference materials. Scoring decisions were based on SME-approved possible courses of action (PCAs) including those actions that must be taken to resolve the situation as presented. For a performance to be acceptable, a candidate needed to present the mandatory courses of action for that scenario. Only those oral responses that depicted relevant behaviors that were observable and could be quantified were assessed in the scoring process.

Candidates were rated on a five-point scale, with 5 as the optimal response, 4 as a more than acceptable passing response, 3 as a minimally acceptable passing response, 2 as a less than acceptable response, and 1 as a much less than acceptable response. For each of the scenes, and for oral communication, the requirements for each score were defined.

For the arriving scenario, the appellant scored a 2 for the technical component, a 5 for the supervision component, and a 5 for the oral communication component. For the evolving scenario, the appellant scored a 5 for each component. The appellant challenges his score for the technical component of the arriving scenario. As a result, the appellant’s test material, video, and a listing of PCAs for the scenarios were reviewed.

The arriving scenario involved a train derailment. For the technical component, the assessor assigned a score of 2, and noted that the candidate failed to establish command uphill and upwind. He also noted that the appellant missed the opportunity to consult or coordinate with available railway personnel to establish a personnel count. These were PCAs for question 2, which asked for specific actions to take after giving the initial report. On appeal, the appellant states that he established command and said he would take into account the wind speed and direction. He states that he addressed the hazardous materials, called DOT for an investigation, and requested EMS. He also requests an explanation of scoring.

In reply, a review of the file indicates that, for the technical component, the appellant missed the actions noted by the assessor. In this presentation, the candidate is the Incident Commander (IC) as he is the highest ranking officer on scene, the wind is blowing east to west at 10 mph, and there is a significant hydraulic leak on the green train which has been contained. At the end of every scenario and prior to the questions, instructions state, “In responding to the questions, make sure your actions directly relate to the scenario. Do not assume or take for granted that general actions will contribute to your score.” In his response, the appellant stated, “Upon arrival, I would establish command and set up a command post in a secure area located for a multi-sided view of this wreck” The appellant received credit for this response in question 1, which asked candidates to provide an initial report using proper radio protocols. However, this response lacks the detail to provide credit in question 2, which asks for specific actions, as it does not account for the wind or terrain. Given the diagrams, it is also not possible establish a command post in an area which would give a multi-sided view of the trains, which are on tracks. The appellant mentioned weather and wind speed as a size-up factor in his initial evaluation of the scene, but did not place his command upwind.

Next, the appellant simply did not consult or coordinate with available railway personnel to establish a personnel count, and credit cannot be given for information that is implied or assumed. The appellant called for many resources and assigned officers. He called for police, EMS, utilities, a Hazmat unit, a RIT team, search and rescue teams, arson for cause and investigation, a cascade unit, and special operations. He then said, “I would order the Department of Transportation to come in for investigation, as well as any federal entities to help resolve this matter.” None of this information indicates that the appellant would consult or coordinate with available railway personnel to establish a personnel count. The appellant missed the actions noted by the assessor and his score of 2 for this component is correct.

As to scoring, every candidate received a Candidate Feedback Report at examination review which explained standardization in layman’s terms and provided all the calculations relative to the individual candidate. These calculations indicate that the appellant received a weighted test score of 0.14840652338. This number included the scores for all six components of the oral examination, and the score for the written examination. This was then standardized and normalized, and those calculations were provided to candidates. Standardization preserves the relative weighting of each of the components of the examination. Under this process, a standardized z-score represents both the relative position of an individual score in a distribution as compared to the mean and the variation of scores in the distribution. Z-scores will form a distribution identical to the distribution of raw scores; the mean of z-scores will equal zero and the variance of a z-distribution will always be one, as will the standard deviation. This places all scores on the same scale, that is, it provides a score that is directly comparable within and between different types of scores. A negative z-score indicates the score is below the distribution mean, while a positive z-score indicates the score is above the distribution mean. These scores are then “normalized.” Standardization allows for the comparison of scores that are from different normal distributions. When an examination has multiple disparate scores, weighting each of them and adding the weights together results in a nonsensical final average, since each score has a different normal distribution. Since the final score was comprised of a multiple-choice examination, oral scores, and seniority, it was necessary to use standardized scoring. There is no scoring error evident in the calculations.

CONCLUSION

A thorough review of the appellant’s submissions and the test materials indicates that the decision below is amply supported by the record, and the appellant has failed to meet his burden of proof in this matter.

ORDER

Therefore, it is ordered that this appeal be denied.

ClosingThis is the final administrative determination in this matter. Any further review should be pursued in a judicial forum.