Joseph Caley, CSC Docket No. 2019-2356

BackgroundJoseph Caley appeals the test administration and his score for the oral portion of the promotional examination for Fire Officer 1 (PM1099V), Jersey City. It is noted that the appellant passed the subject examination with a final score of 80.020 and his name appears as the 87th ranked eligible on the subject list.

It is noted for the record that this two-part examination consisted of a written multiple-choice portion and an oral portion. Candidates were required to pass the written portion of the examination, and then were ranked on their performance on both portions of the examination. The test was worth 80 percent of the final score and seniority was worth the remaining 20 percent. Of the test weights, 31.35% of the score was the written multiple-choice portion, 22.49% was the technical score for the evolving exercise, 7.53% was the supervision score for the evolving exercise, 4.28% was the oral communication score for the evolving exercise, 19.23% was the technical score for the arriving exercise, 7.53% was the supervision score for the arriving exercise, and 7.59% was the oral communication score for the arriving exercise.

The oral portion of the Fire Officer 1 examination consisted of two scenarios: a fire scene simulation with questions designed to measure the knowledge of safe rescue tactics and procedures to safeguard citizens, supervision of fire fighters and the ability to assess fire conditions and hazards in an evolving incident on the fireground (evolving); and a fire scene simulation designed to measure the knowledge of safe rescue tactics and procedures to safeguard citizens, supervision of firefighters and the ability to plan strategies and tactics based upon a building’s structure and condition (arriving). Knowledge of supervision was measured by questions in both scenarios, and was scored for each. For the evolving scenario, candidates were provided with a 15-minute preparation period, and candidates had 10 minutes to respond. For the arriving scenario, a five-minute preparation period was given, and candidates had 10 minutes to respond.

The candidates’ responses were scored on technical knowledge and oral communication ability. Prior to the administration of the exam, a panel of Subject Matter Experts (SMEs) determined the scoring criteria, using generally approved fire command practices, firefighting practices, and reference materials. Scoring decisions were based on SME-approved possible courses of action (PCAs) including those actions that must be taken to resolve the situation as presented. For a performance to be acceptable, a candidate needed to present the mandatory courses of action for that scenario. Only those oral responses that depicted relevant behaviors that were observable and could be quantified were assessed in the scoring process.

Candidates were rated on a five-point scale, with 5 as the optimal response, 4 as a more than acceptable passing response, 3 as a minimally acceptable passing response, 2 as a less than acceptable response, and 1 as a much less than acceptable response. For each of the scenes, and for oral communication, the requirements for each score were defined.

For the evolving scenario, the appellant scored a 4 for the technical component, a 4 for the supervision component, and a 5 for the oral communication component. For the arriving scenario, the appellant scored a 3 for the technical component, a 1 for the supervision component, and a 5 for the oral communication component. The appellant challenges his scores for the technical component of the arriving scenario. He also challenges the test administration. As a result, the appellant’s test material, video, and a listing of PCAs for the scenarios were reviewed.

The arriving scenario involved a train/pedestrian collision. The pedestrian is sprawled on the road and not moving. She is reported to be deceased by a member of the railroad’s company maintenance crew. Some train passengers were thrown forward in the incident and report injuries. Question 1 asked candidates to perform an initial report upon arrival using proper radio protocol. Question 2 asked for specific actions to be taken after the initial report.

For the technical component, the assessor indicated that the appellant failed to ensure all rail service is shut down on the line/flaggers, which was a mandatory response to question 2. He indicated that the appellant missed the opportunity to ensure engines were turned off, which was another response for question 2. The assessor assigned a score of 3 using the “flex rule.” On appeal, the appellant states that he notified PSE&G and turned off power to the train.

Regarding the flex rule, mandatory responses are responses that are requirements for a performance to be acceptable (a score of 3). Sometimes, a candidate states many additional responses but does not give a mandatory response. The flex rule was designed to allow the SMEs to assign a score of 3 to candidates who fail to give a mandatory response but who provide many additional responses. However, the SMEs cannot provide a score higher than a 3 in those cases. All mandatory responses must be given in order for a performance to be acceptable, whether there is one mandatory response or five of them. It is not assumed that candidates receive a score of 5 which is then lowered for lack of responses. Performances that include mandatory responses get a score of 3, and those without mandatory responses get a score of 1 or 2, unless the flex rule is used. Additional responses only increase a score from 3 to 4 or from 3 to 5.

In reply, a review of the file indicates that the appellant indicated stated, “I will also require, er, authorities from the train service. I would request Public Service Gas and Electric to, maybe utilities in the area, especially electric, make sure there is no electric on the train. Make sure the electrical power has been ah, shut down.” At the end of every scenario and prior to the questions, instructions state, “In responding to the questions, make sure your actions directly relate to the scenario. Do not assume or take for granted that general actions will contribute to your score.” The appellant did not ensure that engines were turned off, or stop all incoming trains by ensuring all rail service is shut down on the line or requesting flaggers. Shutting down all track power does not ensure that all rail service is shut down on the line, as diesel trains may be in use. Additionally, the appellant did not mention the engines at all.

As to test administration, the appellant appealed that he could hear the responses being given by another candidate through the walls. He stated that during the arriving scenario he had to pause as it became loud, and he was unable to concentrate. He states that the voices were loud and constant, and he was interrupted by them. He states that this was the reason he was distracted and “unable to answer the question.” He argues that this was unfair, but he does not request a remedy.

In reply, as a result of the appeal, the appellant’s presentation was reviewed in its entirety, including the instructions and the preparation period. Muffled voices can be heard on the video starting from the middle of the instructions given for the first scenario. The appellant made no mention of the test conditions and continued with his presentation. His demeanor was confident, and he gave enough information to receive scores of 4 for the technical and supervision components for the evolving scenario.

At the conclusion of that presentation, the monitor told him he would have five minutes for the preparation, and ten minutes to respond to all of the questions, and he would be given a two-minute warning for the preparation period and the presentation. The appellant was given his papers for the arriving scenario and wrote his applicant identification number on each. He was asked if he had any questions before beginning and he responded, “Just how many minutes I’m going to have. Just again.” The monitor responded, “Yup, you’ll have five minutes for the preparation period,” and the appellant interjected, “Okay.” The monitor stated, “And after that you’ll have ten minutes to respond to all of the questions and I will give you two-minute warnings for both the preparation time and the response time.” The appellant replied, “Very good. Thank you.” The monitor said, “Alright, any other questions before we begin?” and the appellant said, “No, I’m good thank you.” The appellant does not mention the voices through the walls or make any mention of being distracted, or appear to be distracted. After reading the scenario, the monitor repeated, “You have a total of ten minutes to respond to all questions, not ten minutes for each question.”

During the preparation period, the appellant continues to take notes although voices can be heard to start up a little over half way through. At the end of the preparation period, the monitor stated, “You will have ten minutes to respond to all questions, not ten minutes for each,” and the time was started. The appellant took about four minutes to respond to the technical questions, 1 and 2. He provided many responses to these questions and did not appear to be distracted or confused. He concluded with, “Ah, and that’s pretty much ah, the end of my evolving scenario.” The monitor asked, “Is there anything else you would like to add to questions 1, 2 or 3?” and the appellant responded, “No, that’s it.” The appellant did not address question 3, the supervision question, and received a score of 1 for that component.

At the test center, the monitor and Center Supervisor take notes of unusual occurrences. In this case, the appellant filed a test administration appeal after completing his presentations, and this was noted by the monitor. The monitor also noted that the candidate said under his breath that he forgot to answer question three as he left the room. He also stated that the fan was on to create white noise. A review of this presentation indicates that the voices through the wall were heard, but they were not so egregious that the appellant thought he should mention them during his examination. He had plenty of opportunity to indicate that he could not continue under the circumstances or that he was being distracted. He did not state any complaints or mention it. The voices were not louder during the arriving scenario than they were during the evolving scenario. The appellant’s argument that he was unable to concentrate is unpersuasive. The monitor told the appellant four times that he was to answer all the questions in the ten-minute time period, including when the candidate asked about the time, and this information was repeated in the instructions for the evolving scenario. When asked if he had any other questions prior to the start of the arriving scenario, the appellant did not mention the voices through the wall. While the Commission strives to provide the best testing environment it can, candidates have a responsibility to speak up if there is a situation at the test center that they feel affects their performance. Given the situation in its entirety, there is no basis which warrants a make-up examination.

CONCLUSION

A thorough review of the appellant’s submissions and the test materials indicates that the decision below is amply supported by the record, and the appellant has failed to meet his burden of proof in this matter.

ORDER

Therefore, it is ordered that this appeal be denied.

ClosingThis is the final administrative determination in this matter. Any further review should be pursued in a judicial forum.