Frank Phillips III, CSC Docket No. 2023-2090

BackgroundFrank Phillips III appeals his score on the promotional examination for Battalion Fire Chief (PM4444C), Atlantic City. It is noted that the appellant passed the examination with a final average of 89.140 and ranks fourth on the eligible list.

The subject promotional examination was held on May 25, 2022, and 27 candidates passed. This two-part examination consisted of an integrated system of simulations designed to generate behavior similar to that required for success on the job. The first part consisted of multiple-choice items that measured specific work components identified and weighted by the job analysis. The second part consisted of three oral scenarios: Supervision, Administration and Incident Command. The examination was based on a comprehensive job analysis conducted by the Civil Service Commission (Commission), which identified the critical areas of the job. The weighting of the test components was derived from the job analysis data. It is noted that candidates were told the following prior to beginning their presentations for each scenario: “In responding to the questions, be as specific as possible. Do not assume or take for granted that general actions will contribute to your score.”

Each candidate in a given jurisdiction was scored by a team of three different Subject Matter Experts (SMEs), who were trained in current technical scoring procedures. Each of these SMEs were current or retired fire officers who held the title of Battalion Fire Chief (or Fire Officer 2) or higher. Candidates were also assessed by three Commission employees trained in oral communication assessment. As part of the scoring process, an SME observed and noted the responses of a candidate relative to the knowledge, skills and abilities (KSAs) that each exercise was designed to measure. An assessor also noted any weaknesses that detracted from the candidate’s overall oral communication ability. Each assessor then rated the candidate’s performance according to the rating standards and assigned the candidate a technical or oral communication score on that exercise.

In order to preserve the relative weighting of each of the components of the examination, the ratings for each portion were adjusted by a well-recognized statistical process known as “standardization.” Under this process, the ratings are standardized by converting the raw scores to z-scores, an expression of the deviation of the score from the mean score of the group in relation to the standard deviation of scores for the group. Each portion of the examination had a relative weight in its relation to the whole examination. Thus, the z-score for the multiple-choice portion was multiplied by a test weight of 36.53%, the oral technical scores were multiplied by a test weight of 53.91% and the oral communication scores were multiplied by a test weight of 9.56%. The weighted z-scores were summed and this became the overall final test score. This was weighted and added to the weighted seniority score. The result was standardized, then normalized, and rounded up to the third decimal place to arrive at a final average.

On the Supervision scenario, the appellant scored a 5 on the technical component and a 4 on the oral communication component. On the Administration scenario, the appellant scored a 5 on the technical component and a 5 on the oral communication component. Finally, on the Incident Command scenario, the appellant scored a 2 on the technical component and a 5 on the oral communication component.

Appellant's argumentOn appeal, the appellant challenges his score for the technical component of the Incident Command scenario. As a result, the appellant’s test material and a listing of possible courses of action (PCAs) for the scenario were reviewed.

The Incident Command scenario involves a response to a report of fire at a local movie theater, part of which is in the process of a renovation. Question 1 asks what specific actions the candidate would take upon arriving at the scene. The prompt for Question 2 states that the party wall separating two theaters collapses during firefighting operations, trapping two firefighters. Question 2 then asks what specific actions the candidate should now take based on this new information.

The SME awarded the appellant a score of 2 for the technical component of the Incident Command scenario, finding that the appellant failed to identify a number of PCAs, including the mandatory actions of conducting a primary search in response to Question 1 and ensuring the removal of trapped firefighters in response to Question 2, plus the opportunity to ensure the monitoring of the air. On appeal, the appellant argues that he covered conducting a primary search by stating at a specified point that there would be a primary and a secondary search. Additionally, the appellant contends that he addressed the removal of trapped firefighters in response to Question 2 by stating that a rapid intervention crew (RIC) would be activated and rescue and remove the trapped firefighters.

In reply, upon review of the appellant’s appeal, the Division of Test Development, Analytics and Administration has advised that the appellant should have been awarded credit for the subject mandatory responses, but that he was erroneously awarded credit for the additional response of declaring that only emergency radio traffic is to be transmitted on the main fireground frequency1. The Commission agrees with this assessment. Based upon the foregoing scoring changes, the appellant’s technical component score on the Incident Command scenario shall be increased from 2 to 5.

CONCLUSION

A thorough review of the appellant’s submissions and the test materials indicates that the decision below is amply supported by the record.

ORDER

Therefore, it is ordered that this appeal be granted and that the appellant’s score for the technical component of the Incident Command scenario be raised from 2 to 5. It is further ordered that this scoring change be given retroactive effect.

ClosingThis is the final administrative determination in this matter. Any further review should be pursued in a judicial forum.