Kenneth Perez, CSC Docket No. 2023-2069

BackgroundKenneth Perez appeals his score on the promotional examination for Battalion Fire Chief (PM3388C), Newark. It is noted that the appellant passed the examination with a final average of 82.580 and ranks 19th on the eligible list.

The subject promotional examination was held on May 23, 2022, and 39 candidates passed. This two-part examination consisted of an integrated system of simulations designed to generate behavior similar to that required for success on the job. The first part consisted of multiple-choice items that measured specific work components identified and weighted by the job analysis. The second part consisted of three oral scenarios: Supervision, Administration and Incident Command. The examination was based on a comprehensive job analysis conducted by the Civil Service Commission (Commission), which identified the critical areas of the job. The weighting of the test components was derived from the job analysis data. It is noted that candidates were told the following prior to beginning their presentations for each scenario: “In responding to the questions, be as specific as possible. Do not assume or take for granted that general actions will contribute to your score.”

Each candidate in a given jurisdiction was scored by a team of three different Subject Matter Experts (SMEs), who were trained in current technical scoring procedures. Each of these SMEs were current or retired fire officers who held the title of Battalion Fire Chief (or Fire Officer 2) or higher. Candidates were also assessed by three Commission employees trained in oral communication assessment. As part of the scoring process, an SME observed and noted the responses of a candidate relative to the knowledge, skills and abilities (KSAs) that each exercise was designed to measure. An assessor also noted any weaknesses that detracted from the candidate’s overall oral communication ability. Each assessor then rated the candidate’s performance according to the rating standards and assigned the candidate a technical or oral communication score on that exercise.

In order to preserve the relative weighting of each of the components of the examination, the ratings for each portion were adjusted by a well-recognized statistical process known as “standardization.” Under this process, the ratings are standardized by converting the raw scores to z-scores, an expression of the deviation of the score from the mean score of the group in relation to the standard deviation of scores for the group. Each portion of the examination had a relative weight in its relation to the whole examination. Thus, the z-score for the multiple-choice portion was multiplied by a test weight of 36.53%, the oral technical scores were multiplied by a test weight of 53.91% and the oral communication scores were multiplied by a test weight of 9.56%. The weighted z-scores were summed and this became the overall final test score. This was weighted and added to the weighted seniority score. The result was standardized, then normalized, and rounded up to the third decimal place to arrive at a final average.

On the Supervision scenario, the appellant scored a 4 on the technical component and a 4 on the oral communication component. On the Administration scenario, the appellant scored a 3 on the technical component and a 4 on the oral communication component. Finally, on the Incident Command scenario, the appellant scored a 5 on the technical component and a 3 on the oral communication component.

Appellant's argumentThe appellant challenges his scores for the technical components of the Supervision and Administration scenarios. As a result, the appellant’s test material and a listing of possible courses of action (PCAs) for the scenarios were reviewed.

The Supervision scenario involves the candidate’s subordinate, a Fire Captain, violating the departmental leave policy by being absent without leave approval from a supervisor. Question 1 asks what specific actions the candidate, as the Battalion Fire Chief supervising the said Fire Captain, should take. Question 2 indicates that while investigating, the candidate discovers that the same Battalion Fire Chief was disciplined for violating the leave policy in the same manner nine months earlier and asks, based on this information, what specific topics should be discussed in a meeting with the Fire Captain and what specific actions should be taken.

On the Supervision scenario, the SME indicated that the appellant missed a number of PCAs, including the opportunities to get a written report from the Fire Captain and to notify the candidate’s immediate supervisor. Based upon the foregoing, the SME awarded the appellant a score of 4 for the technical component of the subject scenario. On appeal, the appellant argues that he should have received credit for the PCA at issue because he stated that he would get a written report from the Fire Captain at two different points during his presentation.

In reply, upon review of the appellant’s presentation on appeal, the Division of Test Development, Analytics and Administration (TDAA) agrees that the appellant should have been credited with the PCA of getting a written report from the Fire Captain. Additionally, TDAA has advised that the appellant should have been credited with the PCA of notifying his immediate supervisor. Based upon the foregoing, TDAA submits that the appellant’s score for the technical component of the Supervision scenario should be increased from 4 to 5. The Commission agrees with TDAA’s findings on appeal for this scenario.

The Administration scenario involves the candidate investigating an incident between Fire Fighter Hernandez and a Police Officer which culminated in the arrest of Fire Fighter Hernandez at the scene of a car accident where the candidate was serving as the incident commander. Question 1 asks what specific steps the candidate would take to investigate the incident between Fire Fighter Hernandez and the Police Officer. Question 2 states that the candidate has learned that Fire Fighter Hernandez is considering filing a civil suit and asks the candidate what actions should be taken concerning Fire Fighter Hernandez based upon this new information.

On the Administration scenario, the SME indicated that the appellant missed a number of PCAs, including, in part, opportunities to conduct a follow-up meeting with Fire Fighter Hernandez to offer support and to review the National Fire Incident Reporting System (NFIRS). Based upon the foregoing, the SME awarded the appellant a score of 3 for the technical component of the Administration scenario. On appeal, the appellant argues that he covered the PCA of conducting a follow-up meeting to show support with two statements he made at specific points during his presentation. He also contends that he covered this PCA by offering counseling and the Employee Assistance Program. The appellant avers that he covered reviewing NFIRS by stating that he would obtain and review his documents, as NFIRS is an official document.

In reply, a review of the appellant’s response to the Administration scenario fails to demonstrate that he should have been credited with the subject PCAs. As noted above, candidates were told the following prior to beginning their presentations for each scenario: “In responding to the questions, be as specific as possible. Do not assume or take for granted that general actions will contribute to your score.” A general statement about reviewing documents does not specifically convey that one would review NFIRS. As such, the appellant’s argument that he should have received credit for reviewing NFIRS is without merit. As to the PCA of conducting a follow-up meeting with Fire Fighter Hernandez to offer support, it is noted that this was a PCA in response to the candidate learning that Fire Fighter Hernandez was considering filing a civil suit. The statements cited by the appellant all related to meeting with Fire Fighter Hernandez and the arresting officer in furtherance of the appellant’s investigation of the incident. Notably, the appellant’s second statement about a meeting, which came roughly five minutes into his presentation was: “As far as question number 1, I would identify . . . uh . . . uh . . . identify . . . excuse me . . . I would have a meeting with both parties. As far as continuing with my investigation . . . " Thus, it could not reasonably be said to be a follow-up meeting to offer support to Fire Fighter Hernandez based upon the news that he is contemplating filing a civil suit. Accordingly, the appellant has failed to sustain his burden of proof regarding the Administration scenario and his technical component score of 3 is sustained.

CONCLUSION

A thorough review of the appellant’s submissions and the test materials indicates that the decision below is amply supported by the record, and that except for the technical component of the Supervision scenario, as indicated above, the appellant has failed to meet his burden of proof in this matter.

ORDER

Therefore, it is ordered that this appeal be granted in part and that the appellant’s score for the technical component of the Supervision scenario be raised from 4 to 5. It is further ordered that this scoring change be given retroactive effect.

ClosingThis is the final administrative determination in this matter. Any further review should be pursued in a judicial forum.