James Pappas, CSC Docket No. 2023-2064

BackgroundJames Pappas appeals his score on the promotional examination for Fire Officer 2 (PM4200C), Jersey City. It is noted that the appellant passed the examination with a final average of 84.810 and ranks 29th on the eligible list.

The subject promotional examination was held on May 19, 2022, and 39 candidates passed. This two-part examination consisted of an integrated system of simulations designed to generate behavior similar to that required for success on the job. The first part consisted of multiple-choice items that measured specific work components identified and weighted by the job analysis. The second part consisted of three oral scenarios: Supervision, Administration and Incident Command. The examination was based on a comprehensive job analysis conducted by the Civil Service Commission (Commission), which identified the critical areas of the job. The weighting of the test components was derived from the job analysis data. It is noted that candidates were told the following prior to beginning their presentations for each scenario: “In responding to the questions, be as specific as possible. Do not assume or take for granted that general actions will contribute to your score.”

Each candidate in a given jurisdiction was scored by a team of three different Subject Matter Experts (SMEs), who were trained in current technical scoring procedures. Each of these SMEs were current or retired fire officers who held the title of Battalion Fire Chief (or Fire Officer 2) or higher. Candidates were also assessed by three Commission employees trained in oral communication assessment. As part of the scoring process, an SME observed and noted the responses of a candidate relative to the knowledge, skills and abilities (KSAs) that each exercise was designed to measure. An assessor also noted any weaknesses that detracted from the candidate’s overall oral communication ability. Each assessor then rated the candidate’s performance according to the rating standards and assigned the candidate a technical or oral communication score on that exercise.

In order to preserve the relative weighting of each of the components of the examination, the ratings for each portion were adjusted by a well-recognized statistical process known as “standardization.” Under this process, the ratings are standardized by converting the raw scores to z-scores, an expression of the deviation of the score from the mean score of the group in relation to the standard deviation of scores for the group. Each portion of the examination had a relative weight in its relation to the whole examination. Thus, the z-score for the multiple-choice portion was multiplied by a test weight of 36.53%, the oral technical scores were multiplied by a test weight of 53.91% and the oral communication scores were multiplied by a test weight of 9.56%. The weighted z-scores were summed and this became the overall final test score. This was weighted and added to the weighted seniority score. The result was standardized, then normalized, and rounded up to the third decimal place to arrive at a final average.

Each oral examination question, and overall oral communication, was rated on a five-point scale, with 5 as the optimal response, 4 as a more than acceptable passing response, 3 as a minimally acceptable passing response, 2 as a less than acceptable response, and 1 as a much less than acceptable response.

On the Supervision scenario, the appellant scored a 5 on the technical component and a 5 on the oral communication component. On the Administration scenario, the appellant scored a 2 on the technical component and a 5 on the oral communication component. Finally, on the Incident Command scenario, the appellant scored a 5 on the technical component and a 5 on the oral communication component.

Appellant's argumentOn appeal, the appellant challenges his score on the technical component of the Administration scenario. As a result, the appellant’s test material and a listing of possible courses of action (PCAs) for the scenario were reviewed.

CONCLUSION

Initially, it is noted that when a candidate challenges a score on an oral examination component, the Commission reviews that entire component to ensure it has been scored correctly.

The Administration scenario involves the candidate being tasked with investigating an incident and revising the department’s current pre-incident action plan procedures following a call where a pre-action plan failed to reflect a building’s conversion and the addition of partition walls inside of the structure. Question 1 asks what specific steps the candidate would take to investigate the incident and the lack of updated pre-incident action plan procedures. Question 2 asks what should be included in an updated pre-incident action plan standard operating guidelines/procedures (SOGs/SOPs).

On the Administration scenario, the assessor found that the appellant missed a significant number of possible courses of action (PCA), including, in part, opportunities to interview the incident commander who was present on scene and to contact the building owner in response to Question 1 and missed the opportunity to identify information about the building in response to Question 2. Based upon the foregoing, the assessor awarded the appellant a technical component score of 2. On appeal, the appellant argues that by stating he would “meet with the chief,” he covered the PCA of interviewing the incident commander present on scene. As to the PCA of contacting the building owner, the appellant avers that he addressed it by stating that he would set up a meeting with Fire Prevention. In this regard, he proffers that Fire Prevention would be the entity that would evaluate pre-incident planning for fire plans and that Fire Prevention would contact the building owner as part of this process. The appellant also notes that he stated that he would develop a pre-incident plan with a committee. He proffers that such pre-incident planning would include gathering data regarding building information, including contacting the owner. The appellant further describes being marked down for both failing to contact the building owner and failing to identify information about the building as a “double jeopardy mark against [him],” since the owner is identified in the tax records for every building as well. He adamantly argues that bringing in Fire Prevention to evaluate the structure and developing a pre-incident plan with a committee was an all-encompassing answer to both questions for the scenario.

In reply, as noted above, candidates were told the following prior to beginning their presentations for each scenario: “In responding to the questions, be as specific as possible. Do not assume or take for granted that general actions will contribute to your score.” The statements cited by the appellant lacked the specificity needed to clearly establish that he was performing the actions at issue. Stating that he would meet with the chief could be interpreted as a statement that he would meet with the Fire Chief, who may not have been the incident commander present on scene. As to the other two PCAs at issue, the appellant misapprehends the distinction between them. The PCA of contacting the building owner in Question 1 involved the specific building from the fire incident. The PCA of getting all information about the building in Question 2 refers to an item that should be included in departmental pre-incident action plan SOGs/SOPs, rather than the investigation of the incident that precipitated the review presented in the prompt. Beyond this, his statements about bringing in Fire Prevention to evaluate the structure and developing a pre-incident plan with a committee were clearly too general to establish that he was performing the specific actions of contacting the building owner and including all information about the building in pre-incident action plan SOPs/SOGs. Further, a review of the appellant’s presentation fails to otherwise demonstrate that he covered these actions. Moreover, a review of the appellant’s presentation reveals that he was erroneously credited with the PCA of reviewing the latest and past incident reports at the location of the underlying incident and the PCA of reviewing current SOPs/SOGs regarding the pre-incident action plan. As such, the appellant has failed to sustain his burden of proof, his credit for these PCAs shall be stricken and his technical component score for the Administration scenario shall be reduced to 1.

A thorough review of the appellant’s submissions and the test materials indicates that the decision below is amply supported by the record, and the appellant has failed to meet his burden of proof in this matter.

ORDER

Therefore, it is ordered that this appeal be denied. Additionally, it is ordered that the appellant’s score on the technical component of the Administration scenario be reduced from 2 to 1. It is further ordered that this scoring change be given retroactive effect.

ClosingThis is the final administrative determination in this matter. Any further review should be pursued in a judicial forum.