Isaac Feliciano, CSC Docket No. 2023-1958

BackgroundIsaac Feliciano appeals his score on the promotional examination for Battalion Fire Chief (PM3390C), Paterson. It is noted that the appellant passed the examination with a final average of 80.480 and ranks 38th on the eligible list.

The subject promotional examination was held on May 23, 2022, and 45 candidates passed. This two-part examination consisted of an integrated system of simulations designed to generate behavior similar to that required for success on the job. The first part consisted of multiple-choice items that measured specific work components identified and weighted by the job analysis. The second part consisted of three oral scenarios: Supervision, Administration and Incident Command. The examination was based on a comprehensive job analysis conducted by the Civil Service Commission (Commission), which identified the critical areas of the job. The weighting of the test components was derived from the job analysis data. It is noted that candidates were told the following prior to beginning their presentations for each scenario: “In responding to the questions, be as specific as possible. Do not assume or take for granted that general actions will contribute to your score.”

Each candidate in a given jurisdiction was scored by a team of three different Subject Matter Experts (SMEs), who were trained in current technical scoring procedures. Each of these SMEs were current or retired fire officers who held the title of Battalion Fire Chief (or Fire Officer 2) or higher. Candidates were also assessed by three New Jersey Civil Service Commission employees trained in oral communication assessment. As part of the scoring process, an SME observed and noted the responses of a candidate relative to the knowledge, skills and abilities (KSAs) that each exercise was designed to measure. An assessor also noted any weaknesses that detracted from the candidate’s overall oral communication ability. Each assessor then rated the candidate’s performance according to the rating standards and assigned the candidate a technical or oral communication score on that exercise.

In order to preserve the relative weighting of each of the components of the examination, the ratings for each portion were adjusted by a well-recognized statistical process known as “standardization.” Under this process, the ratings are standardized by converting the raw scores to z-scores, an expression of the deviation of the score from the mean score of the group in relation to the standard deviation of scores for the group. Each portion of the examination had a relative weight in its relation to the whole examination. Thus, the z-score for the multiple-choice portion was multiplied by a test weight of 36.53%, the oral technical scores were multiplied by a test weight of 53.91% and the oral communication scores were multiplied by a test weight of 9.56%. The weighted z-scores were summed and this became the overall final test score. This was weighted and added to the weighted seniority score. The result was standardized, then normalized, and rounded up to the third decimal place to arrive at a final average.

For the Supervision scenario, the appellant scored a 5 on the technical component and a 3 on the oral communication component. On the Administration scenario, the appellant scored a 3 on the technical component and a 4 on the oral communication component. Finally, on the Incident Command scenario, the appellant scored a 5 on the technical component and a 3 on the oral communication component.

Appellant's argumentThe appellant challenges his scores for the oral communication components of the Supervision and Incident Command scenarios, as well as the technical component of the Administration scenario. As a result, the appellant’s test material and a listing of possible courses of action (PCAs) for the scenarios were reviewed.

On appeal, the appellant claims that he obtained legal counsel 1 and that counsel agreed that the categories were “subjective,” “graded with extreme bias” and that certain rules, like “not being allowed to remove staples from packets that contain the necessary notes to review,” were unreasonable. The appellant also complains that the testing room had “distracting noises during the exam.”

With respect to the oral communication component of the Supervision scenario, the assessor found that the appellant displayed a major weakness in word usage/grammar as evidenced by the candidate’s use of “um” and “uh” 33 times during his response and his use of words in an inappropriate manner. Regarding the latter, the assessor provided several examples, including the appellant stating “have him write off on it . . .” instead of “sign off” and “stating recommend a suspension and retrain . . .” instead of “retraining.” Based upon the foregoing, the assessor awarded the appellant a score of 3.

Background1 It is noted that the appellant did not use counsel to file the instant appeal.

As to the oral communication component of the Incident Command scenario, the assessor found that the appellant displayed a minor weakness in word usage grammar, as evidenced by the candidate’s use of filler words like “um” and “uh” 46 times. In addition, the assessor cited a minor weakness in nonverbal communication, as evidenced by the appellant’s failure to make sufficient eye contact by excessively referring to notes and looking off to the side or below the camera. Based upon the foregoing, the assessor awarded the appellant a score of 3.

For the oral communication component of the Supervision scenario, the appellant challenges his score, particularly as it relates to grammar. Additionally, the appellant asserts that the record does not support his score of 3 on the oral communication component of the Incident Command scenario, as he made sufficient eye contact during his presentation. The appellant attributes his looking to the left of the camera to a machine in the room making a distracting noise and he attributes his need to look down to being barred from removing the staples to separate his notes from inside of the test booklet.

The Administration scenario involves the candidate investigating an incident between Fire Fighter Hernandez and a Police Officer which culminated in the arrest of Fire Fighter Hernandez at the scene of a car accident where the candidate was serving as the incident commander. For the appellant’s technical component score for the Administration scenario, the SME indicated that the appellant missed a number of PCAs, including, in part, the opportunity to offer the Employee Assistance Program (EAP) and the opportunity to review the National Fire Incident Reporting System (NFIRS). Based upon the foregoing, the SME awarded the appellant a score of 3.

For the technical component of the Administration scenario, the appellant argues that he covered the PCAs at issue by stating during his presentation that Fire Fighter Hernandez had the right to union representation, programs for employees and counsel and by indicating that he would help him obtain programs that could help and assist.

CONCLUSION

RuleN.J.A.C. 4A:4-6.4(c) provides that an examination candidate wishing to challenge the manner in which the examination was administered must file an appeal in writing at the examination site on the day of the examination.

AnalysisAt the outset, the appellant’s appeal of the conditions at the test site, including any distracting noises and the requirement that he leave the staples from the testing book are moot, as they were not raised at the examination site on the day of examination, as required pursuant to N.J.A.C. 4A:4-6.4(c).

With regard to the appellant’s challenge to his score of 3 on the oral communication component of the Supervision scenario, the Commission finds that the appellant has failed to sustain his burden of proof. A review of the recording of the appellant’s presentation for this scenario clearly demonstrates that he displayed a major weakness in word/usage/grammar based upon his use of filler words throughout his response and the grammar issues cited by the assessor. Accordingly, his score of 3 for this component is sustained.

Similarly, as to the appellant’s challenge of his score of 3 on the oral communication component of the Incident Command scenario, the record supports the finding that the appellant displayed a minor weakness in word usage/grammar based upon his use of filler words including “uh” and “um” throughout his presentation. and in nonverbal communication. A review of the appellant’s presentation supports the assessor’s conclusion that the appellant displayed a minor weakness in nonverbal communication given his failure to maintain consistent eye contact with the camera. The Commission acknowledges that a low hum is present in the recording for approximately one minute and 42 seconds during his presentation. However, the appellant regularly looked down at his notes or away from the camera both before and after this background noise ceased. Thus, even if the Commission were to evaluate the appellant’s eye contact only during the periods before and after this room noise were present, it would still support the assessor’s conclusion that the appellant displayed a minor weakness in nonverbal communication. Accordingly the appellant’s score of 3 for the oral communication component of the Incident Command scenario is sustained.

With regard to the technical component of the Administration scenario, the appellant argues that he covered the PCAs of offering EAP and reviewing NFIRS by stating that Fire Fighter Hernandez had the right to union representation, programs for employees and counsel and by indicating that he would help him obtain programs that could help and assist. A review of the appellant’s presentation indicates that the appellant only spoke about the appeals process, due process, hearings and advising Fire Fighter Hernandez about his option to obtain legal assistance. Thus, his statements were insufficient to credit him with the PCA of offering EAP. Further, the appellant did not state that he would review NFIRS. Moreover, the record does not reveal that there were any other PCAs for which the appellant should have received credit. Therefore, the appellant’s score of 3 for the technical component of the Administration scenario is appropriate.

Accordingly, based upon the foregoing, a thorough review of the appellant’s submissions and the test materials indicates that the decision below is amply supported by the record, and the appellant has failed to meet his burden of proof in this matter.

ORDER

Therefore, it is ordered that this appeal be denied.

ClosingThis is the final administrative determination in this matter. Any further review should be pursued in a judicial forum.