R.B., CSC Docket No. 2023-2065

BackgroundR.B., represented by Michael L. Prigoff, Esq., appeals his score on the promotional examination for Battalion Fire Chief (PM3390C), Paterson. It is noted that the appellant passed the examination with a final average of 85.780 and ranks 12th on the eligible list.

The subject promotional examination was held on May 23, 2022, and 45 candidates passed. This two-part examination consisted of an integrated system of simulations designed to generate behavior similar to that required for success on the job. The first part consisted of multiple-choice items that measured specific work components identified and weighted by the job analysis. The second part consisted of three oral scenarios: Supervision, Administration and Incident Command. The examination was based on a comprehensive job analysis conducted by the Civil Service Commission (Commission), which identified the critical areas of the job. The weighting of the test components was derived from the job analysis data. It is noted that candidates were told the following prior to beginning their presentations for each scenario: “In responding to the questions, be as specific as possible. Do not assume or take for granted that general actions will contribute to your score.”

Each candidate in a given jurisdiction was scored by a team of three different Subject Matter Experts (SMEs), who were trained in current technical scoring procedures. Each of these SMEs were current or retired fire officers who held the title of Battalion Fire Chief (or Fire Officer 2) or higher. Candidates were also assessed by three New Jersey Civil Service Commission employees trained in oral communication assessment. As part of the scoring process, an SME observed and noted the responses of a candidate relative to the knowledge, skills and abilities (KSAs) that each exercise was designed to measure. An assessor also noted any weaknesses that detracted from the candidate’s overall oral communication ability. Each assessor then rated the candidate’s performance according to the rating standards and assigned the candidate a technical or oral communication score on that exercise.

In order to preserve the relative weighting of each of the components of the examination, the ratings for each portion were adjusted by a well-recognized statistical process known as “standardization.” Under this process, the ratings are standardized by converting the raw scores to z-scores, an expression of the deviation of the score from the mean score of the group in relation to the standard deviation of scores for the group. Each portion of the examination had a relative weight in its relation to the whole examination. Thus, the z-score for the multiple-choice portion was multiplied by a test weight of 36.53%, the oral technical scores were multiplied by a test weight of 53.91% and the oral communication scores were multiplied by a test weight of 9.56%. The weighted z-scores were summed and this became the overall final test score. This was weighted and added to the weighted seniority score. The result was standardized, then normalized, and rounded up to the third decimal place to arrive at a final average.

For the Supervision scenario, the appellant scored a 5 on the technical component and a 3 on the oral communication component. On the Administration scenario, the appellant scored a 4 on the technical component and a 3 on the oral communication component. Finally, on the Incident Command scenario, the appellant scored a 3 on the technical component and a 3 on the oral communication component.

Appellant's argumentThe appellant challenges his scores for the oral communication components of the Supervision, Administration and Incident Command scenarios and the technical component of the Incident Command scenario. As a result, the appellant’s test material and a listing of possible courses of action (PCAs) for the scenarios were reviewed.

With regard to the oral communication scores, the appellant maintains that his scores should accommodate a condition that he suffers from and submits a letter from a licensed professional attesting to the said condition. N.J.A.C. 4A:4-6.4 requires an examination candidate challenging the administration of the examination to do so in writing at the examination site on the day of the examination. Based upon the foregoing, the appellant’s challenge, as it relates to his asserted condition, is moot as untimely. Regardless, N.J.A.C. 4A:4-2.14(a) provides that otherwise qualified applicants with disabilities may request an accommodation in taking an examination by indicating their request for accommodation on the examination application. Under

N.J.A.C. 4A:4-2.14, if the appellant needed an accommodation due to a disability, he needed to make this request at the time he submitted his application so that this agency could verify the need for the accommodation and make appropriate arrangements for the accommodation. In this matter, the appellant waited until filing his appeal in March 2023 to request an accommodation.

As to the appellant’s substantive challenge to the oral communication component of the Supervision scenario, the assessor found that the appellant displayed a major weakness in organization, as evidenced by eight pauses during the response and a minor weakness in word usage/grammar, as evidenced by the appellant’s use of “um” and “uh” eight times during the response. Based upon the foregoing, the assessor awarded the appellant a score of 3 for the oral communication component of this scenario. On appeal, the appellant argues that he should have been awarded a score of 4. The Commission notes that the appellant’s argument appears to be based upon a mistaken belief that the appellant only displayed a minor weakness in word usage/grammar.

In response, a review of the appellant’s presentation recording demonstrates that a score of 3 was appropriate for the oral communication component of the appellant’s Supervision presentation. Both the appellant’s use of filler words and the multiple long pauses during his presentation were appropriately characterized as minor weaknesses. Accordingly, the Commission finds that the appellant has not met his burden of proof for the oral communication component of the Supervision scenario.

As to the oral communication component of the Administration scenario, the assessor found that the appellant displayed minor weaknesses in word usage/grammar, organization and nonverbal communication. Specifically, the assessor cited the appellants occasional use of “uh” and “um,” and his repeating of words and phrases within sentences as the basis for finding a minor weakness in word usage/grammar. The assessor further stated that the appellant’s long pauses towards the end of his presentation constituted a minor weakness in organization. Finally, the assessor found that the appellant’s talking with his hands and his wringing them throughout his response proved to be a minor weakness in nonverbal communication. Based upon the foregoing, the assessor awarded the appellant a score of 3. On appeal, the appellant argues that he should have been awarded a score of 4. However, the Commission observes that the appellant’s argument for a higher score appears to be based upon an erroneous belief that he displayed only a single minor weakness.

In reply, a review of the appellant’s presentation for the Administration scenario confirms that the appellant showed minor weaknesses in word usage/grammar, organization and nonverbal communication in his Supervision presentation. Accordingly, the appellant’s score of 3 for the oral communication component of the Administration scenario is sustained.

Concerning the oral communication component of the Incident Command scenario, the assessor awarded the appellant a score of 3 after finding that the appellant displayed a minor weaknesses in inflection/rate/volume and nonverbal communication. Specifically, concerning inflection/rate/volume, the assessor cited the appellant’s low and rapid speech at moments throughout his presentation. As to nonverbal communication, the assessor stated that the appellant made distracting hand gestures, including flipping through pages of his notes, and wringing and waving his hands. On appeal, the appellant acknowledges the foregoing but argues that he should be awarded a score of 4 for the oral communication component of this scenario because he “was only cited for one minor weakness for this scenario.”

In reply, a review of the appellant’s Incident Command scenario presentation confirms that the appellant’s low and rapid speech was appropriately characterized as a minor weakness in inflection/rate/volume and that his hand gestures constituted a minor weakness in nonverbal communication. Therefore, the appellant’s score of 3 for the oral communication component of the Incident Command scenario is sustained.

Regarding the appellant’s appeal of his score on the technical component of the Incident Command scenario, the scenario involves a response to a fire at a local auto parts store and auto repair shop. Question 1 asks what specific actions the candidate would take upon arriving at the scene. The prompt for Question 2 indicates that while crews are involved in extinguishment operation, an explosion occurs on Side C, emergency radio traffic has been transmitted by a fire fighter and that structural damage is now visible on Side C. Question 2 asks what specific actions the candidate should now take based upon this new information.

For the technical component of the Incident Command scenario, the assessor awarded the appellant a score of 3, finding that the appellant missed a number of additional PCAs, including, in part, opportunities to perform a size-up/360 and to monitor the air. On appeal, the appellant argues that he should have received credit for the foregoing PCA because he stated “Task Force with meters monitoring air quality” during a specified portion of his presentation.

In reply, when discussing the actions he would have his truck companies perform at the incident scene, the appellant stated that he “would task them with meters to monitor air quality.” Upon review of the appellant’s appeal, the Division of Test Development, Analytics and Administration (TDAA) agrees that the appellant should have been awarded credit for this additional response. Further, TDAA and the Commission find that the appellant should also receive credit for the additional PCA of acknowledging emergency traffic transmission in response to Question 2 for this scenario. Based upon the foregoing, the appellant’s score for the technical component of the Incident Command scenario should be increased from 3 to 4.

CONCLUSION

A thorough review of the appellant’s submissions and the test materials indicates that the decision below is amply supported by the record, and that except for the technical component of the Incident Command scenario, as indicated above, the appellant has failed to meet his burden of proof in this matter.

ORDER

Therefore, it is ordered that this appeal be granted in part and that the appellant’s score for the technical component of the Incident Command scenario be raised from 3 to 4. It is further ordered that this scoring change be given retroactive effect.

ClosingThis is the final administrative determination in this matter. Any further review should be pursued in a judicial forum.