Thomas Flanagan III, CSC Docket No. 2023-2024

BackgroundThomas Flanagan III appeals his score on the promotional examination for Battalion Fire Chief (PM4444C), Atlantic City. It is noted that the appellant passed the examination with a final average of 84.220 and ranks 15th on the eligible list.

The subject promotional examination was held on May 25, 2022, and 27 candidates passed. This two-part examination consisted of an integrated system of simulations designed to generate behavior similar to that required for success on the job. The first part consisted of multiple-choice items that measured specific work components identified and weighted by the job analysis. The second part consisted of three oral scenarios: Supervision, Administration and Incident Command. The examination was based on a comprehensive job analysis conducted by the Civil Service Commission (Commission), which identified the critical areas of the job. The weighting of the test components was derived from the job analysis data. It is noted that candidates were told the following prior to beginning their presentations for each scenario: “In responding to the questions, be as specific as possible. Do not assume or take for granted that general actions will contribute to your score.”

Each candidate in a given jurisdiction was scored by a team of three different Subject Matter Experts (SMEs), who were trained in current technical scoring procedures. Each of these SMEs were current or retired fire officers who held the title of Battalion Fire Chief (or Fire Officer 2) or higher. Candidates were also assessed by three Commission employees trained in oral communication assessment. As part of the scoring process, an SME observed and noted the responses of a candidate relative to the knowledge, skills and abilities (KSAs) that each exercise was designed to measure. An assessor also noted any weaknesses that detracted from the candidate’s overall oral communication ability. Each assessor then rated the candidate’s performance according to the rating standards and assigned the candidate a technical or oral communication score on that exercise.

In order to preserve the relative weighting of each of the components of the examination, the ratings for each portion were adjusted by a well-recognized statistical process known as “standardization.” Under this process, the ratings are standardized by converting the raw scores to z-scores, an expression of the deviation of the score from the mean score of the group in relation to the standard deviation of scores for the group. Each portion of the examination had a relative weight in its relation to the whole examination. Thus, the z-score for the multiple-choice portion was multiplied by a test weight of 36.53%, the oral technical scores were multiplied by a test weight of 53.91% and the oral communication scores were multiplied by a test weight of 9.56%. The weighted z-scores were summed and this became the overall final test score. This was weighted and added to the weighted seniority score. The result was standardized, then normalized, and rounded up to the third decimal place to arrive at a final average.

Each oral examination question, and overall oral communication, was rated on a five-point scale, with 5 as the optimal response, 4 as a more than acceptable passing response, 3 as a minimally acceptable passing response, 2 as a less than acceptable response, and 1 as a much less than acceptable response.

On the Supervision scenario, the appellant scored a 4 on the technical component and a 3 on the oral communication component. On the Administration scenario, the appellant scored a 5 on the technical component and a 4 on the oral communication component. Finally, on the Incident Command scenario, the appellant scored a 5 on the technical component and a 5 on the oral communication component.

Appellant's argumentThe appellant challenges his scores for the oral communication components of the Supervision and Administration scenarios and the technical component of the Supervision scenario. As a result, the appellant’s test material and a listing of possible courses of action (PCAs) for the scenarios were reviewed.

On the oral communication component of the Supervision scenario, the assessor awarded the appellant a score of 3 based upon findings that he displayed minor weaknesses in word usage/grammar, organization and specificity/brevity. Specifically, as to word/usage grammar, the assessor indicated that the appellant “repeated words here and there while responding.” The assessor also stated that the appellant displayed a minor weakness in organization by taking brief pauses to gather thoughts and quickly refer to his notes. Finally, the assessor determined that the appellant displayed a minor weakness in brevity because his response was too brief to effectively demonstrate his oral communication ability. On appeal, the appellant maintains that the recording of his video presentation demonstrates that he spoke with specificity and gave a thorough, detailed response. The appellant avers that because his presentation lasted seven minutes and 31 seconds, it cannot be considered brief. He claims that because he “stated many specific possible courses of action to receive a technical score of 4, only missing a 5 by one thing,” his oral communication score should be reconsidered. The appellant further argues that he should not have been penalized for taking brief pauses to gather his thoughts and quickly refer to his notes because the 2022 2nd Level Fire Supervisor Orientation Guide stated that pausing occasionally to review notes was expected and would not be penalized. Based upon the foregoing, he argues that he should have received an oral communication score of 4.

In reply, the appellant’s reliance on his technical component rating of 4 on the Supervision scenario in arguing that his oral communication score for the same component should have been higher is without merit. The Commission emphasizes that it is this agency’s longstanding policy that technical and oral communication component scores are independent ratings on the examination and that an exemplary or poor technical rating does not have a bearing on oral communication scoring and vice versa. Furthermore, oral communication performance can clearly distinguish candidates, including those delivering presentations with comparable technical details. To wit, it would be disingenuous to argue that the presentation of one candidate who spoke at a low rate of volume, had their speech punctuated by the frequent use of filler words like “ah” and “um,” rarely made eye contact with their audience and routinely made distracting hand gestures would be as understandable, effective and well-received as the presentation of another candidate who gave a speech with a comparable level of detail, but without these same oral communication issues. For these reasons, mere arguments that because a candidate received a certain technical score on a scenario, they should have received a corresponding oral communication rating for that same scenario are invalid. Turning to the appellant's specific objections, a review of the appellant’s presentation confirms that the assessor accurately concluded that the appellant displayed a minor weakness in word usage/grammar based upon repetition in his presentation. As to organization, while the 2022 2nd Level Fire Supervisor Orientation Guide states that “[p]ausing occasionally to review notes is expected and will not be penalized,” that same passage also tells candidates to “eliminate long pauses,” as reviewers can consider such a deficiency in a presentation. The appellant paused at several points during his presentation to review his notes. Many of these pauses were reasonably brief. However, the final 90 seconds of his presentation included a lengthy pause that exceeded 20 seconds. Even assuming, arguendo, that the appellant’s other pauses were occasional, it remained appropriate to find that this lengthy pause was a minor weakness in organization. However, regarding the specific issue of brevity, upon review, the Division of Test Development, Analytics and Administration (TDAA) has determined that the assessor’s finding of brevity was not correct. Nevertheless, even with the foregoing change, the appellant’s rating remains correct. Accordingly, the appellant’s score of 3 on the oral communication component of the Supervision scenario is sustained.

On the oral communication component of the Administration scenario, the assessor awarded the appellant a score of 4, based upon minor weaknesses in organization and nonverbal communication. Specifically, the assessor indicated that by returning to Question 1 later in his presentation, the appellant displayed a minor weakness in organization. Similarly, the assessor cited the appellant’s use of hand movements to express himself towards the end of his presentation as a minor weakness in nonverbal communication. On appeal, the appellant argues that because the monitor’s instructions prior to the beginning of his presentation stated that he would have “10 minutes to respond to all questions,” returning to Question 1 to emphasize an additional point shouldn’t have resulted in a deduction from his score, as the instructions did not state that he had to answer the first question within a specific timeframe or in a particular order. As to nonverbal communication, the appellant contends that the reference to his hand gestures “presents an issue with Equal Employment Opportunity as the rights of a person to compete for a job and/or to be promoted on the basis of knowledge, skills, and abilities, free from unlawful discrimination.” He alleges that the different candidate presentations are recorded from different angles and that in some cases, the camera angles do not show the relative position where his hands were located, meaning that the assessor penalized him for something that might not be captured in other candidates’ videos because of the variation in camera angle.

In reply, the appellant’s arguments regarding his Administration scenario oral communication score are without merit. In terms of organization, after answering Question 2, which asked about “[w]hat should be covered in a mutual aid contract,” the appellant stated that he was returning to Question 1 and that he would conduct a “thorough fact-finding mission, w[ould] review[ ] all files, all mutual, all old mutual aid agreements, all current policies . . . ” Clearly, waiting to detail background research one would perform prior to updating a contract until after describing what would be included in an updated contract is not a presentation of those actions in a logical order. As such, the assessor was correct in finding that the appellant displayed a minor weakness in organization. As to the appellant’s nonverbal communication, the appellant has failed to demonstrate how any purported difference in camera angles constitutes an unlawful employment practice or unlawful discrimination based upon his membership in a protected class1. As to the placement of cameras,

1 See for example, the New Jersey State Policy Prohibiting Discrimination in the Workplace or the

New Jersey Law Against Discrimination for what constitutes a protected category. Here, the appellant does not allege that any differential treatment is based upon his membership in any such protected category.

TDAA states that for every presentation recording, camera operators are instructed to zoom out so the candidate and the desk can be captured in the frame. It is evident that the appellant’s video recording was framed in a manner consistent with this standard. The appellant has not named any other candidates whose recordings failed to adhere to this framing standard. Finally, TDAA has advised that because the appellant displayed two minor weaknesses in oral communication during his presentation, he should have received a rating of 3, rather than 4, for the subject scenario. Based upon the foregoing, the appellant’s score for this component shall be revised to 3.

The appellant also challenges his score on the technical component of the Supervision scenario. The Supervision scenario presents that the new Fire Chief wants to implement a new and well-defined disciplinary policy. After this progressive discipline policy is implemented, the Fire Chief receives reports that a subordinate Fire Captain under the candidate’s supervision is not enforcing the new policy. Question 1 asks what specific actions the candidate would take to investigate the situation with the Fire Captain. The prompt for Question 2 states that the candidate has learned that the Fire Captain has not been implementing the new policy because she is not sure how to enforce it in certain situations, particularly those she has not dealt with previously. Question 2 then asks what specific topics/actions the candidate should discuss/take in an interview with the Fire Captain based upon the new information.

On the technical component of the Supervision scenario, the SME awarded the appellant a score of 4 based upon a finding that the appellant missed several PCAs, including the opportunity to review disciplinary actions that the Fire Captain had taken against her crew. On appeal, the appellant argues that he covered this action by stating at specified points in his presentation that he would “review all personnel files” and “get a written report, ensure [he would] speak with all past supervisors regarding similar actions and behaviors of the captain, as well as speak with human resources.” He contends that a review of all personnel files directly conveyed that he would review all previous disciplinary actions. In support, he submits that Adam K. Thiel, Managing Fire and Emergency Services 434 (4th ed. 2012)2 states that “disciplinary or other infractions” should be included in personnel records. Additionally, he presents that N.J.A.C. 15:3-1.1 makes clear that information related to prior disciplinary actions must be included in personnel files. He proffers that ensuring the Fire Captain gave a written report provided for further review of the incident and created a written record that could be compared against personnel files. Moreover, the appellant contends that speaking with past supervisors allowed him to gain insight into any patterns or behaviors taken by the Fire Captain, which could help shed additional light on the situation and uncover any other questionable

Background2 It is noted that the subject text was not on the suggested reading list set forth in the 2022 2nd Level

Fire Supervisor Orientation Guide and the appellant has not submitted a copy of the cited text on appeal. Therefore, it will not be considered by the Commission.

decisions. He argues that the foregoing covered the response necessary to receive credit for the subject PCA.

In reply, as noted above, candidates were told the following prior to beginning their presentations for each scenario: “In responding to the questions, be as specific as possible. Do not assume or take for granted that general actions will contribute to your score.” The statements cited by the appellant were too general to convey that he was specifically reviewing the disciplinary actions the subject Fire Captain took against her crew. Accordingly, his challenge to the subject technical component score is without merit and his score of 4 is sustained.

CONCLUSION

A thorough review of the appellant’s submissions and the test materials indicates that the decision below is amply supported by the record, and the appellant has failed to meet his burden of proof in this matter.

ORDER

Therefore, it is ordered that this appeal be denied and that the appellant’s score on the oral communication component of the Administration scenario be revised to 3. It is further ordered that this scoring change be given retroactive effect.

ClosingThis is the final administrative determination in this matter. Any further review should be pursued in a judicial forum.