Bertrand Moore, CSC Docket No. 2023-2023

BackgroundBertrand Moore, Jr. appeals his score on the promotional examination for Fire Officer 2 (PM4200C), Jersey City. It is noted that the appellant passed the examination with a final average of 86.030 and ranks 24th on the eligible list.

The subject promotional examination was held on May 19, 2022, and 39 candidates passed. This two-part examination consisted of an integrated system of simulations designed to generate behavior similar to that required for success on the job. The first part consisted of multiple-choice items that measured specific work components identified and weighted by the job analysis. The second part consisted of three oral scenarios: Supervision, Administration and Incident Command. The examination was based on a comprehensive job analysis conducted by the Civil Service Commission (Commission), which identified the critical areas of the job. The weighting of the test components was derived from the job analysis data. It is noted that candidates were told the following prior to beginning their presentations for each scenario: “In responding to the questions, be as specific as possible. Do not assume or take for granted that general actions will contribute to your score.”

Each candidate in a given jurisdiction was scored by a team of three different Subject Matter Experts (SMEs), who were trained in current technical scoring procedures. Each of these SMEs were current or retired fire officers who held the title of Battalion Fire Chief (or Fire Officer 2) or higher. Candidates were also assessed by three New Jersey Civil Service Commission employees trained in oral communication assessment. As part of the scoring process, an SME observed and noted the responses of a candidate relative to the knowledge, skills and abilities (KSAs) that each exercise was designed to measure. An assessor also noted any weaknesses that detracted from the candidate’s overall oral communication ability. Each assessor then rated the candidate’s performance according to the rating standards and assigned the candidate a technical or oral communication score on that exercise.

In order to preserve the relative weighting of each of the components of the examination, the ratings for each portion were adjusted by a well-recognized statistical process known as “standardization.” Under this process, the ratings are standardized by converting the raw scores to z-scores, an expression of the deviation of the score from the mean score of the group in relation to the standard deviation of scores for the group. Each portion of the examination had a relative weight in its relation to the whole examination. Thus, the z-score for the multiple-choice portion was multiplied by a test weight of 36.53%, the oral technical scores were multiplied by a test weight of 53.91% and the oral communication scores were multiplied by a test weight of 9.56%. The weighted z-scores were summed and this became the overall final test score. This was weighted and added to the weighted seniority score. The result was standardized, then normalized, and rounded up to the third decimal place to arrive at a final average.

On the Supervision scenario, the appellant scored a 5 on the technical component and a 5 on the oral communication component. On the Administration scenario, the appellant scored a 4 on the technical component and a 5 on the oral communication component. Finally, on the Incident Command scenario, the appellant scored a 5 on the technical component and a 5 on the oral communication component.

Appellant's argumentThe appellant challenges his score for the technical component of the Administration scenario. As a result, the appellant’s test material and a listing of possible courses of action (PCAs) for the scenario were reviewed.

The Administration scenario involves the candidate being tasked with investigating an incident and revising the department’s current pre-incident action plan procedures following a call where a pre-action plan failed to reflect a building’s conversion and the addition of partition walls inside of the structure. Question 1 asks what specific steps the candidate would take to investigate the incident and the lack of updated pre-incident action plan procedures. Question 2 asks what should be included in updated pre-incident action plan standard operating guidelines/procedures (SOGs/SOPs).

On the Administration scenario, the assessor stated that the appellant missed several possible courses of action (PCA), including the opportunity to identify the strategy and tactics that should be used at each incident. Based upon the foregoing, the assessor awarded the appellant a score of 4 on the technical component for this scenario.

On appeal, the appellant states the relevant actions he identified and he requests that the scoring of his oral presentation be reviewed for accuracy.

CONCLUSION

Initially, it is noted that when a candidate challenges a score on an oral examination component, the Commission reviews that entire component to ensure it has been scored correctly.

In the instant matter, a review of the appellant’s presentation by the Commission and the Division of Test Development, Analytics and Administration (TDAA) reveals that the appellant was erroneously awarded credit for the following PCAs: interview the incident commander; review the latest and past incident reports at this location (e.g., NFIRS, 911 calls, etc.); interview the department personnel/field inspection company responsible for the pre-incident action plan; and review current SOGs/SOPs regarding the pre-incident action plan. Specifically, because the appellant stated he would interview the fire prevention chief, rather than the incident commander who responded to the responded to the specific incident that was the catalyst for this pre-incident action planning review, he should not have been credited with the PCA of interviewing the incident commander. As to the PCA of reviewing the latest and past incident reports at the underlying location, the appellant indicated that he would review previous pre-fire plans, fire prevention planning, zoning and the certificate of occupancy for the structure. However, because he did not state that he specifically would review incident reports from the location of the underlying incident, his response was insufficient to cover this PCA. Concerning the PCA of interviewing department personnel/the field inspection company responsible for the pre-incident action plan, a review of the appellant’s presentation does not demonstrate that he covered this action. The assessor appears to have relied upon the appellant’s statement at the 15:47 mark in his recording that the pre-incident action plan had not been updated and created issues with the response to the incident detailed in the prompt. However, this spoke to the problem, rather than the solution. The Commission notes that the appellant stated at 17:46 that he would “meet with the previous chief in charge of fire prevention.” Nevertheless, this statement was insufficient to cover the PCA at issue. Further, the review of the appellant’s Administration presentation fails to demonstrate that he otherwise addressed this PCA. Finally, as to reviewing current SOGs/SOPs regarding the pre-incident action plan, the appellant spoke to reviewing previous plans for the structure and “fire prevention planning.” However, fire prevention planning would be distinct from the review of departmental policies and guidelines that the PCA contemplated and reviewing previous plans for the specific structure involved in the underlying incident was too narrow in scope to cover a review of departmental SOGs/SOPs. Accordingly, the appellant should not have been credited with this PCA. Therefore, based upon the Commission’s reversal of credit for these PCAs, the appellant’s score of 4 for the technical component of the Administration scenario shall be reduced from 4 to 2.

ORDER

Therefore, it is ordered that this appeal be denied. Additionally, it is ordered that the appellant’s score on the technical component of the Administration scenario be reduced from 4 to 2. It is further ordered that this scoring change be given retroactive effect.

ClosingThis is the final administrative determination in this matter. Any further review should be pursued in a judicial forum.