Gabriele Spallacci, CSC Docket No. 2024-916

BackgroundThe Superior Court of New Jersey, Appellate Division, has reversed and remanded the decision of the Civil Service Commission (Commission) in In the Matter of Melvin Jumper, et al., Police Sergeant (various jurisdictions) (CSC, decided March 24, 2021) and remanded In the Matter of Gabriele Spallacci, et al., Docket No. A-2369- 2 (App. Div. August 7, 2023), for further proceedings. The court did not retain jurisdiction. Copies of these matters are attached and incorporated herein.

By way of background, as discussed in In the Matter of Melvin Jumper, et al., supra, in pertinent part, the subject two-part examination, which was administered on February 23, 2018, consisted of a video-based portion, items 1 through 21, and a multiple-choice portion, items 22 through 85. For the video portion, candidates were provided with 35 minutes and candidates were provided with one hour and 40 minutes to complete the multiple-choice portion. The test was worth 80 percent of the final average and seniority was worth the remaining 20 percent. Candidates for the subject test were provided with two opportunities for review: 1) candidates were permitted to review keyed test booklets (test booklet review); and 2) candidates were permitted to inspect their answer sheets and compare them to the keyed answer sheet (answer sheet review). See N.J.A.C. 3A:4-6.4(a). During answer sheet review candidates were provided with the opportunity to review their answer sheet, the scoring key and an individualized scoring report. The copy of the scoring key provided at review did not provide responses for items 76 through 85. As a result, candidates were not able to compare their responses on their answer sheet to the key for these items. Furthermore, it was at this time that candidates first became aware that the last 10 items were omitted from scoring. Candidates who inquired as to why the items were omitted were informed that the items were removed after a statistical analysis was conducted on how the items performed for the entire population. As noted, in part, by the Commission:

The Division of [Test Development and Administration (TDA)]1 conducted several analyses which included an analysis of the individual items in the examination, as well as an analysis of the performance of subtests and of the entire test. As a result, TDA determined that as the subject test progressed, the number of candidates who did not provide responses to items increased. In addition, as part of the analyses noted above, TDA conducted an adverse impact analysis.2 TDA’s review found that the last ten questions presented evidence of adverse impact in that there was a disparity between the performance of minority candidates as compared to the performance of non-minority candidates. As a result,

TDA determined to omit the last ten items from scoring. Furthermore,

TDA determined that the omission of these 10 items did not render the subject test invalid in that sufficient KSAs were tested in the remainder of the test.

Thereafter, Gabriele Spallacci, Victor Lora, Novar Vidal,3 Lillian Sanchez, Juan Garcia,4 Pedro Borerro, Robert Klein, Juan Cosme, Felipe Diaz, Jose Castellanos, Marquis Brock, Mohamad Diabate, Angel Pared, Valeria Sanchez-Bermudez, and Isabel Reyes, represented by Albert J. Seibert, Esq., pursued an appeal with the Appellate Division in which they claimed, in part, that the omission of the last 10 items was arbitrary and capricious, “adversely impact[ing] the examinees who followed the instructions, managed their time properly, and completed the exam in the allotted time.”

AnalysisUpon its review, the court, in In the Matter of Gabriele Spallacci, et al., supra, noted that during the pendency of this matter, “the Commission provided raw data consisting of several spreadsheets, outlining the 2019 exam and previous

1 TDA is now the Division of Test Development, Analytics and Administration (TDAA).

2 The U.S. Equal Employment Opportunity Commission (EEOC) defines adverse impact as “a substantially different rate of selection in hiring, promotion, or other employment decision which works to the disadvantage of members of a race, sex, or ethnic group.” See 29 CFR §1607.16. In this regard, Title VII of the Civil Rights Act of 1964 (see 42 U.S.C. §2000e et seq.) prohibits the use of discriminatory employment tests and selection procedures. Furthermore, Title VII prohibits the use of neutral test or selection procedures that have the effect of disproportionately excluding individuals based on race, color, religion, sex or national origin. In 1978, the EEOC adopted the Uniform Guidelines on Employee Selection Procedure under Title VII (see 29 CFR Part 1607), which provides guidance to employers on how to determine whether their tests and selection procedures are lawful. See https://www.eeoc.gov/laws/guidance/employment-test-and-selection-procedures. One of the measures of adverse impact utilized by the Uniform Guidelines is the Four-Fifths Rule which is “… a selection rate for any race, sex or ethnic group which is less than four-fifths (4/5ths) or eighty percent (80%) of the selection rate for the group with the highest rate as a substantially different rate of selection.” See Questions and Answers to Clarify and Provide a Common Interpretation of the Uniform Guidelines on Employee Selection Procedures, 44 FR 11998 (March 2, 1979). See also, https://www.eeoc.gov/laws/guidance/questions-and-answers-clarify-and-provide-common- interpretation-uniform-guidelines.

3 A review of available records finds that Mr. Vidal retired effective August 1, 2023.

4 A review of available records finds that Mr. Garcia retired effective February 1, 2021. examination scores. These spreadsheets included, but were not limited to, mean scores for male candidates versus female candidates, as well as score breakdowns across different ethnicities. Petitioners stress the raw data has no corresponding explanation, analysis, or interpretation enabling them to understand, and in turn, challenge the Commission’s final agency decision.” The court determined that “the raw data [provided by the Commission] affords neither petitioners nor us the ability to consider if scoring the final ten exam questions disparately impacted racial minorities, or whether, as petitioners suggest, the remedy adopted by the Commission unwittingly amplified rather than ameliorated the purported disparate impact it sought to correct.” As such, the court ordered the Commission to provide, within 60 days of its decision, the petitioners with “an explanation and interpretation of how the raw data demonstrates racial minorities were adversely impacted, in violation of the consent decree and existing law, if the last ten questions were scored.” The court further provided that “the petitioners may renew their challenge to the Commission’s scoring in accordance with the agency’s guidelines. The Commission must then issue another final agency decision within ninety days of the renewed challenge.”

In a letter dated October 4, 2023, TDAA provided the petitioners with an explanation in accordance with the court’s order as noted above. Specifically, TDAA indicated that pursuant to the Consent Decree, the Commission, in collaboration with a United States Department of Justice consultant, devised a test plan and scoring model that would attempt to reduce adverse impact on minority groups. In this regard, TDAA indicated that the Commission utilized various exam and item analysis methods to identify items influencing adverse impact and reduce it where possible. TDAA noted that the last 10 items were omitted by candidates at a rate that was higher than other subtests in the examination. TDAA explained that typically less than 1% of the testing population omits an item but for the subject exam, between 18% to 28% omitted the last 10 items. Further, the percentage of omitted items was higher among minority candidates (Black and Hispanic) as compared to non-minority candidates (White) which resulted in an increased adverse impact on minority candidates. TDAA provided the adverse impact rates for minority groups with respect to the last 10 items and explained how these rates were calculated. TDAA also indicated that additional adverse impact analysis was conducted and explained the methodology that was utilized. TDAA explained that the results of this analysis indicated that by eliminating the last 10 items, adverse impact was reduced for minority candidates. TDAA further explained that each exam item is intended to measure a particular knowledge, skill or ability (KSA) as defined in the job analysis and test plan. However, for the last 10 items of the subject test, analysis indicated that candidates may have been simply randomly guessing responses. In other words, TDAA indicated that these items were not performing the purpose of assessing the intended KSA. TDAA concluded that after conducting analyses in which the last 10 items were included versus omitting them from scoring, it was evident that removing these items improved the outcome for minority candidates.

In response, in a letter dated October 24, 2023, the petitioners present:

At the outset, the CSC argues that the raw data demonstrates that racial minorities were adversely impacted on the subject examination in violation of the [C]onsent [D]ecree. However, as argued in our Appellate

Brief and explicitly stated in the Consent Decree itself, as well as the

March 29, 2021 Final Administrative Action of the CSC[, i.e., In the

Matter of Melvin Jumper, et al., supra,] that is the subject of this dispute, the Consent Decree expired . . . following the second administrations of the police sergeant exam following entry of the Consent Decree . . . It is undisputed that the second administration of the exam occurred on

January 16, 2016. As such, the Consent Decree was expired on that date pursuant to its own express terms and the CSC cannot now use the

Consent Decree as justification for removal of the final ten questions of the subject examination, which is in direct contradiction with its own instructions provided to the examinees.

The petitioners maintain that “in an attempt to ameliorate the alleged adverse impact on minority examinees, the CSC apparently decided to simply eliminate the final ten items from scoring” but “it is unknown whether the CSC explored any other potential remedies that would have accomplished its goal while still being in compliance with the instructions provided to the examinees before the examination. If so, none of this information was provided.” The petitioners add that the “CSC does not indicate whether the 4/5 Rule was applied and/or violated in the three administrations of the [sergeant] examination prior to the February 23, 2019 administration . . . Accordingly, it is respectfully requested that the CSC provide additional information on the analyses of the other examination results.” The petitioners assert that “there remains a number of unexplained issues and data, as well as certain analyses that are referenced but were not provided.” In this regard, the petitioners argue that the October 4, 2023 letter “did not define or indicate what the other subtests in the examination consisted of and did not provide any analysis as to the number of items omitted or answered incorrectly by the examinees in these other undefined subtests.” The petitioners contend that while “the CSC indicates that individual item analysis was conducted on each of the last ten items on the examination . . . the CSC failed to provide the individual item analysis for each item of the entire examination . . . In other words, no explanation has been provided by the CSC for why the decision was made to eliminate the final ten questions of the examination and why those items specifically had an adverse impact, as opposed to another set of ten questions of the examination, whether consecutive or otherwise.” The petitioners further contend that the decision in In the Matter of Melvin Jumper, et al., supra, indicates that TDA “performed analyses ‘of the individual items in the examination, as well as an analysis of the performance of subtest and the entire test’ after each of the four administrations of the exam. None of these critical analyses were provided by the CSC. Accordingly, the subject officers hereby request copies of the same.” The petitioners further assert that “the CSC did not provide any information regarding this supposed ‘additional adverse impact analysis’ or the underlying data . . . [T]he CSC did not identify or set forth what comprises these subtests or information as how the questions were weighted. Moreover, no information was supplied as to how these subtests were scored or ‘weighted,’ nor were the examinees’ results on the ‘subtests’ provided. The CSC also provided no information as to the referenced ‘job analysis and test plan.’” The petitioners contend that the CSC only offered “several conclusory statements and refers to and relies upon analyses that were not provided” regarding how the passing point for the subject test was established. The petitioners claim that:

[T]he October 4, 2023 correspondence was the first mention of a job analysis and test plan, and the same has not been provided by the CSC.

However, the CSC claims that an analysis showed ‘that it could not be said that [the final ten questions] were effectively measuring the intended KSA.’ It must be again noted that this was the fourth administration of the subject examination and this issue, to the subject officers’ knowledge, had never been raised in any of those prior administrations. In any event, this is yet another conclusory statement made by the CSC without any explanation as to how or why just the final ten question of the examination so happen to not properly test the examinees’ KSAs.

The petitioners note that “what is particularly troubling is that thirteen out of the fifteen officers in this matter are minorities and the elimination of the final ten questions adversely impacted these officers, the exact group of people that the CSC is purportedly attempting to protect by eliminating the final ten questions.” The petitioners conclude that the October 4, 2023 letter, “still falls short. There are ‘further’ and ‘additional’ analyses referenced throughout the correspondence that are not provided or fully explained. The CSC also references ‘subtest’ on the examination, but little to [sic] information regarding same was provided. As a final note, [the Commission’s determination in in In the Matter of Melvin Jumper, et al.] refers to ‘correspondences and communications between the parties and their test developers, consultants, and/or experts’ that the CSC has deemed confidential in nature. We submit that given the nature of the within challenge of the subject examination, these correspondences and communications are germane and should be provided as well.”

CONCLUSION

The petitioners’ argument that the Commission “cannot now use the Consent Decree as justification for removal of the final ten questions of the subject examination,” is misplaced. Regardless as to whether the Consent Degree had expired, the Commission is still required to ensure that examinations are designed to eliminate adverse impact on racial minorities. As such, while the petitioners are correct that the Consent Degree was expired by the fourth administration,5 it does not follow that the Commission is relieved of ensuring that its examinations do not adversely impact racial minorities. Rather, if such adverse impact is identified, it must be remedied, as it was in this matter and as explained initially in the Commission’s decision in In the Matter of Melvin Jumper, et al., supra, and further in this matter pursuant to TDAA’s October 4, 2023, letter.

Regarding the petitioners’ requests for information regarding the prior three administrations of the Police Sergeant examination and for the “correspondences and communications between the parties and their test developers, consultants, and/or experts,” which would only be applicable to the first two administrations,6 the Appellate Division, as noted above, specifically ordered the Commission to provide “an explanation and interpretation of how the raw data demonstrates racial minorities were adversely impacted, in violation of the consent decree and existing law, if the last ten questions were scored.” Based on TDAA’s letter, the Commission has complied with this directive. Thus, the petitioners’ requests fall outside the scope of the Appellate Division’s order.

A thorough review of the record finds that information provided by TDAA in its letter dated October 4, 2023 provided the petitioners with “an explanation and

5 However, while the Commission was not bound by the Consent Decree in terms of test development in conjunction with Department of Justice as in the first two administrations, it was required to provide the examination data to the Department of Justice for the third and fourth administrations as indicated in Paragraph 82 of the Second Amended Consent Decree.

6 As indicated in In the Matter of Melvin Jumper, et al., supra, “correspondences and communications between the parties and their test developers, consultants, and/or experts” is taken from Paragraph 81 of the Second Amended Consent Decree. Paragraph 81 specifically provides:

Due to the importance of test security, along with concerns about the proprietary nature of test development materials, all correspondence and communications between and among the Parties and their test developers, consultants and/or experts in connection with the performance of the obligations set forth in Paragraphs 77 through

80 shall be held confidential and shall not be disclosed to any third party in the absence of a court order compelling such disclosure.

Paragraphs 77 through 80 refer to the first and second administrations. As explained in In the Matter of Melvin Jumper, et al., supra, the February 23, 2019 administration of the Police Sergeant examination was the fourth administration subsequent to the Second Amended Consent Decree. Moreover, as noted above, the Appellate Division did not order the Commission to provide information from the prior administrations of the Police Sergeant exam, including any “correspondences and communications between the parties and their test developers, consultants, and/or experts.” interpretation of how the raw data demonstrates racial minorities were adversely impacted, in violation of the consent decree and existing law, if the last ten questions were scored,” in accordance with the Appellate Division order, as noted above. Accordingly, the Commission finds that there was sufficient basis to omit the last 10 items from scoring as these items demonstrated adverse impact on minorities.

ORDER

The Commission finds that the elimination of the final 10 questions from scoring for the subject examination was justified.

ClosingThis is the final administrative determination in this matter. Any further review should be pursued in a judicial forum.