In its continuing efforts to keep the public informed about the ongoing admissions litigation, the University of Michigan makes these transcripts of the trial proceedings in Grutter v Bollinger, et al., Civil Action No. 97-75928 (E.D. Mich.), available to the University community and general public. As is often the case with transcription, some words or phrases may be misspelled or simply incorrect. The University makes no representation as to the accuracy of the transcripts.
1
1
2 UNITED STATES DISTRICT COURT
EASTERN DISTRICT OF MICHIGAN
3 SOUTHERN DIVISION
4
5 BARBARA GRUTTER,
For herself and all others
6 Similarly situated,
7 Plaintiff,
8 v. Civil Action
97-CV-75928
9 LEE BOLLINGER, JEFFREY LEHMAN,
DENNIS SHIELDS, and REGENTS OF
10 THE UNIVERSITY OF MICHIGAN,
11 Defendants.
_________________________________________/
12
13 BENCH TRIAL - VOLUME 8
14
TUESDAY, FEBRUARY 6TH, 2001
15
16 BEFORE THE HONORABLE BERNARD FRIEDMAN
United States District Judge
17 Theodore Levin United States Courthouse
231 West Lafayette Boulevard, Room 238
18 Detroit, Michigan
19 - - -
20 Appearances:
21
Kirk O. Kolbo, Esq.,
22 R. Lawrence Purdy, Esq.,
23 On behalf of the Plaintiff,
24
25
APPEARANCES (CONTINUED):
John Payton, Esq.,
Craig Goldblatt, Esq.,
On behalf of the Defendants Bollinger, et al,
- - -
George B. Washington, Esq.
Miranda K. S. Massie, Esq.
On behalf of Intervening Defendants.
Joan L. Morgan, Official Court Reporter
Proceedings recorded by mechanical stenography.
Transcript produced by computer-aided transcription.
I N D E X
- - -
WINTESS: PAGE:
MOTIONS 4
MARTIN SHAPIRO
Direct Examination by Ms. Massie 17
Cross-Examination by Mr. Purdy 57
JAY ROSNER
Direct Examination (Cont) by Ms. Massie 89
Cross-Examination by Mr. Payton 127
Cross-Examination by Mr. Kolbo 133
Redirect Examination by Ms. Massie 153
CONCEPCION ESCOBAR
Direct Examination by Ms. Masley 158
E X H I B I T S
RECEIVED
Trial Exhibit Number 170 25
BENCH TRIAL - VOLUME 8
TUESDAY, FEBRUARY 6TH, 2001
4
1 Detroit, Michigan
2 Tuesday, February 6th, 2001
3 9:00 a.m.
4 - - -
5 THE COURT: How is everybody? Relaxed and ready
6 to go. I can see you weren't too relaxed, as I've been
7 reading since 6:00 o'clock this morning, and last night also.
8 Let's start I guess with the Professor Steele matter.
9 Let's get that resolved and go from there.
10 Good morning.
11 MS. MASSIE: Good morning, Judge Friedman.
12 Judge, here's our view of the situation with
13 Professor Steele. Professor Steele's testimony is absolutely
14 essential to our ability to make our case to you about
15 standardized test scores, and academic performance. It's
16 always been essential to us.
17 In fact, when we first filed the motion to
18 intervene, the day -- the day after that, or the day after
19 that, within a couple of days there were three people we
20 contacted about being expert witnesses. He was one of those
21 first three people. The other two were Gary Orfield whom you
22 have already heard from, and Walter Allen whom you'll be
23 hearing from tomorrow. He is at the heart of the questions
24 that you set for trial, and the defendants pointed out in
25 their response when the plaintiffs were trying to get
BENCH TRIAL - VOLUME 8
TUESDAY, FEBRUARY 6TH, 2001
5
1 Professor Steele's testimony excluded there's nothing
2 duplicitous about his testimony. Nobody has said what he
3 would say if he were to testify. And nobody can say. He is a
4 national expert of unparallel qualifications.
5 THE COURT: You didn't take his deposition?
6 MS. MASSIE: His deposition was taken before the
7 Sixth Circuit granting intervention, and we didn't think there
8 was going to be any need to take his deposition because until
9 it was announced on the 19th or that Friday that Professor
10 Steele was not going to be called even though he was on the
11 will-call list, we fully expected to be able to ask him some
12 questions here before you.
13 THE COURT: Okay.
14 MS. MASSIE: In short, Judge, we would have listed
15 him as a witness if the University had not, and under local
16 rules we're entitled to call him. The only question is where
17 that's by deposition or whether you have a right -- whether we
18 can make a video tape and bring it back here. We shut
19 everything down, go make a video tape deposition, de bene esse
20 deposition, and bring it back here and play it for you or
21 whether you have the ability to ask him questions about
22 whatever's on your mind and your concerns.
23 That, Judge, is the only question that we face here.
24 MR. PAYTON: Your Honor, I guess I just want to
25 describe how we got to where we are, then where we are. We
BENCH TRIAL - VOLUME 8
TUESDAY, FEBRUARY 6TH, 2001
6
1 retain Professor Steele almost three years ago, 1998, to be an
2 expert in this case. We have used his expert report in
3 connection with all the summary judgment motions including the
4 last round. We still use his summary judgment expert report.
5 I think it's very important. That's why we put it in. We did
6 place him on our original witness list from whenever it was a
7 couple of years ago, and on our current witness list, and on
8 the witness list that we prepared after the summary judgment.
9 But as things progressed we decided that it was unnecessary
10 for us to call him. We let everybody know on the 19th which
11 is the end of the first week of trial. On the 24h, which was
12 the last day of the second week of trial, Ms. Massie told me
13 she intended to subpoena him for a video deposition. I wrote
14 to her about that, and said that a video deposition in last
15 week would have been quite disruptive to our efforts to
16 prepare for this trial and that in any event discovery was
17 closed.
18 She then -- we talked. She then dropped the video
19 deposition, and on February 1st, wrote me and attached a
20 subpoena for a trial presentation. We got a trial subpoena on
21 February 1st, last Thursday. Last Friday, February 2nd, I
22 wrote her and attached a motion to quash the trial subpoena.
23 I had in response to several queries from her about several of
24 our witnesses let the intervenors talk to a number of our
25 witnesses because they wanted to use them in this case. So
BENCH TRIAL - VOLUME 8
TUESDAY, FEBRUARY 6TH, 2001
7
1 Professor Foner, Professor Sugrue, Professor Lempert, and
2 Professor Steele, we let them contact. They contacted
3 Professor Steele. After they contacted Professor Steele, I
4 spoke with Professor Steele. He -- we now represent him in
5 this last matter, and after talking to him, and after him
6 talking to the intervenors, he has decided that he doesn't
7 want to be a witness. That's why we filed a motion to quash.
8 We represent him.
9 The subpoena that was issued, a trial subpoena
10 simply can't reach him for him to testify in this trial. They
11 started out with a video deposition, they dropped that. They
12 issued a trial subpoena. I think the trial subpoena is
13 clearly is not valid being issued from the Northern District
14 of California at all. They can say that they checked the
15 wrong box. They checked the box that said, trial -- you know
16 to appear at trial. So I think that's where we are. You
17 know, you can't reach him.
18 I have an additional report to make to everyone. I
19 have -- this has satellite litigation in the Northern District
20 of California where we filed the motion to quash. That was
21 filed on Friday in opposition and a reply filed yesterday. I
22 talked to the judge to whom that was assigned, a law clerk.
23 The judge to whom it's assigned is Judge Illston, and the law
24 clerk told me last night that Judge Illston who has the motion
25 to quash is a little in the dark about what just is being
BENCH TRIAL - VOLUME 8
TUESDAY, FEBRUARY 6TH, 2001
8
1 argued and where and what is going on and what is the context
2 here. And, therefore, is waiting to see what this court does.
3 Was not -- thought there was also a motion to quash in this
4 court, thought you had the identical motion. And asked that I
5 tell everyone that until she sorts just what is where and what
6 this court has done with whatever court has she's just
7 standing back. But I think where things are, it's a trial
8 subpoena, you can't -- it's not valid --
9 MS. MASSIE: Judge Friedman, could I be heard
10 briefly?
11 THE COURT: You sure can, but it's going to be one
12 second. I want to see what the plaintiff's position is.
13 You'll have the last word.
14 MR. PURDY: Your Honor, we take no position in the
15 battle between the university and the intervenors on this.
16 But I do want to raise one issue. We are concerned about the
17 effort to conduct a trial testimony by video --
18 tela-conferencing unless -- in other words, we would object to
19 that unless the intervenors assured that all of the trial
20 exhibits and all the deposition and the exhibits to Doctor
21 Steele's deposition were made available. I have no idea
22 whether they have made any arrangements to do that in the
23 event they were to get leave to call him via
24 tela-conferencing. But that would pose a problem if he didn't
25 have those materials. I just want to raise a practical issue.
BENCH TRIAL - VOLUME 8
TUESDAY, FEBRUARY 6TH, 2001
9
1 But as far as the dispute between the university and the
2 intervenors, we take no position. Obviously, our position,
3 our feeling is that his testimony simply isn't relevant.
4 We're not taking an issue about his expertise or anything of
5 that nature. So that's all.
6 MS. MASSIE: And we will make arrangements as to the
7 practicalities that Mr. Purdy is raising if the Court allows
8 us to go forward.
9 Just very briefly, Judge Friedman, essentially the
10 University waived its objection to having the testimony taken
11 by video conference. And that's what the motion to quash
12 means in the Northern District of California is really about,
13 and thats' what this dispute is really about.
14 Mr. Payton told me that he had the same objection
15 to video deposition de bene esse to be introduced at trial, to
16 trial testimony, or deposition testimony presided over by the
17 Court by video conference, the same action to those two
18 things, as he did to Professor Steele appearing in person to
19 testify. He waived his objection to the video conference nex
20 of the testimony we're proposing. I'm not sure we checked the
21 wrong box. The Federal rules don't yet have provisions
22 specifically directed at video conferencing. That's
23 specifically why we raised it. We thought it would be
24 preferable not to just make a tape and bring it here, given
25 that would mean stopping everything and given that you
BENCH TRIAL - VOLUME 8
TUESDAY, FEBRUARY 6TH, 2001
10
1 wouldn't have the opportunity to ask Professor Steele the
2 questions that you might have. And that's what this is really
3 about. It's about whether can we do this by video conferencing
4 or not. I don't think there's any question that we can go out
5 to California, take a video deposition of Professor Steele,
6 and play it for the Court.
7 THE COURT: Okay. Thank you.
8 A couple of things. Number one, is we have a witness
9 here that does not want to voluntarily come. He does not want
10 to testify at this trial at this point. He is represented by
11 counsel, in the form of Mr. Payton who has made that very
12 clear, to the Court, and to everybody here. The suggestion of
13 video conferencing is one I think I brought up at the
14 beginning of this trial, and I brought it up in the context of
15 saving dollars for the parties, and trial dollars, and in the
16 context of where we had a voluntary witness, the witness who
17 is going to get on a plan and voluntarily come to Michigan
18 rather making them involuntarily get on a plane. I thought we
19 save costs for everybody. I thought it was a good idea. I'm
20 an advocate of it, and an advocate of saving trial dollars
21 whenever we can. But I think here it's not appropriate for a
22 couple reasons. Number one is we have a witness that's
23 outside the jurisdiction of this court. I could not subpoena
24 that witness. I could not order that witness to come, more
25 than one hundred miles and so forth. He was made as a will-
BENCH TRIAL - VOLUME 8
TUESDAY, FEBRUARY 6TH, 2001
11
1 call witness by the defendants in this matter, but what that
2 is it's a -- the Pre-Trial Statement to be prepared for what's
3 to be anticipated at a trial. It's not hard and fast thing.
4 It doesn't mean that the party who puts him on as a will call
5 decides that it's not in their best interest or doesn't comply
6 with the rules in terms of relevance, whatever the reason is,
7 that doesn't mean that they have to call that particular
8 person. And that's what happened here. On the 19th of
9 January it became very clear that Professor Steele was not
10 going to be called. And as I indicated I have no ability to
11 compel him to be here. A subpoena for video testimony is only
12 for voluntary witnesses. There's nothing in the rules that
13 provide for it or anything of that nature.
14 So I believe at this point that I can't compel him
15 to be here. In terms of any type of deposition the
16 intervenors had months and months of opportunity to take his
17 deposition. They chose not to for the reasons articulated on
18 the record, but that's not the fault of the defendants in this
19 matter. It's not the fault of the plaintiffs. It's not the
20 fault of the intervenors, but we're in the middle of a trial,
21 and we have a witness who chooses not to make himself
22 available to testify. I have no subpoena power. Therefore
23 the Court -- there's nothing that I can do in relation to
24 providing his testimony here at trial.
25 As to the subpoena in California, I guess California
BENCH TRIAL - VOLUME 8
TUESDAY, FEBRUARY 6TH, 2001
12
1 is going to have to deal with that. It's their subpoena. I
2 suggest that the subpoena whether it be -- whatever reason was
3 for video deposition that I'm not going to permit it at this
4 point because as I say we've got a witness who doesn't come
5 within the spirit of what I thought would be good economy, and
6 therefore, it would be moot out there anyhow the way I see it.
7 So that would be my ruling.
8 MS. MASSIE: Judge, can I say two things?
9 THE COURT: Of course.
10 MS. MASSIE: First, the reason we didn't list
11 Professor Steele as a witness in the Pre-trial Order was
12 because he was listed as a will-call witness and the local
13 rules give us the right to call him without further notice --
14 THE COURT: Let me just indicate one thing, I didn't
15 sign that order -- I don't want to get into the technical part
16 of it, but I don't want -- I think the lawyering in this case
17 has been outstanding, I'm not criticizing anyone. However,
18 that order was not signed until I think the day after the
19 trial started.
20 Anyhow, go on.
21 MS. MASSIE: Well -- and I don't understand what
22 bearing that has. We were relying on the fact that he was
23 listed as a will-call witness. If he had been listed as a
24 may-call witness we would have done something about that too,
25 to preserve our right to get his testimony in this live record
BENCH TRIAL - VOLUME 8
TUESDAY, FEBRUARY 6TH, 2001
13
1 of this trial. There is nobody who can explain some of the
2 aspects of how race and racism effect academic performance and
3 standardized test scores in the same way Professor Steele can.
4 It is absolutely essential to any intelligent understanding of
5 the questions you set for this trial. And we must then take
6 his video deposition. We didn't take it because Mr. -- it had
7 already been taken by --
8 THE COURT: I'm nothing going to reargue this.
9 You've argued this already. He chooses not to testify here
10 today. You didn't take it. It was a discovery matter that
11 you chose. At that point there was no will-call or anything
12 of that nature during discovery in this matter because the
13 order had not been signed. There wasn't even a Pre-Trial
14 Order at that time.
15 And, again, I'm not faulting you. I think you're a
16 good lawyer. I think everybody is case are good lawyers. It's
17 not a question of good lawyering or bad lawyering. Professor
18 Steele chooses not to testify. I have no jurisdiction to
19 bring him here. Discovery is completed. The matter is
20 closed. There's nothing else I can do.
21 MS. MASSIE: It's not a discovery deposition. It's
22 a de bene esse deposition, to preserve his testimony for
23 trial.
24 THE COURT: But we're here in trial, and your case
25 is on. And we're going to continue with trial, period.
BENCH TRIAL - VOLUME 8
TUESDAY, FEBRUARY 6TH, 2001
14
1 I mean, that's as easy as it is. We're going every day. We
2 have a schedule, and we're going to try to do that schedule.
3 MS. MASSIE: But don't we have to be allowed to send
4 somebody out there simultaneously to take his deposition to
5 play it for you. We can't present our case to you adequately
6 without the man's testimony. And one other thing, Judge, I
7 did speak to him before Mr. Payton instructed me I wasn't to
8 speak with him any more, I did speak with him, with Mr.
9 Payton's knowledge, and he did not say at all that he didn't
10 want to testify for us. In fact, that's inconsistent with our
11 discussion.
12 THE COURT: He has a lawyer and his lawyer has made
13 that representation. And I have the highest regard for his
14 lawyer and as I do for you. And I'm sure wouldn't tell me
15 something that isn't correct, and I'm sure that -- in fact,
16 I'm positive you wouldn't, and I'm positive Mr. Payton
17 wouldn't either. That's his client's position at this point.
18 And as I indicated there's nothing else that I can;
19 do. We've known that he wasn't going to since the 19th of
20 January. There's been no secret. In fact, it's probably been
21 before that, but I think that's when it was finally announced.
22 MS. MASSIE: It wasn't before that.
23 THE COURT: The matter is closed. We can argue that
24 all day, but I've made my ruling, and we have to move on.
25 MS. MASSIE: Well, this is completely unfair, and
BENCH TRIAL - VOLUME 8
TUESDAY, FEBRUARY 6TH, 2001
15
1 you're making it impossible for us to make our case.
2 THE COURT: I don't think it is completely unfair.
3 MS. MASSIE: It is.
4 THE COURT: I think it complies with what the law is
5 and the rules.
6 The next motion that we have is motion -- well, we
7 can do this when we don't have a witness, to clarify the
8 record. We'll got to that sometime probably near the end of
9 the day.
10 MS. MASSIE: Call Professor Martin Shapiro.
11 MR. KOLBO: I'm sorry, I have one other matter with
12 the Court. It's a scheduling issue, and point of information.
13 I wanted to bring the Court up-to-date on what we anticipate
14 for our rebuttal case, and scheduling and so forth, and I have
15 a request to make to the Court. I have talked to counsel
16 about this. We anticipate calling at this point two rebuttal
17 witnesses. First of all, we're going to recall briefly Kinley
18 Larntz, to the stand to respond to some of the testimony of
19 Stephen Raudenbush. And secondly, we anticipate calling
20 Professor Gail Heriot who is a professor at the University of
21 San Diego Law School. She's got an expert report on file.
22 THE COURT: I've read it.
23 MR. KOLBO: And she'll be called first thing Monday
24 morning we anticipate based on what we think the schedule will
25 be at this point. Dr. Larntz has asked me to see if I can't
BENCH TRIAL - VOLUME 8
TUESDAY, FEBRUARY 6TH, 2001
16
1 find to accommodate him to get his testimony on this week. To
2 make a long story short, the only day that really works this
3 week, unfortunately, would be Saturday. And I wanted to raise
4 that as a possibility with the Court, as I know that the Court
5 had expressed at some time that there was a possibility --
6 THE COURT: If it's good for everybody else, I would
7 be more than happy to be available Saturday morning, if
8 everyone else has no objections.
9 MR. KOLBO: And I have talked with counsel. What
10 we've talked about for various reasons is to suggest that we
11 start early like at 8:30 on Saturday if that's okay.
12 THE COURT: It's fine with me.
13 MR. KOLBO: I think it will be very short, your
14 Honor. I think that our Direct will be less than an hour.
15 And I would be surprised if the Cross would be that long. So I
16 think we're about a half a day.
17 THE COURT: If everybody agrees, I have no problems
18 with it. If no one has any objection, Saturday morning, 8:30,
19 would be acceptable. Let me just check with my staff. Does
20 anybody have any problems? Steven?
21 THE LAW CLERK: No problem.
22 MR. KOLBO: I know it's not the most popular
23 request.
24 THE COURT: We'll make arrangements. As long as I
25 know, I'll make arrangements with the Marshal's Service, and
BENCH TRIAL - VOLUME 8
TUESDAY, FEBRUARY 6TH, 2001
17
1 we'll do it Saturday morning.
2 MR. KOLBO: Thank you, your Honor.
3 THE COURT: Before you let him go, let me just check
4 with the Marshals to make sure -- the Marshals won't give us a
5 hard time, it's with GSA who owns the building to make sure
6 that we can have a little bit of heat in here Saturday
7 morning. Sometimes they give us a hard time on that. Should
8 it be a problem, we'll work something out though.
9 MR. KOLBO: Thank you, your Honor.
10 THE COURT: Mr. Rosner, you're going to recall
11 after the Professor -- is it Professor or is a Doctor --
12 MS. MASSIE: Professor.
13 I apologize, Professor Shapiro will go first, and
14 then Mr. Rosner.
15 THE COURT: No problem. Professor, you can stand
16 right over here, we'll swear you in.
17 M A R T I N S H A P I R O ,
18 being first duly sworn by the Court to tell the truth, was
examined
19 and testified upon his oath as follows:
20 DIRECT EXAMINATION
21 BY MS. MASSIE:
22 Q Hi, Professor Shapiro.
23 A Good morning.
24 Q What's your job?
25 A Professor of Psychology at Emory University in Atlanta.
BENCH TRIAL - VOLUME 8
TUESDAY, FEBRUARY 6TH, 2001
18
1 THE COURT: Oh, what a nice place.
2 THE WITNESS: Thank you.
3 THE COURT: One of my nieces, in fact, two of my
4 nieces went there. We had an intern last summer that went
5 there. So I had an opportunity to visit the campus. It sure
6 is beautiful.
7 THE WITNESS: Thank you. I've enjoyed being there.
8 A long time now.
9 THE COURT: A long time.
10 BY MS. MASSIE:
11 Q How long have you been there?
12 A I came there in 1964. That's a long time.
13 Q As a professor of psychology?
14 A I came -- actually I was an associate professor of
15 psychology, and I guess four years later was a professor of
16 psychology.
17 Q And four years later you were awarded tenure?
18 A No, actually I came in '64. I had left the previous
19 position which I had tenure so I got tenure back in 1965.
20 Q Tell us, just briefly, about your employment before you
21 got to Emory if you would?
22 A Okay. I'll start the education I guess. I have a
23 bachelor's degree from Yale University with a major in
24 psychology. A PhD degree from Indiana University with
25 concentrations in psychology and mathematics.
BENCH TRIAL - VOLUME 8
TUESDAY, FEBRUARY 6TH, 2001
19
1 After I received my PhD I stayed a year at Indiana
2 as the United States Public Health Post-Doctoral Research
3 Fellow. Then I went to The University of Houston as an
4 assistant professor of psychology. And I became an associate
5 professor of psychology at the University of Houston, and then
6 a visiting associate professor at Baylor Med School.
7 I then moved to Emory as an associate professor of
8 psychology. And except for a sabbatical period at The
9 University of College London I've been at Emory the entire
10 time.
11 Q What areas have you focused on?
12 A Well, a long time ago I did laboratory work with animals
13 in learning and conditioning. But since -- after that I did a
14 fair amount of research in learning in children. And then I've
15 mostly concentrated in test and measurement. During the entire
16 period I've always been at -- on the mathematical statistical
17 side of psychology. So at the moment I would say statistics
18 and test and measurement.
19 Q What is your teaching? What are your teaching
20 responsibilities? What courses do you usually teach?
21 A I teach two courses, one called psychology and law, and
22 the other course is psychological test and measurement.
23 Q Have you offered expert testimony in the past?
24 A Yes.
25 Q Tell us the areas in which you have offered -- bearing
BENCH TRIAL - VOLUME 8
TUESDAY, FEBRUARY 6TH, 2001
20
1 mind we're going to be focusing on testing, educational
2 discrimination in tests?
3 A Well, let me get rid of the others first.
4 And irrelevant area that is irrelevant to this trial
5 that I've done a lot of testifying in is regarding the
6 probabilities associated with DNA, matching primarily in
7 criminal cases some in paternity cases. Again, it's really
8 statistics and probability but it's a different question. A
9 lot of cases in employment discrimination. Again, really the
10 same issues, that is, the issues of selection and the validity
11 of the selection procedures. In fact, the issues are
12 identical, that is does a selection procedure have disparate
13 impact and then is the selection procedure valid in a
14 psychometric sense. some cases dealing with other kinds of
15 statistical issues like targeting by state police in highway
16 stops has racial targeting. But the remainder of the cases
17 have dealt with educational testing, educational selection.
18 Again, the issues being the adverse impact of a selection
19 procedure and the psychometric validity or invalidity of the
20 selection procedure.
21 Q Tell us about some of those cases?
22 A Well, the first one was a case brought by Georgia Legal
23 Services Corporation in which we obtained the Office of Civil
24 Rights, that is, the U. S. Department of Education Office of
25 Civil Rights computers tapes on all of the public school
BENCH TRIAL - VOLUME 8
TUESDAY, FEBRUARY 6TH, 2001
21
1 districts in the state of Georgia. And suit was brought against
2 11 of the districts after we had analyzed the tapes and
3 selected the districts with respect to ability tracking, that
4 is, their use of testing to assign students differentially the
5 classrooms. And the use of testing to assign students to
6 special education.
7 I don't want to make it too long a story but
8 typically in many of the southern districts, of course,
9 Georgia is one of the southern states, the school systems did
10 not use ability tracking until after they were desegregated in
11 1971. So the issue was really a statistical analysis dealing
12 with whether -- within school placement had served as a
13 substitute for jurist segregation that is, where they were
14 segregating within the schools. And that case was in the
15 Southern District of Georgia, Savannah.
16 There was another case in Savannah dealing with the
17 use of an exit exam, that is, after the school system, Tagnal
18 County, Georgia, had desegregated, they then instituted an
19 exit, that is, you can only get your diploma if you passed the
20 exit exam which was which was the California Achievement Test
21 that they used.
22 Q Let me stop you for one second. The first case you just
23 mentioned the tracking case is that George State Conference?
24 A The Conference of Branches -- I'm sorry. The Georgia
25 Conference of Branches of the NAACP versus the state of
BENCH TRIAL - VOLUME 8
TUESDAY, FEBRUARY 6TH, 2001
22
1 Georgia.
2 Q And the other one you were just telling us about?
3 A It's styled as Johnson versus Sikes.
4 There is a slightly motive and really a school case
5 but it's a licensing test -- licensing case.
6 I guess the next case was Golden Rule Insurance
7 Company versus Mathias. Mathias was the commissioner of
8 insurance of the state of Illinois. And the educational --
9 THE COURT: Are these cases that he's testified in?
10 MS. MASSIE: Yes.
11 THE COURT: Do we need all the history of the cases?
12 MS. MASSIE: I think this one will be helpful. We
13 won't go into all of them.
14 THE COURT: Go on.
15 BY MS. MASSIE:
16 A And the Educational Testing Service was joined as a
17 co-defendant in that case. And that's in a sense why it's
18 relevant. And that case got settled. There was formed an
19 advisory committee which oversaw the settlement for a period of
20 five years, and then there were reports for another five years.
21 So that case involved a lot of interaction with the Educational
22 Testing Service after the case as well as before.
23 The other school case -- and I'll try to speed it
24 up, I'm sorry. One was sure Sharif versus the Board of
25 Education of New York which was a challenge to New York
BENCH TRIAL - VOLUME 8
TUESDAY, FEBRUARY 6TH, 2001
23
1 state's use of the SAT as a selection device for the awarding
2 of state -- college scholarships. And the case dealt with the
3 validity of the SAT for that purpose.
4 Another case was the People Care versus Rockford
5 Illinois case. Another case was the City of Little Rock
6 School District versus the Pulaski County School District. It
7 really was a challenge by the City of Little Rock School
8 District but the County was creating a white flight by the
9 testing procedures and the placement of student procedures
10 used by the Pulaski County School System. Pulaski surrounds
11 the City of Little Rock.
12 Then there is the case of Hopwood versus the State
13 of Texas dealing with the affirmative -- a challenge to the
14 affirmative action programs used by the University of Texas
15 Law School. There is also Wooden versus the Board of Regents
16 of the University of Georgia which was a challenge to the
17 University of Georgia affirmative action procedure in the
18 undergraduate college of the University of Georgia Athens.
19 That may exhaust, I'm not sure.
20 Q Some of those cases don't -- not all of them dealt with
21 questions about tests, test bias, test validity?
22 A Yes, they all do in one way or another. Some of them are
23 very specific to university admissions.
24 Q I'm going to try two things at once. If I can get you to
25 turn to Tab 170, in the binder that's in front of you. If you
BENCH TRIAL - VOLUME 8
TUESDAY, FEBRUARY 6TH, 2001
24
1 could take a look at that and tell us if it is the report of
2 your expert testimony that you filed in this case.
3 A Yes.
4 MR. PURDY: Excuse me, your Honor, I'm sorry to
5 interrupt. I apologize I didn't do it earlier, but just so
6 it's clear, we have our continuing objection.
7 THE COURT: Yes. You can do it whenever you like,
8 but your objection is continuing.
9 . MR. PURDY: That's fine.
10 THE COURT: It's a new day and a new witness.
11 MR. PURDY: I apologize.
12 THE COURT: No problem.
13 MS. MASSIE: You don't have to stand up and say that
14 every time. We can concede that you object to the relevance
15 of everything to do race and racism in this case.
16 THE COURT: Well, you know, there's no speeches.
17 He's made his objection, and let's go on.
18 BY MS. MASSIE:
19 Q Doctor Shapiro, that is your expert report?
20 A Yes.
21 MS. MASSIE: I'd like to move the admission of the
22 report and simultaneously ask that Professor Shapiro be
23 certified as an expert on psychometrics, generally questions
24 of test validity, test construction, test assessment.
25 THE COURT: Any objection?
BENCH TRIAL - VOLUME 8
TUESDAY, FEBRUARY 6TH, 2001
25
1 MR. PURDY: Only subject --
2 THE COURT: Continuing objection.
3 MR. NIEHOFF: No objection.
4 THE COURT: The report will be received and he will
5 be able to testify as an expert in that area.
6 MS. MASSIE: Thank you, Judge Friedman.
7 (Trial Exhibit Number 170 received into evidence.)
8 BY MS. MASSIE:
9 Q Professor Shapiro, in your report, your earlier work on
10 this case you wrote that there are four basic parts to the
11 business of putting a standardized admissions tests. I'm going
12 to focus your attention on the first two parts that you
13 identify for your testimony today. But if you could please say
14 what the four parts are and say a little bit about them for
15 everybody's benefit, as an introduction.
16 A The first step I discuss is the definition of the content
17 domain of the test. That is, very simply, the decision as t
18 what will be tested, that is what kinds of material will be on
19 the test. Just to use an example, on the LSAT there's a verbal
20 section, there's a quantitive section. Somebody made that
21 decision. There is no content regarding social science.
22 There's no content regarding art or music. I mean, those are
23 the kinds of decisions that are made optretory when the test
24 is first designed. The second step is the actual writing and
25 selection of the items on the test. That is, once you decide
BENCH TRIAL - VOLUME 8
TUESDAY, FEBRUARY 6TH, 2001
26
1 what the content area is you then have to write the test
2 questions and you have to test or pretest the examination
3 questions. And that is in some sense a very well, as I tried
4 to explain a very critical and important step in the process.
5 And really determines the nature and consequences of the test.
6 I assume we'll go into that in more detail.
7 The first step is an attempt to validate the test,
8 that is, an attempt to compare the results on the test to some
9 criterion, that is, to simply calculate the equation that
10 predicts some criterion from the test score itself. It may be
11 the test score is used in conjunction with other information,
12 for example, we all know that in the case of admissions
13 typically the predictive equation uses both the test score and
14 the prior grade point average. If it's college admission,
15 it's the high school grade point average. If it's
16 professional school it's college grade point average.
17 The fourth step of -- which frequently is not done
18 actually, but the fourth step is to look backward at the
19 predictive equation and ask the question did it actually
20 predict, that is, after the students are admitted into school
21 and have had a year or more of grades to go back and to see
22 whether the predictive equation was accurate, that is, the
23 degree to which it actually did predict grades.
24 Q How many times did test companies, test makes go through
25 this process that you've discussed, this four-part process of
BENCH TRIAL - VOLUME 8
TUESDAY, FEBRUARY 6TH, 2001
27
1 producing a test?
2 A Well, although the content domain has some tendency to
3 have its own inertia and stay rather constant, the process
4 itself is a continuing process. I mean you continually have to
5 come up with new forms of the exam. You have to -- new
6 versions, new questions. So you constantly have to test new
7 experimental items.
8 You then have to constantly look at last year's
9 students and the derive an equation from which you will now
10 try to predict performance by next year's students. And then
11 hopefully, although I say frequently is not done, you have to
12 look at this year's students one year later and say did the
13 regression equation, did the predictive equation that you
14 dragged from last year really work this year. So it's
15 continual. I mean, it's a never ending process. It's what
16 keeps selling tests.
17 Q I want to back you up for just a second. Why do you need
18 new test forms? Why do you need new versions of the test?
19 A Well, obviously -- people would memorize the test
20 questions and -- well, they would become a commodity of market.
21 I mean, you could sell test questions, but furthermore, if that
22 happens then obviously the test results would have no
23 particular meaning other than who could manage to acquire old
24 test scores or their equivalent.
25 Q Why do they have to test the new questions?
BENCH TRIAL - VOLUME 8
TUESDAY, FEBRUARY 6TH, 2001
28
1 A Well, several reasons, but primarily you cannot really
2 predict whether a question is a good question. And that
3 statement contains a lot of pieces. You may design a question
4 that is misleading and not realize it, and that would not
5 become obvious until you pre-tested the question. And you look
6 subjectively at the -- I'm going to use a technical term, the
7 distracters, the wrong choices were chosen. And you might look
8 at the wrong choices and see well, people must have
9 misinterpreted this question to have picked this choice, and I
10 now see what was wrong with the question. So we either revise
11 the question or throw it out or whatever.
12 There is more to it than that simple effect. There
13 is the desire to have a high degree of homogeneity within the
14 test, that is, the test makers strive to have the results on
15 the test items be comparable from one item to another. They
16 want people who get a high score on the test overall, on the
17 total test, to also get -- also have the higher probability of
18 getting this particular item correct. That is somehow the
19 idea is that the test is measuring a single trait, a single
20 attribute and so all the items ought to correlate with each
21 other. But in somebody who gets a high test score after
22 getting all the items correct, and somebody gets a low test
23 score, ought to be getting all items wrong. Well, of course
24 it doesn't come out perfectly like that, but it should tend in
25 that direction.
BENCH TRIAL - VOLUME 8
TUESDAY, FEBRUARY 6TH, 2001
29
1 Q You say in your report as I recall that new test forms
2 have to be equated in addition to the statistical limitation
3 constraints you've described, new test forms have to be equated
4 to old ones. Can you say what you mean by that?
5 A That's a rather real complex process. What you strive to
6 do typically is to have the -- I'll use the term that I don't
7 really think is accurate, but it's used. You want to have the
8 same level of difficulty from test form to test form. Now, by
9 difficulty you have no way of measuring difficulty other than
10 how many people get it right or wrong, okay. But using that as
11 the definition of difficulty rather circular definition, but a
12 definition, you want to have your successive test forms of
13 equal difficulty so that a particular test score on the new
14 form is comparable to a test score on an earlier form because
15 when the students apply they have taken different test forms.
16 And the you want to have some assurance that the test forms are
17 comparable.
18 So without pre-testing, you would have no way of
19 knowing the difficulty level of each time. The difficulty
20 level ends up being more than a hundred people that got it
21 wrong or wrong.
22 So in order to have test equating, and in order to
23 have test homogeneity, internal consistency, the items must be
24 pre-tested. There is no alternative.
25 Q And the test equation requirement if I understand you the
BENCH TRIAL - VOLUME 8
TUESDAY, FEBRUARY 6TH, 2001
30
1 same thing to an admissions officer several years ago? Is that
2 a fair summary or not?
3 A Yes, and the way it's done is there -- although there are
4 different variations of it, at the basic level at some point
5 you must have common items, from form-to-form, because it is
6 possible that the test takers change over time. And it's only
7 by having items that are common to more than one form that
8 you're able to maintain the correction, that is, you're able to
9 maintain the -- any change in how you scale the exam, that is,
10 how you interpret the number of items correct.
11 So here you have a test form which consists of items
12 that been pre-tested on earlier forms, and then this test form
13 is taken by the same students who are taking these new
14 pre-test items. These new pre-test items go into a new form,
15 with newer set of pre-text items then every step along the way
16 you have some items that had appeared in a previous form
17 either as actual items or as pre-test item. So now you can
18 link the forms to each other and equate them.
19 Q So the test kind of become its own measuring stick?
20 A Oh, yes. The whole thing --
21 THE COURT: Yes. From now, you just have answer the
22 questions.
23 THE WITNESS: I'm sorry.
24 THE COURT: You have a very good lawyer questioning
25 you and she'll draw out whatever she needs.
BENCH TRIAL - VOLUME 8
TUESDAY, FEBRUARY 6TH, 2001
31
1 THE WITNESS: Yes.
2 BY MS. MASSIE:
3 Q But Professor Shapiro, you should develop the points that
4 you think need developing technically --
5 THE COURT: I think that's the attorney's rule, to
6 develop points that you believe are relevant. Otherwise, we
7 could be here for decades.
8 BY MS. MASSIE:
9 Q We'll try to work together, Professor Shapiro.
10 Let me turn your attention to the content domain
11 question.
12 A All right.
13 Q How does the domain of a test in terms of its contents
14 get determined?
15 A Very often the test is covering actual academic subject
16 areas. That is --
17 THE COURT: Do you have expertise to the domain. Do
18 you write exams? Do you help select the questions, the areas,
19 things of that nature?
20 THE WITNESS: Well, I have previously with respect
21 to exams. I mean, I don't that as my full-timed occupation.
22 THE COURT: Is that why you're here today to
23 testify? I thought it was a different area.
24 THE WITNESS: Well, it's part of it.
25 THE COURT: Okay, go on.
BENCH TRIAL - VOLUME 8
TUESDAY, FEBRUARY 6TH, 2001
32
1 BY MS. MASSIE:
2 Q Who determines the content of the test?
3 A Well, the test manufacturer. And on some exams the
4 content of the test is given to some extent by the content by
5 the of what you're trying predict. For example, freshmen in
6 college have to take calculus typically. Calculus requires the
7 knowledge of algebra and geometry so not surprisingly the SAT
8 covers algebra and geometry. It doesn't cover calculus because
9 calculus you take as a freshman in college.
10 Other exams like the SLAT, on the other hand, have
11 no real prerequisites to the subject matter that is going to
12 be taught. But if you don't learn law as an undergraduate as a
13 prerequisite as a basis for learning law in law school. So
14 the content of the LSAT is not subject matter driven. It is,
15 in fact, more of an aptitude or construct driven test. It's
16 timed to measure something abstract rather than specific
17 knowledge like the at SLAT to some extend does.
18 Q How does the way of doing that get decided on?
19 A Oh, It's a valued judgment by test constructor.
20 MR. PURDY: Excuse me, your Honor, I have to object
21 just on foundation grounds. If we're going to go into the
22 LSAT or the SAT unless there's some foundation laid that he
23 knows how they go about developing their questions, then I
24 think it lacks foundation.
25 THE COURT: I don't think they got into that
BENCH TRIAL - VOLUME 8
TUESDAY, FEBRUARY 6TH, 2001
33
1 specifically yet, but your objection is well taken. Right now
2 I think the question is general, how does somebody develop it.
3 BY MS. MASSIE:
4 Q Tell us what you know about the test development
5 procedures that go into the making of the SAT and the SLAT, if
6 you would.
7 A Well, the test manufacturers do go put out technical
8 manuals or descriptions of their procedures. The general set
9 of rules regarding the construction of tests is a document
10 referred as to the standards for psychological and educational
11 testing which is put out by the American Psychological
12 Association, and three3 of the educational associations -- or
13 two of them. I don't remember exactly which ones in education.
14 But these specified standards for test construction that are
15 generally accepted throughout the industry and the standards
16 require documentation so they require at test makers to state
17 what their procedures are and what their standards are. And
18 they typically will tell you the distribution of items,
19 difficulties on the test forms. They will tell you the leve of
20 correlation that they require between individual items and the
21 test as a whole.
22 I mean these are not trade secrets. These are --
23 this is information that has to be displayed in order to
24 satisfy the professional standards.
25 Q And have you looked at the literature that produced about
BENCH TRIAL - VOLUME 8
TUESDAY, FEBRUARY 6TH, 2001
34
1 the making of the make of the SAT and the LSAT?
2 A Yeah. Well, all of the admissions tests have some degree
3 of documentation. The SAT and the ACT have more available than
4 the others, but they all have some, enough to be able to
5 ascertain the procedure that's used.
6 Q Are the test development procedures and question
7 selection procedures that you started to talk about, and will
8 be talking about are those pretty much parallel across the SAT,
9 the LSAT, and GMET, the other standardized admissions tests?
10 A Yes. For example, the LSAT was first originally designed
11 and constructed by the Educational Testing Service using the
12 same general format and procedures that are used in the SAT.
13 It was later weaned from the Educational Testing Service but
14 it's not changed in its technical specifications.
15 Q With that let's go back to the question of how the
16 content of the test gest defined. I'm specifically interested
17 in asking whether it ever changes?
18 A Yes, there is a fairly recent interesting example of that
19 which really grew out of the case Sharif versus the New York
20 Board of Education. It is easily shown that the performance of
21 women on the SLAT below the performance of men. And the
22 argument was presented and actually the trial court in the case
23 recognized that argument as being correct, that what that
24 really means is that SAT can not be a valid predictor or across
25 sex, across men and women because, in fact, women get higher
BENCH TRIAL - VOLUME 8
TUESDAY, FEBRUARY 6TH, 2001
35
1 grades in secondary school and get higher grades in college
2 than men do. So if men are performing better on the SAT than
3 women but women perform better in both high school and college
4 well, the SAT is neither predicting college nor postdicting igh
5 school. therefore, the test is by definition invalid.
6 What was done is in order to ameliorate the
7 difference with respect to awarding of scholarships like the
8 national merit scholarship the Office of Education negotiated
9 an agreement with the Educational Testing Service in which a
10 writing section was added to the PSAT which is used for the
11 scholarship slip awards, in order to reduce the gap between
12 and male and female performance. I mean, there was change in
13 the content that was driven by the intended or unintended,
14 doesn't matter, consequences of the PSAT. So it was really
15 such a political solution, not a psychometric solution.
16 Q When you say the consequences of the PSAT, spell that out
17 for us.
18 A Well, if you look at the awarding of National Merit
19 Scholarships, like the awarding of the New York State
20 Scholarships, the majority of the scholarships go to males. I
21 forget the exact number, but it's quite different. I mean, it's
22 more than sixty-forty, for example. And logically, of course,
23 that makes no sense because again women perform better in high
24 school and perform better in college, and yet men are getting
25 the majority of the college scholarships, the National Merit
BENCH TRIAL - VOLUME 8
TUESDAY, FEBRUARY 6TH, 2001
36
1 Scholarships.
2 So the decision was made to try to reduce that
3 difference and the way it was done was to change the contents
4 of the test.
5 Q What was the impact of that intervention of the context
6 of the test, if you know?
7 A It did reduce the gap, and I don't remember the exact
8 numbers. It did not reduce it to fifty/fifty, but it do not
9 eliminate the gap. If you take -- if you consider grades as a
10 valid criterion, of course, the results really should be in
11 favor of women. That is, more women should get the
12 scholarships than men, and that's not true. More men still get
13 them. There is still a difference in male and female
14 performance on the SAT and PSAT.
15 Q Generally speaking, who ends up making these decisions
16 about the content?
17 A Well, it's the decision of the test maker. And the
18 example I just gave demonstrates that, you know, the test maker
19 in response to whatever pressures are put on the test maker
20 either by government or by the users of the test in this case
21 the colleges. The colleges really didn't do that; although
22 some colleges over time did eliminate their reliance on the
23 SAT. I mean, in a sense, that's pressure also as you begin to
24 lose market.
25 Q Could you make standardized admissions tests more
BENCH TRIAL - VOLUME 8
TUESDAY, FEBRUARY 6TH, 2001
37
1 predictive by changing the content?
2 A Very likely. The correlations between the admissions
3 test and actual performance in the first year of school are
4 frequently not very good. In fact -- although they differ from
5 school to school, it's not hard to find universities in which
6 the predictability of the admissions test is very poor. And so
7 it is certainly reasonable to think that if you broaden the
8 content of the test or change the content of the test that
9 would have an effect and many cases the only effect would be to
10 go up because the predictability is really very bad. Although
11 again, it differs from school to school.
12 Q What do you mean broaden the contents of the test?
13 A Well, if you look at the content and see that it only
14 covers a limited of skills or knowledge areas you could broaden
15 it in the sense of covering a larger spread of constant areas,
16 that is, it could well be, although it is speculation, it could
17 ell be, that adding social science or actually science -- I
18 mean, science really isn't on the SAT either -- I mean asking
19 questions about actual science, if you added social science and
20 science it would not be surprising that it might predict
21 first-year performance even better.
22 Q If it might -- if you could increase prediction by
23 tapping into a broader range of skills and knowledge areas why
24 isn't that done?
25 A Well, I think the biggest reason is simply the inertia
BENCH TRIAL - VOLUME 8
TUESDAY, FEBRUARY 6TH, 2001
38
1 but every time you change the contents of the test you loose
2 the ability to equate with prior forms. The test has now
3 changed, and so students who are applying, who took the test
4 three years ago are giving you a different measures than
5 students who are applying this year. Now -- I mean, it's not
6 insurmountable, and periodically the tests do get changed, but
7 it does create an administrative problem, and that's part of
8 the reason for the inertia. So, I mean, in some sense it
9 involves you're saying well, the test wasn't as good before as
10 it could have been. I mean, it's a funny admission of the
11 quality of the prior test. So there's a great tendency not to
12 do it. And the content remains relative stable even though
13 there really is no quantifiable statistical evidence to support
14 it remaining constant over time.
15 Q Let's turn to item selection. Tell us how new test items
16 come into being?
17 A Of course, it's basic level in committees or groups or
18 individuals who are contracted to write items. For example, the
19 Educational Testing Service has big brochure on item writing,
20 that they give new people who have not been involved in it
21 before. And the item writing at some level involves the
22 simplest things like can you use answers like none of the
23 above, or one or two versus three and four, or do you use
24 negation or -- all that kind of thing. At another level item
25 writing is -- becomes a statistical question of how does this
BENCH TRIAL - VOLUME 8
TUESDAY, FEBRUARY 6TH, 2001
39
1 item compare to other items. That, is what, is the
2 relationship between performance on this item performance on
3 the remainder of the test, the other items. As you get into
4 this whole issue that in general falls within the rubric of
5 test reliability, that is, the ability of the test to give you
6 a constant answer or relatively constant answer, not just
7 form-to-form but item-to-item.
8 Q How does an item to the point where it's being pre-tested
9 which as I understand is what you're getting at now How does an
10 item get to that point?
11 A Well, that point is simply writing the time and going
12 through some committee review or logical review. And then
13 those items are placed on different test forms. that is, you
14 need to pre-test lot of items so that the way you do it is if
15 you have, let's say, four hundred new items you might put --
16 you could put ten of them on this many tests that handed out.
17 You could put another ten on another number of tests that are
18 handed out. So you get these smaller samples of individuals who
19 each only are doing ten of these questions but over the whole
20 testing group that day that test administration, you're doing
21 all these four or five hundred test items.
22 Then at the end of the you've got the ability to
23 calculate whether the probability of getting this item, what
24 is the probability of getting this new item correct compared
25 to this other old item, or set of old items. So the ultimate
BENCH TRIAL - VOLUME 8
TUESDAY, FEBRUARY 6TH, 2001
40
1 selection is made on the basis of the item performance across
2 these individual groups of test takers. That's how you
3 generate enough pre-tested items.
4 Q So every item that ends up being scored is pre-tested; is
5 that true?
6 A That's correct. It's put all in a prior form and
7 pre-tested. Now, when I say it's true, it is possible that you
8 might make some slight change in the item and assume that one
9 changes the statistical characteristics. But even that's
10 unusual.
11 Q What information do the test companies get from the
12 pre-testing?
13 A Well, originally the information that was gotten is
14 actually -- let's say it this way: There are actually test
15 strips that are made up. There's a statistical table that is
16 made up for each item. And you can look up the item in its
17 statistics. And originally those statistics were the percent
18 correct for that item. And that would be the average test
19 score of every person who picked Choice A, that is their
20 average performance on the rest of the test. The average test
21 score for a group of people who chose alternative B, okay,
22 that's on the test strip. And then also on the test is some
23 estimate of the item difficulty calculated on the basis of the
24 percent correct on that item versus the percent correct on the
25 other item, that is, some comparable measure, some standardized
BENCH TRIAL - VOLUME 8
TUESDAY, FEBRUARY 6TH, 2001
41
1 corrected difficulty.
2 And then there's a section of the item strip that
3 has to do with the correlation between that item and the rest
4 of the test. And frequently that's done in terms of looking
5 at the lowest performing third of all the test takers, what's
6 the correlation between their performance and the performance
7 on this item looking at the middle third, looking at the top
8 third, and say you get this distribution of correlation.
9 That's the way it was originally done. Later on, as
10 really as a consequence of the Golden Rule versus Mathias
11 lawsuit, those test strips to contain information on
12 statistics that were broken down by racial and sex groups,
13 that is, the statistics would be shown separately for male and
14 female. The statistics would be shown separately for the
15 major racial groups. And then there were also developed
16 techniques creating statistics which purported to show the
17 relative difference between the racial groups in performance
18 on that particular item, that is, a method called differential
19 item functioning was developed by the Educational Testing
20 Service, purportedly to show which items have bigger racial or
21 sex differences than the overall test items, which items stood
22 out as being more impacting on a particular group than the
23 item -- the remainder of the items.
24 Q What do you mean more impacting?
25 A Here's the technique. We take everybody who got a hundred
BENCH TRIAL - VOLUME 8
TUESDAY, FEBRUARY 6TH, 2001
42
1 and fifty on the exam, just to pick a score, it doesn't matter.
2 We'll assume -- I don't agree with this -- but I'm saying it
3 this way -- we'll assume that everybody who gets a hundred and
4 fifty is of equal ability. Now, we'll look at this particular
5 item and we'll ask what was the percent correct for the women
6 and what was the percent correct for the men. Given that we
7 presume that the men and women we're comparing are equally able
8 because they all got a hundred and fifty, we'll ask is the
9 difference between percent correct male versus female on this
10 item greater than we would expect by chance as we'll run a
11 statistical test on it to see whether this item is giving you a
12 sex difference even though we're comparing people who are
13 otherwise equal in test performance.
14 Q Well, how does that work if the test has a different
15 impact on women, for example, than men, black test takers than
16 white ones?
17 A As I say, I don't agree but that method, that method
18 that's called DIF, differential item functioning, actually does
19 look at the job because if you look at the --
20 THE COURT: You don't agree, but there are other
21 experts that have obviously tested it use it as a standard to
22 some extent?
23 THE WITNESS: No, I wouldn't say it that way, your
24 Honor. The testing companies, specifically the Educational
25 Testing Service uses it. I agree that it does pick the most
BENCH TRIAL - VOLUME 8
TUESDAY, FEBRUARY 6TH, 2001
43
1 extreme items. I do not agree that it's sufficiently
2 sensitive to pick items that have less than most extreme
3 differential --
4 THE COURT: But you said whatever it is, Educational
5 Testing, uses it. You also said they have some kind of
6 protocol or something that's distributed in the industry, and
7 academically and so forth. They must have devised this system
8 somehow.
9 THE WITNESS: Educational Testing Service,
10 statistician, Paul Holland, devised it as a result of the
11 litigation with the Golden Rule Insurance Company as an
12 alternative technique for detecting bias items.
13 THE COURT: You suggested a better technique?
14 THE WITNESS: Yes, I think there are better
15 techniques. I then argue that then in the extreme, that is, in
16 an extremely biased item you would detect it by using the DIF
17 Method. I'm simply saying there are items that are biased but
18 not as extremely biased, that the DIF Method is insensitive
19 to.
20 THE COURT: But other than the DIF Method, there's
21 another method they could use in or do that.
22 THE WITNESS: Yes, they do, but they don't.
23 THE COURT: They chose not do.
24 THE WITNESS: Yes.
25 THE COURT: Do you know why they chose not? Of your
BENCH TRIAL - VOLUME 8
TUESDAY, FEBRUARY 6TH, 2001
44
1 own knowledge, not what you've heard, or -- I mean, have you
2 ever approached them, or ever discuss it with them?
3 THE WITNESS: Yes. I was involved in the committee
4 that formulated the Golden Rule settlement with the
5 Educational Testing Service, and I would then -- Paul Holland
6 and I were both involved in the advisory committee that
7 oversaw it. And it was developed as he and I interacted. So I
8 am familiar with it.
9 One of their arguments, I understand. Let --
10 THE COURT: You don't have to get into it. Would it
11 be fair to say you have a disagreement with them in terms of
12 -- they believe one thing and you believe another?
13 THE WITNESS: Yes, that's fair.
14 THE COURT: You may move on.
15 BY MS. MASSIE:
16 Q What's the basis of the disagreement?
17 A Okay. If you take the LSAT, it is clear that at the very
18 top of the distribution that is, let's say, at the very highest
19 score --
20 THE COURT: Why don't you explain -- you say -- you
21 say, "it is clear." Tell us if it clear to both sides or just
22 to you.
23 A It's clear to everyone.
24 THE COURT: Okay.
25 A That the distribution of LSAT scores is such that women
BENCH TRIAL - VOLUME 8
TUESDAY, FEBRUARY 6TH, 2001
45
1 are not in the very highest scores. That is, if you look at
2 the top of the LSAT distribution and these distributions have
3 been published in the past so this is literally at --
4 THE COURT: You're testifying under oath. It's
5 clear to everybody -- differentiate between what's clear to
6 you, and what's usually may be the difference between them. As
7 to the top of the women, everybody agrees is what you're
8 saying.
9 A The numbers clearly show that women are under-represented
10 in the highest, the very, very highest LSAT scores, which means
11 that when you say let's hold the ability constant and you set
12 that constant value for your -- this one group at the very
13 highest score, you can't ask that question because there are,
14 in essence, no women in that group, or if you do it by race.
15 What you get looking at the distribution is although you say
16 you're holding people equal by ability at the very highest
17 levels you can't a statistically significance difference
18 between the performance of Group A and Group B because there
19 are not enough people out of Group B even in the group to get a
20 reliable estimate of the item, the particular item difficulty.
21 So what you have is a method that is measuring relative
22 differences between items, but has no standard for overall test
23 bias. In fact, it incorporates overall test bias by taking
24 that total test score as your measure of ability.
25 Clearly if you use the DIF Method and had an
BENCH TRIAL - VOLUME 8
TUESDAY, FEBRUARY 6TH, 2001
46
1 external validate measure of ability, the method would make
2 sense. But if the measure of ability is the performance
3 internally on the test itself, then obviously the method
4 incorporates the overall average level of test bias, and
5 you're simply evaluating an item against that average level of
6 test bias.
7 Q So to be clear there are groups of people who score lower
8 on standardized admissions tests, and let's focus on the LSAT,
9 women are one such group on average; is that true?
10 A Yes, although with women once you get off the top of the
11 distribution it rapidly becomes consistent looking. So if
12 you're looking at the number of men and women in the 70th
13 percentile you're not to going find a sex difference. If
14 you're looking at the 99th percentile, you would.
15 Q In the aggregate, on average, there is a score gap
16 between women and men on the LSAT?
17 A I don't know if it's statistically significant on the
18 average. In some sense, it's an irrelevant question and
19 because people at the bottom of the distribution don't get
20 admitted. So the real issue is what happens in the upper
21 reaches of the distribution, And yet in the upper reaches of
22 the distribution there's a difference.
23 Q Let's turn to racial groups. Are there aggregate
24 differences in test scores to your knowledge by race on the
25 LSAT?
BENCH TRIAL - VOLUME 8
TUESDAY, FEBRUARY 6TH, 2001
47
1 A Yes, there are.
2 Q Describe them.
3 A Well, it varies from school-to-school. I'm don't think
4 you can come up with a general description, but if you look at
5 the overall distributions -- it would look as though you have
6 one valve-shaped curve for whites, and let's say another valve
7 shaped curve for African-Americans, and it's displaced by a
8 pretty good number of points. I don't have an exact number, but
9 I mean it's not clearly statistically significance difference.
10 I mean, you can see it. It's two different distributions. I've
11 seen a lot of these kind of data. There's no dispute about
12 that.
13 Q I want to tie this back to what you were saying --
14 THE COURT: When you say no dispute, does the
15 Educational Service, do they dispute that?
16 THE WITNESS: No, no one does. I mean, when you
17 look these tests scores it's clear that you get very large
18 differences due do economics. I mean you see that in the SAT
19 where students have fill out a questionnaire before they take
20 form, and you can just plot the distribution for everyone
21 whose income is in this bracket, or the next bracket. And
22 they give out the samples, the data, each year of about a
23 hundred thousand randomly selected test papers. You can buy
24 the CD rom for a hundred and fifty dollars and look at it
25 until your heart is content, by economics, by race, by
BENCH TRIAL - VOLUME 8
TUESDAY, FEBRUARY 6TH, 2001
48
1 urban-rural, by state, by religion. I mean, there's huge
2 differences from religious group to religious group. And it's
3 just in the data base. I mean you can look at it.
4 BY MS. MASSIE:
5 Q Going back previously what you were saying about
6 differential item functioning, how is it that method of
7 attempting to get at bias which is now I assume how ETS frames
8 it, deals with the average gap you've just described by race?
9 A It doesn't.
10 Q Why is that?
11 A Because it's comparing only people who have been equated
12 on totals test score. So if you look at these two
13 distributions when you're looking at the top grades you're
14 containing almost no people at all from this other racial
15 group. And then finally when you get down into a lower range
16 are you comparing let's say blacks to white, but that is
17 assuming that those students black and white are equal in
18 ability just because they got the same test score, but, in
19 fact, you know that's not true.
20 Q Why?
21 A Because the test items are selected to conform with, to
22 correlate with, overall performance. So, let's say, let's take
23 two things that are true. Let's say, we've got African-
24 American scoring as whole lower than whites. Now,let's assume
25 that we get an item in which African-Americans actually do
BENCH TRIAL - VOLUME 8
TUESDAY, FEBRUARY 6TH, 2001
49
1 better than whites. Well, when you calculate the correlation
2 between that item and total test score, that correlation, in
3 fact, would be negative. The students who do worse on the test,
4 the African-American students, are doing better on that item.
5 The students who do best on the test, the white students, are
6 doing worse on the item, you get a negative correlation, that
7 item gets thrown out. In fact, the only items that will get
8 selected are the items that for future use are the items that
9 preserve the difference that existed in the earlier test form,
10 because they're the only items that are consistent the earlier
11 test score. They are the only items that have a high positive
12 correlation.
13 THE COURT: But the same exact principle would apply
14 for economics, for rural versus urban --
15 THE WITNESS: That is right.
16 THE COURT: For religion.
17 THE WITNESS: That is correct.
18 THE COURT: So it's not just race.
19 THE WITNESS: That's correct.
20 BY MS. MASSIE:
21 Q Does racial Impact of the different questions is made
22 known by the pre-testing procedures; correct?
23 A Absolutely. As I said before, it is now standard
24 practice to calculate the statistics for racial and sex groups
25 separately. The data are all there, they're laid out in front
BENCH TRIAL - VOLUME 8
TUESDAY, FEBRUARY 6TH, 2001
50
1 of you.
2 THE COURT: They do it for economics too?
3 THE WITNESS: No.
4 THE COURT: They don't do it for rural or.
5 THE WITNESS: No.
6 THE COURT: Or north versus south.
7 THE WITNESS: No.
8 THE COURT: But you would predict it would be the --
9 THE WITNESS: Absolutely.
10 A The test is designed to preserve the status quo. That's
11 the effect of it. Whoever did before is now going to be
12 carried along by picking items that create a similar group of
13 winners again.
14 BY MS. MASSIE:
15 Q Is there anything external that's used to say whether a
16 question should be thrown out or not?
17 A No. There is an example that is cited by the Educational
18 Testing Service. In 1932, Carl Bringham who originated the a
19 LSAT test did a study in which he compared some antonyms
20 performance or some antonyms questions to actual English course
21 grades six months later. As far as I can tell that is the only
22 time anyone has compared test performance item-by-item to some
23 external criterion, and that was in 1932.
24 THE COURT: But those that may use these exams for
25 whatever purpose are aware of the same thing that you are
BENCH TRIAL - VOLUME 8
TUESDAY, FEBRUARY 6TH, 2001
51
1 aware of or may be if they wish to make -- avail themselves to
2 that information and, therefore, if they're the decision
3 makers they can take that into consideration, in making what
4 use they may use in these exams.
5 THE WITNESS: Your Honor, I agree with you and I'm
6 not trying to sit here and say that the exams are evil. I'm
7 saying that if you understand what the test means, and what
8 the test scores mean then they do provide you with additional
9 information.
10 THE COURT: As long as you understand what they
11 mean. They don't take them at raw scores.
12 THE WITNESS: That is correct.
13 THE COURT: You take into consideration that they
14 may have some bias in terms of race, but they may have some
15 bias in terms of urban versus rural. They may have some
16 religious bias --
17 THE WITNESS: But you can't look at these predictive
18 equations based on test score and grade point average and use
19 them in a mechanical way ignoring all the other attributes of
20 each additional applicant and come up with a reasonable and
21 valid answer.
22 THE COURT: Now, I understand your position.
23 Thanks.
24 BY MS. MASSIE:
25 Q What's the impact on black and minority test takers of
BENCH TRIAL - VOLUME 8
TUESDAY, FEBRUARY 6TH, 2001
52
1 the test development procedures that you've described?
2 A Well, the impact is enormous. The difference in
3 performance among racial groups is large enough that if you
4 have a highly selected institution and you rely exclusively on
5 this predicted first-year GPA you are going to exclude all the
6 members of various minority groups.
7 THE COURT: But only minority groups, but also
8 groups that may be the majority, that are of low economic area
9 --
10 THE WITNESS: That is correct, your Honor.
11 The biggest study I've ever seen was done at UCLA where they
12 actually looked at the economic status of the admitted
13 students, and seventy percent -- this is a state school, after
14 all -- but seventy percent of the students admitted to the law
15 school came from families with yearly incomes of excess of two
16 hundred thousand dollars. That is an extraordinary statistic.
17 They then instituted a program in the admissions procedure
18 trying to get greater economic diversity in their student body
19 by considering other information, by reading the applications
20 more thoroughly, by trying to evaluate them less -- relying
21 less on the predicted first year GPA. And they were able to
22 ameliorate it to some extent. But, yeah, the effect is
23 enormous.
24 BY MS. MASSIE:
25 Q Other than the one instance you've described from 193,
BENCH TRIAL - VOLUME 8
TUESDAY, FEBRUARY 6TH, 2001
53
1 are you aware of any time when the test development companies,
2 the test making companies have tried to use something external
3 to define ability as oppose to simply the results that they've
4 already achieved on a test?
5 A There are two answers to the question. There is an
6 instance on an old version of the LSAT in which they looked at
7 the correlation among the subparts of the exam. The exam had
8 more different subparts, that is, had more homogeneity in it,
9 back then. And they found a reasonably high correlation, and
10 they took that as providing some validity evidence for the
11 exam, but still it was internal, it was the subparts of the
12 exam compared to each other.
13 There are also always done as a matter of course the
14 correlation between the exam and the GPA, that is, the prior
15 grade point average, and that is sometimes taken as some
16 evidence for the validity of the exam.
17 It's a kind of funny argument because, of course, if
18 the correlation were very high you wouldn't need the exam.
19 You've got to be careful using that argument. But in fact
20 they're spared that because the correlation isn't very often
21 very high. Sometimes it's in the middle range, sometimes it's
22 in, you know, you're picking up maybe ten, twenty percent of
23 the variability. That is, there is some relationship between
24 the exam and the GPA. But typically it's very low, in fact.
25 There is an -- sorry to belabor this -- there is a
BENCH TRIAL - VOLUME 8
TUESDAY, FEBRUARY 6TH, 2001
54
1 study done by SLAT or the LSAS on schools that had large
2 enough -- law schools that had large enough minority
3 enrollment. I think the cutoff was fifty minority students in
4 several categories, that is, African-American, or Hispanic.
5 In fact, they broke Hispanic down into Latinos also. But
6 looking at those schools that have relatively large numbers of
7 people in those groups, and they published a study showing the
8 correlations between LSAT and undergraduate GPA within each of
9 the three racial groups for those particular schools are
10 identified by code numbers. And it is interesting in many of
11 the schools the correlation within the minority racial groups
12 between undergraduate GPA and their LSAT scores is very low.
13 In fact, I mean down near no correlation at all. In fact,
14 there are a couple of places where the correlation is actually
15 negative, that is, the students who got higher grades, in
16 fact, got lower SLAT scores. This was published in 1992 by
17 the law schools. LSAT scores there was public 1992 by law
18 school, now the applicant service.
19 Q Doctor Shapiro, what is your view as the way out of this
20 circularity and maintenance of bias that you've described?
21 A Well, I think the Court asked me a question that I gave
22 the answer to before. There is no way out except to look at
23 the other information regarding each applicant and to evaluate
24 the LSAT score and the undergraduate GPA, that is, the
25 predicted first-year average in light of all those other
BENCH TRIAL - VOLUME 8
TUESDAY, FEBRUARY 6TH, 2001
55
1 considerations. The -- to argue otherwise is really to argue
2 that you ought to take the predicted first-year GPA and just it
3 use as an absolute cut, that is, everybody above some level
4 gets in, everybody below it doesn't get in. But that's never
5 bene true. I mean, nobody ever was foolish enough to use that
6 way. In fact, when you at the practices they frequently are
7 about the admissions committee considers other attributes of
8 the student, what is this student's LSAT for somebody at that
9 economic level, for somebody of that race, for somebody from
10 that rural area. I mean, this is what's done and that's what
11 has to be done.
12 Q Do you know of any interventions into the contents of the
13 text like the one you've described for the PSAT, national merit
14 scholarships, on behalf of minority students?
15 A No, it's never been tried.
16 Q Is there any way other than speaking specifically of race
17 and the racially biasing effect of the LSAT, is there any way
18 other than taking race into account, or taking racism into
19 account in admissions that you can counter-act that?
20 A No. Actually if you think of other minorities groups you
21 even get clearer examples of that, I mean other than
22 African-Americans. If you take Hispanics where you've got a
23 large number of people who are bilingual, or for whom English
24 is not their first language, those test scores, and even the
25 GPAs must be -- I mean to make sense, must be evaluated in
BENCH TRIAL - VOLUME 8
TUESDAY, FEBRUARY 6TH, 2001
56
1 terms of that disadvantage. I mean -- in fact, the New York
2 State Commission, the committee that I was on, was looking at
3 English as the first language as well as race and sex. I mean,
4 those are huge variables. Being bilingual is a great
5 disadvantage in these English language multiple choice exams.
6 And, of course, in the real world, being bilingual is certainly
7 and advantage, but not if you're taking the SAT or the SLAT.
8 So I mean, there's no way to use that information
9 without using all these other characteristics of the student.
10 Q Just one last question, Professor Shapiro, in your view
11 does the aggregate difference in scores by race on the LSAT
12 indicate anything about the ability of blacks and other
13 minority LSAT takers to study or to practice law?
14 A I know no evidence that suggest that, no.
15 MS. MASSIE: That's all I have.
16 MR. NIEHOFF: Your Honor, if we could just have a
17 few minutes.
18 THE COURT: I was going to take a break at eleven
19 because I have some sentencings at eleven. Do you want to
20 take a break now?
21 MR. NIEHOFF: If we could.
22 THE COURT: Okay. We will break for lunch around
23 12:30.
24 We will stand in recess.
25 (Court recessed, 10:50 a.m.)
BENCH TRIAL - VOLUME 8
TUESDAY, FEBRUARY 6TH, 2001
57
1 (Court reconvened, 11:30 a.m.)
2 THE COURT: Okay. Ready?
3 MR. NIEHOFF: Your Honor, we don't have any
4 questions.
5 THE COURT: Plaintiff?
6 MR. PURDY: Yes, your Honor.
7 CROSS-EXAMINATION
8 BY MR. PURDY:
9 Q Good morning, Professor Shapiro.
10 A Good morning.
11 Q My name is Larry Purdy. I'm an attorney for the
12 plaintiff. And we had not had a chance to meet until this
13 morning. That is the first time we've actually got a chance to
14 shake hands and say hello to one another. Just a couple of
15 questions, really.
16 Counsel at the end of her examination asked you
17 about whether or not there have been any interventions on the
18 SLAT on behalf of minority students to help eliminate the bias
19 that you've discussed and I believe your answer was none; is
20 that correct?
21 A Yes.
22 Q You said it's never been tried?
23 A Yes, I think that's what I said.
24 Q Why haven't you tried it?
25 MS. MASSIE: Objection. Vague, your Honor.
BENCH TRIAL - VOLUME 8
TUESDAY, FEBRUARY 6TH, 2001
58
1 BY MR. PURDY:
2 Q Have you made an effort to try to make a recommendation
3 to help eliminate the bias on the SLAT as you understand it in
4 terms of its impact on minority applicants in particular?
5 A No, I never had the pretension of thinking that, that I
6 would have an influence on the construction on the LSAT.
7 Q You mentioned earlier to us I thought in connection with
8 the Golden Rule case -- incidentally the Golden Rule
9 case involved an insurance exam?
10 A That is correct.
11 Q As I understood it, and maybe you can correct me if I'm
12 wrong, Professor Shapiro, I thought you indicated you had dealt
13 with an ETS representative on the committee that oversaw that
14 settlement? Was I misunderstanding your testimony?
15 A No, the settlement provided that both ETS and Golden Rule
16 would have representatives on the advisory committee that
17 monitored and oversaw the settlement.
18 Q My question is in terms -- so you had contact in terms of
19 at least the Golden Rule settlement with representatives from
20 ETS; is that correct?
21 A Yes, but before the settlement. I mean, there were
22 months of negotiations. ETS was represented by Wilmer, Cutler
23 and Pickering. Golden Rule was represented by counsel. The
24 state of Illinois was represented by the Attorney General's
25 Office. ETS and Golden Rule brought their statisticians to the
BENCH TRIAL - VOLUME 8
TUESDAY, FEBRUARY 6TH, 2001
59
1 negotiations. And this went on for months of a very protracted
2 period of time. And then after the settlement, the settlement
3 required that the data be given to the Social Security
4 Administration to be broken down by race and analyzed by race.
5 And we had to cooperate in arranging that. And then the over--
6 the supervisory committee contained several people from EST.
7 And also ETS presented to the committee the item-by-item
8 statistical results. The committee actually reviewed items for
9 content and then reviewed the items for their statistical
10 results, and approved or disapproved the test forms constructed
11 by EST. I mean it was very intrusive, the process.
12 Q You mentioned that Wilmer, Cutler and Pickering were
13 representing ETS in this matter. You understand that Wilmer,
14 Cutler and Pickering are the lawyers representing the
15 University in this case?
16 A Yes.
17 Q Do you know what position ETS was taking in the Golden
18 Rule case with regard to whether or not there was racial bias
19 in the test?
20 A It was unknown, actually. The State of Illinois in the
21 application for taking the exam did not include a question
22 regarding race. The underlying suspicion came because the
23 adjoining state, Wisconsin, used the same exam and did, in
24 fact, inquire as to the race of the test takers.
25 I don't think ETS took a position with respect to
BENCH TRIAL - VOLUME 8
TUESDAY, FEBRUARY 6TH, 2001
60
1 Illinois. And, in fact, there were no data as I've just said
2 until after settlement when the ETS computer files which
3 identified test takers by Social Security number were given to
4 the Social Security Administration to do an item analysis that
5 is the very same analysis that the test makers do. But it
6 wasn't until that point after the settlement that there were
7 any hard data concerning racial impact.
8 Q I apologize and maybe my question wasn't clear. Do you
9 know whether or not at any time in the litigation prior to
10 entering into a settlement ETS took a position one way or the
11 other as to whether or not the test they had developed was
12 biased against any minority group?
13 A I don't think they actually a position. The litigation
14 up to that point had been that they had gotten themselves
15 dismissed ETS, because they argued they were not a proper
16 party. They simply constructed the test, they didn't use it.
17 The trial court accepted that argument. They were dismissed as
18 a party. Golden Rule then appealed to the Illinois appellate
19 court and got a reversal, the appellate court saying that
20 because they had counseled the state in the manner that they
21 did that their action was, in fact, state action, and they were
22 brought back in as a party. At that point the settlement
23 negotiations began. I don't there was any actual claim or
24 disclaim by ETS up to that point.
25 Q Do you know what position ETS takes today of whether or
BENCH TRIAL - VOLUME 8
TUESDAY, FEBRUARY 6TH, 2001
61
1 not the SAT, for example, is biased against minority test
2 takers?
3 A Oh, I think ETS said that because of its methods for
4 reviewing items subjectively for discriminatory content, and
5 its use of the DIF, Differential Item Functioning Method, that
6 it has attempted to insure test fairness, that is, to remove
7 unfairness from the test. Unfairness being a somewhat
8 different concept than bias, okay?
9 I do not that Educational Testing Service denied
10 that there is differential item -- I'm sorry, that there is
11 differential predictive validity to the test. That is,
12 specifically with respect to women that the test predicts
13 differently than it does for men. I don't think they deny
14 that. In fact, I believe Doctor Cole who was the president of
15 ETS for a while acknowledged that after the Sharif case.
16 I think you are correct in your implication that
17 they then say -- they don't advertise this is a bias test.
18 You are correct in that.
19 Q Do you know Wayne Comara (sp)?
20 A No.
21 Q Do you know whether or not Mr. Comara who serves in a
22 senior capacity with ETS, was retained by the University in the
23 undergraduate case, for example?
24 A I don't know anything about it.
25 Q I gather you have not been provided with the deposition
BENCH TRIAL - VOLUME 8
TUESDAY, FEBRUARY 6TH, 2001
62
1 of Mr. Comara who was a University expert in the undergraduate
2 case to read what he may have said about whether or not the SAT
3 tests or standardized tests are biased against minorities?
4 A No, I have not read his testimony.
5 Q You made a comment that ETS takes a position, and I want
6 to understand this clearly, that ETS predicts female test
7 takers' performance differently than it does males --
8 A No, I didn't say that provide those predictions. What
9 I'm saying is that they do not deny that the data show that
10 there's differential predicted validity for males and females.
11 The -- I'm going to say the APA standards, the American
12 Psychological Association standards for educational and
13 psychological testing -- psychological and educational testing,
14 I'm sorry, provide that whenever there is a difference in group
15 performance that the test maker is obligated to investigate
16 whether or not there is differential predicted validity. And
17 ETS has done this, and it is confirmed that there is.
18 Q Does the -- let's go back just for a second. What is
19 your position as to whether or not the LSAT or the SAT either
20 one predict the performance of minority students?
21 A It varies from institution-to-institution. If you look
22 at the data published by LSAS where they actually give the
23 correlation coefficient the differences are enormous from
24 institution-to-institution. So I don't think you can come up
25 with a general answer to that.
BENCH TRIAL - VOLUME 8
TUESDAY, FEBRUARY 6TH, 2001
63
1 Q Have you read any of the materials that have been
2 produced in this case by other experts for the University such
3 as Derek Bok and William Boland?
4 A No, I have not read their testimony.
5 Q You don't know what their position is as to whether or
6 not the SAT or LSAT may or may not be biased against certain
7 minority groups?
8 A I mean I'm familiar with them only in the general context
9 of the book which they wrote. I don't know what they said
10 about the test, specifically.
11 Q Do you know whether or not they a position in the book
12 that you just referenced that the tests, the standardized tests
13 are not culturally biased against minorities?
14 A No, I don't know one way or the other. I have not
15 actually read their book.
16 Q Have you read the law review article by former Dean of
17 the law school, Terrance Sandlow, who is now a current
18 professor of the law school? Have you ever read his law review
19 article published in May of 1999 that touched on the subject of
20 whether or not standardized tests were culturally biased
21 against minorities?
22 A No, I have not.
23 Q So you don't know what position Professor Sandlow, former
24 Dean Sandlow, would take on that; do you?
25 A No.
BENCH TRIAL - VOLUME 8
TUESDAY, FEBRUARY 6TH, 2001
64
1 Q Are you familiar with the -- let me go back.
2 You mentioned four areas in your report --
3 incidentally, did you draft, did you actually draft the report
4 that has been marked and entered into evidence?
5 A Did I write --
6 Q That was your work product?
7 A Yes, I sat at the computer.
8 Q Do you recall making revisions to it at some point?
9 A Well, all along -- and I added to it as time went on. .
10 This is specifically true with respect to the fact that there
11 was -- or I was led to understand that there was a deadline for
12 the submitting the report and then at some point that deadline
13 was eased, or changed, or what have you and then I added some
14 and revised it some.
15 Q You outlined four areas and I'm going to -- let me
16 briefly mention them and then I want to ask you a single
17 question about each one, hopefully a single question. You
18 mention content domain. You mention item selection. You
19 mention the attempt to validate the test which in your report
20 is called the best fitting regression equation --
21 A That's correct.
22 Q And then you mention looking backward at the predicted
23 equations, in other words determining whether or not the use of
24 A That's correct.
25 Q Have you ever made any suggestions to -- let's start
BENCH TRIAL - VOLUME 8
TUESDAY, FEBRUARY 6TH, 2001
65
1 with: Who is responsible for developing the LSAT test today?
2 A I don't know the actual name of the group, but it's the
3 -- there are various organizations that have very similar names
4 and they all start with LSA, and it's that group.
5 Q Have you ever undertaken to find out who it is that is
6 currently responsible for preparing the test each year that
7 it's given to prospective applicants to law schools?
8 A You mean an actual identity, no.
9 Q Well, then my question may be simple. I gather it's safe
10 to say you have never written anything to the developer of the
11 test concerning your suggestions about how to improve the
12 content domain to remove the bias you believe is present; is
13 that correct?
14 A As I said before I have not been in direct contact with
15 -- I have not gratuitously offered them my advice.
16 Q Professor Shapiro, don't you believe your advice would be
17 helpful to them?
18 A I don't think there is anything unique about what I have
19 said. I think everything I've said is obvious to any cycle
20 nutrition and, therefore, I have nothing new to tell them that
21 they don't already know. I mean, this process has been going on
22 since around 1930, and there are no mysteries as to what's
23 done. We all agree as to what how the array tests are
24 constructed, and we all know the consequences of it. I don't
25 think I have anything brilliant to add.
BENCH TRIAL - VOLUME 8
TUESDAY, FEBRUARY 6TH, 2001
66
1 Q Do you believe that the developer of the LSAT and the ETS
2 with regard to the SAT that they are intentionally keeping bias
3 against certain groups that you've described -- understanding
4 they disagree with you in that regard -- but do you believe
5 they're intentionally keeping bias against certain groups or
6 genders in the test?
7 A It's a hard question. There is no question that the
8 original intent of the LSAT was a bias attempt. I'm sorry, the
9 -- I misspoke, the SAT. I'm sorry. There is no question that
10 the original version of the SAT was developed by Carl Bingham
11 for a bias purpose.
12 I'll elaborate. The problem was very simple. There
13 were too many Jews in the Ivy League undergraduate classes.
14 And there was an attempt to reduce that number by coming up
15 with an exam that could be used as well as high school GPA
16 because the high school GPA was -- as a criterion was
17 permitting a very large number of Jewish applicants to be
18 admitted. The Jewish student population at Columbia in the
19 undergraduate colleges is forty percent. And so a test was
20 developed that was modeled after the curriculum of the prep
21 schools which I think the idea was -- the idea as it was said,
22 was -- it's difficult for me to say it because it's so absurd,
23 the idea being is that you would be able to admit fewer
24 over-achievers who were just, you know, good students in
25 public high schools and it exceeded and the number of Jewish
BENCH TRIAL - VOLUME 8
TUESDAY, FEBRUARY 6TH, 2001
67
1 students in Columbia College was reduced to twenty percent
2 instead of forty percent. That was the beginning of the SAT.
3 Carl Bingham designed the item analysis technique. It says
4 Carl Bingham did this in the ETS technical handbook. There's
5 no mystery about it. Carl Bingham has a written a book on
6 intelligence of showing in the 1920 by his argument that
7 Nordics were more intelligent than other races. He later
8 recanted it. I don't want to demonize him. But that is the
9 beginning of the SAT and that is the beginning of item
10 analysis.
11 Now whether somebody is sitting in there today with
12 that same attitude, I don't want to think that. But I don't
13 see anybody changing it either.
14 It's a very long answer to your question, but I
15 don't want to call people a bigot. At the same time I don't
16 want to tell you that isn't how it started because that is how
17 it started.
18 Q Do you believe that the SAT test today is biased against
19 Jewish students?
20 A I think that would be very difficult to maintain if you
21 looked at the percentage of students at Ivy League schools who
22 are Jewish.
23 Q But my only question is: So the bias was removed in some
24 form or fashion --
25 A I --
BENCH TRIAL - VOLUME 8
TUESDAY, FEBRUARY 6TH, 2001
68
1 Q Let me just finish -- let me back up.
2 A Maybe that's why Jewish students started to go to prep
3 schools.
4 Q My question is you began by saying that the SAT was an
5 attempt to -- an intentional intent to bias Jewish applicants
6 to college.
7 A It was an attempt to create a content that they would
8 perform worse than another group of students. There's no
9 question about that.
10 Q But that's no longer true today as you just said that, if
11 you look at the --
12 A You know, I -- I don't think -- I can't imagine people
13 sitting around and saying, you know, who can we be bias against
14 today. But they have maintained the same past -- you know for
15 all practical purposes and the same item selection procedure.
16 Q Let me go back to the item. That's the second of your
17 four categories. I gather it's safe to say -- and I don't mean
18 to belabor this -- but you've never made any suggestions to ETS
19 or the group that is currently responsible for developing the
20 LSAT as to how to improve item selection to remove the bias
21 that you've discussed; correct?
22 A That's right. We started out -- I mean my venture in
23 this started out in the late 1970s, almost 1980. There was a
24 big flurry of activity. Changes were made. ETS instituted
25 differential item functioning. They did several things that
BENCH TRIAL - VOLUME 8
TUESDAY, FEBRUARY 6TH, 2001
69
1 were more sensitivity to the racial and sexual issues like the
2 change in the PSAT, and so forth. And things moved along. I
3 moved on to other things. We all stayed busy.
4 And then recently there has been this concerted
5 attack on affirmative action which has the effect of if it
6 succeeds of requiring people to make admissions decisions
7 based purely on numbers. So what was not a problem so much
8 before because you could counter-act the effect of the
9 selection process by considering other aspect of the
10 applications now is a very serious problem.
11 So you are right. I mean, I was sitting there kind
12 of dormant, like everybody else through all this time. But
13 you have now re-raised the issue. That is, the attack on
14 affirmative action now makes all of this relevant and
15 important again.
16 Q Let me ask you this: Would you recommend doing away with
17 the LSAT?
18 A I've never been in favor of getting rid of any
19 information that has any useful purpose. And it's not -- it's
20 not the devil. If you understand what it is and use it
21 appropriately it's all right. I'm not on a crusade to get rid
22 of information.
23 Q Are you in favor of applying different standards in terms
24 of evaluating the LSAT simply based on ones race or ethnicity?
25 Knowing nothing more -- nothing else about a student are you in
BENCH TRIAL - VOLUME 8
TUESDAY, FEBRUARY 6TH, 2001
70
1 favor of using different standards to evaluate standardized
2 test scores based on ones race or ethnicity?
3 A Of course, race and ethnicity is not only race and
4 ethnicity. I mean that's like saying merely or only or
5 exclusively. I don't think I've ever said that. Race and
6 ethnicity in our society like sex and other characteristics are
7 characteristics that affect what we do and how we perform.
8 They're not irrelevant. The alternative is to look at the
9 predicted first-year GPA numerically and ignore everything
10 else. And I think that's stupid. It's not only stupid, it
11 prenicious. But, you know, to look at the other things, yes. I
12 think you ought to look at all of them.
13 Q In fact, you made clear and I believe we can all agree
14 there is no school that admits students simply on the basis of
15 those two numbers; isn't that true? At least not any that
16 you're aware of or that I'm aware of.
17 A That's right. I don't know of anyone who is quite that
18 stupid.
19 Q But my question is if it's a test which you believe has
20 disparate impacts on various groups, be it women, be it on
21 certain minority groups, or certain other groups, certain
22 religious groups --
23 A It's not a question do I believe. It does.
24 Q Then why don't you suggest a way to change to change it,
25 to try to eliminate that?
BENCH TRIAL - VOLUME 8
TUESDAY, FEBRUARY 6TH, 2001
71
1 A Well, I think I am. I think by saying you should
2 evaluate the outcome of have the test score, just like you
3 should look at a student's GPA and evaluate it in terms of what
4 that student is. Was that student working through college, was
5 that student, you know, being an athlete? All of these things
6 matter.
7 Q Professor Shapiro, isn't that true for every student
8 regardless of his or her race or ethnicity?
9 A Of course.
10 Q Have you read Professor Lempert's report in this case,
11 where he has developed an index and analyzed its predictive
12 value for the law students at Michigan?
13 A I have read his report. I don't know exactly what your
14 -- the way you're characterizing -- what it is you're asking
15 me. I have read his report.
16 Q Do you agree that the use of the LSAT and the
17 undergraduate grade point average helps to predict the overall
18 academic performance of students once they enter the Law School
19 of Michigan?
20 A You mean, does it correlate with first-year law school
21 grades?
22 Q No, sir.
23 A I know that.
24 Q The entire law school grade point average after three
25 years?
BENCH TRIAL - VOLUME 8
TUESDAY, FEBRUARY 6TH, 2001
72
1 A Oh, three-year law school grades. In some schools it
2 does, and, you know the correlation is typically lower than
3 with first year. Some schools it doesn't.
4 Q Do you know you and I don't want get into a long
5 discussion because we're going to hear from Professor Lempert
6 later in this case, have you read his report to see what is
7 that he says the predictive nature of the LSAT and the
8 undergraduate grade point average with regard to all students
9 including minority students who enter the Law School of
10 Michigan?
11 A I have read his report. I don't remember the details
12 about it. I mean, you would have to ask me something very
13 specific for me to answer.
14 Q You don't recall whether or not it's Professor Lembert's
15 view that the undergraduate -- that the index, the combination
16 of the LSAT and the undergraduate grade point average is a
17 strong predictor of law school performance academically?
18 A Could you tell me what you mean by strong? That's a
19 subjective term.
20 Q Tell me what you think would be strong. You're a
21 statistician so --
22 A But you see, that's just the problem. There is no
23 absolute standard for validity. Validity unlike its use in the
24 language validity in a statistical sense is a continuum. I
25 mean, it's just a value for -- you know, minus one to plus one.
BENCH TRIAL - VOLUME 8
TUESDAY, FEBRUARY 6TH, 2001
73
1 There isn't a standard.
2 Q Would you disagree with Professor Lempert if he says that
3 the law school admission test and the undergraduate grade point
4 average correlates stronger with law school grades? Would you
5 disagree with that?
6 A It's meaningless unless you tell me what the value of the
7 correlation co-efficient is.
8 Q Well, let's ask: have you ever read the law school's
9 admissions policy, in this case, the Michigan admissions
10 policy?
11 A I think so, yes.
12 Q Do you recall when you read it for the first time?
13 A It's been within the last year or so.
14 Q Was it before you prepared your report?
15 A Yes, I assume.
16 Q Do you recall that the law school admissions policy talks
17 about the index being a combination of the LSAT and the and the
18 grade point --
19 A Yeah, but --
20 Q Let me just finish.
21 A Oh, I'm sorry. I apologize.
22 Q We're trying to make it easier for the court reporter.
23 THE WITNESS: I apologize to you also.
24 BY MR. PURDY:
25 Do you recall that the law school admissions policy talks about
BENCH TRIAL - VOLUME 8
TUESDAY, FEBRUARY 6TH, 2001
74
1 the index explaining twenty-seven percent of the variance of
2 first-year grade point averages?
3 A No, that's a number I recall, the twenty-seven percent.
4 When you say the index of course to me in technical jargon
5 that's the linear regression equation. Yeah, I think that is a
6 valid -- the number that's used twenty-seven percent.
7 Q Is that in your a view as a cycle nutrition a strong
8 correlation?
9 A That is the number that's comparable to a lot of -- I'm
10 going to use a technical term r squared, that is the square of
11 the of the correlation coefficient. That is comparable to a lot
12 of values that you see for various admissions tests in various
13 universities. Whether you want to say that predicting
14 twenty-seven percent of the variance at the cost of huge
15 disparate impact, it is -- is large or good. I mean, one could
16 say that. I don't agree. I think that is a huge social cost
17 to pay for twenty-seven percent of the variance.
18 Q Would you recommend -- strike that.
19 Do you know what weight the University of Michigan
20 gives to the law school -- the LSAT and the undergraduate
21 grade point average in terms of its admissions decisions? Do
22 you know what weight? I assume you don't, but I'm asking, do
23 you know?
24 A You mean weight in the individual committee member how he
25 or she uses that? Of course not.
BENCH TRIAL - VOLUME 8
TUESDAY, FEBRUARY 6TH, 2001
75
1 Q Regardless of how they use it, if it has a -- if weighing
2 either of those factors has a disparate impact in your opinion
3 would you suggest that they use -- apply less weight to those
4 particular indices?
5 A I think we're playing with words. If by using -- if
6 applying less weight you mean moderating them by the use of
7 other student characteristics to make your decision, yes, I
8 agree they should. If by less weight you mean changing the
9 regression equation, no. I mean that is the least square's
10 best fitting regression equation. I mean, I'm saying you
11 shouldn't take it so seriously.
12 Q Why couldn't you -- in your view why couldn't you just
13 eliminate the LSAT from the admissions process at Michigan and
14 still be able to admit a fine class, a talented class of
15 students?
16 MR. NIEHOFF: Well, your Honor, I'm going to object
17 to the extent that the question presumes a foundation of
18 knowledge about what happens at Michigan. If Mr. Purdy wants
19 to ask the question generally, I think he can do so.
20 THE COURT: Why don't you ask it generally?
21 BY MR. PURDY:
22 Q Generally.
23 A Well, again, you see the problem is that the empirical
24 evidence differs from school-to-school, group-to-group. So
25 it's hard to give a general answer. But if you're asking me
BENCH TRIAL - VOLUME 8
TUESDAY, FEBRUARY 6TH, 2001
76
1 should you have a one varied equation, that is, how to predict
2 GPA from let's say, predict law school GPA from college GPA,
3 no. I mean, that just leaves you in the same place except with
4 one variable instead of two variables.
5 The real issue is do you want to be bound by a
6 number that is arrived at statistically, that is,
7 probabilistically, and that accounts for twenty percent of the
8 variance. It wouldn't solve it by reducing the equation to one
9 variable and accounting for even less of the variance, maybe
10 twenty percent instead of twenty-seven percent. It still
11 would leave you with the same question, are you going to make
12 a lock-step decision based on a probability. But in fact, you
13 have all this other information from each applicant if you
14 just read the application.
15 Q Would you agree that all of that other information just
16 like the SLAT and the grade point should be evaluated for each
17 individual irregardless of his or her race?
18 A Yes.
19 Q Let me just ask you finally can you tell us what are the
20 reasons why a white student or an Asian-American applicant
21 might score poorly on the LSAT?
22 A In the first place to the best of my knowledge except for
23 a very limited of nationalities within the group, Asian
24 applicants, don't score poorly on the tests.
25 Q I'm asking what would be the reasons why you might find a
BENCH TRIAL - VOLUME 8
TUESDAY, FEBRUARY 6TH, 2001
77
1 white or Asian-American who would score poorly on the LSAT?
2 A An individual person?
3 Q Yes, sir. An individual. These are individuals that
4 take these tests, are they not?
5 A They sure are.
6 Q So why -- what are the reasons why a white or
7 Asian-American applicant would score poorly on an LSAT? What
8 reasons would you give us for why that would happen?
9 A I don't -- I hope you don't take this flippedly. I got a
10 PhD with concentration in psychology and one of the two, more
11 than -- I'm sorry -- yeah, more than forty years ago. And the
12 one thing I've learned in psychology is I surely am not
13 presumptuous enough to try to understand why any one individual
14 person does anything in their life must less a score on a
15 particular exam, on a particular day. I mean, that's beyond my
16 ability.
17 Q Professor Shapiro, if -- whatever the reasons would be
18 which might explain why a white or Asian-American applicant
19 scoring poorly could those same reasons also be the explanation
20 for why a member of an other racial or ethnic group might score
21 poorly? Won't you agree with that?
22 A Well, you make it sound like people are fungible. I
23 don't want to answer it that way. But I don't know what you
24 mean by that. If I don't interpret your question --
25 Q Well, if you take -- let's take -- I think the Court
BENCH TRIAL - VOLUME 8
TUESDAY, FEBRUARY 6TH, 2001
78
1 earlier today was talking about the impact of being rural
2 versus urban, being a socioeconomic different class, and, of
3 course, all of these things cross racial lines, you understand
4 that; correct?
5 A Well, in different proportions. I mean, they're --
6 Q Here's my question. Let's assume that you have some
7 person who do not get the opportunity for good schooling from K
8 through 12. And on top of that let's that they ran with the
9 crowd. Let's use -- we'll use a white rural student, ran with
10 the crowd, that wasn't interested in academic performance, and
11 when it came time to take the SAT or the LSAT, that showed very
12 little interest in performing well, and, in fact, when they
13 took the test they didn't perform well. Now, that could be an
14 explanation for why they would do poorly.
15 A I don't know of any such person quite frankly who took
16 the LSAT. I mean you have a non -- an academically
17 non-performance person who has no academic interests whose
18 socialization is that academics is unimportant and you've got
19 him taking the LSAT.
20 Q Fair enough.
21 A I don't know how to answer that.
22 Q Let me back up. And let's remove because I was probably
23 unfair to that student. Let's assume that he or she was a
24 victim of poor opportunities in terms of their undergrad -- K
25 through 12 education. And let's assume that even though
BENCH TRIAL - VOLUME 8
TUESDAY, FEBRUARY 6TH, 2001
79
1 they're very interested, they worked hard, they didn't get
2 prepared as well as they should have for the test for they
3 perform poorly even though they wanted to prepare well. Can't
4 that be an explanation for why they did poorly? Poor
5 schooling, poor preparation?
6 A But you see any explanation is possible. So I mean --
7 yeah, anything is possible. There are no impossibilities as
8 human beings.
9 Q And isn't that true for every test taker irrespective of
10 race or ethnicity?
11 A Yes. I'm mean -- not said to read one groups'
12 applications and not read the others. I don't understand the
13 question.
14 Q All right. And you do understand that are minority
15 applicants -- minority applicants whom you have placed in a
16 group where there's this tremendous gap upon average; right?
17 A Yes.
18 Q We talked about that earlier this morning.
19 A Yes.
20 Q There are minority applicants who perform very well on
21 this test; are there not?
22 A What minority.
23 Q Any minority.
24 A Well, that's not true. There are some groups for which
25 there are very, very few, a very small percentage of
BENCH TRIAL - VOLUME 8
TUESDAY, FEBRUARY 6TH, 2001
80
1 individuals getting in the higher percentiles.
2 Q But there are -- in every group. Are you suggesting
3 there is a group where no minority performs at the highest
4 levels on these tests?
5 A The proportion of percentage can be extremely low. I
6 don't want to say no because if we back off of it for a moment
7 we -- there are -- there's even a question of whose a member of
8 a minority group. We get into all kinds of definitional
9 questions. But there are some groups for which the proportion
10 of people getting very high scores is very, very small. It's
11 not zero, but it's very small.
12 Q What is the relevance of what you were just talking about
13 the definition of a minority? What are you referring to?
14 A Well, I mean you're asking me a question in which you say
15 no, none. Okay. So in order to answer an absolute zero
16 question like that we're going to have to first decide who's in
17 that group. Of course, you know the matter is that the persons
18 who are in that group are those who say they are in the group.
19 So, I mean, it's not a good question is all I'm saying because
20 it implies a precision that we don't really have.
21 Q Well, with all due respect I'm not sure I used the word
22 and I apologize if I did. I think I said isn't it true that
23 there are members of every group that perform at the highest
24 levels on these tests. Now, are you taking the position that
25 there are none?
BENCH TRIAL - VOLUME 8
TUESDAY, FEBRUARY 6TH, 2001
81
1 A I'm telling you the number is very, very small.
2 Q And I want to go back, why does the definition of who's
3 in these groups matter to you?
4 A Well, I just thought it matter -- it doesn't matter to
5 me. I thought it matter to you in the way you were asking the
6 question. You know, I don't need to define who's a member of
7 what group.
8 Q The final -- let me ask you if I could, If I could direct
9 to the last page of your report. I think it's 170.
10 A Okay.
11 Q Okay. You say -- let me just read it into the record,
12 and certainly correct me if I'm wrong.
13 Barring the discontinuance of reliance of the
14 regression equation, using admission test score and previous
15 grade point average for admissions decisions, it is reasonable
16 to view affirmative action as the only available antidote to
17 test development procedures which have reduced
18 African-American enrollment to zero or near-zero levels and
19 have substantially reduced the enrollment of other minority
20 groups."
21 A You read it correctly.
22 Q Now there I gather that you are suggesting however that
23 there be a discontinuance on this regression equation, are you
24 not?
25 A No. In a sense of using it I am saying it in the sense
BENCH TRIAL - VOLUME 8
TUESDAY, FEBRUARY 6TH, 2001
82
1 of relying on it as a true valid predictor. So --
2 Q Let me just ask --
3 A I mean we're quibbling about the use of a word
4 "reliance."
5 Q No, no, I'm just reading your language which you said
6 barring the discontinuance of reliance on the regression
7 equation it's reasonable to view of affirmative action, I mean
8 you know in this case, of course, what we're talking about is
9 an explicit consideration of race for certain groups. You
10 understand that, do you not?
11 A No, I don't understand that.
12 Q You don't understand that the consideration of race is
13 considered for three particular groups in the University of
14 Michigan Law School?
15 A I don't understand what you mean by the consideration of
16 race.
17 Q Race is a factor that's considered the admissions --
18 A Among other factors, yes, I understand that.
19 Q For three groups, not all groups, you understand that?
20 A Which --
21 MR. NIEHOFF: Objection, your Honor. There is no
22 foundation that he actually knows about the policy.
23 THE COURT: He's asking.
24 MR. PURDY:
25 Q He said he read the files. You told us you read the
BENCH TRIAL - VOLUME 8
TUESDAY, FEBRUARY 6TH, 2001
83
1 policy.
2 A I did read the policy and my answer I'm trying to
3 reconstruct in my own mind who -- which three you're referring
4 to.
5 Q African-Americans, Hispanics, and Native Americans.
6 A Okay.
7 Q You understand that race is factor considered for those
8 three groups in the admissions policy?
9 A Yeah, that makes sense.
10 Q You say "barring the discontinuance of reliance on the
11 regression equation " you go on, "it's reasonable to use
12 affirmative action as the only available antidote so that
13 certain groups such as African-Americans, and others aren't
14 reduced."
15 That's what you say, correct?
16 A Yeah, reliance meaning this -- where I say "barring the
17 discontinuance of reliance, I'm talking about using it in a
18 lock-step fashion. Let me just say one thing about that. As I
19 understand attacks on affirmative action in general and in this
20 case as well, it essentially is the argument that I've got a
21 higher predicted GPA than this person who was admitted and I
22 wasn't admitted. And that's what I'm talking about here when I
23 say reliance. It is as though that predicted equation, that
24 regression equation is the measure of quality.
25 Q Let me just correct you about this case. That is not the
BENCH TRIAL - VOLUME 8
TUESDAY, FEBRUARY 6TH, 2001
84
1 position have the plaintiffs in this case. The position of the
2 plaintiffs in this case is that whatever factors are considered
3 in other words, for example, if the University wishes to
4 completely eliminate consideration of the SLAT, that's fine
5 with the plaintiffs in this case. If the University wants to
6 consider eliminating entirely the use of the undergraduate GPA,
7 that's fine with the plaintiffs in this case. If the
8 University wants to consider eliminating letters of
9 recommendation or consideration of outside activities, all of
10 those are fine with the plaintiffs in this case. The only
11 question is this case is whatever consideration, whatever
12 factors are going to be considered, ought to be fairly
13 considered irregardless of race. So -- just so you understand
14 now, this case, all right? Now, we're tracking? Do you
15 understand where I'm coming from?
16 A I understand what you're saying. I have trouble telling
17 the distinction between you said and what I said.
18 Q There is a distinction and we can argue about that. My
19 question is: If you assume the University is going to continue
20 to rely as they have in the past on this index are you
21 suggesting then that affirmative action, i.e, the
22 consideration, the explicit consideration of race must go on
23 forever?
24 A Forever? That's a long time.
25 Q Well, assume the university is going to continue to rely
BENCH TRIAL - VOLUME 8
TUESDAY, FEBRUARY 6TH, 2001
85
1 on the index in the fashion that it currently does, can you
2 tell us as an expert how long you think affirmative action will
3 be needed as an antidote to preclude the problems that you set
4 forth in your report?
5 A That's easy. When the selection procedure stops having
6 an adverse impact then you can stop searching for alternative
7 selection procedures with less adverse impact. 3.
8 Q Do you have any --
9 A I have no trouble with that.
10 Q Can you tell us in your role as an expert how long that
11 might be before that would occur?
12 A I don't think it's time bound I think it's data bound,
13 whenever it happens. That is, my understanding, you know, is
14 very simple. A procedure has adverse impact. The question then
15 is, is it a valid selection procedure. The question then is
16 there an alternative less discriminatory valid selection
17 procedure.
18 Q Do you believe there is a less discriminatory valid
19 procedure?
20 A This I think -- you know, very simply using the
21 regression equation together with other information is more
22 valid, certainly not less valid, and certainly has less
23 discriminatory impact.
24 Q Would you be able to use that regression equation in a
25 different manner and exclude the consideration of race and
BENCH TRIAL - VOLUME 8
TUESDAY, FEBRUARY 6TH, 2001
86
1 accomplish what you are after?
2 A I don't know what you mean by the exclude the
3 consideration of race.
4 Q Let's assume you remove the question on the application
5 form so that no one knows when an application comes, the only
6 change you've made is you removed the box that identifies a
7 person's race or ethnicity. If you change the manner in which
8 you relied upon the regression equation, in a way that you
9 would feel comfortable with, would that eliminate the
10 consideration of race in the process?
11 A You know that's fairytale. I can look at a student's
12 name, I can look at what school they went to, I can look at
13 what high school they went to, I can look at the zip code and
14 their address, and I can with fairly good certainty tell you
15 what their race is. You cannot remove as a piece of
16 information in the United States. So to say rely on everything
17 else but, and take it off the application, is a dream. It's
18 just unrealistic.
19 Q Is there in your opinion no way to readjust the manner in
20 which you used the regression equation in order to eliminate
21 the need for consider race in the admissions process Or does it
22 even matter based on what you just said?
23
24
25
BENCH TRIAL - VOLUME 8
TUESDAY, FEBRUARY 6TH, 2001
87
1 A Well, if you got people making the decision, they're
2 going to act like people. And they make the decision based on
3 the information they either actually see or inaccurately
4 suppose. I mean, this is not machine making this decision.
5 Thank goodness it isn't. And you are not going to remove that
6 information and presumably you're not going to keep people from
7 using it however they individually use it, whether there's a
8 policy or not.
9 MR. PURDY: That's all, your Honor.
10 THE COURT: Anything? You may step down. Thank you
11 for coming.
12 Okay. We'll stand in recess until 2:15 p.m.
13 (Court in recess, 12:45 p.m.)
14
15
16
17
18
19
20
21
22
23
24
25
BENCH TRIAL - VOLUME 8
TUESDAY, FEBRUARY 6TH, 2001
88
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
|
|