In its continuing efforts to keep the public informed about the ongoing admissions litigation, the University of Michigan makes these transcripts of the trial proceedings in Grutter v Bollinger, et al., Civil Action No. 97-75928 (E.D. Mich.), available to the University community and general public. As is often the case with transcription, some words or phrases may be misspelled or simply incorrect. The University makes no representation as to the accuracy of the transcripts.
1
1 UNITED STATES DISTRICT COURT
2 FOR THE EASTERN DISTRICT OF MICHIGAN
3 SOUTHERN DIVISION
4
5 BARBARA GRUTTER,
6 For herself and all others
7 Similarly situated --
8 Plaintiff,
9 -v- Case Number: 97-CV-75928
10 LEE BOLLINGER, JEFFREY LEHMAN,
11 DENNIS SHIELDS, and REGENTS OF
12 THE UNIVERSITY OF MICHIGAN,
13 Defendants,
14 And
15 KIMBERLY JAMES, et al.,
16 Intervening Defendants.
17 ------------------------------------/ VOLUME 12
18 BENCH TRIAL
19 BEFORE THE HONORABLE BERNARD A. FRIEDMAN
20 United States District Judge
21 238 U.S. Courthouse & Federal Building
22 231 Lafayette Boulevard West
23 Detroit, Michigan
24 SATURDAY, FEBRUARY 10, 2001
25
2
1 APPEARANCES:
2
3 FOR PLAINTIFF: Kirk O. Kolbo, Esq.
4 R. Lawrence Purdy, Esq.
5
6 FOR DEFENDANTS: John Payton, Esq.
7 Craig Goldblatt, Esq.
8 Stuart Delery, Esq.
9 On behalf of Defendants.
10
11 George B. Washington, Esq.
12 Miranda K.S. Massie, Esq.
13 On behalf of Intervening Defendants.
14
15 COURT REPORTER: Joan L. Morgan, CSR
16 Official Court Reporter
17
18
19
20
21
22
23
24
25
3
1 I N D E X
2
3 WITNESS: PAGE:
4
5 ORAL ARGUMENTS RE: WITNESS, GAIL HERIOT
6 By Mr. Goldblatt 5
7 By Mr. Washington 10
8 By Mr. Kolbo 11
9
10 REBUTTAL WITNESS CALLED ON BEHALF OF PLAINTIFF
11 KINLEY LARNTZ, PhD
12 Direct Examination by Mr. Kolbo 19
13 Cross Examination by Mr. Delery 56
14 Cross Examination by Ms. Massie 101
15
16
17 E X H I B I T S
18 MARKED RECEIVED
19 Trial Exhibit 225 54
20 Trial Exhibit 226 54
21 Trial Exhibit 227 54
22
23
24
25
4
1 Detroit, Michigan
2 Saturday, February 10, 2001
3 (At or about 8:33 a.m.)
4 -- --- --
5 THE COURT: Let's talk first, if we might just take
6 a second, and we have --
7 MR. WASHINGTON: Could we wait just a minute? I
8 think Ms. Massie is having car trouble, and she is going to
9 be handling today, so if we could wait for a minute, I'd
10 appreciate it.
11 THE COURT: Oh, I'm sorry. I didn't notice she
12 wasn't here. Of course, it may go much quicker.
13 MR. WASHINGTON: I understand.
14 THE COURT: I'm kidding. No, we will definitely
15 wait.
16 Can we argue the motion in terms of the witness,
17 Gail Heriot, or is Ms. Massie handling that, also?
18 MR. WASHINGTON: I can argue that, I suppose.
19 THE COURT: It's up to you.
20 MR. WASHINGTON: We are really supporting the
21 University's position, so, yeah, fine. We would be happy
22 to do that.
23 MR. PAYTON: Your Honor, Mr. Goldblatt will be
24 arguing this motion.
25 MR. GOLDBLATT: Good morning, Your Honor.
5
1 THE COURT: Good morning.
2 MR. GOLDBLATT: The Defendants have brought this
3 motion to exclude the proffered expert testimony of Gail
4 Heriot. She is a proposed expert submitted by the
5 Plaintiffs. She has submitted an expert report in this
6 matter. I believe it's Proposed Exhibit 136, Your Honor,
7 and that's a report about diversity.
8 It's a report about the opinions of Professor Heriot
9 that policies like the ones at issue here that involve the
10 consideration of race as a factor in admissions don't bring
11 about benefits, don't improve, in sum, the educational
12 experience.
13 As the Court is aware, the Defendants have prepared
14 a case directed exactly at that issue. We believe it is the
15 most rigorous, comprehensive, thorough-going empirical case
16 proving that students who are brought together on campuses
17 that are diverse in many ways, including with respect to
18 race, receive a better education. They learn better, the
19 schools prepare better citizens, the law schools prepare
20 better lawyers.
21 And that interest, that diversity is compelling, not
22 just as that term is used as a matter of legal doctrine, but
23 in its plain English sense that that interest is compelling.
24 It's a case, Your Honor, that the Defendants and the
25 University are quite proud of. We would be delighted to put
6
1 on that case.
2 And we had, of course, a number of witnesses
3 prepared to speak to that issue, Professor Garin, Darren
4 Bok, Thomas Shagru, Albert Camareo. The Court hasn't seen
5 any of those witnesses, and the reason for that, of course,
6 Your Honor, is that this Court's ruling on December 22nd
7 said that that question was out of this trial.
8 Our testimony, the reports that we put in, are part
9 of the summary judgment record, and everyone agrees, they
10 are going to travel with this case as it makes its way
11 through the Federal Judiciary, and that's the Court's ruling
12 and we will, of course --
13 THE COURT: You mean everyone is not accepting my
14 ruling in this case?
15 I'm just kidding, go on. I thank God for the Court
16 of Appeals every day, so.
17 MR. GOLDBLATT: Your Honor, this is what Mr. Purdy
18 said when we were talking about this issue. He said, "And
19 of course, Your Honor, as everyone has made clear, no one is
20 contesting that there are educational benefits of diversity.
21 It is simply not an issue in this case."
22 Now, what Professor Heriot's report says is that
23 there aren't sufficient benefits to justify policies like
24 these, and in that sense, what the Plaintiff is purporting
25 to do is to put on a witness who is going to say that which
7
1 Mr. Purdy told this Court isn't true. And Your Honor, we
2 just think that the Plaintiffs should be stopped from doing
3 that.
4 But there is another reason, also, Your Honor. It's
5 that Professor Heriot isn't an expert within the meaning of
6 the Federal Rules of Evidence. She shouldn't be allowed to
7 offer opinion testimony in this case. She certainly has
8 opinions, but there are cases, as this Court is of course
9 aware, like Daubert and Cumo Tire that make clear that
10 before those opinions become evidence in a case this Court
11 plays a gatekeeping role, and before she is permitted to
12 offer opinion testimony there are certain standards that
13 need to be satisfied.
14 Professor Heriot doesn't have any degree that's
15 relevant to her testimony at all. She is a lawyer. She has
16 the same degree that many of us have. She teaches court
17 law. She says that she has, quote, "studied the policy and
18 legal aspects of the issue of racial preferences." That's
19 from her deposition, Your Honor. And she said those two
20 things, the policy aspects and the legal aspects, are
21 interconnected. But this isn't a subject that she has
22 even taught.
23 The law is clear, Your Honor, that the testimonial
24 latitude, I think that's a quote from Daubert, that expert
25 witnesses get that allow them to say more than just their
8
1 firsthand factual testimony isn't something a witness earns
2 just by having an opinion. There are standards, and with
3 respect to the specialized knowledge to which an expert
4 witness will testify there needs to be evidence that the
5 testimony is reliable, that it's more than just one person's
6 opinion.
7 This Court mentioned yesterday that there was a copy
8 of the Daubert opinion on the Bench, and it's actually an
9 encapsulation of that opinion and subsequent Supreme Court
10 case that I mentioned, it's the Cumo Tire case, and on
11 pages, I think it's 149 and 150 of that case, the Supreme
12 Court tells us exactly what the questions are that we're
13 supposed to ask here. There are essentially four questions.
14 Is there a theory or a technique that's been tested?
15 THE COURT: Does it say -- I don't have the case
16 here, I have the same thing you have, the four questions
17 right there.
18 MR. GOLDBLATT: Exactly, Your Honor.
19 Is there a theory or a technique that's been
20 tested? Have the opinions been subject to peer review
21 and publication? Are there standards we can look to to
22 determine that the opinions are reliable? Is there some
23 technique or theory that's gained general acceptance within
24 a relevant scientific community?
25 Your Honor, the answer to every single one of those
9
1 questions is no. This isn't a close case under Daubert and
2 Cumo Tire.
3 Another subject that Professor Heriot discusses in
4 her report is Proposition 209 and its effects, but it should
5 be noted, Your Honor, that Professor Heriot doesn't even
6 teach at a public university in the University of California
7 system. She teaches at a private university, the University
8 of San Diego Law School, a university that, by the terms
9 of Prop 209, isn't affected. It's not, Your Honor, the
10 University of California at San Diego that Dean Garcia was
11 discussing, it's a private university not affected by
12 Proposition 209.
13 And in this case, the Daubert qualifications of the
14 witness with respect to that question, the effects of 209,
15 aren't any better than they are with respect to the benefits
16 of diversity. All four questions, again, all four Daubert
17 questions are answered no.
18 Your Honor, this is really just one person's
19 opinion. It's exactly the kind of testimony that cases like
20 Daubert and Cumo Tire say aren't evidence, aren't evidence
21 in the Federal Court under the Federal Rules. I don't doubt
22 that these opinions are genuine, that they are sincerely
23 held, but that doesn't make them competent evidence and
24 Professor Heriot should be precluded from offering them
25 here.
10
1 THE COURT: Thank you.
2 MR. WASHINGTON: Your Honor, we certainly agree with
3 Mr. Goldblatt's remarks on this. I would just add a couple
4 more.
5 I think insofar as Professor Heriot purports to
6 testify as to the effects of 209 on the educational system
7 of the State of California, there is no evidence that she
8 has ever taught at the University of California other than
9 in speeches or forums organized by the proponents of 209.
10 There is not even any indication that she has ever had any
11 connection with the University of California. There is no
12 indication that she has published any studies other than
13 popular articles about the University of California.
14 Her report contains at most some alleged statements
15 about grades at the public university in the City of San
16 Diego, which as I understand is a city of something like
17 three million people, and there is no indication where or
18 how she got that or how she has any particular expertise
19 at all.
20 There is no technique or theory here that this
21 witness has. She has not submitted anything to any peer
22 review. There is no kind of background in education,
23 anything of that nature, and in that regard, I think this
24 is really the rankest of lay testimony by a partisan in the
25 debate brought here to say things which aren't true, but
11
1 more relevant than that, are not competent as testimony.
2 So we certainly think this witness should not be
3 allowed to testify.
4 MR. KOLBO: Your Honor, I guess my response in a
5 few words is, surely they jest. They must be making these
6 arguments, in part, tongue in cheek, and I want to address,
7 first of all, the relevance issue.
8 It's true that we made some objections early in this
9 case to any testimony about the value of diversity, and I
10 suspect if there hadn't been any testimony about the value
11 of diversity, and there has been a lot in this case from
12 both the Defendants and the Intervenors, that we wouldn't
13 be at least putting on Professor Heriot to talk about that
14 issue.
15 And we understand that the Trial Court's issue is
16 limited, but this case, as has been mentioned, is going
17 to be heard on appeal and we think it would just not be
18 appropriate for there to be lots of evidence on the
19 Defendants' side of the case, on the Intervenors' side of
20 the case, about the value of diversity, but none, even
21 from one witness, the one witness that the Plaintiffs have
22 offered in rebuttal on that particular issue. We think that
23 would be just highly inappropriate.
24 And just so I can illustrate my point, Your Honor,
25 when Mr. Goldblatt says that they had a case that they
12
1 wanted to bring with respect to diversity and they didn't
2 bring it, it's true, there are some witnesses that they had
3 on that issue that they haven't called, but they elicited
4 very directly and clearly and over our objection a number, a
5 substantial amount of testimony on the issue of the value of
6 diversity, what critical mass is, and why it's important.
7 They obtained that testimony, Your Honor, not
8 necessarily always from qualified or formally qualified
9 experts, but from people who are educators.
10 President Bollinger testified why he believes
11 diversity is important, why critical mass is important.
12 Dean Lehman testified about why diversity is important, why
13 critical mass is important and how you go about achieving
14 it. Professor Lempert testified on that issue, as well,
15 at length, and of course, they called Professor Severud
16 on that issue, really only on that issue in terms of the
17 importance of diversity and how you get it and so forth.
18 So it just seems absurd to me, Your Honor, to
19 suggest that it's not an issue in the case, and therefore,
20 the Plaintiffs shouldn't be able to call a witness even in
21 rebuttal on those issues. It just doesn't make -- it
22 doesn't pass the smile test, Your Honor, is I guess what
23 I would say about that.
24 With respect to the substance of Professor Heriot's
25 testimony, she is not going to testify just about diversity
13
1 and whether it's important. Her report, Your Honor, is
2 before the Court and I think you have a sense about what
3 she is going to testify about. Her view is, there is
4 nothing wrong with diversity, certainly diversity,
5 intellectual diversity is a good thing. You'd rather have
6 it.
7 But one of the things she is going to testify to is
8 how you go about getting it, and what she says, you don't
9 have to work very hard to get diversity in the classroom,
10 it pretty much comes naturally. If you're going to have a
11 classroom of 30 students, people are individuals, they bring
12 by their very nature different experiences, outlooks and
13 backgrounds, so you don't have to do much to bring it and
14 you certainly don't have to use racial preferences to
15 achieve diversity in the schools.
16 We think that's relevant to the issues in the case
17 and it's certainly, it's certainly relevant rebuttal
18 testimony to the voluminous testimony that the Defendants
19 have brought. And I haven't even mentioned the Intervenors,
20 Your Honor. They have marched student after student up here
21 to talk about their classroom experiences and so forth. We
22 had Professor Garcia yesterday talking about diversity on
23 campuses that he's not even involved with at the University
24 of California system.
25 So I think it's just frankly a little absurd to
14
1 say that the issue isn't in the case, and therefore, the
2 Plaintiffs shouldn't be able to bring a witness to respond
3 to their witnesses. What they are really trying to do,
4 Your Honor, is to suggest that there should only be one side
5 heard on this issue. Their side has been heard and our side
6 should be heard, and from the one witness that we intend to
7 call on this issue.
8 If I may then, Your Honor, address the issue of
9 qualifications, again, I have to say that this brings a
10 little bit of a smile to my face when I hear that Professor
11 Heriot, first of all, is not qualified to testify on the
12 subject of legal education, the very same subject that,
13 again, President Bollinger testified about, Professor and
14 Dean Lehman testified about, Professor Lempert testified
15 about, Professor Severud testified about.
16 Our witness, Your Honor, Gail Heriot, is a professor
17 of law and a full professor of law. She has been teaching
18 since 1989. She was Dean for a year at George Mason
19 University. When we're talking about diversity, as I
20 understand, what I have heard from the witnesses on the
21 other side, we're talking about what goes on in the
22 classroom, what's important, what does diversity bring to
23 the classroom, what does it bring out of the students, how
24 does it affect the teacher's performance and so forth.
25 I don't know how anybody could bring more expertise
15
1 to that subject than someone who is, in fact, a legal
2 educator and I think that's all that's required to pass the
3 expert testimony threshold in this case. That's all it took
4 for their experts on that subject. That's all it took for
5 their lay witnesses to testify on the subject of the value
6 of diversity. So there is just -- it's just not a serious
7 argument to suggest that our witness, a full professor of
8 law and a teacher of education and one who has thought about
9 and researched some of these issues, doesn't have the basic
10 threshold qualifications to present testimony.
11 Now, I understand they think their experts are
12 better qualified. They think more of Professor Severud than
13 they think of Professor Heriot in terms of credentials and
14 they are certainly entitled to think that and they can
15 certainly argue in terms of the weight of the testimony on
16 that. We think Professor Heriot is very highly qualified
17 and her opinions will be or should be well received by the
18 Court, so it's just not a matter of throwing her testimony
19 out.
20 And finally, Your Honor, with respect to the issue
21 of California, again, I have to smile. They are suggesting
22 that a professor in California is not appropriate to testify
23 about the effects of Proposition 209 in California. I
24 didn't hear exactly what Professor or President Bollinger's
25 or Dean Lehman's or Professor Lempert's or Dean Severud's
16
1 qualifications were to testify about California, and they
2 did testify about what they thought would be the dire
3 consequences in Michigan, based on California, if race
4 couldn't be used as a factor, so it's just not a serious
5 argument to suggest that their Michigan professors are
6 better able to tell this Court about California than a
7 California professor.
8 And finally, Your Honor, with respect to that issue,
9 they seem to misunderstand the point that we thought has
10 already been brought out in this case, which is the
11 consequences of Proposition 209 aren't just limited to
12 the Berkeley campus and they aren't just limited to the
13 University of California system, and they are not even just
14 limited to the public schools of California.
15 One of the things that Professor Heriot is going to
16 testify concerning is that Proposition 209 has had positive
17 consequences, and those positive consequences extend to
18 schools that aren't even in the California system, schools
19 like hers, where she has seen improvements in the quality
20 of classroom discussion and so forth as a result of
21 Proposition 209 and what she believes should be attributed
22 to it. So there is a great deal of relevance there and she
23 certainly has foundation to testify in these matters, Your
24 Honor.
25 THE COURT: In this matter, I know, Mr. Goldblatt,
17
1 you would like to say some more, but let me just -- no use
2 taking a lot of time on it.
3 As to qualifications, I will have to listen to what
4 the qualifications are. I hear both sides arguing what the
5 qualifications may or may not be, but I'll tell you in no
6 uncertain terms that I'm going to use the same standard
7 that I have used throughout this case, and there have been
8 witnesses that have testified that have done no studies,
9 there have been witnesses that have testified that they have
10 had no formal teaching and so forth, but that the Court has
11 ruled that they could testify, because the reason being is,
12 that they do have special knowledge because of their
13 position and so forth.
14 For instance, I think Mr. White was very helpful
15 after we heard his testimony and I think he was very
16 credible in terms of what he had to say and the statistics
17 and so forth, though he had no formal training, hadn't done
18 any personal studies or anything of that nature, and I am
19 going to use the same kind of test.
20 Even Dean Severud, obviously, had no formal
21 training, but he had been around a long time and had taught
22 courses, he had done things that perhaps, judicial Daubert,
23 you know, you would say, well, that wouldn't apply.
24 I only give you these examples because I know that
25 this person is coming from a long way and so forth, but I'm
18
1 saying that I can't rule on her qualifications until I hear
2 them, until I have had an opportunity to have the Plaintiff
3 present them and the Defense and the Intervenors voir dire
4 the witness and then I can make that determination. It's
5 hard to do in the abstract, though.
6 As I indicated, I have listened to both sides and
7 I want both sides to know that I'm going to use the same
8 standards that I have used for everybody at this point.
9 My other issue is that certain matters have arisen
10 that were important and I wanted to hear them, and I have
11 heard them, that perhaps the Plaintiff was not anticipating
12 or so forth, and this is rebuttal, and I'm going to treat
13 it as rebuttal and it's going to be limited to -- her
14 testimony, if permitted, is going to be limited to rebut
15 those things that have been put into the record by the
16 Defense and the Intervenors.
17 So with that said, I suspect the Plaintiffs will
18 call and we will have some voir dire as to qualifications
19 and I will make that ruling, and my ruling, should I allow
20 her to testify, will be that it's limited to rebut those
21 issues that have been raised by the Defense or by the
22 Intervenors.
23 Okay. Mr. Goldblatt?
24 MR. GOLDBLATT: No, thank you, Your Honor.
25 MR. KOLBO: Your Honor, we appreciate the Court and
19
1 the Parties' agreement to take our first rebuttal witness
2 out of order. We will recall Professor Kinley Larntz to
3 the stand, Your Honor.
4 THE COURT: Welcome.
5 K I N L E Y L A R N T Z, P h D,
6 having been previously called as a witness herein, and after
7 having been first duly sworn to tell the truth, the whole
8 truth, and nothing but the truth, was examined and testified
9 as follows:
10 DIRECT EXAMINATION
11 BY MR. KOLBO:
12 Q Dr. Larntz, thank you for coming again.
13 In the interim, I think it's been a couple of weeks
14 or so since your testimony, you understand that Professor
15 Steven Raudenbush offered testimony with resect to the same
16 issues, in some respects, in response to your testimony?
17 A Yes, I understand that.
18 Q Now, you weren't here in the courtroom for that
19 testimony, were you?
20 A I was not.
21 Q But have you been able to review a transcript of
22 Dr. Raudenbush's testimony?
23 A I have.
24 Q And do you understand that Dr. Raudenbush took issue
25 with some of -- some aspects of your testimony and report?
20
1 A Yes.
2 Q Have you also reviewed your own transcript, trial
3 transcript testimony in this case?
4 A Yes.
5 Q Were you able to ascertain from reading, perhaps
6 rereading or reading your testimony, as well as
7 Dr. Raudenbush's testimony, were you able to ascertain
8 whether you had already addressed or covered some of the
9 issues that Dr. Raudenbush took issue with in his testimony?
10 A Well, certainly. I mean, we exchanged our reports
11 before and I, in my direct testimony, offered information
12 on a number of criticisms that he talked about in his
13 testimony, yes.
14 Q And did you, in reviewing your testimony -- you
15 understood that you had already responded to some of those
16 or anticipated some of those in your direct testimony?
17 A A good number, I think, yes.
18 Q Did you also come to the conclusion that there were
19 some aspects of Dr. Raudenbush's testimony that you hadn't
20 really addressed or hadn't been explained in your direct or
21 cross examination that occurred?
22 A There are a few points, yes.
23 Q Okay. I want to just limit, obviously, our questions
24 and answers this morning to that, so please keep your
25 answers limited to that respect.
21
1 And I'm going to ask you about some criticisms that
2 Dr. Raudenbush raised and to the extent that you don't
3 believe they were addressed either in your prior direct or
4 in your cross examination, I would like you to respond to
5 some of these criticisms, if you have some opinions.
6 First of all, do you recall that Dr. Raudenbush had
7 a criticism with respect to the fact that there are -- that
8 in determining or ascertaining odds ratios, there are high
9 odds ratios that don't reflect large differences in
10 probability; that is, at higher ranges of probabilities, so
11 they don't reflect large odds ratios?
12 A There were cases presented by Dr. Raudenbush, and also
13 I think in my cross examination relating to odds ratios that
14 may be relatively high where the difference in probabilities
15 is not -- well, depending on how you think, not great.
16 Q Do you understand what the criticism is of your report
17 and testimony in that area?
18 A Well, I think my understanding is that the -- while
19 there may be relatively high odds ratios, there isn't much
20 of a difference in probability, and I think it's important,
21 I think it's very important that the Court understand and
22 that everyone understand the effect of odds ratios on
23 probabilities.
24 And I tried originally to provide some illustrations
25 and they provided some illustrations and I just want to make
22
1 sure that we're very clear on the relationship from a
2 baseline probability how a particular odds ratio would
3 translate into another probability.
4 So for instance, a baseline probability, I think in
5 my direct examination we talked about a baseline probability
6 of 10 percent going to 90 percent from an odds ratio of 81
7 and so I wanted to -- I wanted to make sure we were clear
8 about how that works over the whole range of probabilities.
9 Q And did you read in the reports of Dr. Raudenbush's
10 testimony where he compared relative probabilities of, say,
11 90 versus 99 percent and the effect of going from 90 to 99
12 to 999 and so forth?
13 A Yes, yes, I did read that.
14 Q And what is demonstrated by that?
15 A It's again illustrating for a certain baseline
16 probability how relative odds of a particular value would
17 translate into another probability for another group.
18 Q And I think you have testified, you indicated this
19 morning that you have prepared something to illustrate the
20 range of probabilities?
21 A What I did, what I did is it's in the booklet that I
22 think you have, and the last page of the booklet, what I did
23 was prepare just a table, and we have it as a slide.
24 I prepared a table for a range of probabilities over
25 the whole range, rather than just choose a certain point,
23
1 where I used 10 percent as a base, and they, Dr. Raudenbush,
2 used 90 percent for the whole range of probabilities, going
3 from a base of one percent, a baseline probability of one
4 percent up here in the corner, all the way down to, I think
5 it's 99, down in the bottom.
6 And so what I did is I did it for various relative
7 odds and I chose values that we have seen results of, 5, 10,
8 20, 50, 100, 200. I have chosen -- what I have done is I
9 have said, how does this baseline probabilities translate
10 into the probability of another group that has a relative
11 odds of a given amount.
12 So for instance, for instance, Dr. Raudenbush had
13 90 percent, and I think was looking at 90 versus 99, and
14 that's a relative odds of 11. I don't have 11 on this
15 table, but I have 90. I have 10, and 90 goes to what, 98.9.
16 And then talking about 111, I think, was one of the ones he
17 used, and 90 goes to -- he had 111, I have 100, and 90 goes
18 to 99.89.
19 So these are -- I'm referring to them as fractions,
20 but we often talk in terms of percentages. And other
21 values, well, for instance, 10 percent, I had reported
22 10 percent with an odds ratio of 81 going to 90. Well, we
23 can see 10 percent here going, well, at 100, goes to 91.7.
24 And all I have done on this table is just to make sure we're
25 clear what the effect of a relative odds is on a baseline
24
1 probability. And that's the only purpose of the table, is
2 for clarification.
3 Q Let me ask, Doctor, Dr. Larntz, do you disagree with
4 Dr. Raudenbush's premise that when you start with a high
5 baseline probability like 90 and then have a comparative
6 probability of 99, that odds ratios go up rapidly as
7 those probabilities diverge?
8 A Well, sure, the odds do, the odds ratios do go up
9 mathematically. They are what they are, and that's what
10 this table is trying to just illustrate, what the
11 mathematics is, and I think that's clear.
12 In point of fact, in that particular case, if you
13 look at the percentage denied, obviously they go up or
14 they go from 10 percent to one percent, so that's a big
15 difference in that probability scale.
16 Q So as I understand it, you agree with Dr. Raudenbush
17 on that fundamental principal?
18 A On the calculation, absolutely, there is no
19 disagreement on the calculation. That's why I provided
20 the table, so you can see the whole range of possible
21 calculations.
22 Q And does that mathematical fact, does that change
23 anything with respect to your opinions on the extent to
24 which race is taken into account in the admissions process?
25 A No, no.
25
1 Q Dr. Larntz, were you also -- did you also understand
2 that Professor Raudenbush in his testimony took issue with
3 your -- or criticized you for what he called discarding some
4 of the data in your analysis or selectively attending to the
5 data?
6 A Yes, I did read that, yes.
7 Q Do you have an opinion with respect to the validity of
8 that criticism?
9 A You mean the fact that different parts of the data
10 provide different amounts of information? I certainly have
11 an opinion that that's true, and I'm not quite sure -- in
12 the sense that certain cells in the grid provide more
13 information than other cells, that's true.
14 Q What would be an example of that?
15 A Well, I mean, obviously cells that have no one
16 admitted would provide no information about admission rates,
17 in comparison of admission rates. Cells that have everyone
18 admitted provide no comparative information.
19 And that's what we're talking about here, because
20 we're trying to compare the admissions of minority students
21 to majority students. So the different cell grids in fact
22 do provide different amounts of information.
23 Q Okay. And as I understand it, Dr. Raudenbush has
24 suggested that you did not consider certain cells in the
25 data.
26
1 A Well, in the calculation, every cell entered the
2 calculation and the computer made the judgment of how much
3 information would be -- you know, how much information was
4 there. There are some cells that the computer found that
5 there was no comparative information and other cells where
6 it found comparative information and used all the cells that
7 had comparative information in deriving the summary relative
8 odds ratios that we talked about.
9 Q What would be the kinds of cells that would provide no
10 comparative information, what categories would those fall
11 into, into terms of what you saw?
12 A Well, cells in a grid that would have no one admitted,
13 okay, there is no comparative information, everyone
14 admitted. And there are very few cells like that, but there
15 are a few, okay? Cells, if there are no people in the
16 cells, there is no comparative information.
17 And in fact, the one additional category is if
18 everyone in a cell was a member of one of the ethnic groups,
19 just one, so if everyone in the cell, for instance, if
20 everyone in the cell were caucasian and there were no
21 minority students in that cell, or no member of any other
22 ethnic group, then that wouldn't provide any comparative
23 information, because everyone would be the same.
24 Q What would be an example of a cell that provided some,
25 but not very much information?
27
1 A Oh, there are cells, for instance, a cell where
2 virtually everyone is admitted, but not everyone, then that
3 cell would provide -- may provide comparative information,
4 but the amount of information in how much it contributes to
5 the estimation of the overall relative odds would be small.
6 Q Have you prepared some charts to illustrate what cells
7 provided comparative information in your analysis?
8 A What I have done is for each of the years, and this is
9 also in the book, for each of the years what I have done
10 is -- and we have the first one for 1995. What I have done
11 for each of the years is looked at the original grid, this
12 is for all applicants, and I have highlighted, I have
13 highlighted in yellow the cells that provide comparative
14 information.
15 So all the cells that provide comparative
16 information are highlighted in yellow and the cells that
17 are not, the blank cells, except for the permanent cells,
18 the blank cells in the balance of the table for which there
19 is no comparative information, that is, in the sense that
20 the computer would check and say, well, there isn't anything
21 I can add based on these cells, those are left blank.
22 Q Would there be a way of including the cells that have
23 no applicants, first of all?
24 A Cells that have no applicants?
25 Q Yes.
28
1 A Whatsoever?
2 Q I'm sorry, that have no -- for across racial lines
3 there are no admits, or let's take, for example, no
4 admissions.
5 A Well, there -- if there is no admits, then there is no
6 comparative information, so they -- the computer makes a
7 judgment that there is no comparative information there and
8 so it attributes -- it doesn't actually contribute to the
9 estimation.
10 Q Did Dr. Raudenbush in his analysis use those cells in
11 some fashion, cells in which there are applicants, but no
12 admissions from any racial group?
13 A Oh, I believe that in some of his models he used those
14 cells, yes.
15 Q And does that have any effect, as far as you're
16 concerned, on the validity of the results?
17 A Well, with respect to providing comparative
18 information you would have to have a model that says how
19 they enter, and so based on how his model works, that model,
20 whether that model is true or not depends on how that
21 information would enter. I'll just say it that way,
22 that's true.
23 Q Would it provide a more accurate her less accurate
24 picture of the -- as far as you're concerned -- the extent
25 to which race is taken into account, like including these
29
1 cells in which there are no admissions from any racial
2 group, or would there be a distortion there or what
3 would be --
4 A Well, I actually don't know what the effect is, I
5 mean, as far as that goes. With respect to the statistical
6 principle of looking at cells with comparative information,
7 they don't provide comparative information, so that's what
8 I know.
9 Q And so you have got in front of you what we have up
10 here, 1995?
11 THE COURT: They don't provide any comparative
12 information, in your opinion. If you included them, would
13 there be a downside to including them, even though they have
14 no information or is there just no reason for it or --
15 THE WITNESS: Well, I mean, what you have to do
16 is you have to model the whole grid, okay, and that to me,
17 that's a hard process, okay. I looked at it and took the
18 comparative information and compared cell by cell. To me
19 it's harder to model the whole grid. It's just a more
20 difficult problem and your results would depend upon how
21 you did that modeling.
22 THE COURT: So there would be another level of
23 discretion, because that new level of discretion is now an
24 extended modeling of the whole thing?
25 THE WITNESS: That's the way I looked at it.
30
1 THE COURT: I mean, is that pretty much -- I
2 remember you talked about the less that you have to do in
3 terms of discretion, the better the results sometimes are.
4 THE WITNESS: That's right.
5 THE COURT: That's what you told us.
6 THE WITNESS: Right. The kind of modeling would be,
7 there is some kind of -- you know, I'll try to be as clear
8 as I can. You basically have a surface that you're modeling
9 here. You're trying to model a response surface over this
10 whole grid and you have to make assumptions about how to do
11 that, and what I did is, I didn't do that, and then what
12 you -- I have to use two hands, sorry.
13 THE COURT: That's okay.
14 THE WITNESS: You model, say, the majority students
15 one way and the minority students in another and sort of the
16 difference here becomes the odds ratio, right? So depending
17 on how you do that modeling, depending on how you do that
18 modeling, you could get different answers.
19 THE COURT: And the more modeling you do, the more
20 assumptions you make?
21 THE WITNESS: Yeah, well, sure, yeah.
22 THE COURT: Your position is that you did -- you
23 tried to do as little modeling and as little assumptions as
24 you can to actually get the figures?
25 THE WITNESS: Right. And exactly in this way, what
31
1 I did is I said, let's not try to model the whole surface,
2 let's just look at it one point at a time, and look at the
3 difference one point at a time. And that's basically what
4 I did, was look at the difference one point at a time.
5 THE COURT: Creating a situation where you have to
6 make fewer assumptions?
7 THE WITNESS: I think there are fewer assumptions
8 in that, surely there are fewer assumptions in that, yes.
9 BY MR. KOLBO:
10 Q And you have done this for each one of the years in
11 question?
12 A That's right.
13 Q Can we just take a look at 1996?
14 A I think so.
15 Q Same pattern?
16 A Oh, same pattern. The cells that were no admitted
17 were the predominant cells that were left out and they are
18 the ones that have very low test scores and very low GPA's
19 and those students just, you know, they didn't admit them.
20 So that's true universally. There are a few other cells
21 that don't provide very good information for the other
22 reasons I said, but predominantly, they are the lower grade
23 point averages and the lower test scores.
24 Q And 1997, can we see that same pattern here?
25 A Same pattern.
32
1 THE COURT: And '98, perhaps --
2 THE WITNESS: Same pattern.
3 BY MR. KOLBO:
4 Q And then '99. I wanted to ask you about a cell or a
5 couple of cells for 1999. I don't think these are -- these
6 kinds of cells show up on the earlier versions, but there is
7 a cell, for example, under grade point 3.0 to 3.24 and 170
8 and above in which there are nine applicants and three
9 admissions; correct?
10 A That's correct. That's the cell over on the -- let me
11 see, it's the cell right over here in this corner, 170 and
12 above, and this nine and three here that's not shaded.
13 Q And it's not shaded, meaning that it didn't provide
14 comparative information?
15 A That's right.
16 Q And why in that case was there no comparative
17 information?
18 A Well, I don't recall exactly the group, but all nine
19 of those applicants were from one ethnic group, so they were
20 from one ethnic group.
21 Q That shows no comparative information across racial
22 lines?
23 A That's right.
24 Q Did you -- there has been some testimony, I think,
25 from Dr. Raudenbush about the number of cells or percentage
33
1 of cells or so forth that were used in your analysis or that
2 were -- that, as you have testified, provide comparative
3 information. Can you -- do you have some idea in terms
4 of the number of applicants that contributed comparative
5 information to your analysis across these different years?
6 A Yes, I did calculate the percentage of applicants that
7 are in the shaded areas and I don't remember exactly from
8 each year, but it's between 84 and 88 percent across the
9 six years.
10 Q That all provided comparative information?
11 A That's right.
12 Q Dr. Larntz, then if I can ask you, do you recall
13 that in reading Dr. Raudenbush's testimony that he had a
14 criticism with respect to your reporting of uniform odds for
15 composite odds ratio assumptions?
16 A Yes.
17 Q Do you understand what his criticism is in that
18 regard?
19 A Well, I understand the technical, statistical aspect
20 of the criticism, sure.
21 Q Do you agree or disagree with it?
22 A I disagree on its importance, and I am not sure if
23 that assumption is technically satisfied or not, but in
24 fact, the importance of the assumption is that my composite
25 odds ratios, if in fact that assumption is not perfectly
34
1 satisfied, and no assumptions are ever perfectly satisfied,
2 if that assumption is not perfectly satisfied and there are
3 some cells that have higher odds ratios than others, then
4 the number I gave you would be a composite that would be
5 lower than they would be for the ones that are high and
6 a little bit higher than the ones that are lower.
7 Q Do you have an opinion as to whether Dr. Raudenbush's
8 criticism -- does it change any of your opinions with
9 respect to the extent to which you believe race is
10 considered in the admissions process as reflected by
11 your analysis?
12 A No, it doesn't change my opinion with respect to
13 that at all, no.
14 Q Can you explain why not?
15 A Well, I mean, well, first of all, I think the odds
16 ratios we see are a direct reflection of the cell. We can
17 compare the grid cells and see high odds ratios in the grid
18 cells and the summary numbers that I provided, I think, are
19 a reflection of those, of those values.
20 Q Do you recall Dr. Raudenbush testifying that there are
21 a lot of large odds ratios in the middle ranges of the
22 grids?
23 A Oh, I believe that he said that there are very high --
24 or there are high odds ratios in the middle, yes.
25 Q And do you agree or disagree with that?
35
1 A Yes, if there weren't higher odds ratios in the middle
2 we wouldn't have composite odds ratios. If there weren't
3 high odds ratios someplace, and they're mostly in the middle
4 where there is a difference in admission rates, then we
5 wouldn't -- the composite odds ratios wouldn't be as large
6 as they were.
7 Q Do you understand Dr. Raudenbush's testimony to be
8 that the odds ratios are smaller at the upper end of the
9 grid?
10 A I think I -- I think the implication of what he said
11 was they were smaller in the upper end, that's right.
12 Q Do you agree or disagree with that?
13 A Well, I think in the upper end it's very hard to tell,
14 because there is really -- when almost everyone is admitted,
15 there is -- you can estimate odds ratios, but there is --
16 the amount of comparative information up there is relatively
17 so small, so they don't contribute a great deal to the
18 composite odds ratio, because if almost everyone is admitted
19 the comparative information there isn't great, so I think
20 it's very hard to determine the relative odds up in the
21 corners unless you create some kind of model.
22 Q Where does most of the comparative information come
23 from him?
24 A Well, most of the comparative information is down in
25 this area here where there are clear differences and where
36
1 there is some discretion being made with respect to the
2 admissions process where not everyone is admitted.
3 Q The middle of the, primarily, shaded area?
4 A Well, actually, it's the lower part of the shaded
5 area, if you want to call it, the middle of the grid.
6 Q And then do you recall reading a criticism of
7 Dr. Raudenbush with respect to what he termed, I think,
8 the stability of the estimates over the years?
9 A Yes, I do remember reading that.
10 Q And do you have an opinion as to whether you disagree
11 or agree with his criticisms there?
12 A Well, I certainly think that they are relatively
13 stable over the years and I think that we have to -- I think
14 we have to do -- probably do a little bit of work, a little
15 bit of statistical work in the courtroom, to show that.
16 Q So you disagree with Dr. Raudenbush's conclusions?
17 A I do disagree that there is a -- that there is any
18 kind of major problem with stability over the years, yes,
19 I disagree completely.
20 Q And could you explain the bases, and using the board
21 if the Court will permit?
22 THE COURT: Sure.
23 A Well, a great deal was made of a comparison -- a great
24 deal was made of a comparison between African American
25 relative odds in 1997 and African American relative odds in
37
1 the year 2000, and I think that a great deal was made of
2 that comparison, and I think we need to talk about African
3 American relative odds. I have to say, if you select two
4 years out then you might get a different comparison, so I
5 want to look at all of them across all six years, okay, and
6 so I want -- can I go to the board?
7 Q Sure. Have you prepared some notes?
8 A I have a little note that summarizes things that I had
9 from before.
10 THE COURT: Can I make a suggestion?
11 THE WITNESS: Yes, sure.
12 THE COURT: I didn't know where you were, I thought
13 you were sitting.
14 If we put the board over here just for purposes
15 of this, then everybody can still have a seat and sit and
16 see it.
17 MR. KOLBO: In fact, I think we're done with the
18 projector.
19 MS. MASSIE: Thanks, Judge.
20 (Discussion off the record at 9:15 a.m.)
21 THE WITNESS: What I want to do is, I want to look
22 at all the years, so 1995, 1996, 1997, 1998, 1999 and 2000.
23 And I want to talk about relative odds and I want to talk
24 about comparing relative odds, and I said I have to -- I
25 have to do a little bit of statistics and a little bit of
38
1 math here, so that's just because it's been raised and I
2 want to make sure that we all understand.
3 So the relative odds, and I'll just call it the
4 estimated relative odds that we calculated, the estimated
5 relative odds from the report for 1995 was 257.93. That's
6 1995. This is, we're looking at the relative odds for
7 preference for African Americans, and that's always to a
8 baseline of caucasian.
9 For 1996, it was 313.59. For 1997 it was 53.49.
10 And for 1998 it was 132.16. For 1999, it was 206.45. And
11 for 19 -- whatever it is -- 2000, it was 443.26.
12 So what was talked about, what was talked about was
13 the comparison of this value, the 53, to the 443. Now, I
14 presume, I presume those were chosen because, well, if you
15 look down the list, what's the smallest one? It's 53. And
16 what's the largest one? It's 443. Would there be -- would
17 there always be a largest and a smallest? Well, that's
18 mathematics. There will be a smallest and a largest. And
19 if you're doing comparisons, well, depending, you would
20 typically look at the -- might look at the extreme, that's
21 fine, okay.
22 Now, I also had a number of standard deviations, a
23 Z score, and I need to look at that, and I'll talk about
24 that in a second, but I just want to record the one. These
25 are straight out of the reports and we saw these before, and
39
1 the standard deviation, and I'll need these, so 14.40,
2 13.18, 13.96, 13.46, 12.64, and 12.53 -- or 51, excuse me,
3 51. So this is how this is reported. These were reported
4 in the first report that I did, these first four, and these
5 were in two supplemental reports. And these are the
6 standard deviations which measure the level of statistical
7 significance of how far these are away from one, okay. So
8 this is all summary information.
9 Now, how do we actually do the calculations? Where
10 do they come from? They come from logistic regression. If
11 you recall, that's the technique we used. And logistic
12 regression works in terms, like every regression, works
13 in terms of regression coefficients.
14 And what are the regression coefficients that are
15 behind these numbers, what are the regression coefficients?
16 Well, in fact, these are not the regression coefficients
17 themselves, there is -- in the logistic regression there are
18 coefficients that we can use to calculate these. In fact,
19 the coefficients, the coefficients themselves turn out to
20 be, to be careful, it's the natural logarithm of these
21 values, the natural logarithm, not the log to the base ten,
22 but the natural logarithm. On your calculators, that's the
23 LN function, right?
24 So in fact, the regression coefficients, the
25 coefficients, the coefficients themselves, the coefficients
40
1 themselves are what you use for doing all the inferences.
2 And so what I'm going to do is write down what the
3 coefficients are, and that's what you use for comparison,
4 are the coefficients, because that's the basic underlying
5 calculation mechanism, are these coefficients.
6 And so the coefficients are, and I'll put them in
7 order and say them, the coefficients are, and I'm going
8 to round them to two places, the coefficients are -- for
9 corresponding to 57.93, the natural logarithm is 5.55. For
10 313.59, the natural logarithm is 5.75. For 53.49, the
11 natural logarithm is 3.98. And for 132.16, the natural
12 logarithm is 4.88. And for 206.45 the natural logarithm
13 is 5.33. And for 443.26, the natural logarithm is 6.09.
14 So these are the actual coefficients in the computer
15 output that you would find in the logistic regression
16 computer output.
17 Now, a relative odds of one means no preference. If
18 you take the logarithm of one, you get zero, that's just the
19 way it works, and so these need to be compared to zero. And
20 in fact, the way these standard deviations are calculated
21 is they take the coefficient, this is out of this, and
22 divide it by something called the standard error of the
23 coefficient, and you get the number of standard deviations.
24 Now, I think it's informative to plot these values,
25 because this is the scale for comparison, the comparisons in
41
1 the regression coefficients, so I need to plot these values,
2 and I'll do the best I can, okay. Zero is here, and then we
3 have got one, two, three, four, five, six, and you can see
4 if I did a reasonable job of making those equally spaced. I
5 tried, okay.
6 And then what I would do is plot values, plot the
7 values here, so I better give myself a code, one, two,
8 three, four, five. So 5.55 is here. And 5.75 is here.
9 And 3.98 is here. And 4.88 is here. And 5.33 is here.
10 And 6.09 is here, okay.
11 So this, these are the six years. This is '97 and
12 this is 2000, these are the extremes, but this is where the
13 coefficients fall in terms of the regression coefficients
14 and this is the scale for comparison.
15 Now, statistically what you're doing is comparing
16 each of these values to zero. You're comparing each of
17 these to zero, because that's in the regression coefficient
18 scale and clearly these are clustered relatively far away
19 from zero.
20 In fact, in fact, the standard deviation tells us
21 that for -- well, this standard deviation number tells us
22 how many standard deviations we are away from zero and so,
23 for instance, the year 2000 one is 12.5 standards deviations
24 away relatively. Each one of these has a standard error
25 comparative measurement attached to it, and so we're fairly
42
1 always far away. So when I say that these results are
2 statistically relatively stable, I'm making the statement
3 of how they are in the regression coefficient scale. So
4 that's how to illustrate and say that the variation from
5 year to year is within reasonable statistical standards.
6 Now, there are a couple of more things we could do
7 with this, and I'm not sure how far we should go, but one
8 more thing we could do is we could, for instance, provide
9 confidence limits for these relative odds if we wanted to.
10 I was asked about, could you provide confidence limits, and
11 I think I was criticized for not providing confidence
12 limits, I'm not quite sure. It's not hard to do.
13 So I could provide confidence limits, and if I do
14 one of them, if I do one of them and write down the results
15 of the -- for the others, we can see how that goes and I'm
16 going to give -- so what I need to do to provide confidence
17 limits for the relative odds is I need to determine the
18 standard error of the coefficient, because the way I would
19 provide confidence intervals is, I have to provide a
20 confidence interval for the coefficient and then I have to
21 retranslate it into relative odds.
22 Remember, the coefficient is the logarithm, so
23 what I have to do is provide confidence intervals for the
24 coefficient and then retranslate it, okay.
25 So to get standard errors, I actually -- the
43
1 standard deviation is the coefficient divided by the
2 standard error. It turns out if we take the coefficient
3 and divide it by the standard deviation, we have to get the
4 standard error. So the way the standard can be calculated
5 is we will take this coefficient value and divide by the
6 standard deviation, so 5.55 divided by 14.40, and that will
7 actually -- this is, we're doing this backwards in some
8 sense, because this is actually in the computer output, but
9 just to show where it comes from, this is the standard error
10 of the coefficient, 0.39. That's this value divided by that
11 (indicating). And we can do the same for all of these. So
12 we get 0.44 for 1996, 0.29 for 1997, 0.36 for 1998,
13 0.42 for 1999, and 0.49 for the year 2000.
14 And from this, from those standard errors, we
15 can calculate confidence intervals. The usual way of
16 calculating confidence intervals, I'm going to do it
17 approximately, because it won't make any difference, is to
18 get 95 percent confidence bounds, we take the confidence
19 coefficient, the regression coefficient, and take a plus or
20 minus two standard errors and that gives us 95 percent
21 confidence amounts.
22 So I could do that for this, and so, for instance,
23 in 95 percent confidence, 95 percent confidence interval,
24 then, for the regression coefficient, take 5.55 and subtract
25 two times this, and the lower bound, then, is 4.77 and the
44
1 upper bound is 6.33. So that's a confidence interval for
2 the true value of the coefficient.
3 And I can do that for each of these, and I'm not
4 going to write them all down right now unless you would like
5 me to, but what I can do now is that I can translate these
6 into confidence intervals for the relative odds. And
7 what I do is I use the, let me see, inverse logarithm,
8 exponentiation, so I undo, to get into terms of relative
9 odds, and in terms of relative odds, the confidence interval
10 runs from 118, that's the lower bound, to 561.
11 I think I had indicated earlier that these numbers,
12 I don't believe these numbers exactly, and maybe, you know,
13 in some sense I should never put two decimal points down on
14 something I don't believe too accurately, but we can see
15 that this is a confidence interval for the relative odds.
16 These are all high numbers. You know, I think they are all
17 high values, but these are confidence intervals for the
18 relative odds.
19 So if I might, I think I'll just -- could I write
20 down -- I'll write down the confidence interval for relative
21 odds for each of these, without going through all the other
22 calculations.
23 So the confidence interval for 1996 then becomes
24 130 to 757. Notice they are wide, and they are actually,
25 because of the way it works, they have to be wider on the
45
1 high end, because the relative odds, as they go up, there
2 isn't that much -- as much difference as they go higher and
3 higher.
4 For the 53.49, lower bound is 30, again, and the
5 upper bound is 96, so there is a spread, but these lower
6 bounds are all what I consider big relative odds.
7 THE COURT: Even 30?
8 THE WITNESS: Oh, sure. Oh, sure. Ten is big,
9 okay. I think I said that before.
10 THE COURT: You did.
11 THE WITNESS: Okay.
12 And 64 to 270, 89 to 478, and the last one is
13 166 to 1,176, okay.
14 So the lower bounds are all, you know, in what I
15 consider large relative odds. The upper bounds, of course,
16 are -- well, you know, they just become harder to estimate
17 out at higher relative odds.
18 So they are all -- so this gives you an idea of some
19 variation. So in my estimation, these are all indicating
20 the same thing, that in fact, there is a substantial, a very
21 large allowance for African American applicants for each of
22 the years, and they are consistent.
23 They are consistent, not technically statistically
24 consistent, but substantively consistent, but substantively
25 consistent from year to year. There may be a technical
46
1 difference between them.
2 So that's something I wanted to make sure we
3 demonstrate, the stability in the appropriate scale, which
4 is the regression coefficient scale, and the corresponding
5 confidence intervals that we could generate for relative
6 odds.
7 Q So in short, in a short sentence here, what you have
8 just explained is the basis for your opinion that you
9 disagree with Dr. Raudenbush's conclusion that the relative
10 odds ratios are unstable across years?
11 A Well, that's part of it, yes. That's certainly the
12 basis. This is my thinking of how I would look at this
13 and certainly I believe they are certainly substantively
14 stable, if not technically statistically stable across
15 years.
16 Q Now, I think Dr. Raudenbush criticized you for not
17 reporting confidence intervals as you have just illustrated
18 here for these years for African Americans. Could
19 Dr. Raudenbush have calculated confidence intervals for your
20 relative odds based on the data that he had available to
21 him?
22 A Oh, sure. This is not -- this is straightforward
23 calculations and the computer output was all provided.
24 Everything was there to do this, yes.
25 Q Now, you --
47
1 A But you can do it from the reports, actually, as we
2 just did from that information.
3 Q And you have reported that all of these, as far as
4 you're concerned for these years for African Americans, all
5 of these show very large preferences?
6 A Oh, sure, yes.
7 Q Are you able to tell whether that's true or not for
8 the other races that you did odds ratios analysis on for all
9 these years?
10 A Well, if you computed confidence intervals for
11 relative odds, they would all be, you know, relatively -- or
12 distributed around the estimated values, but they would all
13 show large preferences, yes.
14 Q You know, Dr. Raudenbush testified, I believe, that he
15 believed that the standard deviation -- I probably won't say
16 this correctly so you will have to correct my question,
17 perhaps, and then answer it. I believe he testified that
18 there were 11 standard deviations in different separated
19 odds ratios between 1997 and 2000. Did you understand that
20 he testified along those lines?
21 A Yes, I remember being asked that in cross and I read
22 his testimony to that effect that the difference was about
23 11, I think he said, almost 11 standard deviations.
24 Q Do you disagree with that opinion?
25 A Oh, yes. That's wrong. Excuse me.
48
1 Q Do you have any understanding based on what you have
2 seen from Dr. Raudenbush's testimony and the data that you
3 have as to how he could have come to that conclusion?
4 A Well, if I might, can I do the calculation for the
5 Court, do the calculation in the regression coefficient in
6 the appropriate scale and then show you what the difference
7 is? And then I obviously don't know exactly how he did
8 the calculations, but I have an idea of how he did the
9 calculations, okay?
10 THE COURT: Certainly. Why don't we -- just for the
11 record, why don't we just put some identification on that.
12 MR. KOLBO: I was going to ask if we could mark it,
13 Your Honor, and offer it, as well.
14 THE COURT: Why don't you at least mark it and then
15 we can talk about offering it.
16 MR. KOLBO: 226.
17 THE COURT: What's the next number?
18 MR. KOLBO: Do you want to write 226?
19 THE COURT: Ex. 226, just so we can talk about it,
20 then you can move it later. Just so we have something for
21 the record that we know where it is, okay.
22 THE WITNESS: I need to do the comparison. I have
23 to do the -- compare standard errors.
24 MR. DELERY: We have another easel with another pen.
25 THE WITNESS: That's okay, let me record them, then
49
1 we will do it.
2 1997, the coefficient was what, 3.98, and the
3 standard error was .29. That's okay. We're fine. And it
4 was 6.09, is that right, and the standard error was 0.4,
5 okay. That's all we need.
6 THE COURT: And then if you'd just put on the
7 right-hand bottom corner again, 227, Ex. 227, and the record
8 will reflect that he is now working on 227, proposed. Okay.
9 THE WITNESS: Well, from these values we can see how
10 different they are. And I am going to bring this back just
11 to show you, then I'll move it back over.
12 You can see this value here is what, 12 standard
13 deviations from zero, right, 12 standard deviations from
14 zero, and this one, this one here is what, 14 standards
15 deviations from zero, and what Dr. Raudenbush has testified
16 to is that if we compare these two, we get something close
17 to 11.
18 Well, I mean, statistically, and I think I hope
19 I reacted appropriately when someone asked me, would it
20 surprise you that the difference between these two is 11,
21 given this one all the way down here is twelve and this one
22 all the way down here is 14, would it surprise you that
23 this difference is 11, and the answer is, it would totally
24 surprise me, and the reason is that we're carrying this to a
25 fixed value of zero, and these are compared to each other,
50
1 and each of them has a standard error attached to it, so it
2 doesn't -- it's not reasonable that this difference should
3 be 11.
4 So I went back and did the calculation. And I'm
5 going to show you how to do the calculation now and show
6 you what the number is, okay. What I need is I'm going to
7 compare these two coefficients, so I'm going to compare
8 6.09, the difference between 6.09 and 3.98. That's what I
9 need to compare.
10 In order to compare this difference, I have to get
11 what's called the standard error for the difference, and
12 this is something that we teach in our first course in
13 statistics. And if we have got two standard errors and we
14 combine them to the standard error for the difference,
15 assuming they are computed from different data sets -- which
16 they are, I didn't put anything together, '97 was done
17 completely separate from 2000 -- so the standard error of
18 the difference, actually the way we do it in statistics, we
19 always have to do things in the variance scale, so we have
20 to square these things. So we put them together, 0.29, we
21 square it, and 0.49, we square it, and then to get the
22 standard error we have to go back and we have to take the
23 square root of that. So that's the math. We square these
24 two numbers, add them up, and take the difference.
25 Now, I should have a calculator.
51
1 MR. DELERY: It's .31.
2 THE WITNESS: You got .31? Let's see.
3 THE COURT: He did it in his head, see.
4 THE WITNESS: But he is not testifying.
5 THE COURT: That's okay. For your profession, the
6 calculator is the most important tool. For lawyers, it's
7 their business card.
8 THE WITNESS: Okay. Let's just make sure I do it
9 right. So I have to make sure I do it right, and the value
10 I get, this one is .49, it can't be any smaller than that,
11 it's got to be bigger than that, bigger than the total, each
12 one separately, and it turns out to be 0.57, 0.57, okay.
13 And so we divide this by 0.57. That's the standard
14 error of the difference. And this tells us how many
15 standard deviations these two numbers are apart, and that
16 number turns out to be 3.7, 3.7.
17 Now, that's in the range of big. Two or three is
18 big. This is a big number, but it's not 11. Am I concerned
19 that this number is this big? Well, in fact, I think there
20 probably are some year to year differences, and I think I
21 testified to that before. There probably are. We don't
22 expect these coefficients to be the same year to year.
23 There is variation. There might be differences in the way
24 things are done.
25 Is this a really large number? In the context of
52
1 looking, what, at the most extreme case, right? 1997 versus
2 2000, those are the most extreme cases. Is this a large
3 number? The answer is, it's not large when you consider
4 that it's the most extreme and the fact that we don't expect
5 it to be exactly the same year to year. So this is a number
6 that I think is consistent with the kind of variation we
7 saw on the previous plot, that these are not 11 apart or
8 almost 11.
9 THE COURT: And 11 would be a large number, you
10 would agree with that?
11 THE WITNESS: I think I agreed in my cross
12 examination that I thought 11 was -- I was surprised.
13 I couldn't believe it was 11, okay, but I didn't dare do the
14 calculations right there on the spot. So 3.7, that's given
15 where we are, that's the value. Well, it is the value, so
16 it's reflected in the plot.
17 Q Dr. Larntz, do you have any -- based on what you
18 understand from the data, is there any way you could
19 understand how someone could conclude that the standard
20 deviation was 11?
21 A Well, I don't know exactly what Dr. Raudenbush did,
22 but if, if you didn't go into the regression coefficient
23 scale, if you tried to work in the relative odds scale, if
24 you didn't go to the basic logistic regression coefficient
25 scale and tried to do these directly and used this standard
53
1 deviation, but didn't take this, that we had to work in
2 coefficient scale, if you did the math, the same math that
3 we did over here, but you just took these numbers and
4 assumed that you could take the standard error of this one
5 by dividing by that, if you did that, if you did that,
6 which is --
7 THE COURT: That's not an acceptable statistical
8 procedure?
9 THE WITNESS: Well, it's not right. It's not right.
10 It's not right, because these have more variations. We saw
11 that from the confidences. These have more variations than
12 that. I mean, the standard deviation associated with this
13 isn't about ten, it's much more variation than that, or 11
14 or 15 or whatever.
15 So, but if you did that, if you did the math that
16 way, and did the same, to my understanding, did the same
17 thing, you would get 10.94 if you did it that way. So I
18 don't know what was done, but I know if you did that, you
19 would get 10.94.
20 THE COURT: And you know if that was the way it was
21 done, it's not done directly, in your opinion?
22 THE WITNESS: Oh, absolutely, that's right.
23 BY MR. KOLBO:
24 Q Is that something statisticians could reasonably
25 disagree with, whether it's appropriate?
54
1 A There is no disagreement about that at all. I mean,
2 the logistic regression coefficient scale is the scale for
3 comparison, not the -- and the relative odds are derived
4 from those directly for purposes of understanding the model.
5 Q I just have a few more questions, Dr. Larntz.
6 I think Dr. Raudenbush testified -- I understood him
7 to testify that your analysis really demonstrates only that
8 a race factor was used as a factor in the admissions
9 process. It doesn't demonstrate anything about the extent
10 to which race was used in the admissions process. Do you
11 agree or disagree with that?
12 A Well, I would disagree, and shortly -- can I just
13 briefly describe?
14 I disagree in the sense that these are big
15 representative odds. There is a tremendously large
16 allowance given to race, and it's hard to believe that any
17 other factor could explain that away, unless it were just a
18 surrogate for race.
19 MR. KOLBO: I have nothing further, Your Honor.
20 I would like to offer the exhibits, Your Honor.
21 THE COURT: Any objection?
22 MR. DELERY: No objection, Your Honor.
23 THE COURT: Fair enough.
24 MR. KOLBO: It's 225, 226 and 227. 225 is the first
25 handout.
55
1 MR. DELERY: Oh, the charts, okay. All right.
2 THE COURT: You know, if you remind me Monday, we
3 may be able to see if Court Services -- actually, when we
4 have those and admit them, they have a Polaroid, I think
5 they still have it, and we can take some Polaroids of that
6 so that everybody will -- I don't know how else to get it to
7 a form so everybody will be able to have it, so if somebody
8 reminds me Monday, I'll try to see if they have a Polaroid
9 still up there and take some pictures. If not, I'm not sure
10 how we will do it.
11 MR. PURDY: Just a suggestion, if the Court would
12 prefer, we're happy to have these typed up. We would be
13 happy to do that and then just --
14 THE COURT: That's even better. Nobody objects? We
15 will keep these. You can all sign off on them.
16 MR. PURDY: We won't even take them out of the
17 courtroom. We will just copy them and present them.
18 MR. DELERY: Or mark Dr. Larntz' notes if they are
19 the same thing.
20 THE COURT: I think we should use what he used and
21 preserve those for the official record and type them up and
22 if for some reason you can't do that, or if you can't do it,
23 we will take the numbers, we will get them down here to take
24 pictures.
25 In Judge Taylor's courtroom, they have the white
56
1 board now that you write on and then you push a button and
2 the thing comes out.
3 CROSS EXAMINATION
4 BY MR. DELERY:
5 Q Good morning, Dr. Larntz.
6 A Good morning.
7 Q Welcome back.
8 Like Mr. Kolbo, I think I'm going to -- certainly
9 hope that I stick to what you have covered here this
10 morning. We're not going to go back over the ground that we
11 covered last month when you were here the first time, and I
12 think I'll start where we left off and then work back from
13 there.
14 MR. DELERY: If I may approach, Your Honor?
15 THE COURT: Absolutely.
16 BY MR. DELERY:
17 Q The first two columns here in Exhibit 226 were taken
18 from your expert reports; is that correct?
19 A That's correct.
20 Q The first relative odds numbers came from model one
21 for each year; is that right?
22 A Yes, I think that's right. That's the first set of
23 analyses, that's true.
24 Q You reported three composite odds ratios for each
25 year; correct?
57
1 A That's true. That's true. I was responding to the
2 criticism of this particular comparison.
3 Q Right. I just want to get it clear what these numbers
4 are. These were the model one numbers from each year?
5 A Sure.
6 Q Okay. And then the second column here that you have
7 headed SD are what your report called the standard
8 deviations on the same tables from model one?
9 A That's right.
10 Q And the standard deviations were reported in your
11 report on the log scale; correct? I mean, these are
12 standard deviations in the log scale?
13 A They are standard deviations for the regression
14 coefficient, that's right.
15 Q And the regression coefficients are in the log scale?
16 A Well, the logistic, l-o-g, regression, does analysis
17 on the log odds, so that's correct.
18 Q And the odds ratios, the estimated relative odds in
19 the first column, are not on the log scale?
20 A Oh, that's true, yes.
21 Q And is there a name for not on the log scale, is there
22 a term for that?
23 A You mean on just the normal scale?
24 Q Is that what it's called?
25 A The real scale? I mean, relative odds scale, the
58
1 relative odds scale. It's the relative odds scale.
2 Q It's the normal scale; is that --
3 A I would call it the relative odds scale.
4 Q Okay.
5 A As opposed to the log odds scale.
6 Q And -- okay. The scales are not -- if you line them
7 up, they work differently; correct?
8 A I mean, there is a direct translation one to another,
9 but obviously the relative odds are it's a trans -- they are
10 a direct translation of each other.
11 Q So just so I'm clear, the two numbers reported in your
12 reports for each of these model results were on different
13 scales?
14 A Well, I guess I wouldn't consider it that way. I
15 would consider that, in fact, I gave the relative odds,
16 that's an appropriate way to summarize the value, and this
17 is the significance to understand the relative odds. So
18 I think the standard deviations are directly related to
19 the significance of relative odds.
20 Q The standard deviations are not the standard
21 deviations of the relative odds; is that right?
22 A Oh, no, no. They are an indication of the statistical
23 significance of the relative odds.
24 Q But they are actually the standard deviations for the
25 regression coefficient?
59
1 A For the regression coefficient, that's exactly how you
2 have to do that, yes.
3 Q Okay. And that's the way you reported it in your
4 report?
5 A The report, I reported with those two columns, that's
6 exactly right.
7 Q And then it was the calculations that you have done
8 here that led to the third column for the regression
9 coefficient; is that right? I guess that's actually the
10 fourth column on this page.
11 A The numbers there, I re -- I back calculated, but they
12 are the numbers from the -- if you looked at the computer
13 output for logistic regression, that those are the numbers
14 that you would see in the computer output, yes.
15 Q And so the relative odds is just the -- well, I'll go
16 the other way. To get the regression coefficient, you just
17 take the log of the relative odds?
18 A Of course, the way you get the relative odds is you
19 exponentiate the regression coefficient, that's how, but the
20 regression coefficient is the basic number that you work
21 from and then you exponentiate that to get it in relative
22 odds. That's a standard output in logistic regression, is
23 to report both of those.
24 Q So you can move back -- what you are saying is you can
25 move back and forth between the log scale and the relative
60
1 odds scale?
2 A Well, they represent the same thing.
3 Q Turning now to Exhibit 227, the standard error of
4 the difference here between the odds ratio -- or the
5 coefficient, I'm sorry -- for 1997 and 2000, you calculate
6 as 3.7?
7 A That's right.
8 Q That's a fair statement of what's on this page?
9 A I mean, someone can check the math, but --
10 Q Okay. Am I right that anything over two is considered
11 statistically significant?
12 A Without a selection bias in the sense of looking at
13 extremes, anything over two would have a five percent
14 statistical significance, but we're not doing just any two
15 here, we're doing the most extreme two, so we have selected
16 the two we're looking at out of a group, and I would say
17 that in that case you need a value, I don't know what the
18 value is, but it certainly would range, when you're
19 selecting out of, you know, the most extreme cases, that two
20 doesn't go anymore. You have got to take account of the
21 fact that you're selecting from the most extreme cases.
22 And so typically values of -- you know, you wouldn't be
23 surprised if you got values of three or four when they are
24 the most extreme.
25 Q Okay. Putting aside the most extreme context, two
61
1 ordinarily would be statistically significant?
2 A I mean, you want me to ignore the fact that -- where
3 the data came from?
4 Q For this purpose, I'm just asking you, if --
5 A If I ignore where the data came from, then you have
6 about a five percent chance of getting a value outside the
7 ranges of two if the data comes from a normal distribution.
8 Q So greater than two ordinarily would be considered
9 statistically significant?
10 A In the context of something that wasn't generated as a
11 selection --
12 Q All right.
13 A -- that would be true, in something that wasn't
14 generated as a selected.
15 Q And greater than three would ordinarily be considered
16 highly statistically significant, is that fair to say?
17 A In the same context, where you're not -- where you're
18 just looking straight away, not where you're looking at
19 extremes.
20 Q Okay. Let's go back, if we could, to Exhibit 226.
21 Your view is that all of the odds ratios that you have
22 reported here, both in the first column from your report
23 under the RO column, and then in the confidence interval
24 parentheses, are very large, is that your testimony?
25 A By the standard of statistical practice, they are very
62
1 large, absolutely.
2 Q So in the year 2000, for example, what your results
3 show is that the relative odds for African Americans as
4 opposed to whites could be anywhere from 166 to 1,176?
5 A The data are consistent with those values, yes, at
6 the 95 percent, at the 95 percent confidence level.
7 Q And so we can't really be sure where in that range it
8 falls, but you're fairly confident that it's somewhere in
9 that range?
10 A I mean, the interpretation of a confidence interval
11 now, you want to do that? Okay.
12 Q I mean, have I fairly --
13 A Those are the values that are consistent with the
14 data at the 95 percent confidence interval, confidence
15 coefficient level, that's right.
16 Q Okay. And so we can do the same thing for the other
17 years, 1999, it's somewhere between 89 and 478?
18 A Well, knowing that those statements are made with
19 error rates that are five percent, that's right.
20 Q But the conclusion you draw is that all of these are
21 large, so these differences don't trouble you?
22 A Oh, that's right. They are all substantively
23 consistent, that's what I said, and that's exactly right.
24 Q And then when you plotted it here, you said that the
25 reason that these differences, the differences in the first
63
1 odds ratios that you reported don't trouble you is because
2 they are all quite far away from zero?
3 A They are all quite far away from zero and they cluster
4 together.
5 Q So again, here zero means a relative odds of one, am I
6 right?
7 A That's right.
8 Q Which means that the members of both groups have an
9 equal likelihood of being admitted?
10 A That's true.
11 Q So from these odds ratios you can be quite sure that
12 you have disproved a contention that both groups have the
13 same likelihood of being admitted as a percentage basis?
14 A You certainly can disprove that and you can disprove a
15 lot of other odds ratios, because the lower bounds are far
16 away from one, that's right.
17 Q So you think that you can be confident of more than
18 just that the so-called null hypothesis is disproved from
19 these?
20 A Oh, certainly, certainly, certainly, and from the
21 size of the standard deviations, the number of standard
22 deviations and the corresponding confidence, yes, certainly.
23 Q You believe, in fact, that you have quantified,
24 I believe that you said, the role that race plays in
25 admissions based on these numbers; is that right?
64
1 A Well, I am just a statistician, so I'll be careful of
2 what I did, okay? What I have done is I have described what
3 the admissions office did, okay? And so this quantifies the
4 admissions decisions, okay? And so what I have done is
5 described, quantified the admissions decisions, and this is
6 from their data and this is what their data tells me about
7 that.
8 Q And I want to be clear about that, because we
9 obviously had some fairly lengthy discussions of this issue
10 when you were here last month.
11 Today, Mr. Kolbo asked you some questions about
12 whether anything that Dr. Raudenbush said in his testimony
13 changed your opinions concerning the effect that race plays
14 in the admissions process. Do you remember those questions
15 from Mr. Kolbo?
16 A I'm not sure of the exact wording, but I took that to
17 mean if was there any change in my opinion from previously,
18 and I answered that there was no change in my opinion
19 previously.
20 Q Okay. So putting aside that, now I want to ask the
21 follow-up question. Do you believe that you have expressed
22 an opinion concerning the extent -- and I'm sorry, I guess I
23 may have misspoken a moment ago. I think Mr. Kolbo used the
24 word extent, not effect, if I misspoke, but do you believe
25 that you have expressed an opinion concerning the extent to
65
1 which race is taken into account in the admissions process?
2 A What I have done is expressed an opinion of the size
3 of the allowance that is shown in the data that's from the
4 admissions office for individuals that have the similar
5 credentials, the advantage that's given based on ethnic
6 groups in those for individuals with similar credentials,
7 and that's what I testified to, I hope, and that's really
8 what the conclusion is, is that I have quantified
9 statistically the size of the allowance that's given
10 for individuals with similar credentials.
11 Q And just so we're clear, by size of the allowance, do
12 you mean by an admissions officer sitting down to read a
13 file in making the decision?
14 A I mean describing the admissions decisions, what show
15 up in their -- from their data.
16 Q What shows up as a result of the decisions?
17 A As a result of the decisions, absolutely. It's the
18 results of the decision. I don't think I have ever said
19 or didn't mean to say that I did anything other than to
20 describe. It's their decision. I'm describing what the
21 results are.
22 Q Right. But again, just so we're clear, that's
23 different from describing how the decisions were made;
24 correct?
25 A These are the results of the decisions.
66
1 Q So is the answer to my question yes?
2 A Well, I don't know exactly how decisions are made.
3 Q Okay. And these data don't say anything about how
4 decisions are made?
5 A They say a good deal about the results of those
6 discussions and that's what I would say.
7 Q But not how the decisions were made?
8 A The mechanism for making decisions, that's not
9 statistics.
10 Q The process?
11 A That's not statistics.
12 Q I believe that you said earlier in asking -- answering
13 the questions for Mr. Kolbo that you're confident in the
14 substantive stability of your results, but that the odds
15 ratios were not technically statistically stable across the
16 years, did I get that right?
17 A I think that what I would say is that the substantive
18 results are clearly consistent, okay, clearly consistent.
19 There may be, and I think there probably is, I would expect
20 there to be some technical, in the sense of statistical,
21 variation from year to year. I wouldn't expect the
22 variation from year to year to year, so what I was
23 responding to is I think that the variation from year to
24 year is probably not zero. There is probably some -- in
25 the underlying true scale, that there probably is some year
67
1 to year variation, but it's not large enough to change the
2 substantive conclusions, that's right.
3 Q So the differences among the odds ratios across years
4 don't lead you to alter the basic conclusions that you have
5 reached, that's the bottom line of what you're saying?
6 A Substantively they are the same from year to year,
7 that's what I said.
8 Q I think maybe now I would like to go, if we could,
9 back to the last page of the packet that you have. It's
10 Exhibit 225, I believe, if you could look at that.
11 A I have it.
12 Q The last page was the chart of the relative odds
13 effect on baseline probabilities.
14 A That's correct.
15 Q Am I right to understand that this is basically a way
16 to translate odds to probabilities, is that what this is
17 doing?
18 A It's showing the effect of a relative odds value on
19 certain -- on a range. I tried to do the whole range of
20 probabilities. I think both sides offered their own
21 baseline probabilities. I mean, we both did. And so I
22 thought we should probably, just for clarification, give the
23 whole range of probabilities.
24 Q Okay. So if we look across the top, you have various
25 relative odds, 5, 10, 20, 50, 100, 200?
68
1 A Sure.
2 Q Do you see that?
3 If all we know is an odds ratio number like 100,
4 if that's all we know, we don't know anything about the
5 relative chances of admission of two groups, is that fair
6 to say?
7 A We don't know the probabilities, that's right.
8 Q Okay, because on your chart here, an odds ratio of
9 100 could translate to any of the probabilities that you
10 have listed under that heading as compared to the baselines;
11 right?
12 A Well, it depends on the baseline, yeah, sure.
13 Q But the point is that unless you know, unless you
14 know the underlying probabilities, you can't evaluate the
15 significance of an odds ratio like 100?
16 A Well, it would mean different things for different
17 baseline probabilities, that's what I'm trying to
18 illustrate, exactly.
19 Q Now, I believe you testified when you were here before
20 that an odds ratio of two or three, you would consider
21 large; is that right?
22 A In my work, two or three is a big number in odds ratio
23 terms, yes, absolutely. And I saw one of eight the other
24 day, so everyone was astonished in the room.
25 Q So a select group, I guess.
69
1 A Well, they were clinical medical researchers and they
2 were astonished that it was as big as that. Eight was big.
3 Q And that's -- if you think about it in terms of a
4 medical study for a second, you don't have two or three here
5 on the last page of Exhibit 225, but you do have five.
6 A Oh, yeah, sure, sure.
7 Q Which, if two or three is large, then five is clearly
8 large?
9 A Five is a considerable effect.
10 Q You have, if we look here, a baseline probabilities
11 of .1 or 10 percent for a relative odds of five, the other
12 group's probability would be .35, .36, if you round, is that
13 what this table shows?
14 A I'm not with you. I'm sorry.
15 Q Okay. If we look under relative odds of five, so the
16 first column.
17 A I have that.
18 Q Starting with the baseline probability of .1.
19 A Oh, .1, I'm sorry. I misunderstood.
20 Q Yes, .1 for 10 percent?
21 A So 10 percent, 10 percent translates into -- if the
22 relative odds is five to 35.7 percent, that's right.
23 Q And so that's the kind of effect that you're
24 considering to be really large based on your experience,
25 for example, in medical studies?
70
1 A Well, it is a large effect, yes.
2 Q Okay. And is it fair to say that it's your opinion
3 that in the context of this case, the differences in all of
4 the odds ratios that you have reported are not significant,
5 because they are all so far beyond this two, three, five
6 level that you find really large?
7 A I think that the odds ratios here are large. I think
8 some are larger than others. I don't say that there are no
9 differences between them statistically or substantively.
10 I think that they are larger for some ethnic groups than
11 others, but they are -- but for the selected minority
12 groups, they all looked large, yes.
13 Q Is it fair to say that in order to interpret the
14 significance of a reported relative odds number, you need
15 to look at the underlying probabilities?
16 A Oh, absolutely. And we did that. We looked at the
17 grids and these are just -- the relative odds summaries are
18 just summaries of what goes on in the grids, and the grids
19 give us those baseline probabilities, absolutely,
20 absolutely. I absolutely agree, you need to look at the
21 basic data in the grids, yes.
22 Q Would you also say that it's important to look at
23 the sample size or the pool size of the groups that are
24 contributing to the relative odds calculation in order to
25 understand the significance of a reported relative odds
71
1 ratio?
2 A That's actually taken into account in the statistical
3 significance calculation, so different sample sizes enter
4 into that. The amount of information about a particular
5 ethnic group is accounted for in that, but absolutely, if
6 there are very small numbers, very small numbers in the
7 aggregate, then those would be -- well, correspondingly
8 the confidence bounds for those would be wider.
9 Q So it's fair to say that to really interpret and
10 report estimated relative odds, you need to look at the
11 odds ratio number, the underlying probabilities and the
12 underlying pool sizes, is that fair to say?
13 A Well, I think a good bit of that information is
14 accounted for in the statistical significance, but that's
15 right, that is certainly -- you can certainly say that you
16 could look at those and that would be informative, yes,
17 absolutely.
18 Q Okay. Is it fair to say that if you didn't look at
19 any one of those, in your view, you would be missing an
20 important piece of evaluative information?
21 A I mean, those are things you would look at. Whether
22 you needed to look at each one of them for each particular
23 context, it would depend on the context.
24 Q Okay. I think I understand from what you have just
25 said, in other words, that you might draw different
72
1 conclusions about a reported relative odds number if you
2 were talking about three members of a group as opposed to
3 30 or 300, is that fair to say?
4 A Well, if the data came from three, I'm sure we
5 wouldn't have gotten any statistical significance, and so
6 the size of the standard deviations accounts for the sample
7 size to a great extent.
8 Q But for the same odds ratio like 100 that we were
9 talking about before, you would draw different conclusions
10 from it if it were based on three members of a group as
11 opposed to 30 or 300, is that fair to say?
12 A Well, we would surely have different measures of
13 statistical significance from those cases.
14 Q And therefore, from that, you would -- you would reach
15 different conclusions based on that number?
16 A I would reach different conclusions in the sense that
17 for the smaller group, the confidence intervals would be
18 much wider and so I would be less sure of that, yes.
19 Q Okay. If you could, actually, I would like for you to
20 look at one of your earlier reports. It's Exhibit 138.
21 And if you could look -- do you have it now,
22 Dr. Larntz?
23 A It's the February 21 supplemental report.
24 Q Yes.
25 A Yes.
73
1 Q And if you look, there is a series of 24 pages of what
2 we have called cell by cell odds ratio calculations there?
3 A Yes.
4 Q Do you see that?
5 If you would look at page six of 24 there --
6 MR. DELERY: This is again Exhibit 138, Your Honor.
7 THE COURT: Page 24?
8 MR. DELERY: The last set of the report, the back
9 of the report, is a series of pages, 24 pages numbered
10 page one of 24 and so forth, and I would like to look at
11 page six of 24.
12 THE COURT: Okay.
13 BY MR. DELERY.
14 Q And particularly the second block there of lines for
15 LSAT ranges from 164 to 166, I think it is. Do you see
16 that?
17 A Yes, I see that.
18 Q It's the second paragraph, so to speak, on this page.
19 The first cell there for 3.75 GPA and up has a
20 minority -- three minority applicants and three were
21 admitted, and 131 majority applicants and 56 admitted; is
22 that correct?
23 A That's correct.
24 Q And you have calculated infinite odds ratio for that?
25 A The computer did the calculation, yes.
74
1 Q Right. And then for the next cell down you have again
2 three minority applicants, only two of whom were admitted,
3 and then 216 majority applicants, 73 of whom were admitted?
4 A Yes, I see that.
5 Q And because two out of three minority students were
6 admitted as opposed to all three out of three, the odds
7 ratio goes from infinity down to 3.92; is that right?
8 A That's correct.
9 Q And that's due to the fact that there is such a
10 small sample size of minority students; isn't that right?
11 A Yes. If we're looking at each individual cell, and
12 these cell by cell comparisons, you recall, were not the
13 primary, now, these are supplemental analyses, and I didn't
14 naturally expect to see statistical significance in these,
15 because there would be small sample sizes, and in fact, for
16 each of these two, the P values are bigger than .05, and so
17 in and of themselves, which is -- again, they were never
18 meant to be stand-alone, but in and of themselves, neither
19 one shows statistical significance, that's right.
20 Q But it is fair to say that many of the cells that you
21 have highlighted on the charts that you showed us earlier
22 had very small numbers of minority applicants in them; isn't
23 that right?
24 A But the analysis doesn't do it cell by cell, it
25 generates the information across all the comparative cells
75
1 or comparative information, so that's a very different story
2 than looking at an individual cell and saying there are few
3 numbers there. Your composite estimate combines across all
4 the cells, all the cells with comparative information.
5 So I think we have to make sure you're talking about
6 the cell by cell analysis. This was done specifically to
7 look and see if there was, you know, an effect of odds ratio
8 across these cells, and this is -- this is for these
9 particular cells. They are small, and I don't disagree
10 with that.
11 Q All right. And I actually want to move, use this
12 as sort of a transition into the charts that you put up
13 earlier and how you calculated the composite odds ratios.
14 Actually, maybe we should -- if we could, can we put
15 one of those back up?
16 While we're waiting, I think I can ask you some
17 preliminary questions.
18 You highlighted the cells for whom applicants were
19 included in the composite odds ratio calculation; is that
20 right?
21 A What I did, yeah, is I highlighted the cells that
22 provided comparative information, that's true.
23 Q And the comparative information that they provided was
24 on a cell by cell basis; right?
25 A The comparative information is across the cell by cell
76
1 for -- it's across all the ethnic groups, that's right.
2 Q Across all the ethnic groups within each individual
3 cell; correct?
4 A Well, I mean, I talked earlier about the fact that
5 we're looking at differences in a cell by cell fashion as
6 opposed to modeling it overall, yes.
7 Q Right. So you --
8 A So the composite -- I'm sorry, if I might -- the
9 composite puts together the estimates cell by cell, which
10 may be quite variable, and puts them into a composite that
11 accounts for all the information, all the comparative
12 information.
13 Q But it uses as its basic building block, if I
14 understand it correctly, the cell that's defined on the
15 grid; correct?
16 A Well, the basic building block is to look at people
17 with similar credentials, and what we did is looked at
18 individuals with similar credentials, similar grade point
19 averages, similar LSAT as defined by the law school, and
20 their grid, the grid -- remember, the grid boundaries came
21 from the law school, so they provided these grid boundaries,
22 and we looked at the comparative information in looking,
23 combining it across these individual cells, that's right.
24 Q And in fact, you created a variable for each cell that
25 contributed comparative information; isn't that right?
77
1 A The computer actually created cells -- created one for
2 every cell and then evaluated whether it was comparative. I
3 mean, every cell was included in the analysis and I didn't
4 go in and have it exclude cells, no.
5 Q But I'll come back to the computer here in a minute,
6 but to follow on this point, the cell itself contributed a
7 variable in your analysis; in other words --
8 A What we did is we looked at each cell to see if
9 that -- the computer looked at each cell and the computer
10 determined how much it would contribute and added in the
11 amount of comparative information, depending on what was
12 in the cell, that's always true. That's true of any
13 statistical analysis.
14 Q Okay. You mentioned the computer doing a lot of
15 things here this morning, but the computer only did what
16 you told it to do; isn't that right?
17 A I think that would be -- that would be in the ideal
18 world, yes.
19 Q Do you have any examples of where the computer
20 overrode your instructions and did something on its own?
21 THE COURT: In this case?
22 BY MR. DELERY:
23 Q Putting aside your microwave?
24 A Computers are only as good as the programming that
25 goes into the computer, that's always true.
78
1 Are there errors in the programs that I used? I
2 used programs that I think are reliable. I checked them
3 out. I feel quite confident that it was doing what I wanted
4 it to do. It did a lot more than I wanted it to do, because
5 that's what computer programs do, they provide you with --
6 we used to have a term, a side inch, in the old green paper,
7 we used to call them how many side inches of output did you
8 get, you know, and how many side inches. We don't do that
9 anymore, we save it all on disk, but the number of side
10 inches was considerable in this case.
11 Q You're not trying to distance yourself, though, in
12 some sense, from what your computer did?
13 A Oh, no, not in any sense. No, I don't mean to do
14 that. If I give that impression, it wasn't -- I'm not -- I
15 am not, in any sense, saying that I didn't control what the
16 computer did.
17 Q You weren't helpless in the face of the computer that
18 was excluding this data; right?
19 A I enjoy it when the clerk says the computer lost your
20 reservation, yes, I enjoy that, in the sense that I know
21 it's a person that made a mistake that caused that to
22 happen. I'm very, very cognizant of the fact that computers
23 do what people instruct them to do, yes.
24 MR. DELERY: Okay. So if we could put up one of
25 the -- I guess the 1995 grid, which was the first page of
79
1 Exhibit 225, just so we're all looking, and if you could
2 slide it over so it's all there.
3 BY MR. DELERY:
4 Q Just so we have a sense of what is here, I believe you
5 said -- let me make sure I get this right. You highlighted
6 the cells that contributed comparative information; is that
7 right?
8 A That's what I attempted to do and the highlighting was
9 done by me.
10 Q Not the computer. The computer told you which ones it
11 had from --
12 A No, no. I actually -- I actually went back and looked
13 at the cell grids and determined which ones would have
14 contributed, so I did it.
15 Q Okay. And then I think you said that about -- across
16 these years -- 84 to 88 percent of the total applicants are
17 in the shaded cells; is that right?
18 A I did a calculation and I think that's right, you
19 know, best that I could punch my numbers in. I actually --
20 I actually counted across, and just to be clear, and totaled
21 the number, and I counted down and totaled the number, and
22 then I counted down and across and made sure that they came
23 out to be the same value. So I actually -- I actually did
24 check to see the best I could that -- those numbers that
25 are not highlighted, actually, is what I counted, and then
80
1 calculated the percentage of highlighted.
2 Q And just so I understand that correctly, the reason
3 that you excluded, for example, this cell, which is at
4 3.5 to 3.71 -- 74, hard to tell -- and then I think that's
5 148 to 150, so this cell here, 35 applicants, zero in its --
6 A That's right.
7 Q The reason that that cell is not highlighted is
8 because, although there were members of different racial
9 groups, everybody was denied, and so in your definition of
10 comparative information, that doesn't contribute any?
11 A There would be no comparative information from
12 that cell.
13 Q From that cell, all right. And that's true even
14 though those -- that cell reflects actual decisions by
15 the admissions office?
16 A Right. Decisions that were the same for everyone in
17 that cell, that's right.
18 Q And in your view, I think you said, cells can -- even
19 cells that you shaded contributed different amounts of
20 comparative information?
21 A I think that's -- well, that's certainly true. I
22 mean, every cell contributes a different amount -- well, I
23 mean, I won't say every -- yeah, I think every cell probably
24 contributes a different amount.
25 Q So the way you set up your model, the closer the two
81
1 groups were to being treated the same, the less information
2 was contributed to the composite odds ratio; is that a fair
3 statement?
4 A That's not true, no.
5 Q So what do you --
6 A I mean, that is not necessarily true. I mean, if in
7 fact they were treated the same across the cells, if the
8 admission rates were similar, then we would have gotten odds
9 ratios that would have used those, that information, and
10 calculated odds ratios appropriately.
11 Q Well, you said earlier that for cells where members of
12 majority groups on the one hand and minority groups on the
13 other hand were not treated exactly the same, but were
14 treated close to the same, the cells contributed less
15 information?
16 A Well, I think I better be careful, then, and make sure
17 we understand. In cases where virtually everyone is
18 admitted or virtually everyone is denied, particularly
19 ones where everyone is admitted, then that will contribute
20 relatively small amounts of comparative information. We
21 see in the grids back before, there was not statistical
22 significance in some of those cells.
23 Q So it's not just whether the groups are treated the
24 same, it's whether the relative probabilities are at one or
25 the other extreme; is that right?
82
1 A Oh, absolutely. Absolutely. Absolutely. I mean,
2 what we're doing is doing comparative analysis. We're
3 comparing the relative probabilities. That's what we're
4 doing.
5 Q So where two groups are treated the same and the
6 probabilities are extremely high or extremely low, that
7 contributes little information?
8 A Where they are treated the same --
9 Q And both probabilities, the probabilities for both
10 groups are high or low.
11 A Now, I think it's true. I mean, it's a continuum in
12 the sense that if everyone were admitted, then there is no
13 comparative information. If virtually everyone is admitted,
14 amount of comparative information is small. That's true,
15 okay. I can say that, and that is what I can say.
16 Q Okay. Now, if 84 to 88 percent of the applicants are
17 in the shaded cells, that means somewhere, 12 to 15 percent,
18 16 percent of the applicants are in cells that you excluded;
19 correct?
20 A Cells that provide no comparative information, that's
21 true.
22 Q Okay. Do you consider 12 to 16 percent of the data
23 not providing comparative information to be a high
24 proportion?
25 A Oh, it's not high at all. It's not high at all. I
83
1 mean, it's cells that, particularly when you look at the
2 cells that we see what kind of cells they are, there were
3 cells for the most part, not 100 percent, but for the most
4 part, they are cells where people have presented themselves
5 with very low credentials. I mean, grade point averages
6 less than 2.5, LSATs less than 148. I mean, those are
7 the -- I mean, look where the area is, yeah, here, all the
8 cells, they are all down in this area, right, and up in
9 here, and with an occasional cell here and there, just
10 depending, as we would expect.
11 Q So 12 to 16 percent of the applicants not in cells
12 contributing comparative information doesn't trouble you.
13 What about 20 percent, would that concern you?
14 A It depends on where the -- it depends on where the
15 credentials are. If in fact everyone moved, if in fact,
16 if in fact the admissions office decided we needed higher
17 LSATs and higher grade points and we moved it up, I mean,
18 the cells where they made different decisions for
19 individuals, where they were making decisions other than
20 just all rejects or all admits, but all rejects, in
21 particular, the ones that we're losing here, in that sense,
22 if they moved it up I would be satisfied, statistically,
23 because I want to look at where there is a potential for
24 difference in potential, not whether there is, but potential
25 for a difference in admission rates, that's what I want to
84
1 look at.
2 Q So in fact, the proportion of applicants excluded
3 doesn't really trouble, doesn't concern you for purposes of
4 this analysis, that's not a number that you need to look at
5 in order to evaluate your results?
6 A I mean, it's a number I offered for information.
7 It's not a number in the sense that I really want to do
8 an analysis of the comparative information, so that's what
9 I did.
10 Q Okay. When you were looking at the proportion of
11 the applicants excluded or not providing comparative
12 information, to use you phrase, did you look to see what
13 percentage of the minority applicants fell in the cells that
14 are not highlighted on these charts?
15 A I didn't calculate it separately for each group, no.
16 Q Did you calculate it as an aggregate for all of the
17 minority groups put together?
18 A No.
19 Q For 1995, would it surprise you to learn that there
20 are 245 selected minority -- or let's see, I guess it's a
21 total of 267, 267 minority students, in other words, African
22 Americans, Native Americans and Hispanics who fall into the
23 cells that you haven't shaded?
24 A Well, I'm not going to -- I'm not going to act
25 surprised or not with respect to this statement. It is
85
1 true that minority applicants have on average presented
2 themselves with lower grade point averages and lower test
3 scores, on average, so in that sense, it may be that there
4 are more of them that are in the unshaded areas, that
5 certainly could be true.
6 Q Well, not -- I don't want to say more, because in
7 fact, by our count there are 595 applicants in the
8 non-shaded areas, 267 were minority students, and 328
9 were majority students.
10 A Okay.
11 Q If I represent that to you, does recognizing that
12 you haven't counted it, does that -- is there a reason to
13 correct me right off the bat as you did earlier?
14 A Certainly, I didn't do the calculations, so I'm not
15 going to correct you right off the bat.
16 Q And 267, if we have done the calculations right, comes
17 out, 267 minority applicants comes out to 39 percent of the
18 total number of minority applicants, my question is whether
19 excluding or constructing a model so that 39 percent, almost
20 40 percent of the minority applicants don't contribute
21 comparative information, whether that causes you any concern
22 for the design of your model.
23 A Well, the reason they are -- the reason they don't
24 provide comparative information has to do with the decisions
25 made, the results of the review of their credentials, and so
86
1 this is where we're at. I would -- I think they -- if they
2 are not admitting minority students or majority students in
3 these particular cells, then those cells are people with
4 credentials that they don't offer admission to.
5 Q So if my numbers are right, then your composite odds
6 ratio calculations are based on calculations that don't even
7 look at the information provided by 40 percent of the
8 minority applicants; is that right?
9 A Well, that would be if the percentages -- we can go
10 back and calculate, anyone can go back and calculate,
11 because I have given you all the grids, you can count and
12 see how many are in these cells. And we were looking at the
13 cells with comparative information, and that's what we did.
14 Q But assuming that that percentage is right, that would
15 mean that your composite odds ratios were calculated without
16 looking at the information provided by 40 percent of the
17 minority applicants; is that right?
18 A All the information was looked at. All the
19 information was looked at. I didn't go in and exclude
20 information. All the information was looked at. They
21 didn't provide comparative information on that composite.
22 Q And so the computer, going back to the computer,
23 didn't take that information into account when it calculated
24 the composite odds ratios that we looked at earlier?
25 A It got all the information out of there that was there
87
1 in computing the odds ratio and there wasn't any information
2 for comparison.
3 Q As you define comparative information?
4 A That's true.
5 Q Now, in your medical work, in your drug studies, have
6 you had occasion to reach conclusions about the relative
7 odds of a treatment when, say, 40 percent of the people who
8 took the placebos were excluded from the analysis?
9 A It would depend on what's -- where they were at in a
10 particular study. I don't know that that percentage has
11 ever arisen. I don't think that has ever arisen, but it
12 would depend on where they are at. Certainly, with respect
13 to analyses, we often exclude large numbers of people. For
14 instance, for instance, if we're looking at the effect of a
15 treatment on death, it will turn out --
16 Q Treatment for death? I'm sorry.
17 A A treatment, the effect of a treatment.
18 Q I'm sorry.
19 A The effect on death of a treatment, so for instance,
20 you're giving some kind of medical treatment, a heart
21 treatment or something like that, it turns out that the
22 risk of death is highly related to age, which is, you know,
23 that seems -- there seems to be evidence of that and the
24 data would do that, would relate that, and we would often be
25 in a study where we have hundreds and hundreds of people,
88
1 but for instance, there would be no deaths from people
2 under age 40, say, in the study, and the comparisons there
3 would -- for looking at death would exclude those people for
4 whom there were no deaths, groups from whom there were no
5 deaths, if we took account of age in the analysis.
6 So what I'm saying is, depends on the special
7 circumstances of the analysis and the variables, but for
8 instance, in a medical study such as that, we may wind up
9 doing the same -- we wouldn't -- this is a standard method,
10 so we wouldn't wind up doing this kind of comparison and it
11 may be that large numbers of -- large numbers of individuals
12 would not be included in the composite odds ratio that we
13 would report for a particular effect.
14 Q Is that kind of study that you just described the
15 template that you had in your mind when you designed your
16 work here?
17 A The template I had in mind is doing a good statistical
18 analysis for comparison, using methods appropriate to a
19 binary response.
20 Q Do you equate the cells where no one was admitted,
21 the non-highlighted cells in this particular chart, to the
22 situation that you just described where there were no
23 deaths?
24 A Or where everyone died, either way. I mean, there are
25 cells that don't provide us comparative information for the
89
1 issue at interest, that's right.
2 Q So from your perspective, those two situations are
3 analogous in terms of designing the study?
4 A Well, I mean, it's important to understand that
5 you want to estimate the effects where there will be
6 probabilities, probabilities other than zero and one,
7 I mean, that's what we would do.
8 Q Now, you mentioned a moment ago that you chose the
9 cells that were defined by the law school and we went over
10 those?
11 A Yeah, followed the law school grids, that's right.
12 Q And this was a document from 1995, it's Exhibit 16,
13 the one that you had been given at the outset of your work,
14 right?
15 A That's right.
16 Q And you just took the grid definitions that you found
17 in Exhibit 16 and applied it for the other years; is that
18 right?
19 A These grid definitions came from that exhibit, that's
20 right.
21 Q Did you look at -- did you ever consider whether you
22 should use different LSAT ranges, for example, to define
23 your cells?
24 A Did I ever look at other ones?
25 Q Yes.
90
1 A No.
2 Q So across the top here, and it's a little hard to
3 read, but the cells go -- you know, figure out how to say
4 this, the third column in, for example, has an LSAT of 146
5 and 147, do you see that?
6 A Yes.
7 Q That's how it's defined?
8 A Yes.
9 Q So that's two LSAT points in that cell.
10 The next one to the right has 148 to 150, so that
11 has three LSAT points?
12 A That's right.
13 Q The next one has three again, 151 to 153, am I right?
14 A Oh, I'm sorry, yes.
15 Q And then the one continuing to the right has only two,
16 154 to 155?
17 A That's right.
18 Q And we go back to three, 156, 157, 158.
19 Did you consider, based on this pattern, whether
20 there was a -- whether it made sense to keep this definition
21 of the cells when you were designing your analysis?
22 A Whether it made sense to keep -- did I think from that
23 pattern -- no, I didn't, I didn't worry about that, no.
24 Q Just didn't consider one way or the other whether
25 these particular lines that were drawn in Exhibit 16 should
91
1 be changed for purposes of your analysis?
2 A Well, what I did is I specifically wanted to use
3 someone else's lines, if I might say. I mean, they used
4 these lines, so I used the ones they used.
5 Q Okay. Even though some cells happened to include
6 two LSAT points and some happened to include three?
7 A That's the way they looked at the data.
8 Q Okay. And if you shifted it, obviously, if you put
9 the 153, for example, people with LSAT scores of 153 into
10 the cell to the right of that and went with the 154 and 155,
11 you would change the number of applicants and the number of
12 admits in some of these cells, you would expect?
13 A Sure.
14 Q And you didn't consider whether you should adjust your
15 analysis in any way to take that into account?
16 A No, I didn't adjust my analysis to take into account
17 of that, no.
18 Q When you designed your structure based on Exhibit 16,
19 the grids that you received, did you make any inquiries to
20 try to find out whether the admissions office based its
21 decisions in any way on these particular lines, for example,
22 for LSAT scores?
23 A No, I don't think -- I don't think that that's --
24 I don't think I made any -- I know I didn't make any
25 inquiries, if that's what you're asking.
92
1 MR. DELERY: Okay. If I could, Your Honor, I would
2 like to put up a board which is just a copy of this same
3 page without the highlighting on it.
4 THE COURT: Sure.
5 MR. DELERY: Just so we can see that. I think we
6 can take this down, I don't know. We need to figure out how
7 to turn off the light.
8 BY MR. DELERY:
9 Q Now, I think when you were here before, in your
10 powerpoint presentation -- can you see it? I should perhaps
11 turn it more so that you can have a better view of it. Can
12 you see that?
13 A Right. I see there is a number missing out of that
14 thing. I mean, someone didn't put a number up.
15 Q Okay. Actually, I think it turned out that all of our
16 copies didn't have -- copies that we had originally didn't
17 have the numbers in it, but it's 151, I think.
18 A Yeah, yeah, okay. Right.
19 Q So I'll just write that in. I think in the copying
20 process, when it came over to us, that was missing and so we
21 have figured out that it's 151.
22 But earlier, when you were here the last time, you
23 highlighted a cell here that had -- and I'm going to put
24 a black box around it on this chart, 198 applicants and
25 17 admits. Do you recall that from your powerpoint
93
1 presentation?
2 A Well, I certainly highlighted a cell and --
3 Q I think it's Exhibit 143, if you would like to look.
4 I just want to use this same one as a starting point from
5 where you were here.
6 A Well, okay. We can start there. That's fine.
7 Q It starts on slide 16, you had a series where you
8 took us through a cell, and I think this is the cell.
9 A Yes, I see that. I see that. I see that. That's
10 fine.
11 Q Okay. At any point did you consider what would happen
12 if you had used a larger set?
13 A Whether I used these cells --
14 MR. DELERY: Actually, let me -- if I can put up one
15 other board, Your Honor.
16 THE COURT: Sure, but I'm not sure where you're
17 going. Don't forget, this is rebuttal.
18 MR. DELERY: If you'll bear with me, I'm building
19 back up to the highlighted section, if I could do that for
20 a second.
21 THE COURT: I'm with you.
22 MR. KOLBO: Your Honor, I think it is beyond the
23 scope of cross examination at this point, and I do want to
24 lodge that objection.
25 THE COURT: I'm going to bear with him for just a
94
1 short while, but this is -- as I have indicated, rebuttal is
2 rebuttal.
3 MR. DELERY: Sure. I'll just -- I'll do it very
4 quickly, Your Honor, if I could.
5 BY MR. DELERY:
6 Q You could have drawn a cell, for example, that
7 included the eight cells around the one that you started
8 with?
9 A One could, one could do that.
10 Q And one could look at what the odds ratios would be
11 for that cell, for example, and how much comparative
12 information that would contribute as that definition was
13 made?
14 A Well, I mean, there is going to be comparative
15 information, because they -- all these cells, they are all
16 shaded, I think.
17 Q Okay. Oh, you're right, they are all within the
18 shaded range on the exhibit that you brought today.
19 A Right. If you made cells bigger and included some
20 of the unshaded ones, you would have more cells and more
21 individuals, that's math, yes.
22 Q Okay. Similarly, we could -- you could slide this
23 nine cell grouping up to the right so that you get the
24 uppermost right nine cells, for example, isn't that right?
25 You could calculate an odds ratio for that?
95
1 A One could do that for any combination of cells you
2 want, sure.
3 Q Now, you said earlier that your shaded areas included
4 80-some percent of the total number of applicants?
5 A Right.
6 Q Did you do any analysis to look at where the admits
7 were coming from, which cells contributed proportions of the
8 admitted students to the law school class?
9 A Did I calculate how many -- what percentage of the
10 admits were included?
11 Q Yes.
12 A Well, I mean, we know virtually all the admits were
13 included. Not all, but virtually all the admits were
14 included. All the cases of admissions except for the cells
15 where everyone was admitted or in the few cells where there
16 was only one ethic group, so virtually all the admits are --
17 virtually all are included.
18 Q Okay. And in fact, isn't it the case that about
19 75 percent of the admitted students come from these nine
20 cells in the red box that I have drawn here, the uppermost
21 right-hand nine cells?
22 A Those would be the cells with the students who present
23 with very high credentials and for whom the decisions are
24 made based on those very high credentials and they -- and
25 there are -- it's not -- if I can read that from here, but
96
1 it's not uniform across that, there is quite a difference in
2 admission rates across that, but a large number of admits
3 come with that cell -- that area, excuse me.
4 Q Okay. But you didn't look to see what proportion
5 and whether, for example, it was about 75 percent of the
6 admitted students came from these very uppermost right-hand
7 corner cells?
8 A I never calculated that percentage, no.
9 Q Okay. And I guess similarly, we could draw a cell
10 above that included the students above a 150 LSAT, which is
11 about the 50th percentile, right, and above a 3.0, and could
12 calculate an odds ratio for the admitted students in that
13 larger cell; right?
14 A We can do this for any group of cells. I said that.
15 Q For any combination.
16 Would it surprise you if I represented to you that
17 95 percent of the admitted students in 1995 came from the
18 cells in this grid -- I'm sorry, I have misdrawn the line
19 slightly. I want to drawn it down one.
20 So in this range, that 95 percent of the admitted
21 applicants came from that upper right-hand quadrant, above a
22 150 and above a 3.0?
23 A I mean, you can do some calculations based on the
24 totals over here. Clearly a large percentage of the
25 admissions, admitted students, come in those cells,
97
1 that's right.
2 Q And that's consistent with your -- with your
3 highlighting, because those are essentially the cells, with
4 a few exceptions, that you highlighted this morning;
5 correct?
6 A I think almost virtually all the cells that had
7 admitted students were highlighted. I think we said this
8 already.
9 MR. KOLBO: Are you done with this?
10 MR. DELERY: For the most part. I want to ask one
11 more question.
12 BY MR. DELERY:
13 Q So would it surprise you to learn that the odds ratio
14 for that larger cell, accounting for 95 percent of the
15 applicants, is just over two?
16 A Oh, the odds ratio for a larger grid where you ignore
17 the difference in LSAT and GPA within there?
18 Q Yes.
19 A Would it surprise me that it gets down lower?
20 Q 2.45?
21 A It's going to be lower. It's clear from looking at
22 this grid, it's clear that there are incredible differences
23 in admission rates across those cells, and so combining
24 those would be -- you can do it, and you can calculate an
25 odds ratio, but you're losing a lot of information about the
98
1 individual, the credentials.
2 Q Are you familiar with the testimony of -- that has
3 occurred in the trial from admission officers concerning the
4 range of qualified applicants, what the admissions office
5 considers to be qualified applicants?
6 MR. KOLBO: Your Honor, I just want to object.
7 THE COURT: Your objection is sustained. We're way
8 beyond.
9 MR. DELERY: All right. Thank you, Your Honor.
10 BY MR. DELERY:
11 Q So a cell -- let me just ask you this one question.
12 I think it's back connected to the highlighted tables.
13 The cell that we have drawn there, the large red
14 box, has substantial overlap with the highlighted cells
15 that you identified earlier as contributing comparative
16 information to the odds ratios; correct?
17 A I think I have answered that, yes.
18 Q So the same -- the same cell that you used to
19 calculate an odds ratio above 200, the cells that you used
20 to calculate an odds ratio above 200, taken together, yield
21 an odds ratio of about 2.4?
22 THE COURT: He has already answered. He says he
23 doesn't know exactly, but it would be a different amount.
24 MR. DELERY: Okay. Fair enough, Your Honor.
25 BY MR. DELERY:
99
1 Q Last thing I want to ask you about, Dr. Larntz, is
2 your discussion of assumptions earlier this morning. You
3 indicated in response to some questions from Judge Friedman
4 that a model that took into account all of the data in the
5 cell, in all of the cells, would have more assumptions than
6 your model, which looked at only the ones that contributed
7 what you called comparative information; correct?
8 A In my estimation, my statistical opinion, that would
9 require some kind of assumption connecting the cells and I
10 made no assumption connecting the cells.
11 Q Do you believe that a model that takes into account
12 all of the cells would actually have more assumptions or
13 are you just saying that it would have this additional
14 assumption, but it might have fewer in other respects?
15 Do you understand my question?
16 A When we're counting assumptions, I mean, we're --
17 there is a whole slew of different ways of doing the
18 counting, okay, so I'm -- I don't want to --
19 Q That's exactly what I thought, which is why I asked
20 the question.
21 A More or less, it depends on the particular aspects
22 you're looking at, and so making fewer assumptions is
23 generally better, but you also want to validate the
24 importance of those assumptions.
25 Q Okay. Is it necessarily the case that a model that
100
1 included more of the cells in the table would include more
2 variables, is that necessarily the case?
3 A Oh, no. In fact, typically you would make -- if you
4 connect -- the more variables in the technical regression,
5 logistic regression sense, I mean, the number of variables
6 included in the model would probably be fewer in those -- in
7 that sense, right. I mean, but including variables in a
8 model is not the same as making a set of assumptions.
9 Q But both the assumptions that you make and the number
10 of variables that you include contribute to the reliability
11 of the model; isn't that right?
12 A That sounds like it's got to be true, yes.
13 MR. DELERY: Thank you, Your Honor. I have no
14 further questions.
15 THE COURT: Intervenors?
16 MS. MASSIE: Can we take a very quick break?
17 THE COURT: Yes, I am going to take a break. I
18 wanted to know if you had any questions.
19 MS. MASSIE: Oh, yes.
20 (Recess taken at 10:47 a.m.)
21 -- --- --
22
23
24
25
|
|