In its continuing efforts to keep the public informed about the ongoing admissions litigation, the University of Michigan makes these transcripts of the trial proceedings in Grutter v Bollinger, et al., Civil Action No. 97-75928 (E.D. Mich.), available to the University community and general public. As is often the case with transcription, some words or phrases may be misspelled or simply incorrect. The University makes no representation as to the accuracy of the transcripts.
1
1 UNITED STATES OF AMERICA
2 IN THE UNITED STATES DISTRICT COURT
3 FOR THE EASTERN DISTRICT OF MICHIGAN
4 SOUTHERN DIVISION
5 BARBARA GRUTTER,
6 for herself and all others
7 similarly situated,
8 Plaintiff,
9 -vs- Case Number:
10 97-CV-75928
11 LEE BOLLINGER, JEFFREY LEHMAN,
12 DENNIS SHIELDS, and REGENTS OF
13 THE UNIVERSITY OF MICHIGAN,
14 Defendants.
15 -and-
16 KIMBERLY JAMES, et. al.,
17 Intervening Defendants.
18 ______________________________________/ VOLUME II
19 BENCH TRIAL
BEFORE THE HONORABLE BERNARD A. FRIEDMAN
20 United States District Judge
238 U.S. Courthouse & Federal Building
21 231 Lafayette Boulevard West
Detroit, Michigan 48226
22 Wednesday, January 17, 2001
23 APPEARANCES:
24 FOR PLAINTIFF: Kirk O. Kolbo, Esq.
25 R. Lawrence Purdy, Esq.
2
1 APPEARANCES (CONTINUING)
2 FOR DEFENDANTS: John Payton, Esq.
3 Craig Goldblatt, Esq.
4 Stuart Delery, Esq.
5 On behalf of the Defendants
6 Bollinger, et. al.
7
8 George B. Washington, Esq..
9 Miranda K.S. Massie, Esq.
10 On behalf of Intervening Defendants.
11
12 COURT REPORTER: MARY F. WISNESKI, CSR-0231
13 Official Court Reporter
14
15
16 Proceedings recorded by mechanical stenography.
17 Transcript produced by computer-assisted
18 transcription
19
20
21
22
23
24
25
3
1 I N D E X
2 WITNESS PAGE
3 KINLEY LARNTZ, Ph. D.
4 Direct Examination by Mr. Kolbo 7
5 Cross Examination by Mr. Delery 116
6
7
8 E X H I B I T S
9
10 NUMBER IDENTIFICATION ADMITTED
11 137 Dr. Larntz Report 28
12 142 Fifth Supplemental Expert Report 30
13 68 Dr. Larntz Report 30
14 16 1995 Final Grid 93
15 143 Policy 115
16
17
18
19
20
21
22
23
24
25
4
1/17/01 - BENCH TRIAL - VOLUME II
1 Detroit, Michigan
2 January 17, 2001
3 * * *
4 THE COURT: Ms. Massie called. Is she here yet?
5 I was going to say we'll wait for you.
6 MS. MASSIE: Actually we got to the source of the
7 problem just after.
8 THE COURT: Murphy's Law. That always happens to
9 me. I'm in the middle of a jam of the expressway and as
10 soon as I hang up, it's gone and I'm here.
11 MS. MASSIE: Wait a minute, what you are doing.
12 I'll have to try it next time.
13 THE COURT: Everybody here? Let the record
14 reflect, looks like she's here.
15 MR. PAYTON: Your Honor, I just wanted to say
16 something quickly about the --
17 THE COURT: Sure.
18 MR. PAYTON: Chart that I used yesterday that
19 showed the plot of all of the admitted students in 1997,
20 minority and majority.
21 THE COURT: 182, 183?
22 MR. PAYTON: That's correct.
23 THE COURT: Yes.
24 MR. PAYTON: There were questions about whether or
25 not there are LSAT scores at the range of, say, twenty.
5
1/17/01 - BENCH TRIAL - VOLUME II
1 Let me explain just briefly what we did. Although there
2 aren't LSAT scores of twenty or forty, and most of them are
3 120 or 180, in fact, Law Services does report LSAT scores
4 of zero for everyone who takes a non-standard form of the
5 tests; that is persons who are given it by hand because
6 they are disabled. And that shows up as a data point of
7 zero. And that's in the database. So in the database,
8 there are LSAT scores of zero. And they show on our chart,
9 and if you look at the line, it will be scores on the very
10 bottom line of zero, because there's reported like that.
11 So we try to present a chart that could account for all of
12 the data, including the zeros.
13 THE COURT: Okay.
14 MR. PAYTON: Okay. We also wanted to talk about,
15 and I've discussed that with all the parties, what I just
16 said. We also wanted to question something about how we
17 see the rest of our trial days for the next several days.
18 THE COURT: Great.
19 MR. KOLBO: Just on the last point, Your Honor,
20 I'm going to reserve any comments we have on that graphic.
21 And my understanding is Mr. Payton may come here with
22 another graphic as well. And I just wanted to say that for
23 the Court.
24 THE COURT: Okay.
25 MR. PAYTON: Here's, I think, a rough estimate of
6
1/17/01 - BENCH TRIAL - VOLUME II
1 where we're going to go. We believe that we will spend
2 most of today with Mr. Larntz. We believe that tomorrow we
3 will put on President Bollinger, Professor Lempert. We
4 believe that on Friday we will have our Professor
5 Raudenbush and Mr. Shields. And we believe all the rest of
6 our witnesses will probably be done on Monday. We may push
7 over a little bit, but I'm just saying that.
8 THE COURT: Great. No, I appreciate that. And
9 that also gives the Interveners some advance notice to be
10 able to start lining up their witnesses.
11 MR. PAYTON: That's correct.
12 THE COURT: I'll tell you, and I'll say it
13 probably a hundred times. There's nothing like having good
14 lawyers in the case, and that are civil to each other.
15 It's just such a nice way to preside over a case, I'm sure
16 such a nice way for each of you to practice.
17 Generally we have to fight for those kind of
18 things and here you are all agreeing and giving the
19 courtesy to each other to line up witnesses. And as I say,
20 I'll say it a hundred times because that's not enough.
21 It's really a nice way to do it. Okay.
22 MR. KOLBO: Your Honor, Plaintiffs call as our
23 next witness, Dr. Kinley Larntz.
24 THE COURT: Very well.
25 MR. PAYTON: Your Honor, I would also like to
7
1/17/01 - BENCH TRIAL - VOLUME II
1 introduce my partner, Mr. Delery, who will be conducting
2 the examination.
3 THE COURT: Great. Let me have your name one more
4 time.
5 MR. DELERY: Stewart Delery, D-e-l-e-r-y.
6 THE COURT: Thank you.
7 MR. DELERY: You're welcome.
8 THE COURT: Okay.
9 MR. KOLBO: Thank you, Your Honor.
10 K I L N E Y L A R N T Z, Ph. D.
11 was called as a witness and after having been
12 sworn was examined and testified as follows:
13 DIRECT EXAMINATION
14 BY MR. KOLBO:
15 Q. Good morning, Dr. Larntz?
16 A. Good morning.
17 Q. Could you state your full name, please?
18 A. Kinley Larntz. I'll spell it. It's K-i-n-l-e-y
19 Larntz. Last name is spelled, L-a-r-n-t-z.
20 Q. And what is your -- where are you located? Where are
21 you from?
22 A. I'm a statistician. I reside in Minnesota.
23 Q. And are you currently employed?
24 A. I'm currently self-employed.
25 Q. Okay. And how are you -- in the area of statistics?
8
1/17/01 - BENCH TRIAL - VOLUME II
1 A. As a statistician, yes.
2 Q. Okay. Prior to being self-employed, were you employed
3 by others?
4 A. Yes. I was twenty-seven years a faculty member in
5 statistics at the University of Minnesota.
6 Q. Can you tell me, just briefly run through the history
7 of your career at the University of Minnesota, what
8 positions you held there, maybe starting from the beginning
9 and bringing yourself to date.
10 A. Yes. I started at the university in 1971 as a, as we
11 would say, a lowly Assistant Professor, beginning Assistant
12 Professor. I continued, was promoted to Associate
13 Professor in 1977 and promoted to full professor in 1982.
14 And that's the position I held until I retired from the
15 university in 1998. Actually I still maintain a title.
16 It's nice of them to do that. I'm now referred to as
17 Professor Emeritus.
18 Q. Could you describe just briefly your formal
19 educational background?
20 A. Yes. Starting with college, I was an undergraduate
21 math major at Dartmouth, graduated in 1967, and continued
22 my studies as a graduate student at the University of
23 Chicago and did my Ph.D. in 1971 in statistics.
24 Q. Do you have any professional association or
25 memberships that you belong to?
9
1/17/01 - BENCH TRIAL - VOLUME II
1 A. Well, the two I maintain now are, the American
2 Statistical Association is the primary society for
3 statisticians, professional association for statisticians
4 in the country and the American Society of Quality, which
5 is, serves the same function for people interested in
6 issues of quality control.
7 Q. And have you been involved in any publications in your
8 field of expertise?
9 A. I have published. I have to say, I wouldn't have been
10 an Associate Professor or Full Professor if I had not
11 published, that's for sure.
12 Q. What's the principal journal in your area of expertise
13 in statistics?
14 A. Well, there are several journals. I guess Journal of
15 the American Statistical Association is a major journal.
16 I've certainly published there. Journals, I'm just trying
17 to think. There's a whole series been published by
18 different societies.
19 Q. Have you been involved in any editorial positions with
20 any of these statistical journals?
21 A. Yes. Sure. I served as associate editor for the
22 Journal of the American Statistical Association on several
23 occasions and I also served as editor of another journal of
24 the association called the American Statistician.
25 Q. In addition to your academic post at the University of
10
1/17/01 - BENCH TRIAL - VOLUME II
1 Minnesota, have you, in the course of your career, engaged
2 in outside consulting work in the area of statistics?
3 A. Yes.
4 Q. Can you just give us a description of what kind of
5 consulting work you've done over the years?
6 A. Well, consulting work I've done, I have to say, as
7 part of my duties at the university. I was also a
8 consultant within the university, so I worked quite
9 extensively as part of my job collaborating with other
10 researchers on research projects. And I did do some, and I
11 do a bit more now, consult for companies and government,
12 agencies.
13 Q. Can you give us some examples of the government
14 consulting that you've done over the years?
15 A. Okay, sure. I've been, worked with National Science
16 Foundation, National Institutes of Health and extensively
17 with an agency called, the National Institute of Justice,
18 overseeing a series of experiments that were done in police
19 departments on responses to domestic violence. I've worked
20 as a consultant, actually a member of a Scientific Advisory
21 Board to the Environmental Protection Agency concerning
22 small particulates. And, as a say, on the Scientific
23 Advisory Commitment that made recommendations concerning
24 that.
25 My current activity, with respect to government
11
1/17/01 - BENCH TRIAL - VOLUME II
1 consulting is I'm a consultant, statistical advisor, to the
2 Food & Drug Administration, primarily in the area of
3 orthopedic devices. And so I sit on the panel, typically
4 sit on the panel that advises the FDA on whether the new
5 medical device are to be approved or not.
6 Q. Can you give us, your -- I believe you indicated that
7 you, your post at the University of Minnesota has been in
8 the area of applied statistics?
9 A. Well, universities are organized in different ways, as
10 I'm sure everyone knows. And in Minnesota, we had two
11 departments in statistics, or actually several statistic's
12 departments. But one was called Applied Statistics and
13 that was where my appointment was.
14 And I explained that I did internal consulting
15 work within the university, and that was a responsibility
16 of each person in that department to carry on that
17 collaborative research.
18 And applied statistics as it was defined there,
19 and as I generally define it, is the area of using
20 statistics in subject matter fields. So making sure that
21 when you look at data or gather data, design studies to
22 gather data, that they are done statistically
23 appropriately.
24 Q. Can you just give us an idea of what kind of fields
25 one would use applied statistics?
12
1/17/01 - BENCH TRIAL - VOLUME II
1 A. I mean, statistics is useful everywhere.
2 Q. Yes.
3 A. You know that. And I believe that, sir.
4 THE COURT: That's why I went to law school, so I
5 didn't have to take statistics.
6 A. You had no statistics in law school?
7 THE COURT: None. Otherwise I would have probably
8 been in podiatry school or something.
9 A. I've had no law, of courses, either.
10 Q. Let me ask you, Dr. Larntz, for example, in the area
11 of medicine, medical devices, medical cures, is applied
12 statistics used in that area at all?
13 A. Certainly. I use that in that area. I was going to
14 say before that I, I basically am probably too broad in the
15 sense that I've used statistics in lots of areas across the
16 board, starting from the -- well, let's see, I've worked on
17 the, determining the wealth of the United States in 1775.
18 That's a long time ago, and I worked on that.
19 I worked on the composition of fishery by catch by
20 Japanese fishing boats in the Pacific to see what they take
21 into their nets, to see how many tuna they get and how many
22 other things arrive.
23 I've worked, as you say, in medical devices. I do
24 quite a bit of work in that area now. I've worked in
25 engineering. And I've actually written software that's
13
1/17/01 - BENCH TRIAL - VOLUME II
1 used for designing experiments by engineers.
2 Q. Have you done any consulting work in the past, in
3 connection with discrimination cases involving, say,
4 employment or other areas?
5 A. Yes, I have.
6 Q. Were you asked in this case to take a look at certain
7 issues regarding the use of race as a factor in the
8 admissions process?
9 THE COURT: Can I just ask one question? You said
10 that at the university when you applied the statistics
11 department that you were required to do some internal
12 statistics for the university.
13 A. Yes.
14 THE COURT: What kind of statistics did you do for
15 the university?
16 A. Oh, internal. What I did was I worked with
17 researchers in various areas.
18 THE COURT: I see.
19 A. So someone, say, for instance, who was working -- in
20 fact, a large part of my work was with people in medicine.
21 Someone who was getting a grant or wanted to get a grant,
22 study the effect of a drug on HIV, which I worked on for
23 about seven years.
24 THE COURT: But not academic statistics at the
25 university as part of your --
14
1/17/01 - BENCH TRIAL - VOLUME II
1 A. -- Academic statistics in the sense of?
2 THE COURT: I don't know, whatever they may be?
3 A. You mean in the sense of administrative statistics?
4 THE COURT: Yes. Yes.
5 A. That was not what I worked on, per se at the
6 university, no.
7 THE COURT: Okay.
8 A. I was not part of the administration. I was part of
9 a, I'll call it the Research Academic Corps.
10 THE COURT: Okay. Thank you. I'm sorry.
11 Q. Were you asked in this case to take a look at, from a
12 statistical point of view whether, and to the extent to
13 which the University of Michigan Law School takes race into
14 account in the admissions process?
15 A. I was asked to look at the question of examining data
16 for, from the University of Michigan Admissions Office, I
17 presume that's where the data arrived, concerning the role
18 that ethnicity played with respect to admission's
19 decisions, yes.
20 Q. When did you become involved in looking at those
21 issues?
22 A. I believe it was the fall of 1998.
23 Q. And what were you asked to do, specifically?
24 A. Well, I was asked to, if I would be willing to look at
25 the data and see what the data would say concerning the
15
1/17/01 - BENCH TRIAL - VOLUME II
1 role of ethnicity and admissions.
2 Q. Okay.
3 A. And I wanted to, and I took on that job.
4 Q. What information did you consider in, in doing your
5 analysis?
6 A. Well, when I first started, I certainly looked at some
7 materials that were prepared by the law school concerning
8 their admissions policy. The first thing I asked for, you
9 know, being a statistician, the first thing you ask for is,
10 you know, you want to find out what kind of data are
11 available. That's what we, that's our bread and butter.
12 And so I, I would understand that there would be a
13 large number of applicants. So I wanted to try and make
14 sure I got data in a computerized form. And I eventually
15 got computerized data bases of material concerning law
16 school admissions.
17 Q. Okay. Did you look at any documents with respect to
18 the law school admissions?
19 A. I certainly looked at some documents. I think I
20 looked at some, there's certainly a version of an
21 admission's policy. There was, I think a visitor's report
22 that discussed the law school and some written materials,
23 some tabulations presented by the law school.
24 Q. I think you may have up there with you a free-standing
25 copy of Exhibit 4. Do you see that? Is there such a
16
1/17/01 - BENCH TRIAL - VOLUME II
1 document up there?
2 A. Yes.
3 Q. Can you just take a look at that and tell me if that's
4 one of the documents you took a look at as part of
5 undertaking your analysis?
6 A. Yes. I've looked at this document before.
7 Q. Okay. And was that represented to you to be the law
8 school admission's policy that was in effect for the years
9 that you were going to take a look at the data?
10 A. Well, it says it's admissions policies. That's what
11 it says on here. It's dated 4/22/92, and that predates the
12 database dates. I don't, I don't know that there's another
13 admission's policy after that.
14 Q. That's one of the documents you looked at?
15 A. Yes.
16 Q. In addressing the issues you looked at?
17 A. Yes.
18 Q. How many years of -- you mentioned, I think that you
19 obtained in electronic form the data from the law school,
20 certain data from the law school?
21 A. Yes. Over a period of time, I received large
22 databases, yes.
23 Q. And how many, how many years of law school data did
24 you look at?
25 A. I was given datas, initially from 19, they covered the
17
1/17/01 - BENCH TRIAL - VOLUME II
1 admission years of 1995 through '98. And then subsequently
2 received 1999 admission's data and the year 2000
3 admission's data.
4 Q. Okay. And in what form, you mentioned that it was
5 electronic. Can you just describe the format in which the
6 data was supplied to you?
7 A. It was a Microsoft access database.
8 Q. Okay. And what kinds of data were included in the
9 database? What kinds of information, generally speaking,
10 was contained in the database?
11 A. Well, there were a large number of tables in that
12 database. And they contained information, for instance, on
13 ethnicity, which given the subject of this case, that would
14 be important to have. Information on credentials, grade
15 point average, admissions test score. And there was an
16 index, maybe, I call it a selection index, information in
17 coded form on what school people went to as undergraduates,
18 a variety of information of that sort.
19 Q. Okay. And was the data provided to you in
20 substantially the same form for each of the years that you
21 were provided data?
22 A. Essentially. I mean, there's probably some small
23 changes in coding, but, essentially, the same form, for
24 which I admit I was grateful, so I didn't have to change
25 everything each year.
18
1/17/01 - BENCH TRIAL - VOLUME II
1 Q. Okay. Can you describe just generally, in a very
2 general fashion to begin with, what types of statistical
3 analysis you performed with respect to the data that you
4 received, just very generally.
5 A. In general the terms I -- I used some descriptive
6 statistics to look at the characteristics of the
7 individuals in the database, the applicants, and, and then
8 I also then used, I guess, in general terms, I used
9 statistical methods to examine the odds or chance of
10 admission as a function of credentials of the applicants.
11 And I guess in statistics when we get fancy, we'll call
12 that inferential statistics, but first we call it
13 descriptive statistics.
14 Q. Okay. We'll get into the details then of your
15 conclusions. We'll go through a number of items. But did
16 you form some overall conclusions? You stated some general
17 fashion of some overall conclusions that you came to with
18 respect to your analysis of these databases?
19 A. Yes.
20 Q. And were those conclusions formed to a reasonable
21 degree of certainty in the field in which you practice?
22 A. Oh, I feel quite comfortable with my conclusions, yes.
23 Q. Okay. Can you just generally describe, in general
24 fashion, what conclusions you, you drew from the data that
25 you reviewed?
19
1/17/01 - BENCH TRIAL - VOLUME II
1 A. From the data I reviewed, which is the description, in
2 trying to describe as best I could, as it was done, the
3 admissions that were done by the University of Michigan Law
4 School. And with respect to ethnicity, I found, this is
5 not a statistically technical term, but I found an
6 incredibly large allowance given to members of selected
7 minority groups with respect to their chance of admission.
8 And that all of my analyses that I did confirmed and
9 continued to confirm that for individuals that have similar
10 credentials.
11 Q. And was that true for all the years you looked at, or
12 just some of the years or?
13 A. The basic substantive conclusion. The basic
14 substantive conclusion. Of course the numbers are going to
15 vary from year to year. And if they didn't, I wouldn't be
16 in business, I guess, as a statistician. But the numbers
17 varied a fair bit. But the actual substantive conclusions
18 were exactly the same across over the years, yes.
19 Q. Now, you mentioned before that you did different types
20 of analysis. And I just want to spend a little time on
21 some detail there. You mentioned descriptive statistics.
22 What does that refer to, descriptive statistics?
23 A. Well, it describes the data, how's that? What it does
24 is try to give some summary numbers, and in a couple cases
25 I looked at summary pictures of the data to try to
20
1/17/01 - BENCH TRIAL - VOLUME II
1 understand the characteristics of the applicants, and in
2 particular, the characteristics of applicants who were
3 admitted to the law school.
4 Q. And then you've mentioned that you also employed
5 inferential statistics in your analysis?
6 A. Well, I employed statistics that would allow us to, at
7 least, when I say draw conclusions, as to whether or not
8 the, the things we were seeing are just due to chance or
9 not. And that's what I mean by inferential statistics.
10 And I certainly, the main technique I used in that
11 was a technique called logistic regression, which is a
12 standard technique that we use for looking at a binary
13 response, binary in the sense of admit or not admit, to the
14 law school, and analyzing what, what would effect that,
15 that particular response and particular, the relation of
16 that to grades, undergraduate grades, admissions test
17 scores and other factors.
18 Q. Is an examination of relative odds or odds ratios, is
19 that a form of inferential statistical analysis?
20 A. Well, it is, and logistic regression specifically
21 looks at trying to understand odds as the response, the
22 odds of, say, odds in this case, odds of admission, yes.
23 Q. So relative odds analysis is related to an analysis
24 using a logistic regression?
25 A. Logistic regression, actually the technique itself
21
1/17/01 - BENCH TRIAL - VOLUME II
1 analyzes its response, technically, the log rhythm of the
2 odds. And then when you're doing comparisons between one
3 odds and another odds, and you'll get to that, I presume.
4 We would calculate what I call relative odds or odds
5 ratios.
6 My students always accuse us in statistics of
7 having one concept in fourteen names. It makes taking
8 exams difficult. And odds, and relativity odds and odds
9 ratios are the same, the same thing.
10 Q. Can you just then define for us what in the area of
11 statistics what odds and relative odds mean?
12 A. I think, yeah, it's probably a good idea, because odds
13 is used a lot in, in, well, in our everyday life, so I can
14 do that. Is it possible I can write on the board a little
15 bit?
16 MR. KOLBO: With the Court's permission.
17 THE COURT: I have no problem. The only problem
18 it's going to kind of screw up your Elmo.
19 MR. KOLBO: I think we're actually going to use
20 this over.
21 (Whereupon an off-the-record
22 discussion was had.)
23 Q. Do you have a pen there, Dr. Larntz?
24 A. Okay. Well, what I want to do is just define for you
25 odds. And I know -- I don't presume that you know a lot
22
1/17/01 - BENCH TRIAL - VOLUME II
1 about odds. And I'll just, I'll just.
2 THE COURT: Just in Vegas.
3 A. Well then, I particularly, then I particularly need to
4 define what I mean. And in statistics we talk of the odds
5 in favor of an event.
6 And just to be clear, most odds, as reported in
7 Vegas, and I have to say the odds aren't always, you know,
8 always, they're always put in the form of odds against,
9 typically. And so, and I'll leave my comments out about
10 the Vikings.
11 But we think of the odds as a number, so in
12 statistics, we think of odds as a number. So, for
13 instance, if we have what we think of as fifty, fifty odds,
14 50/50, odds of 50/50, that means that's like a coin flip.
15 THE COURT: If you guys want to sit in the jury
16 box, you're more than welcomed to, or you may stand.
17 Whatever makes you happier.
18 A. I don't think we'll be here forever. And what we do
19 in statistics is we take the ratio of these. So fifty
20 divided by fifty. That is the chance, the chance against,
21 and we divide these. And we call, we make that a number,
22 okay. And so that would be, in our terms, an odds of one.
23 Okay. So what do we do for other chances, or probabilities
24 of admissions? Suppose there's a 25 percent chance; how
25 would we convert that to odds, just so we're clear.
23
1/17/01 - BENCH TRIAL - VOLUME II
1 So that would be what, twenty-five for to
2 seventy-five against. We do the same thing, make it a
3 ratio and then we get twenty-five over seventy-five, and
4 that's a number. The number is one third.
5 Similarly, if we did 75/25, again, whoops, 75/25,
6 that would be a number. And we get the number,
7 seventy-five divided by twenty-five, we get three. So
8 those are, those are how we calculate odds. We can do the
9 same thing for others. And I'll probably take another
10 example in a second.
11 Now, when we're comparing odds, the odds of two
12 events, and that's what we're going to do here a fair bit
13 is compare odds. We take what's call the relative odds or
14 the odds ratio. And the odds ratio then for, say, the
15 event that has probability 75 percent, to the event that
16 has probability of 25 percent, the odds ratio is just the
17 ratio of the odds. And so the odds ratio then would be
18 three divided by one third, and -- and we've divided by
19 fractions at one time in our lifes, yes.
20 THE COURT: I did.
21 A. Okay. And we come up with nine. That's the odds
22 ratio. Take one more example.
23 Suppose the, so an odds ratio of nine corresponds
24 to comparing seventy-five to twenty-five. Another one,
25 say, ten to ninety would give us, what, I think I can skip
24
1/17/01 - BENCH TRIAL - VOLUME II
1 this and just write the odds as one ninth, ten to ninety,
2 say, ninety to ten. That gives is an odds of nine. And if
3 you're comparing an event that has probably 90 percent to
4 an event with the probability of ten percent, we
5 calculation an odds radio, odds ratio, then of nine divided
6 by one ninth, which would be equal to eighty-one. So the
7 concept of odds, essential to comparison of two events,
8 probabilities are odds ratio.
9 Q. Doctor, how would one communicate that in sort of a
10 sentence if one wanted to express the relative odds of a
11 ten percent probability of something occurring, versus a 90
12 percent probability of something occurring?
13 A. In statistical terms we'd say that the odds ratio;
14 that is, the odds of, the odds ratio of the event, whatever
15 we call it, ninety, that has 90 percent probability to the
16 percent that has ten percent probability has an odds ratio
17 of eighty-one.
18 Q. Very good. Do you want to resume, resume the witness
19 stand. And let me ask you, in the area of statistics, does
20 it happen sometimes that matters simply occur by chance?
21 A. Does it happen?
22 Q. Yes.
23 A. In statistics, I think in the real world it happens
24 that things occur by chance.
25 Q. Okay. And in your profession, are there ways in which
25
1/17/01 - BENCH TRIAL - VOLUME II
1 one can summarize in which statisticians summarize the
2 degree to which an observed difference, say, a ten percent
3 probability versus a 90 percent probability might simply be
4 due to chance, as opposed to something else?
5 A. Yes. We often summarize, if we're doing statistical
6 comparison, we want to understand whether or not the events
7 mean, anything can happen. So do the events that we see,
8 are they due to chance alone or not. And we summarize that
9 several ways, quite commonly use what's called a P value to
10 summarize the degree of evidence concerning whether or not
11 something occurred by chance.
12 Q. Is the term, standard deviation, used as well in that
13 context?
14 A. Lots of ways of -- remember in statistics there's
15 often several ways to do the same thing. And so what we do
16 is we have the P value, may be a summary, or we may
17 summarize also in terms of standard deviation. So, for
18 instance, and are we going to use the board a lot? I'm not
19 sure whether these people need to stay up there or not?
20 It's up to you, Your Honor.
21 THE COURT: I think they like it there. They can
22 move whenever they want.
23 A. Okay. I'm going to take a drink of water because I
24 was counting on that when they moved.
25 THE COURT: Help yourself.
26
1/17/01 - BENCH TRIAL - VOLUME II
1 Q. Could you just summarize what standard deviation
2 analysis tells a statistician, just kind of summarize just
3 what kind of values you look at for statistical
4 significance?
5 A. What I think I better do is talk about P value and
6 standard deviation both, and we summarize the, we
7 calculate, if we're describing an outcome, whatever that
8 outcome is. We calculate the probability that we see that
9 outcome or a more significant outcome than that, if chance
10 alone were operating.
11 And in statistics we often talk of P values. We
12 have magic numbers -- I shouldn't say that too loud, there
13 might be other statisticians listening. But magic numbers
14 of .05, five percent.
15 So if an event has a five percent or less chance
16 of occurring under chance, we usually call that
17 statistically significant. That's a term that arises. We
18 get, people like us to summarize succinctly, so if it has
19 something, an event has a one percent chance or less, we'd
20 often call that highly statistically significant. And I
21 think we've stopped at that particular point.
22 Now, statistics that we look at we can often
23 summarize the departure from chance in terms of numbers of
24 standard deviations. And this is also done quite often
25 with lots of summary statistics. And so many of the tests
27
1/17/01 - BENCH TRIAL - VOLUME II
1 we do are summarized in terms of a statistic Z score, which
2 gives the number of standard deviations that the outcome is
3 away from, the chance alone outcome.
4 A two standard -- well, I should say, I'll be
5 precise, 1.96 standard deviation corresponds to a five
6 percent P value, so 1.96, we get a little sloppy. We say
7 two, okay, in statistics. And so we'll say about two
8 standard deviations corresponds to an event that's
9 statistically significant.
10 And if we summarize in terms of standard
11 deviations, a one percent event is, see if I remember the
12 number, I once forgot this number in court so I better
13 remember it today. The one percent value is 2.576 and so
14 2.57, 2.58 would correspond to a significance of one
15 percent. So, in general, in statistical terms, events that
16 have, now I'll say it in more summary forms, events that
17 correspond to departures that correspond to two or three
18 standard deviations are generally considered statistically
19 important. Events that have, that further departure are
20 obviously much more statistically significant than, than
21 that.
22 Q. For example, what would a standard deviation of five
23 signify?
24 A. Five?
25 Q. Yes.
28
1/17/01 - BENCH TRIAL - VOLUME II
1 A. Well, five is, and this is actually literally true for
2 most normal tables, it's off the chart in the sense that
3 its, we believe events that would correspond to a five
4 standard deviations have very, very minuscule chance of
5 occurring by chance alone. Calculation -- I don't have
6 that calculation with me, probably less than one in a
7 million is the probability associated with five or more.
8 Q. Now, I think you've indicated that you wrote some
9 reports in connection with the work that you did in this
10 case?
11 A. One of the things that this witness did was write
12 reports, that's true.
13 Q. Can you take a look, or Dwayne, can you show the
14 witness the books that have Exhibit 137 through, I think
15 it's 142. When you get to 137, can you let me know, Dr.
16 Larntz?
17 A. Okay. I am at 137.
18 Q. Yes. Is that a copy of your report?
19 A. It appears to be. It's dated December 14, 1998,
20 expert report of Kinley Larntz.
21 MR. KOLBO: I'd offer Exhibit 137 at this time,
22 Your Honor.
23 THE COURT: Any objection?
24 MR. DELERY: No objection, Your Honor.
25 THE COURT: Received.
29
1/17/01 - BENCH TRIAL - VOLUME II
1 Q. Will you take a look at 138 to 142 and just tell me,
2 we can speed things up a little bit if you can confirm to
3 me that those are copies of all your remaining reports
4 generated in this case?
5 A. One thirty eight, 139, 140, 141, yes. Those are the
6 additional reports that I entered in.
7 Q. Okay. And then if you could take a look at exhibit,
8 and this will be in a separate book, I think, at this
9 point, Exhibit 68.
10 A. Yes, I see Exhibit 68.
11 Q. Is that data that you assembled from the database that
12 you reviewed in connection with this case?
13 A. Yes. These are spreadsheets that I, that summarize
14 the data that was in the databases for 1995 through '98
15 concerning law school admissions, yes.
16 MR. KOLBO: Maybe to get things in order here?
17 Your Honor, I will at this time offer Dr. Larntz as an
18 expert witness in this indication. I have to do that?
19 THE COURT: Any voir dire? Anybody have any
20 objection to him testifying?
21 MR. DELERY: No, Your Honor.
22 MS. MASSIE: No.
23 THE COURT: Very well.
24 MR. KOLBO: I don't think I've then offered then
25 exhibits -- it's just been pointed out to me, I'm not sure
30
1/17/01 - BENCH TRIAL - VOLUME II
1 that it makes any difference for our purposes, but 137 also
2 contains Exhibit 1 -- or 137 also contains Exhibit 68, so
3 there's some duplication there.
4 A. Yes. Yes. I actually saw that as I went through.
5 MR. KOLBO: Okay. At this time, then, I would
6 offer the remaining of Dr. Larntz' reports, Exhibits 138
7 through 142, as well as, I guess, Exhibit 68, as long as
8 it's already marked.
9 THE COURT: Any objection?
10 MR. DELERY: No, Your Honor.
11 MS. MASSIE: No Your Honor.
12 THE COURT: Received.
13 Q. Dr. Larntz, in addition to the written reports we have
14 here, have you assembled something in the nature of a video
15 presentation that will help explain your conclusions and
16 analysis that you performed?
17 A. We prepared, picked out some slides which are, for the
18 most part, with a couple of exceptions, copies of the
19 tables and figures from the reports, yes.
20 Q. Let's take a look at the first slide, if we can. Can
21 you tell us, first of all what table one represents? Let
22 me back up a little bit. We're going to go through a
23 number of these slides. Can you tell us, for the most
24 part, where these slides came from? Are they derived from
25 some other records?
31
1/17/01 - BENCH TRIAL - VOLUME II
1 A. This is table one from the first report that I did, so
2 this is from the first, December, 1998 report, table one.
3 Q. For the most part, these, these tables, although maybe
4 in slightly different form, are, in terms of the graphics,
5 the way they're graphically displayed, these are contained
6 in your various reports?
7 A. Exactly, yes.
8 Q. As we go through these?
9 A. Exactly.
10 MS. MASSIE: Mr. Kolbo, can I stop you there for a
11 second? This was not dropped off at our office. And I
12 haven't objected until now because I understood that it was
13 exactly the same as the materials contained in Dr. Larntz'
14 report. If it's not, I'd like a copy so that I can look at
15 what's different.
16 MR. KOLBO: Your Honor, this is an exhibit that
17 was actually delivered with the witness books. It's
18 exhibit what, Dwayne?
19 MS. MASSIE: 143.
20 MR. KOLBO: 143.
21 MS. MASSIE: It was not in the materials that we
22 received. If I can just a have a few minutes to look at
23 it? I assume it's substantially the same?
24 MR. KOLBO: It should have been. I apologize if
25 it wasn't.
32
1/17/01 - BENCH TRIAL - VOLUME II
1 THE COURT: Okay. It's 143 in the book.
2 MR. KOLBO: One forty three in the book.
3 THE COURT: Take your time. Take a look at it.
4 MR. KOLBO: You have it there.
5 MS. MASSIE: We do have it. I apologize.
6 MR. KOLBO: We have an extra copy, Miss Massie, if
7 you'd like.
8 MS. MASSIE: No. You can go ahead. I apologize.
9 Q. Can you tell us, Dr. Larntz, what, what table one
10 summarizes or represents?
11 A. Yes. This first line, which is table one from the
12 first report that I did summarizes by ethnic group
13 classified in exactly the same manner as the variable
14 ethnicity was classified in the law school database.
15 It summarizes by ethnic group the number of
16 applicants for law school for the years 1995 through 1998.
17 So we can see there are about, well, you can see exactly if
18 we read, 4,147 applicants in 1995, and similarly 3,500,
19 about 3,537 applicants in 1998. And by each ethnic group,
20 we can count the number of applicants, so.
21 Q. Dr. Larntz, do you have a pointer there if you want to
22 use one? I'm not suggesting that you have to, but if it's
23 going to be helpful to you to make specific points?
24 A. I do have one, yes.
25 Q. Okay. Can you tell me what, what, if any, general
33
1/17/01 - BENCH TRIAL - VOLUME II
1 conclusions you drew from just table one here?
2 A. Well, these are the numbers of applicants, the raw
3 data. The conclusions are, and by the way, the ordering is
4 exactly in the same order as the variable was in the law
5 school database. I just used that, so if someone asks
6 where did the order come from, that's the order that was
7 used in the code. It just gives the number of applicants.
8 So for instance in 1995 there were 4,147
9 applicants, 45 of the applicants were listed as Native
10 American by ethnicity. 404 African-American, 2,316
11 Caucasian American, 98 listed as Mexican American, 115 as
12 other Hispanic American, Asian Pacific Island American,
13 470, not too dissimilar from the number of
14 African-Americans applicants, Puerto Rican, twenty, small
15 number, foreign, classified as 412, and unknown, 567. So
16 the largest groups with caucasian American, Asian Pacific
17 Island, African-Americans, well, except for the unknowns.
18 Q. Okay. Let's go to the next slide.
19 A. That's similar for all the years, and I think we don't
20 have a prepared a slide for 1999 or 2000, but there were
21 about 3,400 applicants in each of those years and there's a
22 similar distribution.
23 Q. Let's take a look at the next slide.
24 A. Yeah.
25 Q. Can you summarize what this, what information this
34
1/17/01 - BENCH TRIAL - VOLUME II
1 slide provides.
2 A. This is part of the descriptive statistics that I was
3 preparing. And so what I've done here is for the applicant
4 group as a whole; that is. For all applicants, I've
5 summarized the median undergraduate grade point average.
6 So, for instance, in 1995 the overall median was
7 3.49 and the median for Native Americans was 3.14, somewhat
8 lower. These are among applicants, 3.03 for
9 African-Americans, 3.55 for Caucasian Americans, compared
10 to 3.49. 3.31 for Mexican American applicants. 3.56 for
11 other Hispanic American, Asian Pacific Island applicants
12 3.48, 3.14 for Puerto Rican applicants 3.46 for foreign
13 applicants, 3.53 for those unknown ethnicity. And so, and
14 you can see that we've given that, those numbers for 1995,
15 '96, '97, '98, and I think a couple more slides will give
16 '99 and 2000.
17 Q. Let's go to the next slide then.
18 A. And similarly, let's just make sure we see what we're
19 seeing. The median, the median, I guess given the way
20 we're doing things I better make sure I define what a
21 median is.
22 A median is the value in which half the applicants
23 are at or below that value and half are at that value or
24 above. So there are applicants that are higher, higher and
25 lower, but it's the one that splits.
35
1/17/01 - BENCH TRIAL - VOLUME II
1 So, in fact, in 1999 we have actually similar
2 patterns. What we see is with respect to the overall
3 median of 3.52, Caucasian American applicants are, are at
4 3.57, Asian Pacific Island, 3.46, and then Native
5 Americans, 3.37. Probably -- I'm not trying to do this
6 particularly in order, but I'll just point out
7 African-Americans are at 3.15, somewhat lower than, with
8 the median somewhat lower than the overall. Mexican
9 American applicants, 3.36 and Puerto Rican 3.20.
10 Q. Let's go to the next slide down. Is this just for the
11 same information for the next year?
12 A. Exactly. With the same pattern of median.
13 Q. All right. Next slide please, Dwayne. What does that
14 table three summarize there?
15 A. I did the same analysis, looking at LSAT score in
16 calculating the median. These are for applicants as they
17 presented to the, to the law school. And so the median,
18 the median overall for 1995 was 162. And we can see that
19 Native Americans applicants that year had a median that was
20 eight points lower at 154, African-Americans applicants had
21 a median that was 150, twelve points lower, Caucasian
22 Americans applicant had a median that was one point higher,
23 163. Mexican American applicants had a median LSAT of 155,
24 seven points lower than the overall median. Hispanic,
25 other Hispanic American applicants had a median of 156, six
36
1/17/01 - BENCH TRIAL - VOLUME II
1 points lower than the overall. Asian Pacific Island
2 Americans had 161, which was within one point of the
3 overall, Puerto Rican applicants, 155, seven points lower,
4 and you can see foreign and other unknowns also their
5 values down there.
6 Q. And without going into detail you've reported the same
7 information for '98, '96 and '98, correct?
8 A. In '96, 1997, '98 in this, showing, showing similar
9 patterns, yes.
10 Q. Let's go to the next slide down.
11 A. And this is the same, the same display for 1999.
12 Q. Okay.
13 A. And if we look at the next slide, we get the same
14 display for the year, the year 2000.
15 Q. What was the next step in your analysis then?
16 A. Well, well, I just summarized this. I mean, in some
17 sense, the summary of this, we saw, we see that the, a
18 number of ethnic groups had median scores that are lower
19 than the average, consistently across there, and that's
20 among, among the applicants. I just let that be a summary.
21 The next step was, and we're first looking at all
22 applicants, and this is, in some sense, who presented
23 themselves to the law school for admission. Those are the
24 applicants. The next step we looked at is what are the
25 characteristics of the individuals they selected; that is
37
1/17/01 - BENCH TRIAL - VOLUME II
1 the accepted applicants. So we looked at, I looked at,
2 prepared exactly identical tables for accepted applicants.
3 That's the next step.
4 Q. Should we go to the next slide then?
5 A. Sure.
6 Q. And what information is, is summarized just briefly,
7 what information is contained here?
8 A. Sure. This is, I'm sorry. This is the same display,
9 using the median, comparing the actual accepted applicant
10 GPAs. And we can see, for instance, in 1995 the, among the
11 accepted applicants, the average, or not the average,
12 excuse me, the median. The median is a form of average, so
13 I better -- but the median was 3.64, and the Native
14 Americans were at 3.36, somewhat lower, African-Americans
15 were 3.33, among accepted applicants, again, somewhat
16 lower. Caucasian Americans were 3.68, slightly higher.
17 Mexican American applicants were 3.50, and Asian Pacific
18 Island applicants were 3.6, again, within, about a
19 hundredth of a point of the overall and Puerto Rican
20 applicants were 3.3 point, somewhat lower.
21 And so what we see higher, and we see throughout
22 this is that among accepted applicants, Caucasian and
23 American and Asians and non-American applicants had median
24 undergraduate GPAs that were similar or slightly higher
25 than the overall median.
38
1/17/01 - BENCH TRIAL - VOLUME II
1 And applicants from ethnic groups of Native
2 American, African-American, Mexican American, and Puerto
3 Rican, ethnic groups had averages, had undergraduate,
4 median undergraduate GPAs that were somewhat lower.
5 Q. Did you do similar analysis for the later years and
6 have similar conclusions?
7 A. For 1999 and 2000 we did exactly the same thing, yes.
8 Q. Can we have that next slide?
9 A. And that's a report in 1999. And the next slide
10 reports 2000.
11 Q. What was -- let's go to the next slide, Dwayne.
12 A. And then we continued in the same summary manner that
13 we did before for LSAT scores. And LSAT scores for 1995,
14 the overall median was 168. In fact, the overall median
15 was 168 for all four years, for all accepted applicants for
16 '95, '96, '97 and '98.
17 Native American applicants, or accepted
18 applicants, Native American accepted applicants were six
19 points lower the first year, and seven points the second,
20 and seven the third, and eight the fourth year, lower than
21 the overall median.
22 African-Americans accepted applicants were nine
23 points lower the first year, nine points lower the second
24 year, eight points lower the third year and nine points
25 lower the, in 1998.
39
1/17/01 - BENCH TRIAL - VOLUME II
1 Caucasian American applicants were all, either at
2 or one point above their median scores were at or one point
3 above. Mexican American applicants, accepted applicants
4 were eight points lower in '95, five in '96, seven in '97,
5 and eight in '98.
6 Asian Pacific Island American applicants were at
7 or one point above the median for all the years given here,
8 and Puerto Rican applicants, accepted applicants were nine
9 points lower in '95, eight points lower in '96, four points
10 lower in '97 and seven points lower in '98.
11 Q. Let's go, did you do the same analysis then for the
12 later two years again?
13 A. Yes. We have 1999 and we have 2000.
14 Q. Slides twelve and thirteen are for 1999 and 2000?
15 A. That's correct.
16 Q. The same LSAT analysis for accepted applicants?
17 A. The same, reporting the median of accepted applicants,
18 yes.
19 Q. All right. Let's go to the next slide, Dwayne. Can
20 you tell us?
21 THE COURT: What's the difference between "mean"
22 and "median"?
23 A. The median is exactly the value at which half or at
24 that value or above and half or that value or lower.
25 That's the median. Splits the day in half.
40
1/17/01 - BENCH TRIAL - VOLUME II
1 THE COURT: Right.
2 A. We also in percentile terms, that's the fiftieth
3 percentile.
4 THE COURT: Okay.
5 A. The arithmetic mean, the average, if you want to call
6 it that, basically what it does is it sums up all the
7 values and divides by how many you have. Okay. So the
8 average, the arithmetic, we typically call the mean, is, is
9 precisely that calculation. It's just another way of
10 summarizing. We could have done the same thing with
11 averages.
12 THE COURT: How, if you use "mean", how would the
13 charts differ? Other than numbers, would the spread be any
14 different, or how would it look if you used "mean" instead
15 of "median"?
16 (Whereupon an off-the-record
17 discussion was had.)
18 A. Well, I don't recall the exact numbers, Your Honor. I
19 would -- my feeling is it would be similar in, in
20 difference, but I don't. I don't have the actual numbers.
21 THE COURT: I see. That is would probability, if
22 you chart it, it would be about the same?
23 A. In the case what would be different would be if there
24 are extreme values and extreme values would tend to pull
25 the mean up or down, depending on how extreme the value is.
41
1/17/01 - BENCH TRIAL - VOLUME II
1 May I give an example?
2 THE COURT: Sure.
3 A. Not in this area.
4 THE COURT: Yeah.
5 A. The example I used to use in class, I used to use an
6 example of, and I'm try to make it so I can do the math in
7 my head although I've got a conductor here if I have to.
8 THE COURT: Sure.
9 A. So my wife worked in a dental office, okay, and we
10 looked, I looked at the salaries. I'm making up this
11 example, by the way. She did work in the dental office but
12 I'm making up the numbers.
13 THE COURT: Yeah.
14 A. And the individuals working at that dental office, the
15 salaries of the individuals working in the dental office
16 were $20,000 a year, $20,000 a year and $260,00 a year.
17 There were three individuals working in this dental office.
18 Do you see what I'm saying?
19 THE COURT: Uhm-uhm.
20 A. Two at twenty, and two sixty, okay. The arithmetic
21 mean of that is a hundred thousand dollars a year. So
22 that's the average salary in a dental office. And someone
23 might say, that doesn't really summarize what's going on in
24 that office with respect to salary. The median actually in
25 this case would be $20,000, so.
42
1/17/01 - BENCH TRIAL - VOLUME II
1 THE COURT: I see. And why, you decided to use
2 the median here instead of the mean?
3 A. Well, I probably could have used either in this case,
4 but when there are extreme values, the median is not
5 effected by those extreme values.
6 THE COURT: I see.
7 A. And that's, generally, why I would report the median
8 typically, although I don't think there's any evidence in
9 this case that the mean would give any different answer,
10 although I don't have those numbers calculated in front of
11 me.
12 THE COURT: But because there's no extremes like
13 in your example, the median you thought was the way to do
14 it?
15 A. That's what I thought would be the way to do it. You
16 know, there are some reported zeros in the GPAs and the
17 LSAT. I think you actually heard about those earlier, and
18 the median would be effected very little by, if we
19 concluded or excluded those values, and I can't even
20 actually remember whether I did include or exclude those.
21 The mean obviously would be effected quite a lot. It would
22 lower those values quite a bit.
23 THE COURT: Okay. You've answered my question. I
24 appreciate it.
25 Q. Can you tell us in what kinds of analysis you
43
1/17/01 - BENCH TRIAL - VOLUME II
1 performed that you displayed with respect to slide fourteen
2 on your presentation?
3 A. Well, the median isn't the whole story. I wanted to
4 look at the actual distribution of scores and this is, this
5 is a standard summary technique that is used in statistics,
6 part of many statistical passages. A summary that consists
7 of, there are a lot of applicants, let me just say. There
8 are a lot of applicant.
9 And so displaying every single value isn't very
10 useful. And so we need to display a summary, and so what
11 we've done here is display the distribution of the
12 undergraduate grade point averages for accepted applicants
13 and we're displaying this as a function of ethnicity. So
14 let me just try to tell you what this box plot means.
15 What we have here is the, for Native Americans
16 accepted applicants, we have a box and some lines and
17 brackets and another line below, and the values that are
18 highlighted here, if I might, the white space in the
19 middle, that corresponds to the median, okay. So exactly
20 the same values that we have before, the median here would
21 be that white space in the middle of the box. The box
22 extends up to, remember the median was the fiftieth
23 percentile. And what we did is take the value that 75
24 percent of the applicants at or below the seventy-fifth
25 percentile so the box extends out the seventy-fifth
44
1/17/01 - BENCH TRIAL - VOLUME II
1 percentile on the lower and the upper side and the
2 twenty-fifth percentile on the lower side.
3 And then this technique, this box, there's several
4 ways to make them, but what it does, it doesn't report any
5 other values out beyond that unless the values are far away
6 from the box in a relative sense. So what it does is it
7 takes up the other values, the other, let's see there's 75
8 percent here, so about 25 percent of the data up here, so
9 about 25 percent of the accepted applicants in this range
10 here, about 25 percent are in the range below, down to the
11 bracket, unless a value is considered more extreme,
12 relative to its own box. And so there is one value down
13 here, see a bar down here, a line. That corresponds to the
14 undergraduate GPAs of one accepted applicant that's found
15 at that value.
16 Q. Is that one bar down there just representing one
17 applicant?
18 A. I believe it does. It actually could, given the way
19 the computer works. If there were two that had exactly the
20 same score, it would override them. That's true of all the
21 plots that you see, Your Honor. Further sometimes if
22 they're exactly the same values, they're going to overprint
23 unless we do some special, and I didn't do anything to do
24 that.
25 Q. What does the term, outlier, mean in statistics?
45
1/17/01 - BENCH TRIAL - VOLUME II
1 A. Well, there are whole books written on that, none of
2 which I've written, by the way. Outlier means a value
3 that's considered far away from the normal pattern or range
4 of the data. And this particular, this particular figure
5 is constructed so that these values that are printed
6 outside are termed outliers. That's the term that's used.
7 So they're further away from the bulk of the data than the
8 rest of the values.
9 Q. The lines outside the two brackets at the top and
10 bottom are outliers?
11 A. Well, yes, that's what this plot refers them as, yes,
12 outliers, yes.
13 Q. I'll ask you just summarize what your conclusions were
14 based on the analysis you did here for the median GPAs for
15 applicants accepted in 1995?
16 A. Yeah. The GPAs from the plots here, you can see that
17 in looking at these box plots, you can see that the Native
18 Americans box, the African-American box that, they're in
19 the same order as we had for ethnic groups before. And to
20 some extent the Mexican American box, that is the box
21 contains the, between the seventy-fifth and twenty-fifth
22 percentile. And the Puerto Rican box are lower than the
23 other groups. You can see the boxes are all lower.
24 We saw that before, but the boxes themselves are
25 somewhat lower, and Caucasian Americans and Asian Pacific
46
1/17/01 - BENCH TRIAL - VOLUME II
1 Americans have values, and their range of accepted, of
2 accepted applicants is somewhat higher with respect to
3 GPAs. And that's also true of other Hispanic Americans and
4 foreign and unknowns.
5 Q. Now, did you do this similar information, or did you
6 construct similar grids for the later years, 1996 through
7 2000?
8 A. My reports contain plots like this for every year,
9 yes.
10 Q. We don't have all those on your presentation here
11 today, the slide presentation?
12 A. I think it's probably best we don't show all those,
13 but they're all in the reports.
14 Q. Could you summarize in a very conclusionary fashion
15 whether your conclusions were the same, were the same or
16 similar with respect to what you found in these later
17 years, 1996 to 2000?
18 A. There's no substantive difference in the conclusions
19 for each, each of the substantive years.
20 Q. You can find these box plots in your reports?
21 A. Yes. The four for 1995 through '98 are in the first
22 report. And the one for 1999 is in the report referring to
23 1999 analysis. The one for 2000 is in the report referring
24 to the 2000 applicants analysis.
25 Q. Can you just explain why it is that, that your, this
47
1/17/01 - BENCH TRIAL - VOLUME II
1 data is contained in several different reports? Just give
2 is an explanation for that?
3 A. Why it's in several reports?
4 Q. Several different reports for several different years?
5 A. Well, when I constructed the first report, I only had
6 the data for 1995 through '98. Later on I got the data for
7 1999 and conducted a report for that and later I got the
8 data for 2000 and conducted a report for that.
9 Q. Let's go to the next slide. What was the next step in
10 your analysis?
11 THE COURT: Excuse me. Hey, Len, if you guys want
12 to move over to the jury box, too, you're more than welcome
13 to.
14 MR. NIEHOFF: Thank you, sir. I appreciate it.
15 THE COURT: Excuse me. I saw them struggling with
16 the book.
17 MR. KOLBO: We have more copies of this
18 presentation.
19 Q. Can you tell us, Dr. Larntz, what the next step of
20 your analysis was, as reflected by slide fifteen?
21 A. Right. This is a box plot constructing the same way
22 for accepted applicant LSAT scores. And we can see the,
23 the same pattern in the sense that there are four boxes
24 that are lower than the others; those for Native Americans
25 accepted applicants, African-Americans accepted applicants,
48
1/17/01 - BENCH TRIAL - VOLUME II
1 Mexican American accepted applicants and Puerto Rican
2 accepted applicants.
3 And in fact, I guess in this case, the boxes
4 themselves are all at or below the lower end of the boxes
5 for Caucasian Americans and Asian Pacific Island American
6 applicants. That means in terms of what we're talking
7 about, the seventy-fifth percentile, the upper one of the
8 box, the seventy-fifth percentile for these four groups
9 individually are lower than the twenty-fifth percentile of
10 accepted applicants for Caucasian Americans, Asian Pacific
11 Island Americans.
12 Q. And did you construct similar grids and do similar
13 analysis for the later years as well?
14 A. Yes. Lots for 1996, 1997, 1998, 1999 and 2000 are
15 contained in my reports.
16 Q. And the conclusions there, are they the same or
17 similar?
18 A. The substantive conclusions stay the same. Obviously
19 the numbers and the actual plot positions change somewhat,
20 as we'd expect to vary from year to year, but the
21 substantive conclusions remain the same, yes.
22 Q. Okay. Let's go to the next slide. Dr. Larntz, can
23 you tell us whether slide sixteen reflects the next step
24 that you took in your analysis?
25
49
1/17/01 - BENCH TRIAL - VOLUME II
1 (Whereupon an off-the-record
2 discussion was had.)
3 Q. And the question again is if you can tell us what,
4 what your next step in the analysis was and how it's
5 reflected on slide sixteen.
6 A. Well, what I did, this was actually a grid. I call it
7 an admission grid of LSAT and GPAs for all applicants in
8 1995. And this is a, this is actually my construction
9 duplication of a grid that was given to me when I
10 originally got materials for the case of an admission grid
11 that was constructed by, I presume, by the admissions
12 office of a law school.
13 And what the grid gives is for, it cross
14 classifies individuals by LSAT score. And there are a
15 number with no LSAT score in the range that we think of
16 from 120 to 180, and then classifies individuals by LSAT
17 score for categories 120 to 145, out through, well, this
18 doesn't look very good on here, but it's 170 and above.
19 Q. It's the last category.
20 A. So it classifies applicants by their LSAT score in
21 relatively small ranges of LSAT, and their, cross
22 classifies them by grade point average. So the first line
23 for grade point average is 3.75 and above, and then by
24 quarter grade points down until we get to below two. So
25 for instance, and I don't know, I don't know if you can see
50
1/17/01 - BENCH TRIAL - VOLUME II
1 the lower right-hand corner or not.
2 THE COURT: I can see it on my own chart.
3 A. Okay. In the lower-right hand there are all together
4 4,147 applicant.
5 THE COURT: Right.
6 A. And we cross classified the individual applicants, and
7 I also put in this chart the number admitted, and the
8 number right below is the number admitted. So there are
9 1,130 admitted. And for individual cells, we can go at,
10 for instance, at a combination and we highlighted one,
11 individuals with a 3.25 to 3.49 GPAs and LSAT of 161 to
12 163, in that range. There were 198 applicants overall,
13 198, 17 of whom were admitted.
14 So with my charts from before, you could calculate
15 the odds by just looking at, it would be a ratio of 17
16 divided by, I'll do my math, 181, so you could actually
17 calculation the, the observed odds of admission for a case.
18 Q. Can I ask you, Dr. Larntz, how it was you decided to
19 display your materials in this graphic form? I think you
20 may have alluded to it, but could you give a more detailed
21 explanation as to how it is you came to explain the
22 information in this format.
23 A. Thank you. Well, this is exactly the format that I
24 received. I think we, it was referred to me as Exhibit 16
25 from the law school. I'm not sure what exhibit number that
51
1/17/01 - BENCH TRIAL - VOLUME II
1 was or where it came from.
2 Q. Dwayne, could you show the witness Exhibit 16, if he
3 doesn't have that in front of him.
4 A. I have it in front of me here.
5 Q. And do I understand that you did your own work with
6 database to construct a grid in the same fashion as is
7 displayed on Exhibit 16?
8 A. Yes. I used the database, the computerized database
9 that was provided to me and reconstructed Exhibit 16 to my
10 own satisfaction that these numbers were exactly the same.
11 Q. And was it with the use of Exhibit 16 that you decided
12 upon the manner in which the different LSAT combinations
13 and grade point averages should be combined or put together
14 in a cell?
15 A. I used exactly the same categorization as Exhibit 16,
16 yes, I used, that's what I used.
17 Q. If we go to the next slide then, Dwayne.
18 A. Exhibit --
19 Q. Or tell us what slide 17 represents.
20 A. In the, in Exhibit 16 there are further breakups of
21 the all applicants into various sub-groups, and sub-groups
22 by ethnic group and sub-groups by gender and combinations
23 of ethnic group and residency. And so what, and the next
24 page in Exhibit 16 is the admission grid for Native
25 American applicants only. You can see there are, what, in
52
1/17/01 - BENCH TRIAL - VOLUME II
1 this case, 45 applicants, fourteen of whom were admitted
2 overall. And in the cell we were following, there were two
3 applicants, two of whom were admitted. So both were
4 admitted.
5 Q. And how would you summarize the odds for that
6 particular, that particular cell?
7 A. If I were calculating odds, now we're into
8 mathematics. So it's two versus zero, right, two admitted
9 and none denied. And so technically, the odds in that
10 case, if we divide by zero, they turn out to be infinity.
11 Obviously that's, that's a big number, if I want
12 to say that, in statistical terms, but, in fact, we know
13 that with small numbers that we have here, we're going to
14 see numbers like that, numbers like infinity. I would
15 believe that if there were many more applicants, and there
16 weren't. But if there were many more applicants, we would
17 probably see some accepted and some denied. The odds would
18 be different than infinity.
19 Q. The next slide there. What does slide 18 demonstrate?
20 A. This is the, again, in the next, Exhibit 16, chart was
21 for African-Americans applicants and they're in this order
22 because that's the order at which the variable is in the
23 database. And that's the order in which they were
24 displayed in Exhibit 16.
25 And so we have African-Americans applicants.
53
1/17/01 - BENCH TRIAL - VOLUME II
1 There were 404 applicants, 106 of whom were admitted. And
2 in the cell we're following, there were four applicants,
3 four of whom were admitted.
4 Q. Next slide?
5 A. This is the cell for Caucasian American applicants.
6 Again, remember we had 2,316 applicants, 668 of whom were
7 admitted. And in the cell we're following, there were 126
8 applicants, five of whom were admitted.
9 Q. And what did you calculate the odds for that
10 particular?
11 A. If I calculated the odds, it's five divided by 121,
12 five divided by 121. I could do that with a calculator.
13 I'm not going to try to do in my head.
14 THE COURT: You can use it if you wanted to.
15 Q. Why don't you go ahead and tell us what that number
16 represents.
17 THE COURT: He doesn't have to. I just thought if
18 he wanted to.
19 MR. KOLBO: I'd just as soon, Your Honor.
20 A. Five divided by 121 is 0.041 so in odds of 0.041.
21 Q. Okay. Then if we could proceed to the next slide.
22 What does this information tell us?
23 A. The next slide is for Mexican American applicants.
24 There are 98 total applicants, 41 of whom were admitted.
25 And in the cell that we're just going through to summarize,
54
1/17/01 - BENCH TRIAL - VOLUME II
1 there was one applicant and that applicant was admitted.
2 Q. And the next slide?
3 A. Other Hispanic Americans, as classified in the
4 database, 115 applicants, 15 of whom were admitted. And in
5 the cell that we're looking at, seven applicants, one
6 admitted.
7 Q. And the next slide.
8 A. This is Asian Pacific Island Americans, as tabulated
9 in the database. There was 470 applicants, 111 of whom
10 were admitted. And in the, again, the grid cell we're
11 following, there are twenty applicants, two of whom where
12 admitted.
13 Q. And could you calculate the odds on this?
14 A. This one I can actually calculate, because that's,
15 that corresponds exactly to what we had before, isn't that
16 right, a ten percent chance of, ten percent chance, so two
17 to eighteen, so the odds are one ninth, one ninth.
18 Q. And the next slide.
19 A. This would be for Puerto Rican applicants. And there
20 are a few, twenty, not a lot of individuals classifieds as
21 Puerto Rican Americans. Twenty were, there were twenty
22 applicants, five of whom were admitted. And in this
23 particular cell there weren't any. So as far as
24 information about Puerto Rican American applicants, this
25 cell doesn't tell us anything, because there aren't any.
55
1/17/01 - BENCH TRIAL - VOLUME II
1 Q. Okay. And next.
2 A. Foreign applicants, overall twelve applicants, twelve
3 of whom were admitted, and the cell we're following, three
4 applicants, none of whom were admitted.
5 Q. Okay. Let's go to the next cell, or next slide then.
6 A. And finally, this is the last one in this series that
7 give, that give the nine, by ethnic groups. There are 560
8 individuals classifieds as unknown ethnicity, 158 are
9 admitted, and in the cell we're following, 35 applicants,
10 two of whom where admitted.
11 Q. Next, Dwayne. What does this slide summarize?
12 A. This slide is just a summary, looking at that, the
13 particular cell that we happened to follow through in the
14 database. The 3.25, 3.49 GPAs, LSAT range of 161 to 163,
15 and this just summarizes the number of applicants for each
16 of the ethnic groups and the total, and the number of
17 accepted applicants for each of the ethnic groups, among
18 those who applied.
19 Q. Okay. Let's go to slide 27 then. Can you?
20 A. Well, this.
21 Q. Tell us what the next step in your analysis was, Dr.
22 Larntz.
23 A. If we wanted to compare, say, two ethnic groups with
24 respect to their chance of admission with similar
25 credentials; that is, similar GPAs and similar LSAT scores,
56
1/17/01 - BENCH TRIAL - VOLUME II
1 classified in the same way that the law school did in their
2 Exhibit 16, we could, for instance, put the tables side by
3 side, and try to read them, which would be a nice exercise.
4 I don't know how readable they are in the written report.
5 But we could, we could, for instance, look at,
6 say, well, I think we have it set up to bring up the, the,
7 the actual comparison by, say, all the individuals with
8 LSAT scores of 161 to 163 for both groups.
9 So we could, for instance, look at the, at the
10 comparison of African-Americans applicant, and their
11 admission to Caucasian American applicants and their
12 admission. So, for instance, for individuals with that
13 range of LSAT score and grade point averages in the highest
14 category, 3.75 and above, for African-Americans applicants,
15 it was three admitted out of three. And for Caucasian
16 American applicants, it was eight out of 93.
17 In the next cell down, 3.5 to 3.74 grade point
18 average, among African-American applicants that there were
19 six applicants, five of whom were admitted and among
20 Caucasian American applicants, there were 161 applicants,
21 14 of whom where admitted.
22 Going down to the next grade point average for
23 African-Americans, four out of four were admitted. And
24 Caucasian American applicants, five out of 126. That's the
25 cell we were following before, exactly the same, so the
57
1/17/01 - BENCH TRIAL - VOLUME II
1 numbers are the same.
2 To the grade point average range from three to
3 3.24, seven out of eight African-Americans admitted, and
4 two out of 42 caucasian Americans admitted. For 2.75 to
5 2.99, African-Americans, there were four applicants. All
6 four were admitted. Caucasian American applicants, there
7 were 14 in that combination, none of whom were admitted.
8 Grade point average of 2.5 to 2.72,
9 African-Americans admissions were two out of three,
10 Caucasian Americans, zero out of seven.
11 And in the next one, which is 2.25 to 2.49 there
12 was one African-American applicant who wasn't admitted, and
13 five Caucasian American applicants, none of whom were
14 admitted. And so, in fact, in this cell, there's really no
15 ability to compare admission rates in the sense that
16 they're both the same, but there's no, no one was admitted
17 in that particular, no one is admitted in that particular
18 cell.
19 In order to make a comparison, at least in terms
20 of an odds ratio, we have to have individuals that two,
21 each ethnic group in the class, in the cell, and we also
22 have to have some admitted and some denied. And so in this
23 case, both were denied. The next cell has no applicants
24 and no, well, you can see what goes on down below.
25 Q. If you have a, if you have, if you're comparing two
58
1/17/01 - BENCH TRIAL - VOLUME II
1 groups and all are admitted from one cell and all are
2 admitted from that same cell in the other group, can you
3 compute relative odds for that, for those two groups?
4 A. Actually the relative odds is, it could be calculated
5 as being either an infinity, overinfinity in the case
6 you're saying, if all were admitted, and we wouldn't, and
7 we would say that that cell didn't have any comparative
8 information with respect to relative odds.
9 Q. And the same is true if you have cells in which no
10 one's admitted from either group?
11 A. Again, in that case if no one is admitted, the odds
12 are zero for each of the cells. Then it would be a zero
13 over zero. We wouldn't define that either. We would say
14 that also gives no comparative information.
15 Q. Okay. Where there's comparative information, that's
16 when you calculate, when you can calculate relative odds?
17 A. Where there's comparative information, you can
18 calculate relative odds. Although you may wind up as we
19 did in our example with some examples with small counts of
20 infinities or corresponding into zeros.
21 Q. Okay. For example, the first, at the very top there,
22 three over three, and 93 over eight, would that calculate
23 out to infinity because in the case of the African-American
24 applicants one hundred percent of them were admitted?
25 A. That's correct. You'd wind up with three over zero,
59
1/17/01 - BENCH TRIAL - VOLUME II
1 which would be infinity, divided by 91 over 85, which is
2 not, and so we'd wind up with an infinity there, yes.
3 Q. And the cell below that where there are, where it's
4 less than a hundred percent for either category, could you
5 calculate relative odds for the six over five and the 161
6 over 14?
7 A. Yes, you could.
8 Q. Using the same method that you've explained earlier?
9 A. Right. It would be five over one, so there would be,
10 five would be the odds, five would be the odds. They
11 observe odds of African-Americans admits and five over 121
12 would be the odds of admission for Caucasian American
13 applicants.
14 Q. Actually it's the cell above?
15 A. I'm sorry. I'm glad someone's checking me when I read
16 these. I apologize. It's, you're right. It's 14 out of a
17 hundred and, now I lose my train of thought, 149, I
18 believe.
19 Q. All right. Let's go to the next slide then. Can you
20 do the same, you've illustrated these comparisons with
21 respect to a column of LSATs. Can you do the same thing,
22 illustrating a column, a horizontal column with grade point
23 average ranges?
24 A. Exactly. I think we're set to look at those for the
25 same range that we did before.
60
1/17/01 - BENCH TRIAL - VOLUME II
1 Q. And what, what information is summarized here?
2 A. Are we waiting for another one to come up?
3 Q. Sure.
4 A. Thank you. What we've done here is for the grade
5 point average of 3.25, 3.49, we've looked at the, across
6 LSAT score combinations as laid out in the Exhibit 16
7 categories and we've compared African-American admission
8 chances to those of Caucasian applicants with the same
9 grade point average, and then we can see how it changes is
10 a function, or as LSAT changes.
11 So in essence here, if we look across the row for
12 African-Americans applicants, we can see that those
13 African-Americans applicants were very low LSATs
14 relatively. There's 120 to 140. There's, well, none of
15 them were admitted. There were 15 applicants. And
16 similarly in the next category, none were admitted.
17 And as we go across, two out of six in the next
18 cell were admitted, three out of seven in the next one,
19 four out of five. And then actually in this case, once you
20 get to 156 and above, it looks like in this case, all the
21 African-American applicants with LSATs in that range, and
22 that value or above, ten out of ten, three out of three,
23 four out of four, one of one, two out of two, all of the
24 were admitted.
25 So you can see as LSAT increases, the chance of an
61
1/17/01 - BENCH TRIAL - VOLUME II
1 African-American applicant being admitted increases. And
2 that's consistent with, I think, the stated law school
3 policy. And similarly we can look at the same, at the same
4 increase for caucasian applicants.
5 It turns out that, that all Caucasian American
6 applicants below 155 are not admitted except for those for
7 whom there was a no-LSAT category, so at least all of those
8 that had numerical value from 120 to 155, none of those
9 were admitted. And then admissions started in the 156
10 range, exactly the same category in which all
11 African-American applicants then are admitted.
12 And then we had one out of 51 compared to ten out
13 of ten, one out of 61 compared to five out of 126, compared
14 to four out of four, eleven out of 92, compared to one out
15 of one. 38 out of 78, compared to two out of two. And
16 actually the last cell has 55 over 74, but there weren't
17 any African-American applicants in that cell.
18 Q. Let's go to the next slide down. You spent some time
19 taking a look at these cells and some comparisons. If you
20 could just remind us again how relative odds would be
21 calculated looking at those comparisons.
22 A. All right. This should be a summary exactly of the,
23 of what I did on the board earlier; that is, that if we
24 look at calculating relative odds, relative odds comparing
25 a 75 percent probability of admission to a 75 percent
62
1/17/01 - BENCH TRIAL - VOLUME II
1 probability of admission corresponds to a relative odds of
2 nine, just exactly the same way we did before. And
3 similarly, a relative odds of a 90 percent probability of
4 admission, two out of a ten percent probability of
5 admission the relative odds would correspond to 81.
6 Q. You can go to the next slide then, Dwayne. Can you
7 tell us, Dr. Larntz generally what kind of analysis then,
8 perhaps you can draw us some detail about what kind of
9 overall analysis we did to assess the relative odds of
10 acceptance for the different racial groups that you studied
11 or other groups that you studied, as well.
12 A. Well, what we wanted to do next, or what I wanted to
13 do next, and what I did next was to say, try to provide a,
14 what I'll call a composite measure of the relative odds.
15 Many of these cells are very small, relatively small. They
16 have small numbers of counts and small numbers of
17 individuals. Some cases there are cells where there are no
18 admissions. Some cases there are cells where there are a
19 lot of admissions.
20 And so what I did was construct a statistical
21 model that allowed us to calculate a composite relative
22 odds combining all the cells, all the cells in the grids
23 that have comparative information. So in all the cells
24 where there was comparative information, they would
25 contribute to a composite relative odds.
63
1/17/01 - BENCH TRIAL - VOLUME II
1 And what I wanted to do was summarize, I mean, the
2 relative odds that we saw before, some were very, some, a
3 lot were infinity. Some were other numbers. Can we put
4 those together and the statistical method I used which was
5 logistic regression allows us to create an estimate of the
6 relative odds comparing the odds of admission from one
7 ethnic group to another ethnic group.
8 And the goal of that analysis was to summarize
9 the, the weight or the effect in terms of relative odds
10 that are given to members of various ethnic groups with the
11 same grade point average range as we did in Exhibit 16 and
12 the same LSAT range. So individuals that have the same
13 credentials with respect to those two variables.
14 Q. Now, you've mentioned that there are occasions when
15 you get a calculation of infinity because, for example, all
16 applicants from one minority group might be admitted and
17 you have less, you have some number of majority applicants
18 who are denied admission and you'll get an infinity value,
19 correct?
20 A. That's correct.
21 Q. And how is that accounted for in your assessment of
22 trying to summarize in overall fashion the relative odds of
23 different racial groups when you have cells that shows
24 infinity as a value for relative odds?
25 A. Well, those cells contribute information. They're
64
1/17/01 - BENCH TRIAL - VOLUME II
1 not, they're not no-information cells. They certainly
2 contribute information. And so the statistical technique
3 that's used, logistic regression makes those a composite,
4 makes a composite estimate from those cells accounting for
5 all the information, all the cells that have comparative
6 information. And so, for instance, if I might use a very
7 mundane example, and I apologize, Your Honor, if you're not
8 a Chicago Cubs fan, but I go to a Chicago Cubs game every
9 year.
10 I was a graduate of the University of Chicago and
11 they collapsed in 1969 when I was there. And so I followed
12 them from then on. And I have a son who was born in
13 Chicago, doesn't live there any more, lived there only a
14 couple years, but he said that's his hometown, so we have
15 to go back.
16 THE COURT: Oh, yeah. I had a law clerk who
17 wanted to visit every single -- I'm not a baseball fan, and
18 she wanted to visit every single baseball stadium in the
19 country. So we traveled quite a bit hearing cases all
20 over. So I've been to many stadiums. I don't think it was
21 a Cub game. Steve, you want went to a Cub game, didn't
22 you?
23 (Whereupon an off-the-record
24 discussion was had.)
25 THE COURT: But not with Barb. She'd always say I
65
1/17/01 - BENCH TRIAL - VOLUME II
1 need to, assign it here, because I hadn't seen that
2 stadium, and then they started building new stadiums. She
3 just went crazy because, you know, we'd have to go back to
4 cities again. Anyhow, go on. I've been to more games than
5 I wanted to with her.
6 A. Well, I have to do as a father, of course. But, and
7 so I sacrifice myself for that, but I don't -- anyway.
8 At any rate, we've sometimes watched Sandy Sosa
9 play. And we are very curious while he plays each day.
10 And sometimes he does very well. And for instance, he, he
11 bats thousand in some individual games. That's true. I've
12 seen him do that, hit three home runs and bat a thousand.
13 That's a pretty good day for Sandy Sosa. That's a pretty
14 good for most baseball players, right.
15 And so those a thousands are like, are like
16 infinities, in some sense. They're sort of off the scale.
17 And we've also seen Sammy Sosa on other days, I have to
18 admit, strikes out every time at bat, and that's like a
19 zero, nothing, no. And so neither one of those probably
20 represent his batting average overall, right, because, you
21 know, a batting average overall is a composite of a large
22 series of games.
23 And these numbers that I have here, and we
24 certainly would include the games where he, he did very
25 well, and the games where he didn't do very well. And
66
1/17/01 - BENCH TRIAL - VOLUME II
1 these estimated relative odds that we have here are
2 somewhat composites of the relative odds that we see in
3 the, in the individual cells.
4 Q. So in calculating Sammy Sosa's batting average, you
5 don't throw out the games where he bats a thousand, do you?
6 A. No.
7 Q. You don't throw out the games where he bats a zero?
8 A. No, you don't.
9 Q. You include those and they get factored into the
10 analysis?
11 A. That's correct.
12 Q. Is that the same here with respect to the analysis
13 that you did?
14 A. In some sense that's the same. We've got an infinity
15 and a zero, corresponding to the same kind of thing in our
16 case. They correspond to boundary cases, that is, in some
17 cases, a minority sub-cell where we had a hundred percent
18 admitted. That would correspond to an odds ratio of
19 infinity. We certainly include all that information in
20 this composite.
21 Q. Now, you've provided some information about how
22 relative odds are calculated. And we've seen some
23 information about that on the board.
24 Can you tell us how, outside the context of a case
25 like this, how would the other areas where you have
67
1/17/01 - BENCH TRIAL - VOLUME II
1 performed statistical work, whether it's regulatory work or
2 medical work, medical devices, how are relative odds used
3 in those fields, generally speaking?
4 A. Well, actually, in a clinical study, say of a new
5 medical device or a new drug where we would be comparing
6 the new drug or new device to a standard therapy or a, a
7 drug case may be a placebo pill or a current active pill,
8 we would compare the results in, in exactly the same way.
9 We would use our individual cells, in this case
10 would be hospitals, hospitals, for each hospital. If we
11 did a multi-center study of many hospitals, and I've been
12 involved in studies that include, you know, fifty or a
13 hundred hospitals, each hospital contributes a, some number
14 of cases to the study, maybe not very many, just like a
15 cell contributes some number of cases. And we would look
16 at, in each hospital, the number of individuals in the,
17 that took part in the study, that received, say, the
18 standard treatment and seeing what happened to them,
19 whether or not they were cured or had some complication or
20 something like that.
21 And we'd also look at the number with a new
22 device, the number that took part in that particular
23 hospital, and the number that were, say, cured, or had a
24 complication. And many times, many times these studies
25 result in a number of fairly small cells with not very
68
1/17/01 - BENCH TRIAL - VOLUME II
1 large numbers, very similar to what we have here.
2 And we put together, using this, the technique I'm
3 using here is a technique we use to get a composite
4 relative odds of how useful the new device or the new drug
5 is compared to the standard. And we, we often summarize
6 that in terms of relative odds.
7 Q. Can you give some examples of what are, in, say, in
8 the field of working with medical devices or new medicines,
9 what kind of odds are deemed significant? What kind of
10 relative odds are deemed significant in those fields?
11 A. Well, I mean, if I were designing a study, if I were
12 designing a study, we'd often make, you know, some kind of
13 estimate of what was an important relative odds with
14 respect to patients.
15 And depending on the area, relative odds, I'd
16 certainly design a study where we're looking for a relative
17 odds of say, two, or one and a half. Or, I read about Dick
18 Cheney's angioplasty, using a stint. And it's an area I
19 work in, actually, coronary stints, and there the, the
20 relative odds of heart disease versus no heart disease for
21 someone, say, receiving Aspirin is about 1.3, 1.4.
22 So relative odds that might be small numbers
23 greater than one are common. Odds -- I can think of a
24 medical example I had once, if I might continue, where we
25 were looking at historical data of people who were at very
69
1/17/01 - BENCH TRIAL - VOLUME II
1 low blood pressure, very low cholesterol, and their chance
2 of a heart attack, or heart disease, showing itself, to
3 those with high cholesterol and high blood pressure. And I
4 calculated relative odds in a case like that, the actual
5 observed relative odds was, well, if I remember right,
6 about nineteen. We used that as an example in my classes,
7 because that's such a big number.
8 Nineteen's an enormous relative odds, and so that
9 would. To me, it gave me advice on how to handle my diet,
10 of course, and things like that, but it was some of the,
11 some of the odds we're seeing, if you change relative odds
12 by factors of one and a half, two, twenty would be
13 incredibly large value in medical studies I've dealt with.
14 Q. Well, in the relative odds analysis that you performed
15 in this case with respect to the University of Michigan Law
16 School, can you summarize what your conclusions and
17 findings were for comparisons to the different racial
18 groups?
19 A. This slide 30 is a result in my statistical analysis,
20 using logistic regression, using the technique that I would
21 use in standard medical studies to, to compute composite
22 relative odds or odds ratios. And when you're comparing,
23 when you're comparing two probabilities, you have, you
24 compare one probability one odds to another. And you have
25 to have a baseline of comparison. And so I chose as my
70
1/17/01 - BENCH TRIAL - VOLUME II
1 baseline for comparison, it's typical of the analysis we
2 do, we choose, say, the largest, the largest group as a
3 baseline.
4 So what I did is largest group which is Caucasian
5 Americans I use as a baseline, and they would be given a
6 relative odds to themselves of one. That's by definition.
7 So the one that's given here is just our baseline for
8 comparison. And what I've done is computed the composite,
9 estimated, the composite relative odds for Native Americans
10 to Caucasian Americans as about 61. And African-Americans,
11 the estimated relative odds, this is looking at the
12 relative odds of acceptance, a composite measure across all
13 these grid cells. So we're controlling for, we're looking
14 at combining the information from each combination of GPAs
15 and LSAT.
16 So putting those together for African-Americans,
17 the number is 257. For Mexican American, the estimated
18 relative odds is 81. Other Hispanic Americans it's 1.03.
19 One corresponds to basically a relative odds,
20 which is similar probabilities. One is what you get for a
21 relative odds that are, that they're basically the same.
22 1.35 is the number for Asian Pacific Island Americans, 37
23 for Puerto Rican Americans. 0.5, less than one seems to
24 indicate that foreign applicants, although it's not a very
25 big amount less than one, would be, have a smaller chance,
71
1/17/01 - BENCH TRIAL - VOLUME II
1 compared to caucasian Americans. If the relative odds is
2 below one, then it would indicate that there would be a
3 smaller chance compared to Caucasian Americans.
4 And those unknown the estimated relative odds is
5 1.18. So what I see here in this, in this example, 1995, I
6 see that four of these values are very, very large, very
7 large by all my experience. And it just, how do I say
8 this. These are in non-technical terms, enormous values.
9 There's four that are very, very large. Those for Native
10 Americans, African-Americans, Mexican Americans and Puerto
11 Rican Americans.
12 And so given the same credentials, given the same
13 credentials, that is the same GPAs, LSAT cell, it, there is
14 a tremendous advantage, or allowance made. I'm reporting
15 just what the admissions committees have done with respect
16 to admission. There's a tremendous advantage given to
17 Native American, African-American, Mexican American, Puerto
18 Rican American applicants, compared to, well, Caucasian
19 American, other Hispanic, Asian Pacific Island Americans
20 and foreign and unknown ethnicity.
21 Q. Dr. Larntz, can you give us, can you state your
22 opinion as to how the size of these relative odds compares
23 to generally other kinds of odds you've seen in the kinds
24 of work you've done, either as a consultant or as an
25 academic in your thirty years or more as a statistician?
72
1/17/01 - BENCH TRIAL - VOLUME II
1 A. These numbers are just big, I mean they're giant. I
2 can't recall any particular examples of any data set of any
3 size that, where I found relative odds of this magnitude.
4 They're large.
5 Q. Okay. Incidentally, is the data that, size that
6 you're working with here, would you consider it a large
7 sample, small sample, how, for each of these years, and
8 then for the years overall?
9 A. I should let my computer answer that question. We
10 have a lot of data. We have a large number of applicants.
11 The amount of comparative information we have, although the
12 cells themselves are small, the amount of comparative
13 information we have, for instance, for African-American
14 applicants, in particular, is considered a large data cell.
15 Q. Now, you report on slide 30, to the right there, some
16 information on standard deviations. Can you tell us what
17 the significance of those values are here?
18 A. Well, as I said, I tried to explain earlier, we want
19 to see, there's always a chance of variation in the world.
20 That's life. I said it earlier. I'd be out of business as
21 a statistician if there weren't chance of variation, if
22 everybody responded exactly the same. So I like chance
23 variation, okay. It gives me a livelihood.
24 And now what we want to do is say, these relative
25 odds that we see here, if we redid them for a different
73
1/17/01 - BENCH TRIAL - VOLUME II
1 year, or we redid them with the new group, they would
2 change. There's no question about that. We'd get
3 different numbers. That's -- there's variation. And we
4 wouldn't expect them to be the same. And the question is,
5 is there evidence that these are bigger than the baseline
6 of one. Are they really bigger.
7 And so we can do a test to compare what's the,
8 what's the likelihood, what's the chance that we would get
9 odds this big if, in fact, chance alone were operating, if
10 chance alone were operating. So could we get numbers like
11 this. Sure, I mean, you can always get any number. Now,
12 how likely is it that chance alone is operating.
13 So, what I did in the standard deviation column is
14 summarize the degree of statistical evidence in terms of
15 standard deviations, so we're thinking, something
16 statistically significant, if it's two or three, or larger
17 in magnitude, and so what I did is summarize the degree of
18 statistical evidence that, concerning each of the relative
19 odds and how likely it is that chance alone was creating
20 this.
21 Q. And what were your conclusions?
22 A. Well, it's clear there are, in this particular slide,
23 there are four values that are bigger than the usual two or
24 three, and we're talking about. There are four, and those
25 are for Native American, African-American, Mexican American
74
1/17/01 - BENCH TRIAL - VOLUME II
1 and Puerto Rican applicants. And so that's, in fact, the
2 smallest one of those is 4.95, the largest is fourteen.
3 Those are, again, sort of strong, strong indications that
4 it's not just chance alone that's, that's causing these,
5 this difference in relative odds.
6 Q. Now, again, did you perform the same kinds of analysis
7 here of relative odds for acceptance controlling for GPAs
8 and LSAT grid cell for the later years, 1996 to 2000?
9 A. Yes.
10 Q. And we don't have those here.
11 A. I think we do actually have the '96, '97, '98.
12 THE COURT: Let me just ask you a question. In
13 terms of relative odds, can that be translated into how
14 many times more likely it is that one group will be
15 admitted compared to another group?
16 A. It's directly in terms of odds, Your Honor. So if, in
17 fact, so, if -- let's just take the example that we had.
18 So if, say --
19 THE COURT: Using the figures that are on the
20 board.
21 A. Okay. Okay. Sure. Can I use the Mexican American
22 figure?
23 THE COURT: Whichever one you want, just.
24 A. Well, that number there, you see what number that is?
25 THE COURT: Sure.
75
1/17/01 - BENCH TRIAL - VOLUME II
1 A. It's 81. And we've worked with 81 already. So I can,
2 I can do it in that terms, and I think it makes it easier
3 for me to explain. So, for instance, if, if we, if a
4 Caucasian American applicant in a particular combination of
5 GPAs and LSA had a ten percent chance, ten percent chance
6 of being admitted, then the odds, relative odds of 81 for
7 Mexican American applicants would take Mexican American
8 applicants in the same combination of LSAT and grade point
9 average. They would be boosted from ten percent chance to
10 90 percent chance. Okay. Okay. And so that's --
11 Q. Excuse me. That's where the 81 comes in?
12 A. Numbers that are smaller than that would have a
13 smaller change, but, you know, still, I mean, numbers,
14 changing odds by tens, I consider that pretty big, more
15 than pretty, big, that's, that's enormous.
16 THE COURT: So when you get to a figure of the
17 African-Americans up there, it would be --
18 A. Two fifty seven is such they would change, you know,
19 in the same kind of context. And I'm just going to be, I'm
20 not going to be exactly precise, because I don't want to
21 take out my calculator but that would change probably a six
22 or seven percent chance to 93, 94.
23 THE COURT: And you think that that really
24 translates in the real world, is that, these figures?
25 A. These are a summary of the information from the
76
1/17/01 - BENCH TRIAL - VOLUME II
1 admission grids. So these are, these are.
2 THE COURT: Is this the real world?
3 A. Well --
4 THE COURT: Assuming the grids are correct?
5 A. It's from the database. These are, these are not
6 anything, other than me trying to summarize the information
7 in what I consider an appropriate statistical way for the
8 advantage given to what are terms selected minority groups
9 compared to other ethnic groups.
10 THE COURT: Okay. Thank you. You may proceed.
11 Q. Is there a way, Dr. Larntz, that it would be helpful
12 to illustrate how one would illustrate the change from,
13 say, having a ten percent chance of probability to, to a 90
14 percent chance of probability in terms of relative odds?
15 Is there way to illustrate that with a drawing, or, or
16 would that be helpful?
17 A. Sure. I could do it if you want me to. I mean,
18 again, if you think it would be useful to reiterate the,
19 the point. I could probably do it with a drawing,
20 something I do when I teach.
21 Q. Why don't you just briefly do that, if you could.
22 A. Okay.
23 Q. Give us a physical illustration of what it means to
24 have relative odds of 81, for example?
25 A. Okay. I'll do that and use the flip chart.
77
1/17/01 - BENCH TRIAL - VOLUME II
1 THE COURT: Why don't we put the flip chart over
2 here and then everybody.
3 MR. KOLBO: That's fine.
4 A. And this is an example I use in class, just to, it
5 sometimes helps to have physical models. And in class when
6 I'm teaching, and I'm retired from teaching, but I enjoy
7 teaching, believe it or not.
8 THE COURT: I can tell.
9 A. Thank you. I appreciate that. So what I would do in
10 class with my students is I would actually bring in M & M's
11 in the class. That's actually quite effective. It gets
12 their attention and we use the demonstration, and then they
13 get to eat the results. How's that, fair enough.
14 THE COURT: Yeah.
15 A. And so they, they enjoyed this. And this is, this is
16 an actual illustration that I used in class. Now, is this
17 one, this is red. Okay. So, for instance, let me get the
18 black. I would demonstrate this particular case. And it's
19 just a demonstration of probabilities and relative odds,
20 which I think you probably understand. But I'll just
21 demonstrate it. I've got, say a big jar. And in class I
22 would have an actual jar I'd put M & M's in.
23 And I would say ten percent, ten percent chance of
24 admission, if I were using that in this example. Ten
25 percent chance would be, I would have in the, in the jar an
78
1/17/01 - BENCH TRIAL - VOLUME II
1 M & M. One, fair enough. And what's ten percent.
2 Let's see, I got to make sure I do this right,
3 one, two, three, four, five, six, seven, eight, nine, if I
4 counted right, okay. Then what we would do is we would
5 take this jar, we'd mix them up and say, well, a ten
6 percent chance which is, is, is a chance of drawing in this
7 jar and picking out an M & M. And if it's green, then the
8 event occurred, say admission. If it's red, then the
9 student was denied. Fair enough. And so, so we do that.
10 Now, we want to compare that to a relative odds of
11 81, which corresponds to 90 percent which, as you know
12 already. What does that correspond to? How many M & M's
13 do I have to dump in to make that, make that the relative
14 odds of 81. Well, I've already got one in here, right. So
15 in order to get a relative odds of 81, what I have to do is
16 I have to dump in -- well, I've got to dump in 80 more.
17 And so what I would do then in class, I would take
18 the 80 M & M's, I prepared then, dumped them in and then
19 show them. That's the increase in relative odds by going
20 from one to 81, going from ten percent to 90 percent. And
21 I won't draw 80 here, but the idea is I wind up with many,
22 many, whatever in this thing, and think of, think the jar
23 now containing 81 green ones and nine red ones. That
24 actually corresponds to 90 percent, 81 out of ninety. So,
25 so that's a demonstration of how I would do it in teaching,
79
1/17/01 - BENCH TRIAL - VOLUME II
1 to get people to think about the physical, the physical
2 value of relative odds of 81.
3 THE COURT: Okay.
4 A. Okay.
5 Q. I think Dr. Larntz reminded me that you do have
6 analysis power point recitation as well as in your report?
7 for later years of relative odds acceptance?
8 A. Yes. That's given for, this particular analysis is
9 given for 1996, is the next slide.
10 Q. Slide 31?
11 A. Yes, and '97, and '98, and '99 and 2000.
12 Q. Okay. And again, the numbers change somewhat, but can
13 you see what your overall conclusions are with respect to
14 the relative odds analysis that you did controlling for GPA
15 and LSAT over the course of these, of this six-year period?
16 A. Yes. The numbers do change. And that's what I would
17 expect. It's not the same set of applicants every year.
18 But, in fact, for all four years the, the relative odds for
19 Native American, African-America, Mexican American, Puerto
20 Rican Americans, are all large, and given, very large,
21 compared to those of, in particular, other large groups,
22 Caucasian Americans and Asian Pacific Island Americans.
23 Q. Let's go to the next slide. What was the next step in
24 your analysis then, your next method of analysis then, Dr.
25 Larntz?
80
1/17/01 - BENCH TRIAL - VOLUME II
1 A. Well, in doing a statistical analysis, that, that was
2 a summary and provided a summary, looking at the
3 information of grade point average and LSAT.
4 Now, I do know and, that there are other factors
5 that the law school says in their policy that they would
6 consider using, and so I want to do an analysis that
7 accounted for those other factors in some way.
8 In particular in statistics, if you can find other
9 factors that change or, again, this is not a statistical
10 term, but washes away, takes away the effect, you want to
11 know if there are other factors that explain the effect.
12 And so, and in the case I have here, I want to look at
13 factors that would, that were in the database.
14 I have to say I have to work with, as best as I
15 can, with objective information that's in the database and
16 I, I chose some of the those factors for a further
17 analysis.
18 Q. Dr. Larntz, could you purport to study all of the
19 factors you thought might be considered in the admissions
20 decision-making process?
21 A. I surely didn't do use all the factors, no. I mean, I
22 didn't have the quantitative information on all of the
23 factors. And I chose the ones I thought that I could get
24 reliable information on and had, had in the data. I'm sure
25 there are other factors. I mean, I don't perfectly explain
81
1/17/01 - BENCH TRIAL - VOLUME II
1 the admissions process. The probabilities that we get are
2 not a hundred percent or zero percent. So I, there's
3 surely other factors, yes.
4 Q. How was it, we've seen some focus here on GPAs and
5 LSAT scores. How was it you came to part of your analysis,
6 at least, on the study of those factors in your relative
7 odds analysis?
8 A. GPAs and LSAT?
9 Q. Yes.
10 A. Well, it was clear from the admissions policies that I
11 looked at that those were credentials that were considered
12 very important, and useful for each, in making a decision.
13 It was clear from looking at the data itself. It was clear
14 from looking at the data itself that four all ethnic
15 groups, that the higher you were, an individual was, with
16 respect to GPAs and LSAT that the better their chance of
17 admission. So LSAT and GPAs were stated criteria, and
18 there's strong statistical evidence that, in fact, both
19 LSAT and GPAs were used in, as a part of the admissions
20 decisions for all ethnic groups.
21 Q. Did you make an assumption that some factors, other
22 than LSAT scores and GPAs were factors in the admissions
23 decision-making process and that could be dispositive
24 factors in the admissions process?
25 A. You mean, were there other factors beyond?
82
1/17/01 - BENCH TRIAL - VOLUME II
1 Q. Did you assume that there were?
2 A. Well, I mean, I don't, I don't know that I assumed
3 there were. I presumed that that might be different,
4 statistically. I didn't assume there were other factors,
5 but I assume, I presume there probably were, yes.
6 Q. Well, if we could take a look at slide 36, can you
7 summarize what information you sought to present there?
8 A. Well, I found, or constructed variables on other
9 factors I thought might be somewhat explanatory with
10 respect to the admissions process. The first one I looked
11 at is Michigan residency, and that's clearly stated in the
12 admissions policy, that Michigan residents should be given
13 preference over other non-residents. And that's clearly
14 stated, and so I wanted to include Michigan residency,
15 control for it in the statistical sense, that is included
16 in the analysis.
17 Now, I don't believe I saw anyplace in the policy
18 where they said that they would use gender as another
19 factor, but my experience in other aspects of doing
20 litigation is that gender sometimes is a factor, whether
21 it's explicit or implicit. And so I decided I would
22 include gender, and that's certainly in, if I might say an
23 aside, in a non-statistical expert role, in my own
24 university, gender was an important factor in some of the
25 decisions that were made with respect to admissions.
83
1/17/01 - BENCH TRIAL - VOLUME II
1 I also was looking in some sense for another
2 factor that would somehow indicate something about the,
3 what do I want to say, economic status of the individuals.
4 And there was a variable in the, in the variable in the
5 database called fee waiver. And I have to admit that I
6 have no justification, other than as title for using that
7 variable. But I thought that I would, individuals that had
8 received a fee waiver in the admissions process, I thought
9 that there might be some, there might be some explanatory
10 natures, if they took advantage of, if they gave any kind
11 of preference to individuals that, that received fee
12 waivers.
13 Also, if you recall the cell grids, they had,
14 well, the LSAT grid was like 161 to 163, and the GPAs was
15 3.25 to 3.49. I mean, their boundaries, that's how Exhibit
16 16 was constructed, and, you know, I said to myself, I have
17 to say, what if one someone were at the upper levels of
18 those, if they were at the higher end of the LSAT and the
19 higher end of grade point, would they make have a better
20 chance than someone at the lower boundaries of those.
21 And so what I did, I included within, within cell
22 GPAs and within cell LSAT, that's, you know, where
23 individuals were relatively within the cells. And I
24 included that as a possible explanatory factor in, in doing
25 that in doing a logistical regression analysis.
84
1/17/01 - BENCH TRIAL - VOLUME II
1 (Whereupon an off-the-record
2 discussion was had.)
3 Q. I think, Dr. Larntz, you were now talking about within
4 cell GPAs and why you did that analysis.
5 A. I think I completed the explanation that I thought in
6 deposition within the grid cell, you know, with the upper
7 boundaries they probably had a slight advantage over
8 individuals that were at the lower boundaries. And that's
9 why I included those. Those are the fine expert factors
10 that I wanted to allow for, allow for in doing the logistic
11 regression analysis.
12 Q. And what conclusions did you draw from your analysis?
13 A. Well, you can see the results here, this is from 1995.
14 Again, with respect to what we're looking at now is the
15 effect of ethnic group, allowing for these other factors,
16 so allowing for whether or not a person was a resident,
17 allowing for whether or not an individual, a female or
18 male, whether they'd received a fee waiver, and whether
19 their relative position was within the cells. What we can
20 see, actually I think it might be instructive just to see
21 what these other effects were.
22 The relative odds for Michigan residency that we
23 estimated was 6.5. That's big. Okay. That's a big
24 relative odds. That's to say that Michigan residents at
25 the same level of these other factors are given an
85
1/17/01 - BENCH TRIAL - VOLUME II
1 advantage. And so that's, and you can see the statistical
2 significance, the standard deviations correspond to ten.
3 So, in fact, the data showed that residency is a big
4 factor. That is a relative odds of 6.5.
5 If we look at being female, in fact, the relative
6 odds there is 1.9. Now, that's -- that's not giant
7 compared to the numbers we're seeing, but that's of the
8 same size I'm used to seeing as far as a factor that would
9 be considered an important factor.
10 So, in fact, it looks to be, at least for 1995,
11 and the statistical significance backs that up, standard
12 deviations apply, a relative odds of 1.9 indicates that, at
13 least the admissions decisions that were made in 1995 for
14 females seemed to be favoring females.
15 And just to say, just to make sure, Your Honor, a
16 relative odds of two would change, say a 50 percent chance
17 to a 66 percent chance. Just, that's a relative odds of
18 two. We go from fifty to 66, actually 66 and two thirds,
19 right. So that's what it would do.
20 For fee waiver, the relative odds turned out to be
21 very close to one and not statistically significant, which
22 indicated that, at least the variable I chose, whatever it
23 indicated, it didn't seem to be taken into account with
24 respect to the admissions statistics made.
25 And the, within cell GPAs and LSAT, these are
86
1/17/01 - BENCH TRIAL - VOLUME II
1 scaled a little differently, so the numbers look small, but
2 they're both on the order of showing some significance,
3 which is to say there's some advantage to being at the
4 upper corners of these cell grids, compared to the lower
5 corners. That's basically what it's saying.
6 So the extra factors I put in, except for fee
7 waiver were explaining, helping explain the, the admissions
8 decisions. And then once we've done that, we also then can
9 see the corresponding estimated relative odds for the
10 ethnic groups that are included in the analysis.
11 And, again, the ones for Native American,
12 African-American, Mexican American and Puerto Rican
13 Americans are all large. They're all large. And compared
14 to those for Caucasian American -- well, Caucasian
15 Americans automatically won because of the baseline but
16 Asian Pacific Island, 1.56, may be an indication of a
17 slight advantage, but not a lot given these other factors
18 and the other groups.
19 Q. So do I understand correctly that then, Dr. Larntz,
20 here what you've done is, in addition to holding constant
21 in the way that you have, GPAs and LSAT when comparing the
22 relative odds of, say, one of the minority groups to one of
23 the majority groups, what you've also done is controlled or
24 held constant these other factors like residency at the
25 same time you're controlling for GPAs and LSAT. Is that
87
1/17/01 - BENCH TRIAL - VOLUME II
1 fair?
2 A. That's correct. It's what I call a statistical
3 control. Instead of having each individual cell grid, we
4 use a statistical variable to negate that control, but
5 that's what we're doing is, the best we can, to hold
6 constant residency gender, fee waiver status and position
7 within the cell grid.
8 Q. Okay. Can we go to slide 37. What was your next step
9 of the analysis here?
10 A. Well, this, this slide just compares the relative odds
11 for 1995 only for, on the left side we're doing the
12 original analysis, just controlling for GPA position and
13 LSAT grid cell. And on the right we're looking at the
14 controlling, after we've controlled for these additional
15 five factors.
16 So what I did is just compare the estimated
17 relative odds in those two cases, and we can see, you can
18 see the numbers that, the numbers there are big in both and
19 they intend to be big in each. The numbers are big in the
20 original analysis, are big in the analysis adjusting for
21 the other factors. In fact, they look a little bigger.
22 You know, that's, the 80, the 61 became 116. 257 became
23 513. 81 became 183. That's for the Mexican American
24 applicants, and Puerto Rican applicants, 37 became 72.
25 They got larger.
88
1/17/01 - BENCH TRIAL - VOLUME II
1 I don't want, I would not say that that's a big
2 change in the way these things go around. They, you know,
3 once we get up to a high number, there isn't a great big
4 effect, but they all look a little larger in this example.
5 Q. Okay. What is the significance of that, the fact that
6 they are larger?
7 A. Well, if, and I didn't test this per se, but if it
8 were consistently larger, that would say that, that when we
9 take account of these additional factors, we take account
10 of residency, we take account of gender, we take account of
11 fee waiver status, although it didn't have much of an
12 affect when we take a account of position within the cell.
13 When we take an account of these additional factors
14 decisions were made if they were consistently larger for
15 the selected minority groups, say, that would say that as
16 we made the, the, the comparison, you know, finer and
17 finer; that is, we got more and more similar credentials,
18 then it looks like ethnicity is actually a larger factor
19 than it was apparent, just from taking a count of the two
20 the GPAs and LSAT.
21 Q. Now, did you do this -- I don't think we have slides
22 for the later years of this analysis, correct?
23 A. I don't think we do. They're all in the reports, and
24 similar, similar -- the nice thing about doing the
25 computing is that once you've programmed it for one year,
89
1/17/01 - BENCH TRIAL - VOLUME II
1 you can do the other years and we did that.
2 Q. And can you tell us whether the conclusions you drew
3 for the later years for 1996 or 2000, whether they're
4 substantially different from the conclusions you've
5 testified here to this morning with respect to 1995?
6 A. No. The conclusion, the conclusions are the same.
7 There's, they, the allowance I see, for, if I call it
8 selected minority status, which is one of the, the Exhibit
9 16 categories for Native American, African-American,
10 Mexican American, Puerto Rican applicants, there's a very,
11 very large allowance given with respect to admissions
12 decisions for individuals with similar credentials.
13 Q. Let's go to the next slide then. Can you tell us what
14 slide -- I can't read it, 38, represents.
15 A. In the admissions policy and in the database there is
16 a reference to an index, which is -- well, that's what it
17 is for 1995. It's an equation. And the, the index, or, I
18 guess I refer to it as a selection index. But the index is
19 a summary measure that, of GPAs, undergraduate GPA and
20 LSAT. So it's a summary measure that is in the database.
21 It's interesting. I guess I wasn't, I wasn't too
22 clever in all my work. I, this formula that we have here,
23 I may have been given that materials, but I didn't find it
24 in the materials, and so I used the database itself to
25 derive this formula and then was quite gratified when the
90
1/17/01 - BENCH TRIAL - VOLUME II
1 interrogatory answer came back and said this is the formula
2 for the index. So I shouldn't be proud of things, but I
3 was kind of proud that I, at least agreed on the index
4 formula.
5 But what the index formula does, if you look at
6 it, it's just an equation, and -- just an equation, he
7 says. Well, he's a statistician. That's why he says just
8 an equation.
9 It's an equation, and the equation, the
10 co-efficients for the equation, the co-efficients for the
11 equation are the key for understanding the equation. The
12 co-efficient for GPAs is .324. So that means this index
13 value goes up .324 for every grade point average up. So
14 someone that had a grade point average of 2.5 compared to
15 someone that had a grade point average of 3.5, their index
16 score would be .324 higher, case.
17 And similarly for LSAT, the co-efficient, it's a
18 decimal, right. It's 0.0320. This decimal says for every
19 point of GPA, then this index score goes up by .03. That
20 doesn't sound, looks a lot, but, in fact, ten points it
21 goes up ten times that, all right.
22 Q. Dr. Larntz, you said GPA. Did you mean LSAT?
23 A. Did I? I'm sorry. I misspoke. I'm surprised I
24 didn't do it earlier. Okay. So with respect to LSAT, ten
25 points of LSAT, an increase of ten points of LSAT would
91
1/17/01 - BENCH TRIAL - VOLUME II
1 increase about the same amount, .30, ten points of LSAT and
2 one point of GPA would each increase this about .32.
3 That's what the formula, that's what the formula does, and
4 so individuals are scored on this, and it's, it was in the
5 database.
6 Q. All right. Apart from the relative odds, the
7 different relative odds analysis you did, did you do some
8 additional kinds of comparative statistical analysis with
9 respect to the issues you were looking at in this case?
10 A. With respect to selection index, I wanted to see, and
11 I, I assumed, and I assumed from admissions documents that
12 it was intended that individuals with large, higher index
13 scores would have a higher probability of admission. And
14 what I did is I tried to calculate what the probability of
15 acceptance for an individual would be as a function of;
16 that is, as index score changed.
17 So what I tried to do, what I did was construct
18 graphical display that showed an estimated probability
19 using a statistical technique that would allow me to get a,
20 call it the maximum likelihood estimate for the probability
21 function of admission versus selection index. What I did,
22 it's just a curve, it's actually a step-function curve that
23 shows us what the probability of admission is versus
24 selection index for each ethnic group.
25 Q. Can we see the next slide, Dwayne.
92
1/17/01 - BENCH TRIAL - VOLUME II
1 THE COURT: Why don't we take a break here.
2 (Whereupon a recess is had.)
3 MS. MASSIE: I've talked to both Mr. Payton and
4 Mr. Kolbo about this. We're going to file a response to
5 the plaintiff's motions in limine, hopefully this
6 afternoon, but maybe tomorrow morning.
7 THE COURT: That's fine.
8 MS. MASSIE: And if we could maybe argue it
9 Friday, we were all thinking --
10 THE COURT: That's fine. I just, you know, didn't
11 want to rush you. I know you're in trial. As soon as you
12 can get it, you know get it to us. Friday is fine.
13 MS. MASSIE: Great. Thanks.
14 MR. KOLBO: Before we jump --
15 THE COURT: Okay. Can we do the lights?
16 MR. KOLBO: Before we jump back into the
17 presentation, I want to cover a couple of housekeeping
18 matters, too. At this time I would offer Exhibit 16, which
19 was the 1995 law school grid.
20 THE COURT: I think. Didn't you already?
21 MR. KOLBO: I think fifteen. Fifteen was --
22 they're very similar. Sixteen today.
23 THE COURT: Any objection?
24 MR. WASHINGTON: No.
25 MR. DELERY: No, Your Honor.
93
1/17/01 - BENCH TRIAL - VOLUME II
1 THE COURT: Received.
2 Q. And before getting back to the power point
3 presentation, Dr. Larntz, apart from the work that you did
4 with respect to the reports that you generated, was there
5 anything else that you were asked to do in connection with
6 your work on this case, that's not reflected in the actual
7 reports themselves?
8 A. Oh, right. I was, there was, I was requested by
9 attorneys to make a selection of cases, I believe,
10 applicant cases in order to request cases from the
11 admissions files, so I, I did that work.
12 Q. If I could have the witness shown Exhibit 120 through
13 125, and if I could explain briefly.
14 MR. KOLBO: Your Honor, I've had a conversation
15 with Mr. Payton about this, or Mr. Goldblatt, I think this
16 morning, and Mr. Delery. Exhibits 120 to 125 have been,
17 basically, reserved in the event that we decide later on to
18 produce to the Court a number of application files that
19 were requested in the course of discovery in this case.
20 There were, approximately, a hundred files for each year,
21 from nineteen, I think 19 -- I'm not sure of the first,
22 perhaps 1995 all the way to 1997. Those files, we have not
23 decided yet whether to offer them. They're not actually in
24 the exhibit books yet.
25 THE COURT: Well, I see. I'm looking. Okay. Go
94
1/17/01 - BENCH TRIAL - VOLUME II
1 on had.
2 MR. KOLBO: I'm going to ask Dr. Larntz to
3 authenticate the documents that were used to select those
4 files in the event.
5 THE COURT: In the event you need it. Okay.
6 MR. KOLBO: Exactly.
7 THE COURT: Just to make a record at this point.
8 MR. KOLBO: Exactly.
9 THE COURT: Fine.
10 MR. KOLBO: My understanding is there's no
11 objection to that.
12 Q. So Dr. Larntz, could you describe what Exhibits 120
13 through 125 or the portions that you have of those
14 exhibits?
15 A. They're lists of numbers. And they, I actually can't
16 verify that these are numbers I selected because I don't
17 remember the numbers of the cases. But they're lists of
18 numbers that are of the form that I gave to indicate random
19 selection, selections I made from accepted minority
20 applicant's files and non-selected majority applicant
21 files.
22 Q. Okay. And did you provide that list to me?
23 A. I provided lists, again, I can't tell you that these
24 are the exact numbers, but I provided lists like this to
25 you.
95
1/17/01 - BENCH TRIAL - VOLUME II
1 Q. Provided a list like that to me?
2 A. Yes.
3 Q. And what do the numbers represent, as far as you know?
4 A. The numbers represented are the ID numbers. I think
5 they're substitute Social Security ID numbers that were in
6 the database for applicant files.
7 Q. They're fictional files that correspond with an
8 applicant?
9 A. That's my understanding, yes.
10 Q. And how did you go about, first of all, deciding to
11 select a hundred files for each of the years? How did you
12 decide upon that number?
13 A. Well, I guess it was a mixture of how many should we
14 look at and how many, how many should we go for. And
15 statistically, I probably said that you needed a
16 representative number in each year. I don't think I did an
17 exact calculation. I think it was a practical number of
18 what would be a reasonable number to select.
19 Q. And how were the -- can you tell me something about
20 the manner in which these hundred files were selected?
21 A. Yes. I can tell you exactly how I did it. In the
22 sense I took a random sample of, of selected minority
23 applicants that, who were accepted. And then I matched, in
24 the sense of chose a candidate from Caucasian or Asian
25 Pacific Island Americans with the same grid cell of LSAT
96
1/17/01 - BENCH TRIAL - VOLUME II
1 and GPA who was not accepted, if that was possible to do.
2 Q. So a hundred files all together, fifty majority
3 students, or fifty students from selected minority groups
4 and fifty students from other racial groups, is that right?
5 A. Yes, fifty, fifty each.
6 Q. Okay. And you do the same thing for each of those
7 years?
8 A. I did it actually for all six years, 1995 through
9 2000.
10 MR. KOLBO: Okay? I would offer then, Your Honor,
11 that portion of Exhibits 120 to 125 that Dr. Larntz has in
12 front of him.
13 THE COURT: Any objection for the limited purpose,
14 should they desire to get those in at a later time?
15 MR. DELERY: No, Your Honor. The exhibit list and
16 the pre-trial order includes a note about how the parties
17 have agreed. Will proceed with the files if we get that
18 far. And subject to that we have no objection.
19 THE COURT: I suspect the same.
20 MS. MASSIE: Neither do I.
21 THE COURT: Very well.
22 Q. Okay. Dr. Larntz. Let's continue. I think you were
23 on slide 39 of your presentation. And I don't think we got
24 to the point of you describing exactly what this, what this
25 kind of a graphic represents, if you could do that with
97
1/17/01 - BENCH TRIAL - VOLUME II
1 using the example of slide 39.
2 A. Yes. If you recall, we put up the formula for a
3 selection index. That was the previous slide. And what
4 we, for 1995. And what we've done here is, on the
5 horizontal axis, we have selection index. On the vertical
6 axis we have probability of acceptance.
7 And what I've done is for each ethnic group, I
8 estimated a, the probability of acceptance as a function of
9 selection index. So as we see down here at the, at the
10 selection index scores that are around two, we'll see that
11 there, that the probability is right at zero. That means
12 individuals with selection indicis that low, actually quite
13 less than 2.3, none of those individuals were admitted, so
14 that that probability of acceptance is zero.
15 The curve, I'm tracing the curve with the asterisk
16 on it. The curve that's estimated, and I say curve, it's a
17 step function. But that actually estimates the probability
18 of acceptance for individuals with the corresponding
19 selection index value, so, for instance, up at the top
20 then, in the curve with the asterisk for selection index
21 values, if I can see it. It's hard to tell. I think
22 that's a value, selection index of about three there.
23 And then from that point on, this is for Native
24 American applicants. Native American applicants with
25 selection indicis; that is a combination of LSAT and GPA
98
1/17/01 - BENCH TRIAL - VOLUME II
1 that corresponded to an index value of three, a hundred
2 percent of those individuals were selected. Now, the
3 values go up, as we see. That's an indication that
4 acceptance probability increases as selection index
5 increases, just as we might expect because selection index
6 is a function of GPA and LSAT.
7 The other curve that's given on the plot, and I've
8 used the same baseline curve. The other curve that's given
9 on the plot, the one without the asterisk, that is the
10 corresponding curve function for Caucasian American
11 applicants. So that's, that's the curve that's to the
12 right. And, and so for Caucasian American applicants,
13 again this angle's hard, but certainly somewhere beyond 3.5
14 or so, then we have a hundred percent selection. And
15 certainly down here, less than selection index of about
16 2.7, 2.8. That's a zero. There's no, no probability. So
17 this curve tries to estimate directly the acceptance
18 probability for each ethnic group.
19 I took the Caucasian American curve on each of
20 these for comparison purposes, so rather than just draw
21 each one and try to compare it, I've drawn the Caucasian
22 American curve on with the Native American curve in this
23 case.
24 And we can see, let's just say if you look at the,
25 at the probability of acceptance of .5, let's just say, you
99
1/17/01 - BENCH TRIAL - VOLUME II
1 could go across and see that that's a value 38.
2 Well, it turns out that if you go down, it's about
3 an index of three for Native American applicants. For
4 Caucasian American applicants it's about 3.2. So this
5 distance here would say that this, this distance here
6 corresponds to the difference, the gaps correspond to the
7 difference in selection index at each probability of
8 acceptance.
9 So this distance here is about, I'd say it's
10 approximately two-tenths of a point in the selection index.
11 And if you recall from what we did last, looked at just
12 before the break, one grade point average is about three
13 tenths of a point or ten LSAT points is about three tenths
14 of a point, so that tells the amount of advantage that's
15 given in terms of, at least those LSAT and GPA points for
16 comparing Native American acceptance probabilities to
17 Caucasian American acceptance probabilities.
18 Q. Can we go to the next slide, please?
19 A. This is the same comparison now done with
20 African-American applicants to Caucasian American
21 applicants. And it has the same structure and the same
22 form. I guess the main substantive difference is that the
23 gap is bigger than it was for the difference between
24 American Indian, for Native American and Caucasian American
25 applicants. And so the gap is bigger, which means that
100
1/17/01 - BENCH TRIAL - VOLUME II
1 with respect to the selection index, we've wound up with
2 similar acceptance probabilities with larger differences in
3 selection index that would correspond to larger differences
4 in GPA and LSAT.
5 You could also look in down here, for instance,
6 say in a value of three, say, you could go across and get
7 the probability, I guess. And again, I can't read it very
8 well, but it looks like it's somewhere between five and ten
9 percent probability for a selection index of three for the
10 chance of Caucasian Americans being accepted. For
11 African-Americans, going up then to three, it looks like
12 the chance is somewhere between 90 and 95 percent.
13 So we actually see our relative odds can be
14 displayed without making any assumptions, at least with
15 respect to selection index. You can see that there's a
16 large gap in the acceptance probability for similar
17 selection index values.
18 Q. When you say you can make, draw some conclusions here
19 without making any assumptions, what do you mean by that?
20 A. Well, these acceptance probability curves, these
21 acceptance probability curves, are computed without making
22 any, any assumption about relative odds, or that there's
23 any, it's just, essentially, a descriptive summary of the,
24 of the, how acceptance probability was a function of
25 selection index for each ethnic group.
101
1/17/01 - BENCH TRIAL - VOLUME II
1 Q. And you did one of these for each one of the ethnic
2 groups for 1995?
3 A. For each ethnic group for 1995 we did the same graph
4 comparing Caucasian American to each other ethnic group.
5 So the next slide, for instance, would be Mexican
6 American applicants. Again we see a gap of the same
7 nature, indicating that for given selection index values,
8 there's a higher probability for Mexican American
9 applicants to be accepted.
10 And the next line I think is other Hispanic
11 American applicants. As, other Hispanic American, just to
12 be clear, all these categories are exactly the categories
13 that were spelled out in the database that was given. So
14 that this categorization was given, we can see -- actually,
15 we see a fair bit of overlap in this one. This one has a
16 fair bit of overlap which indicates that there isn't a
17 great deal of difference in the acceptance probabilities.
18 Q. The database you saw contained a designation for
19 Mexican Americans which we looked at earlier, I think?
20 A. Yes.
21 Q. And then this is a separate one for other Hispanic
22 Americans, and those are designations that you found in the
23 database yourself?
24 A. Those were directly the categories that were in the
25 database in all the years, yes.
102
1/17/01 - BENCH TRIAL - VOLUME II
1 Q. Okay. There was a code, in other words, you were able
2 to identify each applicant by one of those ethnic
3 designations?
4 A. That's right.
5 Q. Okay. Can you tell me what, if one were to apply
6 every combination of LSAT score and, or I should say in the
7 case of these graphics, every selection index for every
8 applicant, would it always be a straight line? Would there
9 be any variation at all along those lines in the positive
10 direction that they're headed?
11 A. Would it always be? Would it always go up and to the
12 right?
13 Q. Yes.
14 A. Well, the technique I used to estimate these, these,
15 the probability function assumes and was estimated under
16 the statistical assumption that it was, that probability of
17 acceptance would be increasing as selection index
18 increases. That's, that's the way the technique was,
19 that's -- this is the best fitting curve under that
20 assumption, and it's clear. There are -- it's clear that
21 for the most part, that's exactly how the decisions were
22 made.
23 Q. There's some variation around that?
24 A. Well, sure, these slides don't go straight out and
25 then straight up. So they aren't using just selection
103
1/17/01 - BENCH TRIAL - VOLUME II
1 index. And in fact, the variation in decisions of course
2 is indicated by the fact that not everyone at a particular
3 value, until you get to a certain point is admitted, and
4 certainly not everyone is denied. So the curves, if only
5 selection index were used, then the curve would go across
6 to one point and go straight up to one and go across, but
7 there's clearly variation, yes.
8 Q. Let's see the next slight then?
9 A. This is the slide comparing Asian Pacific Island
10 Americans to Caucasian Americans, and, well, there's
11 considerable overlap, indicating no particular degree of
12 preference.
13 Q. And slide 44?
14 A. Puerto Rican applicants versus Caucasian American
15 applicants. And again we see considerable separation to
16 indicate that Puerto Rican applicants with the same
17 selection index have higher probabilities of acceptance
18 compared to caucasian applicants with the same selection
19 index value.
20 Q. And I think there's one more slide here?
21 A. I think there are two more, but the next one is for
22 foreign applicants versus Caucasian Americans. This is
23 interesting. Remember the, whatever group we have is with
24 an asterisk. And in this particular case, the asterisk is,
25 at least the top the line of the asterisk is to the right
104
1/17/01 - BENCH TRIAL - VOLUME II
1 instead of to the left of the Caucasian Americans which is
2 an indication of, at least in these higher selection index
3 values, there may be -- it's not that significant, but
4 there may be some preference for Caucasian American
5 applicants compared to foreign applicants.
6 Q. Okay. And one more?
7 A. I think the last one is for unknown. And, well,
8 there's certainly considerable overlap.
9 Q. Okay. Now, again, did you do the same kind of
10 analysis and display the same kind of graphics with respect
11 to the year 1996 through 2000?
12 A. For the years 1996, 1997, 1998, 1999, 2000. There's a
13 plot. There are plots like this for each year, yes.
14 Q. In --
15 A. In my reports, in their respective reports for the
16 analysis of those years.
17 Q. Can you just summarize whether there are any
18 significant or substantive difference in terms of your
19 overall conclusions, with respect to those years compared
20 to 1995?
21 A. With respect to, looking at these plots, it's clear
22 that in all the years there was considerable preference,
23 and allowance made, that is higher acceptance probability
24 for given selection index values for Native American,
25 African-American, Mexican American and Puerto Rican
105
1/17/01 - BENCH TRIAL - VOLUME II
1 applicants, compared to Caucasian American applicants and
2 Asian Pacific Island American applicants and, in fact, the
3 other categories as well.
4 Q. Okay. And did you do any further statistical analysis
5 with respect to the issues that you looked at?
6 A. Yes.
7 Q. Can we go to the next slide? Can you tell us what the
8 next step in your analysis was, the next method of your
9 analysis?
10 A. This, this grid is actually from my, the supplemental
11 report to my analysis, I mean, original analysis.
12 What we've talked about already is what was
13 contained in my original set of reports. This particular
14 grid was constructed to allow comparison of selected
15 minority applicants. And selected minority applicants
16 consisted of, in Exhibit 16 there was a grid for selected
17 minority applicants. And in selected minority applicants
18 in constructing this, there was four groups. Ethnic groups
19 that were included in that were Native American applicants,
20 African-American applicants, Mexican American applicants,
21 and Puerto Rican applicants. And so this is a comparison
22 of selected minority applicants. This is a grid.
23 But broken down, this time it wasn't in the
24 original grid, but broken down by residency, and so this is
25 actually a 1995 Michigan non-residents. This is the grid
106
1/17/01 - BENCH TRIAL - VOLUME II
1 for Michigan non-residents for selected minority
2 applicants.
3 And the grid was constructed in the same way as
4 the grids for Exhibit 16 was before. There were 474
5 applicants, selected minority applicants, in 1995 Michigan
6 non-residents, 142 of whom were admitted.
7 Q. Can I just ask you, with respect to the categorization
8 of selected minority applicants that you found in the
9 database, selected minority applicants did not include in
10 the law school's categorization of a database Asian
11 Americans?
12 A. That's correct.
13 Q. Did it include the category that's also identified
14 there as, other Hispanic?
15 A. Well, I had the Exhibit 16 for 1995 and that, they
16 were not included in that. In order to replicate, let me
17 just say, in order to replicate the results, it was clear
18 that the four ethnic groups that I named were the ones that
19 corresponded to selected minority applicants.
20 Q. Okay. Any further information from this slide that
21 needs to be summarized?
22 A. Well, they're just in the same way. There's other
23 grids, except now we're looking at non-residents and not
24 just, not combining the two, so for instance, in this we
25 highlighted the cell, just for comparison, 3.5 to 3.74 GPA,
107
1/17/01 - BENCH TRIAL - VOLUME II
1 154, 155. And in this case, selected minority applicants
2 there were eight of eight selected minority applicants,
3 four of whom where admitted.
4 Q. Next slide?
5 A. We did the same thing for comparison purposes. For
6 comparison purposes we did the same thing for non-resident
7 majority and non-selected minority applicants. That's
8 everybody else, essentially. There were 3,008
9 applications, 795 were admitted. And in the cell that we
10 were looking, at the highlighted cell, there were forty
11 applicants, three of whom were admitted.
12 Q. Next slide, Dwayne?
13 A. And this just compares side by side those two grid
14 cells, and so we can bring those up just for comparison
15 purposes. We could compute odds ratios and relative odds
16 for these cells, if we so desire. Okay.
17 Q. And is this the next slide?
18 A. Right. And what we're doing here is in this next
19 slide is we just wanted to highlight other cells that we
20 just, we picked out those. In fact, these two cells that I
21 originally talked about are in my report. And so just to
22 show you other cell combinations. For instance, the 3.5 to
23 3.74 GPA and LSAT of 161 to 163, among selected minority
24 applicants, eleven out of twelve were admitted, compared to
25 other applicants was twelve out of 207.
108
1/17/01 - BENCH TRIAL - VOLUME II
1 And similarly, for the other, for the other cell
2 comparison, I think it's clear, fourteen out of sixteen, in
3 this particular cell. I'm not going to read the
4 designation, compared to one out of 57. And similarly, in
5 this other one, nine out of nine compared to one out of 51.
6 This is, this is a, just highlighting the, if we, no matter
7 where we pick the cells, at least in the range where there
8 are decisions being made, we can do comparisons like this.
9 Q. Next slide, Dwayne? And what does this slide
10 summarize?
11 A. Well, we've seen, we've seen already that residency is
12 a factor in admission, and it's a specified factor. And
13 Michigan residents are supposed to, and do receive
14 preference with respect to law school admission.
15 And so this particular slide actually looks at
16 1995 selected minority non-residents; that is individuals
17 who wouldn't receive the residential preference and
18 compares them to other, that is majority non-selected
19 minority resident applicants to see how they, how resident
20 majority applicants compare to non-resident minority
21 applicants. And so these grids, we can look at that same
22 cell. It was four out of eight for selected minority
23 non-residents. Among residents who are not selected
24 minorities, it turned out in that figure cell to be zero
25 out of twelve accepted.
109
1/17/01 - BENCH TRIAL - VOLUME II
1 Q. All right. Dwayne are there some more examples there,
2 yes.
3 A. Yes. And we've highlighted four other cell
4 comparisons the same cells. And so for instance in this
5 particular cell, which is, there were eleven out of twelve
6 selected minority non-residents accepted compared to seven
7 out of twenty-four majority residents, residents who were
8 accepted. And similarly the comparisons here, sixteen,
9 fourteen out of sixteen versus zero out of eighteen, nine
10 out of nine versus two out of thirteen.
11 Q. And again one could compute if we took the time of
12 that relative odds of these different cells?
13 A. Yes, we could compute the relative odds, yes.
14 Q. Could we go to the next slide. What was the next
15 method of your analysis, Dr. Larntz, in comparing the
16 relative odds?
17 A. Well, in a, in a report that was critical of my
18 original report, and there was one. The, it was mentioned
19 that, that it would be inappropriate to combine the
20 relative odds for the individual cells into a combined or
21 composite relative odds. And so the reason I actually did
22 this, and in that report the categories compared were
23 selected minority and versus majority of non-selected
24 minority, in exactly the same way we're using here.
25 And I was criticized, and the report said that, in
110
1/17/01 - BENCH TRIAL - VOLUME II
1 fact, the relative odds would vary, would change as a
2 function of LSAT and GPA, that, in fact, there would be
3 some shift. And so what I did was I actually went cell by
4 cell in this analysis, and for each cell, each combination,
5 for instance, that cell we were looking at the highlight to
6 begin with, the 3.5 to 3.74, non-residents to 1995 with
7 LSAT of 154 to 155, there were eight minority applicants
8 and four were admitted.
9 There were forty majority applicants, three of
10 whom where admitted. And so I just computed all the odds
11 ratios or relative odds. In fact, the odds ratio there is
12 12.33. That's if we did the math. And so I computed
13 those, because I wanted to see, I wanted to see if there
14 was a pattern, a pattern, that showed that this odds ratio
15 was different as a function of LSAT score and GPA.
16 And what I found when I did in my summaries here,
17 and is that, well, a lot of these are small cells, so we're
18 going to get infinities and other numbers like that because
19 we're dealing with small numbers. But there was no
20 consistent pattern. And so I wanted, I wanted to convince
21 myself that there was no consistent pattern, and there
22 wasn't any consistent pattern.
23 In addition, I have to say, statisticians faced
24 with data like this will immediately jump to do a test of
25 significance to see whether or not there, it could have
111
1/17/01 - BENCH TRIAL - VOLUME II
1 arisen by chance that this discrepancy, even these small
2 cells. I didn't originally do any tests with respect to
3 the individual cells.
4 But I, I, when I was looking at this, I decided,
5 well, why don't I see if there's a statistically
6 significant difference between the admission rates, even in
7 these cells, these small cells. I didn't expect to find a
8 lot, but are there statistically significant differences
9 here. And so I reported the P value, the value that we
10 talked about before. What's the chance that this
11 difference, this odds ratio would have arisen, that one or
12 a bigger one, if chance alone where operating.
13 And in this particular cell would be highlighted,
14 the P value is .01. So that would be considered
15 statistically significant evidence of higher probabilities
16 of acceptance, even for, for minority applicants, even for
17 this cell alone. So this P value summarizes the
18 information for this cell alone, this small cell, and, and,
19 frankly, I was surprised to get a series of significant P
20 values in this particular case because the cells are small.
21 Q. Okay. Can we go to the next slide then?
22 A. And what I did, what we've got here, the slides that
23 we're reporting here is we're reporting all the odds ratios
24 for all the cells that we have. I notice some of them are
25 blank. Some of them are blank down here. And those are
112
1/17/01 - BENCH TRIAL - VOLUME II
1 cases where we have -- well, in this particular one, one
2 minority applicant and one majority applicant, but no one
3 was admitted, so there's no comparative information there.
4 For every cell that has comparative information we
5 compute the odds ratios and the corresponding P values, and
6 there you can see, there are, you know, not everyone is
7 significant, but ones that are statistically significant,
8 at the five percent level of significance, we, I put in
9 bold. So I think we have more pages of these in the slide,
10 don't we, for LSAT, 159 and 160 non-residents for 1995.
11 And we probably have some more slides of these, which show
12 the same, the same kind of patterns, and in the reports we
13 have table after table of this for 1995, 1996, 1997, 1998,
14 1999, 2000 for both residents and non-residents.
15 Q. Can you just summarize what conclusions you drew,
16 based upon the cell by cell analysis that you did for each
17 of these years?
18 A. Well, what I did was I actually just did a very simple
19 thing. And to me it was simple. I just looked at the
20 number of these comparisons that are statistically
21 significant. So I just looked to see. I didn't expect to
22 find very many, but I looked at the number, and for each
23 year it turned out to be -- well, I don't remember the
24 exact numbers, year by year, but between 17 and 22 of the
25 comparisons were actually statistically significant, even
113
1/17/01 - BENCH TRIAL - VOLUME II
1 though we're working with small cells, okay. So I looked
2 at those. And, and then what I did is I looked at the odds
3 ratios associated with those that were statistically
4 significant.
5 And I looked at the odds ratios for each of those
6 that were significant. And it turned out in every year for
7 every comparison that it was statistically significant, if,
8 if the decisions were made, if they were being made so that
9 there were, approximately equal probabilities of acceptance
10 we'd exceed, about half of the odds ratio would be small,
11 less than one, about half would be greater than one. But
12 in this case every single cell, every single cell that
13 showed statistical significance, every cell-by-cell
14 comparison that showed significance showed preference for
15 minority over majority applicants.
16 Q. Is that the last slide there?
17 A. I think we might have more of the same.
18 Q. This is just illustrative for the, for 1995?
19 A. Yes. And as I said, these are done for every other
20 year, and every combination of residents and non-residents.
21 Q. From all of the analysis that you've done, Dr. Larntz,
22 and you've shown us examples of here this morning, and
23 based on all the information that you looked at, in
24 connection with the work that you were asked to do, could
25 you provide us your overall conclusions and summary with
114
1/17/01 - BENCH TRIAL - VOLUME II
1 respect to what your findings were?
2 A. Yes. What I've concluded in looking at all of the
3 analysis I've done, and looking at applicants with similar
4 credentials is, that is GPA and LSAT scores that are
5 approximately the same, residents, or not.
6 What I found is that there is, how do I say this
7 in a non-statistical way; there's an incredibly large
8 allowance given to selected minority applicants,
9 particularly African-American, Native American, Mexican
10 American and Puerto Rican applicants are given an
11 incredibly large allowance when it comes to admissions
12 decisions for individuals with similar credentials compared
13 to, in particular, Caucasian Americans and Asian Pacific
14 Island Americans. So there's a very, very large preference
15 given.
16 And to be honest, you can see that from the
17 original grids. I mean, the original grids show that
18 preference if you look at the cell by cell. You can look
19 individually at the grids and see exactly the same effect.
20 My statistical analysis basically has tried to quantify
21 what was in those grids, but, in fact, the original grids
22 show that exactly the same conclusion.
23 MR. KOLBO: I have nothing further, Your Honor. I
24 would offer Exhibit 143 at this time.
25 THE COURT: Any objections?
115
1/17/01 - BENCH TRIAL - VOLUME II
1 MR. DELERY: What is exhibit 143?
2 MR. KOLBO: That's the policy.
3 MR. DELERY: No objection.
4 THE COURT: Received. Why don't we break for
5 lunch and then we'll give you an opportunity to wait. I
6 will be happy to go now. You know how you're all excited
7 to start your cross examination. I would be if I were you,
8 but let's break for lunch and then we'll reconvene at, by
9 1:20, 1:30.
10 (Whereupon a recess is had.)
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
116
1 _ _ _
2 (Court back in session.)
3 THE COURT: We've got everybody back,
4 we waiting for anybody.
5 MR. DELERY: I don't think so, your
6 Honor.
7 THE COURT: Doctor, you can take the
8 stand if you would like to. You've obviously been
9 in court before, you know the next thing is
10 cross-examination. Not always as easy as direct.
11 A. Yes, sir.
12
13 CROSS-EXAMINATION
14 BY MR. DELERY:
15 Q. Good afternoon, Dr. Larntz?
16 A. Good afternoon.
17 Q. We have met before, isn't that right?
18 A. That's correct.
19 Q. I took your deposition in this case back in February
20 of 1999? A long time ago?
21 A. I'm sure that's right.
22 Q. You said this morning, I believe that the purpose of
23 your analysis was to look at the role that race
24 plays in admission to the Law School, is that a fair
25 statement of your approach?
GRUTTER -vs- BOLLINGER, ET AL
117
1 A. Role in a statistical sense, yes.
2 Q. In your view, is that the same as extent to which
3 race is considered in admission?
4 A. Well, I'm not sure I understand an awful lot of
5 difference. What I'm saying is that what I did was
6 try to understand and describe Admissions decisions
7 and understand to the best that I could, I call it
8 role.
9 And I'm not sure, I guess I should be
10 an English major to understand the extent of
11 difference.
12 Q. The reason I ask is, you maybe you haven't been told
13 this, it's one of the questions the Court has put to
14 the parties for trial here, is the extent to which
15 race is used in the Law School admission process,
16 are you aware of that?
17 A. I wasn't aware of the specific questions, no.
18 Q. All right. You believe that your analysis
19 quantifies the role that race plays in the
20 Admissions process, is that right?
21 A. That's exactly what I think I did, was to try to
22 describe the Admissions decisions, and the role in a
23 statistical sense, and again I'm doing this in a
24 statistical sense.
25 The statistical sense describe the
GRUTTER -vs- BOLLINGER, ET AL
118
1 effect when called a statistical purpose calls the
2 effect of race on the chance of admission, that's
3 right.
4 Q. Okay. You said that's what you tried to do. In
5 your view, in your odds ratio analysis actually does
6 that?
7 A. I think it does a good job of summarizing the data
8 that we have.
9 Q. You also said this morning that one of the things
10 that you tried to do is analyze how likely it is the
11 results that you found would occur by chance, is
12 that right?
13 A. With respect to particular odds ratios, and I
14 calculated P values for numbers of standard
15 negations. And they specifically addressed--the
16 question is, how likely is this outcome that we see
17 compared to an outcome that was due to chance alone.
18 Q. That was the purpose of your inferential analysis?
19 A. That certainly summarizes what is given by the
20 standard negations.
21 Q. You're aware that the Law School has a written
22 admissions policy, is that right?
23 A. I think I was given a policy, yes.
24 Q. And you're aware that that policy says that race is
25 considered as a factor in the Admissions process,
GRUTTER -vs- BOLLINGER, ET AL
119
1 correct?
2 A. That's correct.
3 Q. So the policy itself indicates that these results
4 are not occurring by chance?
5 A. The policy indicates that these results are not
6 intended to be occurring by chance.
7 Q. So, all you've done is, in this respect, is confirm
8 that the policy is actually having the effect that
9 it says that it intends to have?
10 A. In the sense that race is a factor and if it was
11 intended to be that, then certainly my analysis does
12 show that race is a factor.
13 Q. I believe your reports, certainly your first report,
14 says that the odds ratio analysis--let me rephrase
15 this.
16 The odds ratio was the main tool for
17 comparison in your analysis, is that a fair
18 statement?
19 A. I think the odds ratio is a tool for comparison. It
20 summarizes the individual's odds ratio, yes.
21 Q. You believe that your odds ratios give an accurate
22 picture of the role that race is playing in
23 admission?
24 A. With respect to the individuals with similar
25 credentials, GPA and LSAT and residency and gender
GRUTTER -vs- BOLLINGER, ET AL
120
1 and fee waiver status, I think the odds ratio gives
2 a good composite measure which summarizes the extent
3 to which--extent, I use your word, to which race is
4 being used, yes.
5 Q. Is it your opinion that the odds ratio does not give
6 an accurate picture for some group of students other
7 than the ones that you just mentioned?
8 A. No.
9 Q. So, for all students in the applicant pool, you
10 believe that your odds ratios are giving an accurate
11 picture of the role that race is playing in
12 admission?
13 A. I'd better make sure I understand the question. Are
14 you saying for each individual does the odds ratio
15 apply? I'm trying to understand, and I really do
16 understand what the question is.
17 Q. Okay. Let's take the individual first. Is it your
18 view that the odds ratios that you reported give an
19 accurate view of any particular individuals relative
20 on to admissions?
21 A. The odds ratio is a summary of what happens to a
22 group of individuals. And individuals themselves
23 will, in fact, have all kinds of factors and affect
24 with respect to them. But it summarizes for the
25 group as a whole what the affect was.
GRUTTER -vs- BOLLINGER, ET AL
121
1 Q. And as a summary statistic, you believe that it's a
2 fair and accurate picture of the role that race is
3 playing for all applicants, as opposed to only some
4 subset of the applicants, that's what I'm asking?
5 A. The analysis I did was for the group of all
6 applicants. And so I think it does summarize the
7 measure of the role of race for that group of
8 applicants, yes.
9 Q. Now, this morning Judge Friedman asked
10 whether--actually if we could put up slide 30 from
11 this morning. I think that's what was up when the
12 question came.
13 This morning Judge Friedman asked
14 whether this 81 odds ratio for Mexican Americans
15 here, meant that Mexican Americans were 81 times
16 more likely to be admitted than white students, do
17 you remember that discussion?
18 A. Yes.
19 Q. And, in fact, the answer is no, that does not mean
20 that the Mexican Americans are 81 times more likely
21 to be admitted than the white students, isn't that
22 correct?
23 A. If we mean times likely in terms of 81 times
24 probability if that's what you're meaning, then
25 that's correct. It's, in fact, an odds multiplier
GRUTTER -vs- BOLLINGER, ET AL
122
1 and so it, in fact, multiplies the odds depending on
2 what the base line odds would be.
3 Q. Okay. Let's go back to this, your drawing from this
4 morning, if we could. I think you said that this 81
5 for Mexican Americans approximates basically the
6 lower half here of what you sketched out, is that
7 right?
8 A. That's correct. 81, yes.
9 Q. So, the 81 odds ratios translates to a ratio of
10 probabilities of .9 to .1, is that right?
11 A. Ten percent probability using an odds ratio of 81
12 would become 91 probability, that's true.
13 Q. So, to plus that information into the sentence that
14 we used before, for these numbers we would say that
15 the group with the 90 percent probability of
16 admission is nine times as likely to be admitted as
17 the group with the ten percent probability, is that
18 right?
19 A. If you're using it in terms of probability, that's
20 in terms of probability.
21 Q. Okay. So, the likely language refers to
22 probability, in your view, as commonly used by
23 statisticians?
24 A. No, as commonly used by statisticians, statisticians
25 will work in terms of odds. If I might say, I'm
GRUTTER -vs- BOLLINGER, ET AL
123
1 sorry. With respect to analysis of binary response
2 we work in terms of odds and odd multipliers or odds
3 ratios.
4 Q. You work in terms of odds, but let me go back to the
5 sentence earlier. Is it fair to say that this
6 example here with the 81 odds ratio, indicates that
7 the group with the 90 percent probability is nine
8 times as likely to be admitted, say, the group with
9 the ten percent probability?
10 A. What I would say it's nine times the probability,
11 that's what I would say. We particularly don't use
12 likely, because it's subject to all kinds of
13 misinterpretation.
14 Q. All right. And just to take another example, the
15 one that you have up here at the top half, the 75
16 percent chance or 75 percent probability versus 25
17 percent, the odds ratio comes out to nine.
18 But we would say that the probability
19 is only three times greater for the group of 75
20 percent chance, isn't that right?
21 A. In terms of probability, that's right.
22 Q. Okay. I think I just want to get this clear,
23 because as you say there's room for
24 misinterpretation. And you would agree that we
25 should try to be precise in the language that we
GRUTTER -vs- BOLLINGER, ET AL
124
1 use.
2 So when we're talking about odds, we
3 should use odds type of language?
4 A. As best I can, I will try to be specific as that,
5 yes.
6 Q. Let me do another example just to get a sense of the
7 relationship between probabilities and odds. If one
8 group has a .99 probability of getting admitted, the
9 99 percent, and a second group has a 90 percent
10 probability of getting admitted, so .99 versus .90.
11 Am I correct that the odds ratio for
12 that is eleven, or about eleven?
13 A. Well, we can do the math.
14 Q. Let me get a pen here. .99 versus .90, let me just
15 make sure I have this right. The formula would be
16 for the odds ratio .99 over .01 all divided by .90
17 over .10, is that right?
18 A. That looks good.
19 Q. Okay. And this comes out to about eleven, doesn't
20 it? You can check me with your calculator, if you
21 like?
22 A. I think eleven is a good number.
23 Q. So, if we get an odds ratio of eleven even though
24 the probabilities are very close, right?
25 A. The probability of acceptance is close, and the
GRUTTER -vs- BOLLINGER, ET AL
125
1 probability of denial is quite different.
2 Q. The probability of acceptance is--let me just put it
3 this way. Both groups are highly likely to be
4 admitted in that case?
5 A. Highly likely to be admitted?
6 Q. Yes. .90 versus .99?
7 A. Yes.
8 Q. Okay. And yet you end up with an odds ratio of
9 eleven?
10 A. That's true.
11 Q. When you indicated that earlier two or three in your
12 experience is a very large odds ratio?
13 A. Yes. And if the Court permits I can explain. The
14 reason, of course, is that you also can look at the
15 chance of denial. And the chances of being denied
16 admission for these are one percent versus ten
17 percent.
18 And so there's a symmetry with
19 respect to that, and that's why we use odds in
20 statistics. And so if you look at the chances of
21 denial, it's one percent versus ten percent which is
22 quite discrepant.
23 If we were in a situation in a
24 medical study, a ten percent complication is greater
25 than one percent would be quite big.
GRUTTER -vs- BOLLINGER, ET AL
126
1 Q. Am I correct that your opinions concerning how large
2 a particular odds ratio number is, is based on your
3 experience with this technique, correct?
4 A. That's correct.
5 Q. So, if in your experience you encounter, and the
6 kinds of work that you do, you encounter odds ratios
7 of two or three in contacts where that would be
8 large, that forms the basis for your opinion about
9 what constitutes a large odds ratio?
10 A. It certainly informs from my experience as a
11 statistician over the period of time I've been a
12 statistician.
13 Q. Okay. Am I correct that none of your prior
14 publications relate to work on issues on higher
15 education?
16 A. I don't have my--in front of me, but I don't recall
17 any particular applications of higher education in
18 my publication list.
19 Q. And before your work on this case, you had never
20 worked with admissions data?
21 A. In a litigation setting, that is true.
22 Q. In a statistical setting?
23 A. As examples for classes and things like that, sure
24 we have used them.
25 Q. Before your work in this case, you had never
GRUTTER -vs- BOLLINGER, ET AL
127
1 designed regression models concerning professional
2 school admissions, is that right?
3 A. That's correct.
4 Q. And you had never given expert testimony on
5 Admissions in any other content?
6 A. That's correct.
7 Q. So, your approach to this case and your opinions
8 about the sizes of the effects that you found, based
9 on your experience which did not include experience
10 with admissions data?
11 A. I think I said I hadn't had experience with
12 Admissions data with respect to working in the
13 litigation setting, or with respect to
14 administrative work.
15 I've certainly worked at Admissions
16 data with respect to examples that we used in
17 classes and so on that. So, I certainly have seen
18 Admissions data examples, and I have looked at those
19 before, yes.
20 Q. But not in litigation, not in this context?
21 A. Not in litigation context, that's true.
22 Q. If you look back at the slide, which I think again
23 is 30 from your presentation this morning, the odds
24 ratio for African Americans there is given as
25 257.93. I just want to get a sense of what that
GRUTTER -vs- BOLLINGER, ET AL
128
1 means, going back to our earlier discussion.
2 Am I correct that you're not saying
3 that any African American student has 257 times the
4 odds of admission than any white student?
5 A. On an individual basis?
6 Q. Yes.
7 A. No, this is on an aggregate basis for the whole
8 group.
9 Q. And that has to be the case, right, because not all
10 of the African American students were admitted?
11 A. If they were all admitted it would have been
12 infinity.
13 Q. And if close to all of them were admitted--well, let
14 me put it this way.
15 You agree that the Law School only
16 admits about a third of all of its students, of all
17 of its applicants, right? Somewhere in that range?
18 A. We saw numbers this morning, yes.
19 Q. And, in fact, you understand that the proportion of
20 the so-called majority students who were admitted,
21 is actually larger than the proportion of minority
22 students that are admitted, am I correct?
23 A. I don't have the numbers in front of me.
24 Q. You don't know one way or another?
25 A. I don't know one way or the other. Specifically I
GRUTTER -vs- BOLLINGER, ET AL
129
1 don't recall.
2 Q. When you were approaching your work in this case,
3 did you consider those basic probabilities of
4 admission in evaluating your results?
5 A. Did I consider those basic probabilities of
6 admission? I considered looking at comparing
7 individuals with similar credentials, and that's
8 what I would do as a statistician.
9 Q. But you didn't use the overall probability of
10 admission for majority and minority students as a
11 check on the reasonableness of your odds ratio
12 estimate?
13 A. I think I--I didn't do it, I would only do that if
14 they had similar credentials.
15 Q. But as a general matter for the applicant pool as a
16 whole, you did not do that, you didn't use the
17 applicant admission?
18 A. I did not.
19 Q. Now, you have mentioned a couple of times, and
20 certainly many times this morning, the idea that you
21 wanted to look at applicants with similar
22 credentials.
23 Is it fair to say that that was one
24 of the basic principals of your analysis, you try to
25 identify the students with similar credentials and
GRUTTER -vs- BOLLINGER, ET AL
130
1 compare those?
2 A. I think I took as a basic principal of my analysis
3 that I would use, the groupings as set by the
4 Law School itself to define groups like that, and
5 that's where I started from.
6 Q. And when you say groupings as defined by the
7 Law School itself, you mean in the grids in
8 Exhibit 16 that you were given?
9 A. That's correct.
10 Q. You didn't get those groupings from any other
11 source?
12 A. The groupings came directly out of Exhibit 16.
13 Q. For example, you didn't review the deposition
14 testimony of the Admissions officers who actually
15 makes the decision when deciding on the structure
16 for analysis, did you?
17 A. I don't recall such review.
18 Q. And when you use the term credentials here today,
19 you generally mean GPA and LSAT scores, isn't that
20 right?
21 A. For the most of our analysis GPA, LSAT. In some
22 sense we did analysis involving residents, gender,
23 fee waiver.
24 Q. Did you consider residents or gender or fee waiver
25 to be credentials when you used the term?
GRUTTER -vs- BOLLINGER, ET AL
131
1 A. They're characteristics of the applicants.
2 Q. But when you talk about credentials, you're talking
3 about grades and test scores, isn't that right?
4 A. That's what I'm talking about.
5 Q. And you chose those credentials, to use your term,
6 because that's the data you got, right?
7 A. That certainly was the data I have.
8 Q. You didn't have statistical data on the quality of
9 the essays, or the quality of the letter of
10 recommendation and so forth?
11 A. That's correct.
12 Q. But you have examined the Law School's admission
13 policy, right?
14 A. Yes.
15 Q. So, you understand that the policy contemplates
16 consideration of many factors, other than grades and
17 test scores?
18 A. Of course.
19 Q. It considered things like the essays and letters of
20 recommendation and strength of curriculum and the
21 like?
22 A. Of course.
23 Q. Those factors while they're considered by the
24 Admissions Office, are not included in your model,
25 is that right?
GRUTTER -vs- BOLLINGER, ET AL
132
1 A. I didn't have the data, I didn't include it in my
2 model, that's correct.
3 Q. I would like to look now at your odds ratio analysis
4 in a little bit more detail for some background.
5 Am I right that you you basically
6 did--you computed two kinds of odds ratios. You
7 computed them cell by cell, and then you computed
8 what you called a composite odds ratio, isn't that
9 right?
10 A. Certainly I did that in various parts of the
11 reports, yes.
12 Q. I want to look at the cell by cell part first where
13 you ended this morning.
14 You computed odds ratios for, I
15 guess, for applicants in small groupings of grades
16 and test scores, right?
17 A. That's correct.
18 Q. In fact, you broke the applicant pool down into a
19 total of 240 cells for each year, isn't that right?
20 A. I believe that's correct. I would have to go back
21 and count, but I believe that's correct.
22 Q. Well, we counted. And assuming that we counted
23 correctly, it's 240.
24 A. I believe you.
25 Q. Okay. So, then you computed an odds ratio for each
GRUTTER -vs- BOLLINGER, ET AL
133
1 of those individual cells?
2 A. For each cell for which there was comparative
3 information, that's true.
4 Q. Okay. And you excluded some of the cells from your
5 analysis, is that right?
6 A. I think I just said what I did. Which was to
7 compute an odds ratio from each cell that had
8 comparative information.
9 Q. And so in your view a cell that had no applicants at
10 all in it, didn't have comparative information,
11 right?
12 A. I would agree with that.
13 Q. And in your view, cells that had only applicants,
14 only majority applicants, for example, but no
15 minority applicants, that cell didn't have
16 comparative information?
17 A. That's true.
18 Q. Okay. But you also excluded cells where all of the
19 applicants in both groups were admitted, isn't that
20 right?
21 A. That's true.
22 Q. And you excluded the cells where none of the
23 applicants in either group were admitted?
24 A. That's true.
25 Q. Because in your view, those cells don't have
GRUTTER -vs- BOLLINGER, ET AL
134
1 comparative information?
2 A. Based on statistical principals of computing odds
3 ratios and getting comparative information and using
4 standard techniques, that's what we would do.
5 Q. So, from your statistical point of view, they don't
6 have comparative information?
7 A. They don't have comparative information.
8 Q. You'll agree though that one could look at the two
9 cells and compare them, right?
10 A. They're put out in my reports, one can certainly
11 look at them.
12 Q. And it is a form of comparison to say that all of
13 the applicants in both groups with admitted or
14 denied?
15 A. It's a form of comparison.
16 Q. But it didn't fit within the requirements of your
17 statistical approach, right?
18 A. They didn't fit in the requirements with respect to
19 getting a composite odds ratio, because they give no
20 comparative information with respect to the odds
21 ratio.
22 Q. And so because they didn't fit in your approach, you
23 set them aside?
24 A. Because they didn't give comparative information
25 with respect to the odds ratio, they were not
GRUTTER -vs- BOLLINGER, ET AL
135
1 computed.
2 Q. Okay. I'd like for you, if you would, to look at
3 Exhibit 138.
4 MR. DELERY: I believe it's in binder
5 five, your Honor.
6 BY MR. DELERY:
7 Q. It's one of the reports that you introduced this
8 morning, I believe that's your February 21, 2000
9 report.
10 So, the reason, just to clarify
11 something from this morning, the reason that there
12 are a series of reports, is that because the case
13 has been pending for a while we've passed through
14 several additional admission seasons, right?
15 A. That's correct.
16 Q. And as the data became available it was provided to
17 you?
18 A. I don't know that, I know I got the data.
19 Q. I'd like for you, if you would, to turn to the last
20 series of tables here in the report that have the
21 heading Cell by Cell Comparison and Admission
22 Rating.
23 There's 24 pages of them and I would
24 like for you to turn to the one that says page one
25 on 24 at the bottom.
GRUTTER -vs- BOLLINGER, ET AL
136
1 A. I have that.
2 Q. Do you have that, okay. Pages one through six of 24
3 are the 240 cells for 1995, is that right?
4 A. That's correct.
5 Q. And the slides that we saw at the end of your
6 testimony this morning are some of these cells from
7 these pages, did I have that right?
8 A. That's correct.
9 Q. Would you agree that for the vast majority of cells
10 on these pages, you don't calculate any odds ratio?
11 A. I don't know about vast majority, but there
12 certainly are a good number of cells with a
13 calculation of odds ratios.
14 Q. In fact, you only calculated numbers, numerical odds
15 ratios in twelve cells?
16 A. On the six pages?
17 Q. Yes.
18 A. Numerical values, you mean you're not--you're
19 assuming, you're saying infinity doesn't count.
20 Q. Okay.
21 A. If you're saying infinity doesn't count, then that
22 may be true. That there are twelve that are
23 infinity, there are a good number that are infinity.
24 Q. Okay. Let's just look at the first page and we'll
25 turn further if we have to. On the first page, all
GRUTTER -vs- BOLLINGER, ET AL
137
1 of the cells are blank except for four near the
2 bottom, right?
3 A. That's correct.
4 Q. So you calculated four odds ratios on this page and
5 all four were infinities?
6 A. That's correct.
7 Q. Okay. And looking at the first of the infinity
8 cells, you've got one minority applicant who was
9 admitted and ten majority applicants one of whom was
10 admitted?
11 A. That's correct.
12 Q. And so those probabilities of admission gives you
13 this infinity odds ratio?
14 A. The observed odds ratio is infinity, that's correct.
15 Q. That's not the same as saying that the minority
16 student were infinitely more likely to get in than
17 the majority student, is it?
18 A. That's saying this particular cell the individuals
19 that we saw, that's the observed odds ratio.
20 Q. But it's not the same as saying that in this cell a
21 minority student was infinitely more likely to get
22 in than a majority student, is it?
23 A. I don't believe that's true, and statistically I
24 think I've said that before. These infinities
25 represent the observed information. I believe that
GRUTTER -vs- BOLLINGER, ET AL
138
1 if we had many more students in these cells, we
2 would not get infinity.
3 Q. So, in some sense the odds ratio what you find are a
4 function of the small pool sizes in a lot of these
5 cells, particularly for minority students?
6 A. I said these are small pool sizes, that's absolutely
7 true. And the infinities are a result of that, yes.
8 Q. Okay. Just so we're clear. The reason that you get
9 infinity, which seems like a big number, is just
10 that you're dividing by zero and that's a
11 mathematical construct?
12 A. In this case the odds of minority admit are
13 infinity, and that's a one divided by zero. And the
14 odds of majority admit are one in nine, that's
15 right.
16 Q. But these infinities are, when we use the term
17 infinity, it's a mathematical construct, it's not an
18 actual infinite likelihood of getting in, right?
19 A. I don't think such exist.
20 Q. I am just trying to make sure that we're all on the
21 same page. Let's turn to the second page.
22 Page two you've got nine infinite
23 odds ratios, right?
24 A. That's correct.
25 Q. And then you got one for which you can actually
GRUTTER -vs- BOLLINGER, ET AL
139
1 calculate the odds ratio, near the bottom 4.33?
2 A. I think I would say I got one for which you can give
3 a value other than infinity. Yes, that's true.
4 Q. Do you view the infinite odds ratios as computable
5 odds ratios.
6 A. They certainly are computable, yes. They are the
7 observed odds ratio.
8 Q. And mathematically, you think that that's
9 computable?
10 A. Are we into the foundations of mathematics now? It
11 certainly add a representation of the numbers that
12 are seen. There certainly is comparative
13 information there, and infinity is certainly a
14 representation of that comparative information.
15 Q. Where you have a cell where all of the applicants
16 are admitted in both groups?
17 A. Yes.
18 Q. Do you believe that that is a computable odds ratio?
19 A. Where all are admitted?
20 Q. Yes.
21 A. In fact, in that case you're going to wind up
22 infinity divided by infinity if we're doing the
23 mathematics. And I would say that that's not
24 something that we could compute, because there's no
25 comparative information there.
GRUTTER -vs- BOLLINGER, ET AL
140
1 Q. I just want to understand the difference between
2 that what you say is not computable, and a result of
3 infinity which is computable?
4 A. I think the important thing is that we computed
5 value for every cell with comparative information.
6 And that's all.
7 Q. Okay. The first of the infinities, it's a little
8 hard to see on here. I think the first of the
9 infinities on this list is in bold, does that
10 indicate significance?
11 A. We're on page two of the report?
12 Q. Page two, yes.
13 A. It's the third line from the top down?
14 Q. Yes.
15 A. And corresponds the GPA of 3.25, 3.49, LSAT 151 to
16 153?
17 Q. Right.
18 A. And residents in 1995?
19 Q. Yes.
20 A. And minority applicants on two out of two were
21 admitted, and majority of nonselected minority zero
22 out of nine were admitted, that's right?
23 Q. Right.
24 A. And so that would be calculated as infinity.
25 Q. Okay.
GRUTTER -vs- BOLLINGER, ET AL
141
1 A. As the observed odds ratio.
2 Q. And am I right that you're indicating that you feel
3 that odds ratio is statistically significant?
4 A. What we're doing here is comparing the rate of
5 admission which for minorities observed rate is a
6 hundred percent. To the rate of admission for
7 majority and non-selected minorities which is zero
8 percent.
9 And even with this small sample size
10 the test that's used to do that comparison indicates
11 that there is a statistically significant difference
12 between those admission rates.
13 Q. Okay. And you have listed in bold in these tables
14 the cells that you find--the cells for which you
15 find a statistically significant odds ratio,
16 correct?
17 A. As best I could I bolded the lines that have P
18 values less than .025. That's what I intended to
19 do, if I didn't do that then I made a mistake.
20 Q. And where P value is less than .05, in your view
21 that indicates statistical significance of the odds
22 ratio that you're reporting?
23 A. In the particular cells that we're looking at. On
24 that cell by cell basis, that's correct.
25 Q. Okay. So, the cells that for which you report odds
GRUTTER -vs- BOLLINGER, ET AL
142
1 ratios but they're not in bold, that's an indication
2 that you don't believe that those odds ratios are
3 statistically significant, correct?
4 A. The cells that are not in bold that have P values
5 greater than .025, means that from that cell
6 information alone, that cell information alone,
7 there isn't sufficient information that indicates
8 that it is statistically significant from the
9 information from that cell alone.
10 Q. So, let's continue, look at page three on this
11 table. You've got one numerical odds ratio and six,
12 I believe, infinities?
13 A. That's correct.
14 Q. On the next page you have three more infinities?
15 A. That's correct.
16 Q. And pages five and six you actually have a number of
17 them. I count on page five, eight numerical odds
18 rations and 14 infinities. And two odds ratios on
19 page six and 16 infinities?
20 A. You want me to count to confirm? That's about
21 right.
22 Q. Feel free to count if you want.
23 A. I'll count if you want me to.
24 Q. Okay. So, in some, am I right, that you've
25 calculated in these 240 cells twelve numerical odds
GRUTTER -vs- BOLLINGER, ET AL
143
1 ratios and 52 infinities?
2 A. I didn't sum that up, but I presume that's about
3 right. I forgot, you tell me twelve that had values
4 less than infinity?
5 Q. Yes.
6 A. And?
7 Q. 52 infinities?
8 A. 52 that had values of infinity. That would--it's
9 probably about right.
10 Q. Okay. And by my count you found seven of the
11 numerical odds ratios to be statistically
12 significant. And 14 of the infinities to be
13 significant, does that sound about right?
14 A. Well, I think I testified this morning there would
15 be between 17 and 22 for each year. And so you said
16 21 altogether?
17 Q. That's about right, yes, 21?
18 A. That would be consistent, right.
19 Q. So, to sum up, you found statistically significant
20 odds ratio in only 21 of the 240 cells in 1995?
21 A. 21 of the total number of cells. I don't know how
22 many that is of the cells with comparative
23 information.
24 Q. And again you define comparative information to mean
25 that there are more applicants than admits in both
GRUTTER -vs- BOLLINGER, ET AL
144
1 groups, that's your definition of comparative
2 information?
3 A. Well, I'll give my definition just to be clear. In
4 order for a cell to have comparative information,
5 there have to be minority applicants in the cell,
6 there have to be majority applicants in the cell.
7 There have to be some applicants that
8 are admitted and some that are not admitted. So
9 that's the condition that's required.
10 Q. Okay. At any point, did you look to see how many of
11 the applicants were in the cells for which you found
12 statistically significant odds ratio?
13 A. Did I look to see how many were in those cells?
14 Q. Sure.
15 A. No, I didn't calculate that.
16 Q. Would you be surprised to learn that in the seven
17 cells was statistically significant in numerical
18 odds ratio, there are a total of 601 applicants in
19 your table?
20 A. Would I be surprised, no.
21 Q. Okay. Or that there were 894 applicants in the 14
22 infinity cells that you find to be statistically
23 significant?
24 A. So that's about 1400 overall in the two groups?
25 Q. Yes.
GRUTTER -vs- BOLLINGER, ET AL
145
1 A. No. I would think that would be the size of the
2 group that would have comparative information, yes.
3 Q. And there were just over 4000 applicants this year,
4 right?
5 A. Certainly.
6 Q. By my math, you can check me. If there's 4,147
7 applicants and 1,495 applicants appear in the cells
8 that you find to be statistically significant, that
9 means that your statistically significant odds ratio
10 cover only about 36 percent of the total number of
11 applicants, does that sound about right?
12 A. Those are the applicants that are in the cells with
13 comparative information, I think that's probably
14 right.
15 Q. All right. And so I just want to be clear. You're
16 drawing your conclusions in this odds ratio analysis
17 based on cells with just over a third of the total
18 number of applicants in the pool?
19 A. That's correct. And ones information whom there's
20 comparative information.
21 Q. So that means that nearly two-thirds of the students
22 are in cells that you have excluded from your
23 analysis?
24 A. Two-thirds of the students had credentials where
25 they didn't give any comparative information, that
GRUTTER -vs- BOLLINGER, ET AL
146
1 would be correct.
2 Q. When you say that you feel that cells where all the
3 students are admitted don't have comparative
4 information, you're aware that the University or the
5 Law School could decide not to admit all of those
6 students, right?
7 A. I'm sure if they had a larger pool that we couldn't
8 find them all admitted, that's true.
9 Q. So to some extent those cells reflect choices made
10 by the Law School Admissions officers?
11 A. I think that every cell indicates choices made by
12 Law School Admissions officers, that's true.
13 Q. And similarly cells where all of the students were
14 denied admission, the University could have decided
15 to admit some of the minority students, for example,
16 from the cells, right?
17 A. Certainly. These are displaying the Admissions
18 decision.
19 Q. And so the University simply in this year chose not
20 to admit any of those students, correct?
21 A. In those particular cells?
22 Q. Yes.
23 A. Primarily the ones with the low GPA and low LSAT,
24 they may not have admitted any students with those
25 particular combinations of credentials, that's true.
GRUTTER -vs- BOLLINGER, ET AL
147
1 Q. So, the cells that you have excluded from your
2 analysis include information about decisions that
3 the Law School is making, correct?
4 A. Certainly.
5 Q. Okay. But you're not considering those decisions
6 when you report your odds ratio?
7 A. Those cells don't give comparative information, so
8 they're not included in the odds ratio calculations.
9 Q. I'd like to turn now to what you call the composite
10 odds ratios, the numbers taking the various cells
11 put together. And I want to talk a little bit about
12 how you got those odds ratios.
13 Am I right that you generated those
14 using the regression analysis?
15 A. I use the technique called logistic regression,
16 that's true.
17 Q. And your regressions used models that represented,
18 in some sense, the admissions process, correct?
19 A. I think the model that I used allowed us to control
20 the cell grids, and took whatever admissions
21 proportions were seen in those cell grids and
22 analyzed those, that's true.
23 Q. I'm asking a more basic question then that. And
24 that's just that your regressions used models to
25 represent the admissions process, right?
GRUTTER -vs- BOLLINGER, ET AL
148
1 A. Models in the statistical sense, I did a model term
2 in my regression analysis the answer is yes, of
3 course.
4 Q. Because obviously you can't replicate the actual
5 admissions process, statistically?
6 A. I'm not sure I understand the question.
7 Q. To do a regression analysis, you have to use a model
8 because given the data you had, you can't actually
9 replicate the admissions process that the Law School
10 uses?
11 A. I'm afraid you've used the same words, and since I
12 didn't understand the first time I probably don't
13 understand it now.
14 Q. Okay. Let me try it this way. Your regression
15 analysis was designed in some sense to approximate
16 the admission process that the Law School use, is
17 that fair to say?
18 A. No.
19 Q. You didn't get the individual applicant files and
20 sit down and read them obviously?
21 A. If you mean that I was supposed to go through the
22 files and replicate--that's what you mean a
23 replication, did I read 4,500 files and make
24 decisions on that. The answer is, I did not do
25 that.
GRUTTER -vs- BOLLINGER, ET AL
149
1 Q. You tried to use the statistical technique to draw
2 conclusions about the Admissions decisions without
3 actually going through the same process that the
4 Admissions officers had gone through, right?
5 A. If I'm understanding now, did I look at the files
6 and make decisions, and the answer is I did not.
7 And so I don't think I was trying to replicate the
8 process, what I was trying to do is understand the
9 results of the Admissions process and summarize
10 those. And that's what I was doing.
11 Q. And just trying to understand. You're doing that
12 though with something that you have constructed, in
13 other words, using this statistical technique that
14 you call logistic regression?
15 A. I use the standard technique for analyzing data
16 controlling for, in this case, LSAT and GPA. That's
17 what I did.
18 Q. And that's something that you constructed, you
19 didn't get it from the Law School?
20 A. What do you mean, did I decide to do this
21 statistical analysis? I made the decision, yes.
22 Q. And you decided how to do it?
23 A. How to do it?
24 Q. Yes.
25 A. In the sense that I decided I would use logistic
GRUTTER -vs- BOLLINGER, ET AL
150
1 regression?
2 Q. Yes.
3 A. I did decide to use logistics regression, that's
4 correct.
5 Q. And in doing your logistic regression, you made
6 various choices along the way?
7 A. Very few choices, but I suppose I must have made
8 some choices.
9 Q. You made some choices. And, in fact, you had to
10 make certain assumptions along the way in designing
11 your models, isn't that right?
12 A. This model makes very few assumptions, but there
13 always are assumptions. There's a saying, it's not
14 a saying, George Bakke the professor says, how can I
15 say this? All models are wrong, some models are
16 useful.
17 So, I certainly made a choice of
18 looking at particular models, and I think I looked
19 at ones that I thought were useful.
20 Q. You'd agree that it's important in order to evaluate
21 your results to understand the assumptions that you
22 made as you designed your models?
23 A. As best we can, sure.
24 Q. Okay. And without a clear sense of the assumptions,
25 we can't evaluate the accuracy or reliability of the
GRUTTER -vs- BOLLINGER, ET AL
151
1 results, isn't that right?
2 A. Well, I think we can look at the assumptions, I'll
3 be glad to.
4 Q. All right. One of your assumptions was that the
5 association between race and admissions does not
6 vary with differing levels of grades and test
7 scores, isn't that right?
8 A. I think I said this morning that the composite
9 estimate provides us a value that goes across--that
10 summarizes the individual cell odds ratios.
11 So, in fact, the model that did the
12 computation, then assumes that it would be the same
13 across, that's correct.
14 Q. Okay. And so in order for your results to be
15 useful, that assumption needs to be correct, isn't
16 that right?
17 A. No, that doesn't have to be absolutely correct. No.
18 Q. If it's wildly wrong, you think it doesn't affect
19 your results?
20 A. I think the composite estimates still summarizes the
21 odds ratio for this cells that we have, and it's a
22 summary.
23 Q. You would agree that for applicants with low grades
24 and test scores regardless of race, those applicants
25 are extremely unlikely to get in, correct?
GRUTTER -vs- BOLLINGER, ET AL
152
1 A. The evidence is clear that the Law School use grades
2 and test scores. And applicants with low grades and
3 low test scores have very little or no chance to get
4 in. Many of those cells have all of those students
5 denied, that's correct.
6 Q. And that's true without regard to race?
7 A. Absolutely.
8 Q. So, for those applicants whether or not they're a
9 member of a minority group, is really unimportant
10 for deciding or for predicting whether they're going
11 to get in, is that fair to say?
12 A. There certainly is a range of grades and test scores
13 where students are not admitted, that's true.
14 Q. And for those students whether or not they're
15 members of a minority group, is unimportant for
16 predicting whether they'll get in, is that correct?
17 A. That would be true, yes.
18 Q. Let's look at the other end of the scale. At the
19 very upper end of grades and test scores say, you
20 know, 4.0 and LSAT of about 170, applicants in that
21 range are extremely likely to get in regardless of
22 their race, isn't that right?
23 A. That's correct.
24 Q. And for those students whether or not they're
25 members of a minority group, is unimportant for
GRUTTER -vs- BOLLINGER, ET AL
153
1 predicting their chances of getting in?
2 A. Actually I think the evidence contradicts that.
3 Because I think if you look at those upper ranges,
4 there are, in fact, still--there's still information
5 in most of those upper ranges.
6 Because there are non-minority
7 students denied in most of those cells, not a
8 hundred percent. And where there's students that
9 are both admitted and denied, we can look for
10 comparative information.
11 Q. But if students in those ranges are extremely likely
12 to get in, we're back to our .99 versus .90, for
13 example. Those probabilities are true without
14 regards to race, they're at the extreme high end?
15 A. The probabilities may be high, but if you look at
16 just in your example you have .99 to .90. If you
17 had .999 to .90, then you would have an odds ratio
18 of 111.
19 And that was still in the sense of
20 comparative information that odds ratio could still
21 be high, but now it's effecting not the admitted
22 cohort but the denied cohort. The ones that are
23 chosen not to be in.
24 Q. So, in that situation where you go from .99 to .999,
25 you're getting ever increasing odds ratios even as a
GRUTTER -vs- BOLLINGER, ET AL
154
1 practical matter that the Admissions decisions are
2 coming out and saying, isn't that right?
3 A. I have to say I don't think they're the same. If
4 the students that--you can ask the students who are
5 denied in those cells if they think the decisions
6 are the same.
7 Q. So, in your view as you understand how the process
8 work, do you think that there is a real world
9 difference between a .99 chance of admission and a
10 .999 chance of admission?
11 A. Now, you're switching, I'm sorry. Say it again so I
12 can make sure I've got the numbers right.
13 Q. Do you believe that there is a real world difference
14 for applicants between a .99 chance of admission and
15 a .99 chance of admission?
16 A. You know, I think we're probably in an area of about
17 99 percent, where given the size and samples where
18 we probably wouldn't have much of a difference.
19 But in the examples you gave of 90
20 percent versus 99, I probably think there is a
21 difference, yes.
22 Q. And going back to the issue of levels of grades and
23 test scores, is it your view that the role that race
24 plays is the same at the very high levels of grades
25 and test scores, as opposed to the middle range?
GRUTTER -vs- BOLLINGER, ET AL
155
1 A. Well, I'll tell you what I know about that if you
2 want it. What I know is that I was criticized with
3 regard to that issue. And I went back and looked at
4 the odds ratios in all of the cells and the
5 particular case where I was criticized, and I found
6 no pattern, no pattern that related to grades or
7 LSAT.
8 And admittedly the amount of
9 comparative information may be small in those upper
10 cells, so I don't have real information on it. But
11 there is no evidence, statistical evidence I have
12 that there is a strong, strong effect one way or the
13 other.
14 Q. So you don't have a view one way or the other about
15 whether the role is the same in those two?
16 A. I'm saying the information, the data itself, doesn't
17 inform me that there is information. So, I'm saying
18 when I looked, I looked for evidence to see whether
19 the odds ratios varied by grade point average and
20 test score. I looked at that information, because I
21 was criticized for that.
22 And when I looked and looked at the
23 odds ratios, I couldn't find any consistent pattern
24 that showed that it change, for instance, in the way
25 that you said, which was that it would be higher at
GRUTTER -vs- BOLLINGER, ET AL
156
1 one area or the other.
2 Q. I want to go back to, I think now, my question a few
3 questions ago.
4 A. Sure.
5 Q. In your opinion, is the role that race plays in the
6 admission process, the same for applicants at the
7 very high levels of grades and test scores, as it is
8 for students in the middle range?
9 A. With respect to odds ratio, we have no statistical
10 evidence that it is different, it's a statistical
11 issue. I can't say it's the same, because we don't
12 have enough information to say it's the same.
13 What I have is in statistics we often
14 do things backward. What I have is, I have no
15 evidence that there are differences of this sort
16 that we're talking about.
17 Q. Am I right that that means you also have no evidence
18 that it's the same, based on your odds ratio
19 analysis?
20 A. My evidence is that the data are consistent with it
21 being the same.
22 Q. All right. So just trying to make sure I understand
23 your opinion. Your best opinion on this matter is
24 that, from what you have seen, the role and based on
25 your analysis, the role that race plays in
GRUTTER -vs- BOLLINGER, ET AL
157
1 admissions is the same for students in the middle
2 ranges of grades and test scores as it is for
3 student at the upper range?
4 A. I'll say what I know and I'll try to say it as
5 clearly as I can. What the data itself does not
6 give me evidence that there are differences. I have
7 done statistical tests to try to check that
8 assumption.
9 The statistical test give no evidence
10 of differences, that's what I can say. You can
11 never say, you can never state, and I don't want to
12 be James Bone.
13 You can never say never, but because,
14 in fact, because in fact, the data are consistent
15 with a range of differences in odds ratio.
16 Q. Dr. Larntz, I just want to understand your opinion.
17 You indicated this morning that you were expressing
18 an opinion, attempting to quantify the role that
19 race plays in admissions.
20 And my question to you is, in your
21 opinion, is the role that race plays in admission
22 the same, or approximately the same, for students in
23 the middle ranges of grades and test scores, as it
24 is for students in the upper range?
25 A. And I'll answer it in the same way I can, because
GRUTTER -vs- BOLLINGER, ET AL
158
1 I'm going to answer you with what I know. And what
2 I know is that the data are consistent with it being
3 the same. The data are consistent with it being the
4 same.
5 In statistics we can't say that we
6 have proven it when the data are consistent, but we
7 can say that the data don't contradict that
8 assumption, and that's true.
9 Q. And that's as far as you can go with what you have?
10 A. As far as I can go is my composite estimates if they
11 are different, if they are different, my composite
12 estimate gives me a value that if they are different
13 and they are higher or lower in one range or
14 another, then the composite estimate is greater then
15 some of them and less than some others. So the
16 composite does apply as a summary over the entire
17 table.
18 Q. So, the composite then is something like a weighted
19 average?
20 A. The composites--I have to been very careful. The
21 technique used doesn't do weighted averages. It's a
22 technique called maximum likelihood, and what it
23 does is it chooses the value that is the value that
24 is most consistent with the data.
25 Q. Is it fair to think about the weighted average, even
GRUTTER -vs- BOLLINGER, ET AL
159
1 if that is not what is computed?
2 A. Well, what the computer did is nothing like a
3 weighted average, and I'm not sure that I
4 would--there's no way I could compute a weighted
5 average. What I think was weighted average and get
6 the numbers. It doesn't work that way.
7 The computer tries different values
8 and sees which one is the most consistent with the
9 data. So this is the value that is most consistent
10 with the data.
11 Q. Data as a whole looking across all the cells?
12 A. Looking across all the cells with comparative
13 information.
14 MR. DELERY: Your Honor, I think this
15 is sort of a logical stopping point, if that is good
16 for you.
17 THE COURT: Okay. We'll take a
18 recess in this case.
19 (A brief recess was taken.)
20 (Court back in session.)
21 THE COURT: Back on the record.
22 Okay, you may proceed.
23 MR. DELERY: Thank you, your Honor.
24 BY MR. DELERY:
25 Q. Dr. Larntz, I was going to take this down to get it
GRUTTER -vs- BOLLINGER, ET AL
160
1 out of their way, but I wanted to ask one more
2 questions about this example that you had, I think I
3 learned something earlier.
4 You're saying that if we go from .99
5 to .999 versus .90, the odds ratio goes up to 111
6 from eleven?
7 A. The .999 to .90 it goes to 111, I think that's
8 right.
9 Q. And if we go to .999 more of them, do we add another
10 one here so that it's a 1,011?
11 A. It turns out in that range of odds and probabilities
12 and the corresponding conversions to odds, I think
13 that's correct.
14 Q. So the odds keeps growing?
15 A. The odds ratios keep growing, yes.
16 Q. The odds ratio, correct, I'm sorry. I'd like to go
17 back, if we could, to assumptions that you made
18 during your regression analysis.
19 Am I correct that another assumption
20 that you made is that the variables that you
21 included in your model were not related, not
22 correlated with the variables that were not in your
23 model?
24 In other words, the other admissions
25 factors for which you didn't have any data?
GRUTTER -vs- BOLLINGER, ET AL
161
1 A. Did I make that assumption?
2 Q. Yes.
3 A. No.
4 Q. When a factor not in the model is related to the
5 factors in the model, am I right, that's called
6 confounding?
7 A. When a factor--I'm sorry, would you say it again so
8 I make sure I got your situation clear in my mind.
9 Q. Okay. When a factor that's not in a regression
10 model is related to one of the factors that is in
11 the model, am I right that that's called
12 confounding?
13 A. The technical term confounding is when two factors
14 are related.
15 Q. Okay.
16 A. And it either can be in or out of the model where it
17 could be confounding effect.
18 Q. So it's a more general term then just whether it's
19 in or out of the model?
20 A. Sure.
21 Q. Okay. Did you make an assumption that the
22 admissions factors that the Law School considers but
23 for which you didn't have data, were not confounded
24 with the factors that are in the model?
25 A. No.
GRUTTER -vs- BOLLINGER, ET AL
162
1 Q. You didn't make any assumption one way or the other
2 about that?
3 A. I didn't make any assumption one way or the other.
4 Q. If it turned out, for example, that minority
5 students were comparatively over represented in the
6 group of applicants with, say, strong leadership
7 experience but leadership experience isn't in your
8 model, isn't the effect of that that your model
9 would overstate the effect that race is playing in
10 admissions?
11 A. What would happen in that case is that the model we
12 have measure the terms that are in there, and if
13 another factor were related to minority status then
14 some of the effect of that other variable would go
15 into the minority status effect, that's true.
16 Q. Okay. So, if that were the case, then the model
17 would be over stating the role that race is playing
18 in admissions?
19 A. It could go either way depending, you know. But the
20 models, the coefficients we have are for the terms
21 that we look at. So, the coefficients we have with
22 respect to looking at the grid cell, GPA and LSAT,
23 those are the effects of minority status given
24 those.
25 The ones looking at the other
GRUTTER -vs- BOLLINGER, ET AL
163
1 variables, coefficient change obviously, and so they
2 are representing the effect given those other
3 variables, that's correct.
4 Q. And if it turned out that the minority students were
5 over represented in the group of applicants with
6 better than average leadership experience, then an
7 effect of that would be that your model overstates
8 the role that race plays in admissions?
9 A. The effect of other variables that are used as
10 they're related to race in the model, would mean
11 that if you include those other variables in the
12 model, then you would get a different odds
13 multiplier for race, that's true.
14 Q. And the different odds multiplier would be lower
15 than it is in your estimate?
16 A. It could be, it depends. When we added other terms
17 such as residency, gender and fee waiver, the odds
18 multiplier went up. But it could go either way.
19 Q. Did you do anything to evaluate whether the
20 admissions factors that you couldn't include in your
21 models were, in fact, related in some way to race?
22 A. I had no data to do it that way.
23 Q. So you didn't do any of that?
24 A. That's correct.
25 Q. I believe you said before that you selected your
GRUTTER -vs- BOLLINGER, ET AL
164
1 general model, cell by cell model inspired by
2 Exhibit 16, is that right?
3 A. Inspired?
4 Q. Taken from, that's your point of departure?
5 A. It was provided to me, I wanted to understand the
6 effects based on that exhibit. If I'm ever inspired
7 I suppose. Maybe that's true.
8 Q. All right. And I think you said this morning also
9 that you used the model that you chose, or that the
10 model that you chose was the standard approach that
11 you would take in a medical study, is that right?
12 A. It's a standard approach I would take if I were
13 trying to evaluate, as carefully as I could, the
14 effect of the one factor versus another in any
15 study.
16 I have done this kind of study
17 whether it be a medical study, I gave that as an
18 example. This is the area that I did my research
19 in. So, this is the kind of analysis I would do
20 across a broad range of applications.
21 Q. It's the kind of analysis with which you're most
22 familiar, is that fair to say?
23 A. The kind analysis which I'm most familiar?
24 Q. Yes.
25 A. I don't know if I can rate all the analysis I know
GRUTTER -vs- BOLLINGER, ET AL
165
1 and call one most familiar. It's one that I
2 certainly have used and feel comfortable using, yes.
3 Q. Am I right that when you created the model, you had
4 the computer create a variable for each of the GPA
5 and LSAT cells?
6 A. We're talking now about the actual computations that
7 are done?
8 Q. Yes.
9 A. The actual computations, the effect of those is that
10 you want to look at the composite estimate across
11 the cells with comparative information.
12 So, if you want to make sure that
13 you're controlling for the cells with comparative
14 information, then you would include what we call
15 indicator variables for each of those combinations.
16 Q. So, you created an indicator variable, well, the
17 computer did, for each of the cells?
18 A. The effect of that in the model is so that you can
19 estimate the odds ratio controlling for, that is
20 taking account without making any particular
21 assumption about the effects of LSATs and GPA.
22 Q. Are these variables sometimes called dummy variables
23 in your field?
24 A. We have cute things for things, yes. Did I use the
25 term indicator variable?
GRUTTER -vs- BOLLINGER, ET AL
166
1 Q. Yes.
2 A. And you use the term dummy variable?
3 Q. Yes.
4 A. I suppose since they both indicate the same thing,
5 it would be a matter of one's persuasion to which
6 one you prefer I call it.
7 Q. Okay. I just wanted to make sure that they're the
8 same things, we can use indicator variables if you
9 like that.
10 And the purpose of using the
11 indicator variables was to control for GPA and test
12 scores?
13 A. To allow us only to use the comparative information
14 in getting our composites. That's the only purpose
15 in doing that.
16 Q. Am I right that you ended up with more than a
17 hundred variables in each of your models?
18 A. As many cells as there were in comparative
19 information.
20 Q. Do you know whether it was more than hundred in each
21 of these years?
22 A. Well, in the basic grid there's 120 cells. So, if
23 what you're saying now is I used more than a hundred
24 indicator variables, that mean I used more than a
25 hundred cells, which may be true.
GRUTTER -vs- BOLLINGER, ET AL
167
1 Q. You don't remember how many variables were in there?
2 A. Precisely?
3 Q. Yes.
4 A. Of course not.
5 Q. Am I right also that the computer excluded cells
6 from this analysis if they were empty, if there were
7 no applicants?
8 A. It excluded all cells without comparative
9 information.
10 Q. So, those cells included cells with no applicants,
11 am I right?
12 A. Certainly.
13 Q. Okay. It also included the cells where all of the
14 applicants in both groups were treated the same as
15 we discussed earlier?
16 A. I want us to be clear. I think you were mixing up
17 analysis, so I want to make sure the court is clear
18 and everyone is clear.
19 In the models where we use the
20 controlling for LSAT and GPA, the cell grid, okay,
21 and the analysis that compared the various ethnic
22 groups, okay?
23 Q. Yes.
24 A. Then in that particular model we control for the
25 individual cell combinations. So, we did have, as
GRUTTER -vs- BOLLINGER, ET AL
168
1 you would say, there are 120 such cells? Is that
2 you counted them, and you were right?
3 Q. I believe on your cell by cell analysis you had 240.
4 120 for Michigan residents and 120 for
5 non-residents?
6 A. Exactly. What you're doing is using the cell by
7 cell analysis when the odds ratios weren't computed
8 from that cell by cell analysis, they were computed
9 from the initial--analysis.
10 And in that one there are actually
11 120 cells. And what you're saying, which I believe,
12 is that of those 120 cells probably a hundred or so
13 contributed to the model.
14 Q. I'm not saying, I'm just asking you whether you
15 remember or know how many variables went into the
16 model?
17 A. Well, certainly if there are indicator variables
18 from each cell for which there were applicants, and
19 for which there were applicants from, at least, two
20 ethnic groups, which is what it would have to be.
21 All the applicants couldn't have been
22 in one ethnic group. And if there were individuals
23 both admitted in the volume cell, I would not be
24 surprised if there were about, in that analysis, at
25 about 80 percent of the cells, a hundred out of 120
GRUTTER -vs- BOLLINGER, ET AL
169
1 would contribute.
2 Q. But just so I'm clear. The computer excluded cells
3 that didn't fit into the three categories that you
4 just mentioned? Or cells that didn't have the three
5 characteristics, I should say, that you just
6 identified?
7 If a cell didn't have applicants from
8 both groups?
9 A. From two groups of the nine. There were nine.
10 Q. Two of the nine?
11 A. So, it had to have at least some representation from
12 two of the nine categories of ethnicity.
13 Q. Okay. And then there had to be both admitted and
14 rejected applicants, and at least two groups, am I
15 right?
16 A. There had to be--no, there had to be admitted and
17 rejected applicants.
18 Q. In the cell?
19 A. In the cell.
20 Q. And for cells where that was not the case, the
21 computer excluded the cells?
22 A. For cells not the case, in that case they did not
23 contribute to the odds multiplier estimation, that's
24 true.
25 Q. And for cells that if you just look at a cell by
GRUTTER -vs- BOLLINGER, ET AL
170
1 cell comparison would yield infinity odds ratio,
2 those cells were included by the computer, am I
3 right?
4 A. There's comparative information, so they would
5 certainly be included.
6 Q. So, the computer excluded cells from which you say
7 there was never comparative information, and
8 included, at least, some cells where the odds ratio
9 was infinity, right?
10 A. Where the observed odds ratio was the infinity?
11 Q. Sure.
12 A. Sure.
13 Q. And the computer did that because that's the way you
14 programmed it?
15 A. Yes, that's the way I arrange for the computer to do
16 it. That's true.
17 Q. You didn't have to choose that methodology?
18 A. If I didn't choose that methodology, then I would
19 have had to make some other assumption about the
20 effect of LSAT and GPA on admissions.
21 And I did not, I wanted to do an
22 analysis where I made as few assumptions as possible
23 about LSAT and GPA. Because it was clear from the
24 grid cells that that was a fairly complicated
25 process relatively.
GRUTTER -vs- BOLLINGER, ET AL
171
1 Q. But I just wanted to be clear, that the computer did
2 what it did because that's the way that you set it
3 up?
4 A. I do my own computing, I don't have a grad student
5 or anyone else, I do all of my own work. So, if the
6 computer did something it's because I made
7 instructions that would have it do it, that's true.
8 Q. And you could have chosen something else, this is
9 the way you decided to approach the answer?
10 A. Absolutely. It was my decision.
11 Q. Now, earlier this morning you talked about a
12 baseball analogy, of calculating a batting average.
13 And when a batting average is calculated, as I
14 understand it, all of the games are included, isn't
15 that right?
16 A. All of the games.
17 Q. All of the games that a particular person has played
18 are included?
19 A. Sure.
20 Q. Okay. But if some of the cells were not included in
21 your model, that's the equivalent of excluding some
22 of the games from the batting average, correct?
23 A. There's a flaw in my analogy, I agree.
24 THE COURT: In your analogy?
25 A. Well, in the analogy in the sense that--
GRUTTER -vs- BOLLINGER, ET AL
172
1 THE COURT: (Interposing) The
2 analogy in the sense of the baseball game?
3 A. In the sense that this situation is more complicated
4 than the baseball game, okay. And so in this case
5 we have cells of various sorts. There are cells
6 that basically where there's no comparative
7 information, everyone is admitted or everyone is
8 denied, everyone.
9 Here we're including only cells that
10 have comparative information. So, that is cells in
11 which there are, at least, members of two of the
12 ethnic groups. And cells from which some applicants
13 are admitted and some are denied.
14 But it doesn't matter if ones
15 particular group is all admitted, or one particular
16 group is all denied. So it's a more complicated
17 analogy and I agree with that.
18 BY MR. DELERY:
19 Q. Now, in terms of the way the cells were constructed,
20 I believe you said this morning that you took them
21 from Exhibit 16, what is now known as Exhibit 16?
22 A. And I knew it as Exhibit 16 I think then, yes.
23 Q. You could have picked cells that were larger then
24 what is reflected, isn't that right?
25 A. I could have.
GRUTTER -vs- BOLLINGER, ET AL
173
1 Q. You could have picked cells that were smaller?
2 A. I could have.
3 Q. And the odds ratios that you report would have been
4 different if you had selected different cells, isn't
5 that right?
6 A. It would really be amazing if we did the
7 computations that came out to be exactly the same if
8 we made different cell boundary choices. That's
9 absolutely correct.
10 Q. So the odds ratio numbers that you report would be
11 different if you used different cells?
12 A. Absolutely.
13 Q. Am I right that as a general matter as you expand
14 cells beyond what you have, you would expect that
15 the observed difference between the racial groups
16 would decrease?
17 A. Please be more specific.
18 Q. Okay. For example, if you looked at the whole
19 applicant pool, and essentially took the whole
20 applicant pool as one being cell, the odds ratio
21 then would, in fact, be below one, isn't that right?
22 A. So, if we ignored credentials, ignored GPA and LSAT
23 and we said, okay, one big cell. Then we get a
24 different odds ratio, and that odds ratio you say
25 would be less than one, it would depend on the
GRUTTER -vs- BOLLINGER, ET AL
174
1 admission rate?
2 Q. If the admission rates for minority students were
3 lower than the admission rates for non-minority
4 students, then the odds ratio would be less than
5 one?
6 A. If we made it one big cell, which means we're
7 ignoring LSAT and GPA credentials and calculated,
8 which you could do, then if what you say is true and
9 I don't have any reason to doubt it, then the odds
10 ratio calculated would be less than one, that's
11 true.
12 Q. So, if we start with your cells and expand them in
13 the direction of the whole pool being one cell, then
14 the odds ratio is going to move in the direction of
15 the odds ratio for the whole applicant pool, isn't
16 that right?
17 A. There's no guarantee that you would get exactly that
18 as you step along. But in the end you would
19 obviously wind up there, that's true.
20 Q. And based on your work with these data sets, you
21 don't have a view as to what would happen as you
22 move in that direction?
23 A. Well, actually my other analysis informs us, at
24 least, somewhat on that. The analysis where I
25 accounted for their thin cell GPA. So, as we move
GRUTTER -vs- BOLLINGER, ET AL
175
1 to groups that are less specific with respect to
2 credentials, that is wider groups, that is so there
3 is more of a mix of individuals within.
4 It appears that the odds ratio in the
5 case where I control for the grid cell the odds
6 ratios went up. So, if we did the opposite of that,
7 it would seem if you use broader, that is made
8 groups that are of larger groups of credentials,
9 that you'd probably wind up with odds ratio that are
10 smaller. I think that's probably right.
11 But I didn't do that calculation to
12 know for sure that's exactly what would come out in
13 this case.
14 Q. All right. I believe you said a number of times
15 that your goal in this analysis was as much as you
16 could, to look at students who were similarly
17 situated in terms of credentials as you call it,
18 correct?
19 A. That's the goal of statistical, I call it principal
20 statistical fair comparison, that's what I would
21 call it.
22 Q. Okay. And if we could put up slide 37 from this
23 morning. If you look at the African American line,
24 the second one down here on the left, the left side
25 is your model that controlled only for grade point
GRUTTER -vs- BOLLINGER, ET AL
176
1 average and LSAT scores, correct?
2 A. That's correct.
3 Q. And if we look at the line for African Americans,
4 your odds ratio there is 257.03, right?
5 A. Well, I'm not going to worry about the decimal, you
6 can't.
7 Q. 257, okay. On the right is your second model where
8 you added in some additional factors in addition to
9 GPA and test scores, right?
10 A. That's correct.
11 Q. And so you were controlling for more factors in the
12 model reflected on the right?
13 A. That's correct.
14 Q. So, in your view, the applicants being compared in
15 the model on the right, were even more similarly
16 situated then the applicants on the model on the
17 left?
18 A. I believe that's true, yes.
19 Q. And the African Americans odds ratio for the model
20 on the right goes up from 257 on the left to 513 on
21 the right, correct?
22 A. Correct. I mean those are both giant numbers and I
23 don't want to say they're very different as far as
24 factors goes.
25 Q. So, when you control for more factors you're getting
GRUTTER -vs- BOLLINGER, ET AL
177
1 a larger odds ratio, right?
2 A. In this particular case we control for these
3 additional factors. I got a larger odds ratio
4 in--my expectation was that it may be that they
5 could go down.
6 And in many analysis where we do
7 control for additional factors, they go the other
8 direction.
9 Q. In other context in your experience they go down?
10 A. In statistical context that I have worked on, which
11 is a greater variety of context, yes, they can go in
12 either direction. I didn't say they would go down,
13 I said they could go either direction.
14 Q. In this case in each year the odds ratios go up, is
15 that right?
16 A. I'm not sure that they're in uniformity, I think
17 that's probably the case.
18 Q. At least in 1995?
19 A. Well, certainly n '95 we have that, and we can look
20 at the reports if you want to for the other years.
21 Q. Suppose that you had numerical information or all of
22 the factors that the Admissions Office considers, so
23 that you can bill all of the factors in the model
24 and control for them.
25 In that situation, wouldn't you
GRUTTER -vs- BOLLINGER, ET AL
178
1 expect the odds ratio to approach infinity?
2 A. In what sense? I guess if I had additional
3 numerical factors, additional factors that we
4 control for?
5 Q. Yes.
6 A. You know, I just don't know.
7 Q. In other words, you wouldn't expect that in your
8 model as you add in other factors so that you got to
9 the point where you were controlling for everything
10 the Admissions Office considers other than race,
11 that the resulting odds ratios would not be
12 infinity?
13 A. I think the odds ratios may get large in this case.
14 I don't know if I had--you're talking in a very
15 hypothetical way, since we can't do this analysis in
16 only a hypothetical way.
17 But, in fact, boy if you ever have
18 such models, you've described the process perfectly,
19 that's right. So everything would be infinity in
20 sense of odds ratio.
21 Q. And the reason for that, and I'm just trying to
22 understand the way your approach works. The reason
23 for that is if you control for all the other factors
24 and are looking at people who are identical except
25 for the fact that some are minorities and some are
GRUTTER -vs- BOLLINGER, ET AL
179
1 not, then any difference in the admission in those
2 two groups, the model contributes to race, isn't
3 that right?
4 A. If that's the only factor left and that's how
5 decisions were made, then that's what would happen.
6 Q. And the resulting odds ratio in that context would
7 be infinity, correct?
8 A. It would be large, it could be infinity. In our
9 hypothetical if we could make all the odds ratio
10 infinity for everything.
11 Q. Okay. And the reason it would be large or infinity,
12 is that the only factor left to explain any of the
13 difference would be race, right?
14 A. If that were the deciding factor, sure.
15 Q. And the odds ratio in that situation would be the
16 same, no matter how much race had been taken into
17 account by the person actually making the decision,
18 isn't that right?
19 A. If we have all the other factors that went into the
20 process, if we had that, which we don't here
21 certainly. But if we did have that, then I think,
22 in this hypothetical example you would wind up in
23 that situation.
24 Q. Okay. And this odds ratio analysis then, can't tell
25 us about how much race is taken into account by the
GRUTTER -vs- BOLLINGER, ET AL
180
1 people making the decision, right?
2 A. It can't, is that what you're saying?
3 Q. I'm asking you if whether I'm correct that it
4 cannot?
5 A. It measures with respect to just what we have here.
6 It measures that when we take in account grade point
7 average, LSAT grid cells, how much race is taken
8 into account with respect to explained decisions
9 beyond those. That's what it explains, no more than
10 that. It's a description of the Admissions process.
11 Q. You think that this model is saying something about
12 how heavily race is being weighed by the person who
13 sits down and reads the admission file?
14 A. What I think is the aggregate effect of the
15 decisions made in the Law School with respect to
16 admissions. The aggregate effect is--well, for
17 instance if we look at what we can't see there, the
18 effect of residents, as far as making decisions.
19 That the effect of race, for
20 instance, is much greater and has a stronger effect
21 then Michigan residents.
22 And when we compare non-resident
23 minority applicants to resident majority applicants,
24 you can see that, in fact, decisions were made
25 strongly in favor of the minority applicants. So,
GRUTTER -vs- BOLLINGER, ET AL
181
1 with respect to that, which is that factor, I could
2 say.
3 Q. You just said that the odds ratio say something
4 about the effect of taking race into account?
5 A. The effect that we see, right. You asked me a
6 different question, I'm sorry.
7 Q. I'm sorry. I want to make sure that I understand.
8 The effect that you see after the fact, looking back
9 at the decisions that have been made, because that's
10 what you're doing?
11 A. I don't think I prefer to do anything else other
12 than look at and try to understand the decisions
13 made by the Admissions Office, that's correct.
14 That's correct.
15 Q. In looking at the effect of a factor on the
16 decision, looking at them after they have been made
17 is different, you would agree from trying to
18 quantify how much a person sitting down to read a
19 file is weighing race?
20 A. I'm not sure there's a statistical way to quantify
21 what you're talking about. So, different, but
22 nothing that I can measure.
23 Q. So, statistically we cannot measure the weight that
24 a person sitting down to read a file accords race,
25 is that right?
GRUTTER -vs- BOLLINGER, ET AL
182
1 A. What we see is the effect of whatever they have
2 done, however they have done their rating, however
3 we see the effect of those decisions. And what I'm
4 reporting is the effect of those decisions.
5 Q. Okay. And therefore, you are not reporting the
6 extent to which a person sitting down to read the
7 file is taking race into account, is that right?
8 A. Well, let me say just to be very clear. To be very
9 clear, and you can stop me if I'm not answering the
10 question appropriately.
11 But what I'm going to say is that, if
12 in fact, what we're seeing is in my estimation an
13 incredibly large effect with respect to race,
14 incredibly large.
15 If decisions were made, if decisions
16 were made by individuals that didn't take race into
17 account, to a great extent if I may use your word,
18 and use it in an English contest, then I can't
19 imagine that we would ever see effects this large.
20 And so what I'm saying is, that in
21 fact, what we're seeing are enormous effects. And
22 these enormous effects would be really difficult to
23 imagine that they were not made--decisions were not
24 made without taking race into account.
25 Q. I don't think anybody's disputing that the
GRUTTER -vs- BOLLINGER, ET AL
183
1 University and the Law School take race into
2 account. My question is, whether the odds ratios
3 that you report are reporting the extents to which
4 the people sitting down and reading the admissions
5 file are taking or weighing race, or taking it into
6 account?
7 MR. KOLBO: Your Honor, I think I'm
8 going to lodge an objection. I think that's been
9 asked and answered.
10 THE COURT: He's answered it. You
11 think you have answered it?
12 A. I was willing to repeat the same thing.
13 THE COURT: No, that's all right.
14 BY MR. DELERY:
15 Q. I don't want you to repeat the same thing. Do you
16 have an opinion on how your odds ratios would
17 change, say, if the Admissions Office took race into
18 account half as much as it does right now?
19 A. As you well know, I can't give a numerical answer to
20 that, the odds multipliers would be smaller. I
21 think the way to think about that, and maybe there's
22 lots of ways to think about that and you can decide,
23 is that the range at which there are changes, such
24 as when we saw on the Admissions LSAT for a specific
25 grade point average that I showed you this morning?
GRUTTER -vs- BOLLINGER, ET AL
184
1 If they took it into whatever half
2 means, I have to be careful and I'm sure you
3 understand exactly what you mean and I understand
4 exactly what I mean, and they may not exactly match.
5 But what we would have is we would
6 have more overlap in the area in which there
7 were--and the odds ratios would be smaller.
8 There would be more overlap in the
9 area where minority applicants were admitted and
10 rates at which they would be admitted would be
11 closer to the same rates as majority applicants.
12 Q. Let me move to a related point, but not exactly the
13 same point. I think you said this morning that these
14 odds ratios that you reported as a general matter,
15 are about the largest you have seen in your
16 experience?
17 A. They're big, yes.
18 Q. Largest you have seen in your years as a
19 statistician?
20 A. I don't recall a set of consistent analysis that
21 showed odds ratios like that, that's right. In this
22 large setting, that's right.
23 Q. And you think that they're really large, even though
24 only about a third of the minority students are
25 being admitted, isn't that right?
GRUTTER -vs- BOLLINGER, ET AL
185
1 A. In the area in which decisions are made, these are
2 large effects, that's right.
3 Q. What would the odds ratio be if, instead of
4 admitting about a third of the minority applicants,
5 the University or the Law School admitted about half
6 of the minority applicants?
7 A. In the sense that they would consistently, and I'm
8 going to ask you, because I want to make sure I
9 understand how you're thinking.
10 That they would take individuals in
11 the grid and then go further down in the grid for
12 admission in the way, and then result in about half
13 of the minority students being admitted?
14 Q. They may have to go to some extent further down the
15 grid, or maybe they would take more minority
16 students from cells, you know, comparatively higher
17 up?
18 A. Sure. And so in general what would happen in that
19 case, I think, and I'll wait to see how you follow
20 up to decide whether I agree with that. In general
21 the odds ratios would go up.
22 Q. Okay. So, they're already as large as you've ever
23 seen, and if we admitted 20 percent more minority
24 students they would get--
25 A. (Interposing) I think they would get bigger. They
GRUTTER -vs- BOLLINGER, ET AL
186
1 would get bigger in the sense that if the same
2 pattern of admission goes on with respect to using
3 grade point average and LSAT which is clearly done,
4 then they would get bigger.
5 Q. Okay. Could we put up slide 36, please. These are
6 the relative odds for the model that included the
7 factors beyond grades and test scores. We talked
8 about this this morning.
9 If an odds ratio of two or three is
10 really large in your experience, what is the
11 significance of the 6.5, it looks like, for Michigan
12 residents?
13 A. I think I said two or three are the kind we design
14 for and think were important. I didn't say that I
15 have not seen a 6.5 before, in fact, I gave you an
16 example of 20, 19 or so in a typical example.
17 So, the 6.5 I think I said this
18 morning, that that is a pretty big effect.
19 Q. Okay. Would you call it a large allowance for
20 Michigan residents?
21 A. Would I?
22 Q. Yes.
23 A. I think it's a good size allowance for Michigan
24 residents, absolutely.
25 Q. I think this morning you mentioned that you
GRUTTER -vs- BOLLINGER, ET AL
187
1 calculated composite odds ratio for each of these
2 groups for each of the years, for 1995 through 2000,
3 is that right?
4 A. That's correct.
5 Q. And I think we churned through some of those slides
6 pretty quickly this morning. But if I could, I
7 would like to go back to a couple.
8 Before I do that I should ask you, as
9 you understand it the Admissions policy that you
10 have reviewed adopted in 1992, has been in effect
11 throughout this period, 1995 through 2000, is that
12 right?
13 A. I don't know if anyone told me any different, I have
14 not seen another Admissions policy. So I don't know
15 anything to the contrary.
16 Q. Was that your understanding when you did your
17 analysis?
18 A. I think when I did my analysis I knew that I had
19 this Admissions policy. I didn't know if it was
20 necessarily the effect of Admissions policy, I
21 looked at the data in trying to understand the
22 description of what went done.
23 Q. Did counsel for the Plaintiffs discussed whether
24 they had stipulated that the policy had been in
25 effect during that time?
GRUTTER -vs- BOLLINGER, ET AL
188
1 A. I'm sure I never heard that.
2 Q. Would you agree also that the number of applications
3 from members of the various racial and ethnic groups
4 was fairly stable over this period?
5 A. I think it is fairly stable, that's true.
6 Q. There's some fluctuation, but it's fairly stable?
7 A. These are data. If there's not fluctuation, then we
8 wouldn't believe the data.
9 Q. So, over the same time period as you calculate them,
10 the relative odds have varied quite substantially
11 for members of the minority groups?
12 A. I think that's the way I would expect, they would be
13 very substantial.
14 Q. Okay. If we could put up slide, I think it's 35,
15 which is the 2000 Relative Odds of Acceptance. We
16 have for African Americans 443 are the odds that you
17 calculate, is that right?
18 A. That's right.
19 Q. Okay. And if we then go to slide 32 for 1997,
20 instead of 443 you have 53.49?
21 A. That's correct.
22 Q. So, that means that the odds ratio for African
23 Americans in 2000 was about eight time greater than
24 it was in 1997, is that right?
25 A. That's correct. I have to say once odds ratios are
GRUTTER -vs- BOLLINGER, ET AL
189
1 high they're high. And 50 is high, 400 is high.
2 And I would have to say very clearly that I think
3 that there's not a substantive difference when odds
4 ratios get as high as those.
5 Q. So you don't think there's a substantive difference
6 between 53 and what was it, 443?
7 A. I think they're both big, okay. And I think the
8 substance which is what we are talking about, is we
9 have to very careful and statisticians don't get
10 bogged down in numbers and, in fact, allow us to
11 understand the substance. In a substantive way,
12 these are both big numbers.
13 Q. Interesting to hear that statisticians say don't get
14 bogged down in numbers. But did you do anything to
15 look at whether the differences in the odds ratios
16 for these two years 443 versus 53, were
17 statistically significant?
18 A. I did not make a formal comparison.
19 Q. There is a formal test that could be done to test
20 the significance of that difference, is that right?
21 A. It's possible to do such a test.
22 Q. And you haven't done that?
23 A. I did not carry that test out.
24 Q. Okay. Would you be surprised to find out that the
25 difference between these two numbers, 443 and 53 is
GRUTTER -vs- BOLLINGER, ET AL
190
1 more than eleven times the standard error of the
2 difference?
3 A. Yes, I would be surprised. I can't imagine given
4 the size of these that that difference is there. I
5 can do the calculation, if you would like.
6 Q. If it turned out that the difference between those
7 two odds ratios is more than eleven times the
8 standard error of the difference, would that concern
9 you as a statistician?
10 A. Would it concern me?
11 Q. Yes.
12 A. In saying that there is a statistically significant
13 difference between the Admissions policies in these
14 two years?
15 Q. Yes.
16 A. And thus say if I were concluding that I would say
17 that in 1997 they didn't give so much preference to
18 African Americans, and in 2000 they decided they
19 would give more preference?
20 Q. Exactly.
21 A. Would that concern me?
22 Q. Yes.
23 A. In the substantive part of this case, that doesn't
24 concern me because these are all large preferences.
25 Q. Okay. You don't think that that kind of instability
GRUTTER -vs- BOLLINGER, ET AL
191
1 in the odds ratios would call the validity in your
2 model to question?
3 A. Actually the fact that we got large ratios no matter
4 if they're as different as you said, and the
5 consistency of them in the cross years actually
6 makes me feel very comfortable with the substantive
7 conclusions that we have drawn into these models.
8 Q. I believe you said near the end of your testimony
9 this morning, that you thought you had quantified
10 the role that race plays in Admissions, did I hear
11 that right?
12 A. Did I say I had quantified it?
13 Q. Yes.
14 A. I'm not sure I said that as a direct quote, I may
15 have said something to that effect.
16 Q. Do you believe that you have quantified the role
17 that race plays in Admissions?
18 A. I think what I have shown is that race plays--the
19 individual selected minority groups are given a
20 large allowance with respect to Admissions
21 decisions.
22 Q. How big is that allowance?
23 A. How big is that allowance?
24 Q. Yes. Can you put a number on it?
25 A. I can give you examples of odds ratio for similarly
GRUTTER -vs- BOLLINGER, ET AL
192
1 situated individuals at various levels of GPA and
2 LSAT.
3 Q. But my question is, is it 443, or is it 53 for
4 African Americans?
5 A. And I would say that that specific number is in the
6 sense not important statistically, because they both
7 represent large allowance.
8 Q. So, we can't hang on any one of these numbers as
9 representing what the allowance that you say you
10 found actually is?
11 A. I mean I think it's difficult from year to year,
12 it's estimated to be different from year to year and
13 I think that's the nature of the process.
14 Q. Did you calculate competence in the roles around the
15 odds ratios for any particular year?
16 A. I probably did at some point look at competence
17 levels, yes.
18 Q. You don't remember what they are?
19 A. Well, I remember they're wide and they're fairly
20 wide and that's to be inspected.
21 Q. So, in other words, you have given us an odds ratio
22 on these charts, and maybe we should put the last
23 one back up for 2000, 443.
24 But really all we can know, or all
25 that your model can tell you is that it's likely to
GRUTTER -vs- BOLLINGER, ET AL
193
1 be 443 plus or minus something, right?
2 A. I think there would be a range, of course, that's
3 what we would expect if we get in competence levels
4 we get a range. And that range would be, I think,
5 fairly large. But that range would be far away from
6 one.
7 Q. Okay. The fact that the range is far away from one
8 is important, because that tells you that this is
9 not happening by chance, correct?
10 A. Certainly.
11 Q. Okay. But other than saying that "true odds ratios"
12 is something far away from one so that this effect
13 is not happening by chance, you can't really tell us
14 where around 443 we can be confident with the odds
15 ratio of one?
16 A. We can do calculations and present those, we can do
17 that. We didn't do that in our report.
18 Q. Okay. And you'd would agree that you really need to
19 know the competence intervals in order to know how
20 much weight we can put on a number like 443,
21 correct, in 2000?
22 A. I think that you can calculate competence intervals,
23 I think it's reasonable to look at competence
24 intervals. Yes, I agree with that.
25 Q. But you haven't reported that?
GRUTTER -vs- BOLLINGER, ET AL
194
1 A. I did not report that, correct.
2 Q. I'd like to talk for a minute about how the policy
3 of taking race into account in Admissions, affects
4 the non-minority student, the so-called majority
5 student.
6 Are you saying that the current
7 policy has a big effect on the likelihood of
8 admission for majority students?
9 A. Go back to the word likelihood again.
10 Q. Okay.
11 A. So, you're saying does it change the probability of
12 admission for a majority student by a lot?
13 Q. Much better way to ask the question.
14 A. I apologize for correcting, for suggesting. And I
15 think that, in fact, with respect to the aggregate
16 numbers, I think that it would not affect the
17 aggregate numbers a lot if the policy were changed.
18 The aggregate numbers.
19 Q. And that's because there's so few minority
20 applicants that we're talking about, right?
21 A. I mean there are a fair number of minority
22 applicants. I mean there's hundreds of minority
23 applicants.
24 Q. I should say so few of minority students who are
25 admitted?
GRUTTER -vs- BOLLINGER, ET AL
195
1 A. Well, it's the simple math is if a majority student
2 is denied admission in favor of a minority student,
3 that affect that majority student. And it's a one
4 for one.
5 And now, when we're talking about
6 percentages with different denominators, which is
7 what you're doing. It's surely true that if you
8 change 50 students, then the group with the larger
9 denominator will have a smaller percentage change.
10 And I think that's all you're saying.
11 Q. I think that's right. But just to make sure, let me
12 do it this way.
13 You're aware that there are only
14 between something like 165 and 190 minority students
15 who are admitted in any of these years, right?
16 A. I don't remember the numbers but that sounds in the
17 right range, that's true.
18 Q. Okay. So, even if every minority student were
19 rejected so they took nine, at most, that would mean
20 say 190 additional majority students who could be
21 admitted?
22 A. I think we're talking the same language. It's one
23 for one, yes. Absolutely.
24 Q. So, given that that's 190 majority students out of
25 2700 or 3000 majority applicants, the possible
GRUTTER -vs- BOLLINGER, ET AL
196
1 change and the probability of admission for the
2 majority student is small?
3 A. The proportion of majority students who would be
4 admitted would go up by a smaller amount, that's
5 right. Well, the denominator is bigger so the same
6 number means smaller percentages.
7 Q. Did any of your, I guess, I should do it this way.
8 Your analysis did not look at what would happen if
9 the University did not take race into account in
10 Admissions, is that right?
11 A. That's correct.
12 Q. Well, I think now I have only one more thing to ask
13 you about on odds ratios, and then we'll look at
14 some of your other graphs. But I want to go back to
15 this idea of what these odds ratio numbers really
16 mean.
17 If we look back at the 443 number, is
18 it fair to say based on this number in your view,
19 that an African American had a more than 400 times
20 greater chance of admission than a white student?
21 A. The odds of admission for, the odds of admission in
22 terms of odds, the odds of admission for an African
23 American student situated at the same credential of
24 LSAT cell, GPA cell.
25 That, in fact, there's evidence that
GRUTTER -vs- BOLLINGER, ET AL
197
1 there's about 400 times greater observed admission.
2 These are descriptions of what actually occurred,
3 and so that that's apparently what happened in those
4 cells.
5 Q. If the lawyers for the Plaintiff said that based on
6 this number, an African American had a more than 400
7 times greater chance of admission than a white
8 student, would that be a correct statement?
9 A. No.
10 Q. If we could put up, I want to turn now to the
11 curves, the graphs of select index versus
12 probability of admissions that we talked about. If
13 we could get slide 39, I think it is, David.
14 We obviously talked about these
15 earlier, and I'm going to try to be brief about
16 this.
17 The probabilities of admission that
18 you have on the vertical access here, those are not
19 the observed probabilities of admission for people
20 with a given index score, are they?
21 A. No. I described it was an estimation of the
22 function of probability admission versus selection
23 rate, yes.
24 Q. That prediction is from one of your regression
25 models?
GRUTTER -vs- BOLLINGER, ET AL
198
1 A. This technique actually has the name regression in
2 it, it's called isotonic regression. So they're
3 estimates from an isotonic regression model.
4 Q. So, the probability of acceptance that you're
5 graphing, I think you said this morning is really
6 the maximum likelihood estimate for the probability
7 function of Admissions, is that right?
8 A. That's actually right, I think.
9 Q. So, you didn't just look at the people who had a 3.0
10 index score and figure out what proportion of those,
11 that's not what this graph reflects?
12 A. This graph estimates the probability of admission as
13 a function of select index in a way that the
14 function will go up and to the right.
15 Q. And it's not the observed data?
16 A. The observed data?
17 Q. Yes.
18 A. Well, I mean, there aren't many applicants with
19 specific scores. So it would not be the observed
20 data.
21 Q. The index score across the horizontal axis, I think
22 you explained this morning is a combination with
23 some coefficient of grades and test scores, right?
24 A. That's correct.
25 Q. Is it your understanding that the Law School
GRUTTER -vs- BOLLINGER, ET AL
199
1 Admissions officers actually look at this index
2 score when they're making decisions?
3 A. Is it my--what I know about the index is it
4 certainly referred to an Admissions policy. And
5 it's included in the data base of information I was
6 given.
7 So, that index is in the Admissions
8 policy, and I actually don't know what Admissions
9 officers or decision makers have in front of them,
10 no.
11 Q. You haven't actually looked at the application files
12 that went with those numbers that you showed us
13 earlier?
14 A. I have looked at some application files, that's
15 true. I haven't looked at--for instance, for 1995 I
16 did not look at 4500 application files.
17 Q. But you looked at some of the samples of the files
18 that were produced, is that right?
19 A. I have looked at some files, yes.
20 Q. Did you see the index score reflected anywhere in
21 that sample file?
22 A. I should remember that, sorry. The answer is I
23 don't recall whether it's there or not, I really
24 don't know.
25 Q. I think you said earlier that your understanding, as
GRUTTER -vs- BOLLINGER, ET AL
200
1 a general matter, is that as the index score
2 increases, the probability of admission should
3 increase, is that right?
4 A. I think that's actually the Admissions policy.
5 Q. And so these graphs actually reflect that
6 assumption, don't they?
7 A. Sure. Actually these graphs reflect the data which
8 follow that assumption, yes.
9 Q. But you, I think, the phrase is kind of constrained
10 these curves to be non-decreasing, is that right?
11 A. I constrained these curves to be monotonic. That
12 means, if in fact, the policy--if it turned out that
13 fewer people were admitted as a function of selected
14 index, then they would go down into the right
15 consistently, and if more they would go up into the
16 right.
17 And we can see that there's variation
18 in the middle, certainly in the places where there's
19 zero and ones. All individuals beyond those points
20 were either admitted or denied.
21 Q. Okay. Now, I thought you said this morning that, in
22 your view, these graphs don't reflect any
23 assumptions, did I misunderstand that?
24 A. Any assumptions, I see. And you're going to say--
25 Q. (Interposing) I'm wondering, because monotonicity
GRUTTER -vs- BOLLINGER, ET AL
201
1 sounds like an assumption to me. And I thought you
2 had said earlier that the advantage of these graphs
3 is they didn't reflect any assumptions?
4 A. So, if in the sense that I did assume that as the
5 Admissions policy said, that however the admission
6 was related to monotonically, I didn't say how.
7 And so, these are what we call
8 non-parametric curves, they make a minimum of
9 assumptions. But as you have indicated, all of the
10 techniques involved some assumptions and there is
11 one there.
12 Q. And so the effect of your monotonicity strength, if
13 that's the right phrase, is that we don't ever see
14 the curve jog downward, right?
15 A. That's what monotonicity means, yes.
16 Q. But, in fact, from looking at the grids, we know
17 that some students in high cells, comparatively
18 higher cells are denied admission, whereas more
19 students in a lower cell might in one year happen to
20 be granted admission, right?
21 A. That's certainly in the grid cells, that certainly
22 occurs.
23 Q. So, these grids simplify the Admissions pattern in a
24 way that removes a lot of the detail, don't they?
25 A. In fact, the steepness of them indicates there is a
GRUTTER -vs- BOLLINGER, ET AL
202
1 monotonicity assumption, but, in fact, they go up
2 relatively sharply.
3 So, there is variation, but that
4 variation is indicated by the lack of straight line
5 up. I think I said that this morning.
6 Q. And doesn't your monotonicity assumption also
7 exaggerate the consistency of the spread between the
8 two curves?
9 A. I don't think that's true.
10 Q. In other words, if you graph the actual observes
11 probabilities of admission, wouldn't you see some
12 situations in which the curves are closer to each
13 other than the ones you have here?
14 A. Further and far apart, yes, both. We would see more
15 variation, see a very jagged curve, yes.
16 Q. We talked earlier about whether the effective race
17 in admissions varied depending on where a particular
18 applicant was, along the range of grades and test
19 scores, do you recall that?
20 A. Sure.
21 Q. Isn't it true that this graph suggests that that
22 effect is greater in the middle than it is on either
23 end?
24 A. Greater in the middle in what sense?
25 Q. In other words, this graph shows as you have created
GRUTTER -vs- BOLLINGER, ET AL
203
1 it, a larger gap between the two curves in the
2 middle range of selection index, than in either the
3 low end or the high end, isn't that right?
4 A. I mean at the very extremes there's no gap in the
5 sense that there's a hundred percent. But if we
6 look at the horizontal distance for various
7 probabilities, if you do that, I don't have a laser
8 pointer anymore.
9 But if we looked at the horizontal
10 difference, I mean look at the ones down at the
11 bottom. The horizontal difference, that's a pretty
12 big gap down there. And then there's a jump up and
13 then there's a gap. Actually in the middle it's not
14 very far apart.
15 And then at the high end the
16 horizontal gap gets bigger. So, in respect to the
17 selection index, there's actually, you know, it
18 varies and it's going to vary, because these are
19 real data.
20 But I don't see that their
21 necessarily much closer together at the top than
22 they are in the middle. In fact, it looks to me
23 like they might even be further apart in this
24 particular example. But it's going to look
25 different for different groups.
GRUTTER -vs- BOLLINGER, ET AL
204
1 Q. The vertical gap is the one that reflects the
2 difference in the probability of admission, correct?
3 A. The vertical gap sure, with respect in the
4 probability scale. Not in the on scale, but in the
5 probability scale.
6 Q. Correct. So, up until about 2.75 here on the left
7 side of the graph, the probabilities of admission
8 for the two groups are quite similar?
9 A. Up to 2.75?
10 Q. Yes.
11 A. You mean in the sense that there are no Caucasian
12 Americans admitted up to about 2.75? If I may
13 follow with that, there are no Caucasian Americans
14 admitted up to about 2.75. And from 2.3 on there
15 are Natives Americans admitted.
16 So, in fact, the odds ratio in that
17 case is infinity, even though the probabilities are
18 relatively close. Ten percent versus zero percent.
19 Q. So ten percent versus zero percent is the same in
20 this content as in infinite odds?
21 A. That's what we're seeing in this particular plot.
22 Q. And then at the upper end above, say, 3.4 certainly,
23 it's a little hard to tell from here. Both groups
24 have a probability of admission over 90 percent?
25 A. That looks to be true, yes. Correct.
GRUTTER -vs- BOLLINGER, ET AL
205
1 Q. Whereas in the middle range there is more of a
2 difference between the probabilities of admission?
3 A. Both are admitted. But, in fact, for Native
4 Americans is a hundred percent admission rate from
5 three on. So, with respect to looking at odds,
6 again we're in a situation where there are no denied
7 Native Americans who is beyond, say, three.
8 Q. Now, this graph doesn't tell us anywhere how many
9 applicants fall at any particular point along the
10 index score, right?
11 A. On the graph?
12 Q. Yes.
13 A. No, that's not displayed, the number.
14 Q. Here we don't know how many applicants are in the
15 range where there's a separation between the two
16 curves, and how many are in the ranges where there's
17 no or very little separation, correct?
18 A. From the graphs, that's true.
19 Q. If you could now, David, I would like to look at
20 slide 14. Which is one of your box plot for grades,
21 is that right?
22 A. Yes.
23 Q. Would you agree, Dr. Larntz, that the grades for
24 admitted students of all ethnic groups are quite
25 high?
GRUTTER -vs- BOLLINGER, ET AL
206
1 A. Quite how compared to what?
2 Q. Quite high along the standard 4.0 grading scale?
3 A. I guess I'm not familiar with anything other
4 than--these certainly look like good grades. Yes, I
5 agree with that.
6 Q. And, in fact, the median GPA for the majority
7 students runs in the range of an A minus, doesn't
8 it? Around 3.6 something?
9 A. Is that an A minus? I have to be careful. In
10 Minnesota we didn't use plus and minuses, so I have
11 a limited familiarity with that.
12 But they did put it in the year I
13 left. That wasn't the reason I left.
14 Q. But you would agree that all of these grades were
15 quite high, this is a qualified applicant pool
16 across these ratios?
17 A. That I would agree it's qualified?
18 Q. Yes.
19 A. This is a comparison plot, and what the plot says is
20 how the grades compare. And they compare--the
21 overall level of the grades are what they are,
22 they're numbers.
23 Q. So, you would leave it to the Admissions Office to
24 determine whether, in fact, all of these ranges are
25 what they would consider to be highly qualified?
GRUTTER -vs- BOLLINGER, ET AL
207
1 A. Well, I certainly am not going to say that I know
2 that they're highly qualified or not, I don't know
3 that for a fact.
4 Q. So that issue is to be left to others?
5 A. Highly qualified in a statistical sense, I haven't
6 done no analysis to quantify that.
7 Q Fair enough. Let's look at the next one, I think,
8 slide 15. Same thing for LSAT scores, have you
9 looked at where the medians, the various medians
10 that you report fall on the percentile scale for the
11 LSAT?
12 A No.
13 Q So, you have no view, again, about how "high" all of
14 these test scores are?
15 A These are comparative plots, that's true.
16 Q Did you consider how the gaps between the medians or
17 between the boxes compare to the arrow of
18 measurement of the LSAT?
19 A I don't think that's relevant to looking at the
20 comparison. In fact, I understand that there would
21 be a test retake variation in taking the test. But
22 what is summarized here are the coretiles and the
23 medians, in particular, of a large, mostly, for a
24 large group of individuals that were admitted.
25 Q But you understand that at some point fine
GRUTTER -vs- BOLLINGER, ET AL
208
1 differences between two applicant's LSAT scores
2 don't mean very much, right?
3 A What I know which is something about this, but not a
4 great deal. What I know is that Admissions
5 decisions are made based on these LSAT scores, I do
6 know that.
7 And I do know that the Admissions
8 decisions seem to be taking account of specific
9 points of the LSAT, we saw that in the analysis.
10 Do I know substantively how much a
11 difference of a certain amount of point needs, the
12 answer is, I don't know.
13 Q Could we go back, David, to slide 14. Did you do
14 anything to look at the absolute numbers that fall
15 within, say, the lower ranges of these box plots?
16 That's a bad question, I'll rephrase it and be more
17 specific.
18 If you look here at the second
19 column, which is for African Americans, I think you
20 said this line is the 25th percentile?
21 A That's correct.
22 Q And this bracket reflects the lower range, the lower
23 end of the range of normal value, is that right?
24 A That's how I describe it, yes.
25 Q And is that about another 15 percent of the
GRUTTER -vs- BOLLINGER, ET AL
209
1 students?
2 A Well, I think it's probably closer to another 25
3 percent.
4 Q Well, I guess I should ask it this way. There's a
5 total of 50 percent of the admitted students in the
6 colored part of your chart, is that correct?
7 A That's right. The box represents a group that
8 corresponds to the 75th percentile and 25th
9 percentile. So, half of the students are within the
10 box, and thus half of the students are outside the
11 box.
12 Q Okay. And the other half of the students, for the
13 most part, are within the brackets, correct?
14 A Oh, sure, absolutely.
15 Q Except for the occasional outlier, I think?
16 A And the outliers are defined relative to be actual
17 box themselves. So, once an outlier in one column
18 differs from once an outlier in the other.
19 Q So basically the entire group of accepted applicants
20 is between these two brackets?
21 A It looks like with one exception. Again, let me be
22 very clear. It's possible there were two or three
23 at that particular level, but it's doubtful. It's
24 doubtful, it's probably wrong.
25 Q Did you do anything to compare, say, the number of
GRUTTER -vs- BOLLINGER, ET AL
210
1 applicants in this 25 percent range here, as against
2 the number of outliers over here, for example?
3 A The wrong number?
4 Q Yes, the wrong number.
5 A And I'm going to have to--my recollection is there
6 are about 450 African American applicants in this
7 year, so there would be about 25--I can't do the
8 math, sorry. About a hundred in that range down
9 there, about a quarter of them, 25 percent.
10 And then you count the number
11 of--now, of course, the number of applicants of,
12 Caucasian applicants, is over 2000. It was like
13 2300 or something, I think that's right.
14 And so did I do any comparison of the
15 actual numbers of Caucasian applicants that are in
16 that same range with the hundred lowest African
17 American GPA?
18 Q Right.
19 A No.
20 Q You would agree that based on your analysis
21 undergraduate GPA and LSAT scores are very important
22 in the Admissions process?
23 A Certainly.
24 Q And that's what you've concluded based on your
25 models, right?
GRUTTER -vs- BOLLINGER, ET AL
211
1 A Actually I concluded that based on looking at the
2 Admissions policy and also looking at the grids to
3 begin with, I think the grids basically tell the
4 story. They may tell all the story. But it's
5 certainly clear from the grids.
6 Q Your models tell the same story, don't they?
7 A With respect to those?
8 Q To the importance of grades and test scores in the
9 Admissions process?
10 A The models make virtually no assumption about that.
11 So I don't want to say that my models say that. I
12 do know from things that weren't presented that LSAT
13 and GPA are very important in the Admissions
14 process. But not from what you have presented
15 today.
16 Q But for the work that you did for this case, your
17 models leads you to conclude that grades and test
18 scores are very important in the Admissions process,
19 right?
20 A It's very clear they were.
21 Q And, in fact, maybe not in what you presented here
22 today, but in the models that you report, your
23 models show that grades and test scores have the
24 strongest assocation with Admissions decisions,
25 isn't that right? Strongest association of any
GRUTTER -vs- BOLLINGER, ET AL
212
1 factor?
2 A I'm sure that that's true, even without quantifying
3 it specifically that their grades and LSAT are
4 important for all the applicants, yes.
5 Q For all applicants, regardless of racial--
6 A (Interposing) Certainly if you look at the grids,
7 you can see within each racial group, you can see,
8 each ethnic group, you can see the effects and the
9 effects are very strong within each group. Although
10 the point at which admissions are done are different
11 for the two groups.
12 Q And, in fact, you believe from your models, don't
13 you, that grids and test scores have stronger
14 associations with admissions than an applicant's
15 race, is that right?
16 A It's certainly true if you look at the grids and
17 from my models, that individuals that are at very
18 low grades and very low LSAT will not be admitted,
19 no matter what.
20 Individuals with high grades and high
21 LSATs will almost certainly be admitted if they are
22 members of a selected minority group. And have a
23 higher chance, but not the same certainty as an
24 admission for a majority non-selected minority
25 group, that's right.
GRUTTER -vs- BOLLINGER, ET AL
213
1 Q Would you agree that based on your models, grades
2 and test scores are the most important factor in
3 Admissions?
4 A The most important factors?
5 Q Yes. Is that what you conclude from your model?
6 A I don't know that I quantified it specifically, but
7 I think they are very important.
8 Q In fact, didn't you conclude based on your models
9 that grades and test scores are more important than
10 race in the Admissions process?
11 A Grades and test scores are more important than race?
12 Q In the Admissions process?
13 A In a statistical measure?
14 Q Yes. Based on your models?
15 A I would think that probably if I did a quantitative
16 measure which I didn't do particularly, that I would
17 find that that's true. In the sense that what my
18 models are doing is looking at the effected race
19 beyond the grid cells, and I assume that the grid
20 cells were important to begin with.
21 Q So, the results of your statistical analysis are
22 that grades and test scores are more important in
23 the Admissions process than race, is that right?
24 A I would say in my statistical analysis starts with
25 the assumption that we want to look at the
GRUTTER -vs- BOLLINGER, ET AL
214
1 individuals with the same grades and test scores.
2 That premise does say that grades and test scores
3 are, in fact, important. And I believe they're
4 important.
5 To say they're more important in given
6 situations, I looked at comparisons of ethnic groups
7 by combination of grades and test scores.
8 I did not look--I did also look at
9 grades and test scores by ethic groups, but I don't
10 want to say if I ever looked at the Admissions
11 process without taking grades and test scores into
12 account.
13 Q I just want to make sure that I'm understanding you.
14 Are you saying that grades and test scores, according
15 to your models are, are not more important than race
16 in the Admissions process?
17 MR. KOLBO: Your Honor, I need to
18 pose an objection. This has been asked and answered
19 THE COURT: Sustained, it has been.
20 BY MR. DELERY:
21 Q I think, Dr. Larntz, you ended your testimony this
22 morning by saying that you conclude that you found an
23 incredibly large allowance given to selected minority
24 applicants in the Admission process; is that a fair
25 summary?
GRUTTER -vs- BOLLINGER, ET AL
215
1 A In comparison to other applicants, that's true.
2 Q And you think you have quantified that allowance?
3 A I think that allowance is there, it's large, and I
4 have quantified it in a statistical sense year by
5 year. And the substance of the conclusions remain
6 year by year, yes.
7 Q And you quantified it based on your models?
8 A Yes, based on my models and the examination of the
9 grids and the whole composite of that. The whole
10 set of analysis I did, yes.
11 Q Do you think you quantified it even though your
12 models don't include all the factors the Admissions
13 Office take into account?
14 A Even though we don't include all factors--which by
15 the way, all statistical analysis there--I don't
16 know of any statistical analysis that account for all
17 possible factors.
18 Q You think that it is a quantification of the role
19 that race places into admission, even though you have
20 excluded the number of cell from the model?
21 A I think what I did is and an appropriate statistical
22 analysis for the comparison. Most of the cells that
23 were included, most of the individuals that we talked
24 about are in very low combination of the LSAT and
25 GPA.
GRUTTER -vs- BOLLINGER, ET AL
216
1 There are very few at the upper end
2 that are included, most are at the low end and where
3 there were no students admitted. So there was no
4 comparison made.
5 Q And do you think that you have quantified the role
6 that race plays in admission, even though the number
7 that you give for that roll varies from year to year?
8 A Certainly. I think that the degree which varies
9 from year to year is not surprising. And certainly
10 I think that the fact that it's substantively the same
11 from year to year is confirming.
12 Q And that's true despite the fact that the Admission
13 policy hasn't changed in these years?
14 A The Admission policy hasn't changed. But, in fact,
15 what I'm trying to describe is admission to the curve
16 during those years regardless of what the policy was.
17 Q And you have done that?
18 A I have done the best I could, yes.
19 MR. DELERY: Thank you. I have no
20 further questions, your Honor.
21 THE COURT: Okay. You want to start
22 today or you want to do it in the morning?
23 MS. MASSIE: The morning.
24 MR. KOLBO: Your Honor, I do want to
25 alert the Court to the fact that Dr. Larntz has a
GRUTTER -vs- BOLLINGER, ET AL
217
1 plane he has to get tomorrow at 1:00. And I assume
2 that's not going to be a problem.
3 We're happy to continue tonight if the
4 Court wants to do that. But I didn't want to
5 surprise the court tomorrow.
6 THE COURT: I'd love to do it tonight,
7 I thought we would finish him today. I have made
8 some plans and it's too late to cancel right now.
9 Let's start at 8:30 tomorrow morning.
10 MR. KOLBO: Is it my understanding,
11 your Honor, that Dr. Larntz will be able to leave at
12 least by noon tomorrow?
13 THE COURT: Yes, so he can get to the
14 airport on time. If there is some reason we don't
15 finish with him, we'll do it by video conference
16 somewhere during the trial, since we have had a
17 chance to see him.
18 We can give him the exhibit books, we
19 can work it out. I think the federal court--where
20 do you live?
21 A I will be traveling a good deal for the next few
22 weeks.
23 THE COURT: Okay, we will set it up
24 for a video conference. All of the federal courts
25 have them now. We can start him at 8:30 tomorrow and
GRUTTER -vs- BOLLINGER, ET AL
218
1 then we'll make sure that we can get him out.
2 MR. KOLBO: I appreciate that, your
3 Honor.
4 THE COURT: I will be here, so it's up
5 to you guys.
6 (Court adjourned at 4:45 p.m.)
7 - - -
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
GRUTTER -vs- BOLLINGER, ET AL
|
|