Ep. 4-Dr. Itiel Dror-Science of Bias

Dr Dror researches expert performance in the real world, examining medical surgeons, military fighter pilots, front line police, and forensic analysts. His research provides insights into the inherent trade-offs of being an expert. In the forensic domain he has demonstrated how contextual information can influence the judgments and decision making of experts; he has shown that even fingerprint and DNA experts can reach different conclusions when the same evidence is presented within different extraneous contexts. 

Originally aired Sep 17, 2020

Transcript

00;00;30;05 - 00;00;49;10
Unknown
Hello, everybody, and welcome. My name is Eugene Liscio and welcome to episode four of Forensics Talks. Okay. So, we're going to get started today with our guest. I have Dr. Itiel Dror and since 2005, he's been the principal consultant of cognitive consultants in the UK.

00;00;49;21 - 00;01;06;29
Unknown
Since 2012, he's been an honorary senior researcher for the Center of Forensic Sciences University College, London. From 2019 to 2021, he was appointed by the Governor of Massachusetts as a member of the Forensic Science Oversight Board. He's been awarded many grants.

00;01;07;16 - 00;01;29;29
Unknown
I let him know just before that he has 144 papers, just so in case he wasn't sure. He's also been author or coauthor of four books. He has an extensive travel regime with workshops, training and conferences. I first met him all probably way back in 2000, and I'm going to say it's 2014 or something at the Canadian

00;01;30;05 - 00;01;45;01
Unknown
Society of Forensic Science Conference, and his CV is absolutely too long to talk about. So, I'm just going to bring him on. Dr. Dror, thank you very much for being here. My pleasure. All right. Well, I usually like to start at the beginning.

00;01;45;11 - 00;01;58;27
Unknown
I want to ask you, what first prompted you, if you can go back and remember when you were an undergraduate or, you know, early academic, what was your interest or your passion about the human mind or what prompted you to move in this particular discipline?

00;02;00;17 - 00;02;20;11
Unknown
The human mind. And this discipline is not exactly the same. But if you wanted me to go very back, you can tell me when to stop. I a my interests go off on philosophy, philosophy of mind, trying to understand what underpins the human mind and how we philosophy formation.

00;02;20;19 - 00;02;43;12
Unknown
And I remember that I was basing it on a children's fairy. I remember reading Pinocchio in the story of Pinocchio. You know, the carpenter builds a doll, but it's not a life until, you know, the fairy comes and puts kind of dust or some kind of very spiritual life material, you know, on the dark to make it

00;02;43;12 - 00;03;06;06
Unknown
from an inanimate, inanimate object. And then I contrasted with the story of Frankenstein, where Frankenstein and the doctor was building a living creature, not because the fairy came and gave it to thought or mind, but because of the physical way in which he produced Frank and the Monster.

00;03;06;16 - 00;03;28;14
Unknown
So, this is kind of very two different constructing perspectives, old story of the human mind. And I was interested in philosophy of mind and studied philosophy, and that's where I started. And maybe I shouldn't say that I was going to tell you between you and I, but then other people, honestly, I don't know if a God.

00;03;28;15 - 00;03;44;11
Unknown
Jerry Springer. And it's a secret to the entire nation, but it's not only when I go, but my real interest is in philosophy. Okay. Interesting. Well, you know, everybody has something, I think, that that drives them. And I think that's very interesting.

00;03;44;11 - 00;04;05;26
Unknown
And certainly, I think your study of the cognition, human mind and everything I think falls into that. We're going to be talking about bias, obviously. So, I want to start with the problem and the frequency of bias or how big of a problem do you feel bias is in the forensic sciences?

00;04;07;01 - 00;04;25;26
Unknown
First of all, to say we're going to talk about bias, obviously. Well, I don't know why, obviously, because I study expert decision making, not only in forensic science, they work a lot in aviation, in finance, in the medical domain and in forensic.

00;04;26;04 - 00;04;47;08
Unknown
It's not only about bias. How do people make decisions, however, bias. And even before the last few months or years, where bias is really turned into a hot topic in society. Bias just got a lot of attention. Not because it's a sensitive topic that too, but a lot because the forensic examiner claims they were unbiased.

00;04;47;15 - 00;05;07;11
Unknown
They were all impartial then. Well, infallible. But my research is not about bias. It's about human decision making. When people make mistakes, when they don't make mistakes, human factors, cognitive factor and bias is only one part of it that gets a lot of attention because of the response of the for in the community.

00;05;07;11 - 00;05;20;27
Unknown
You know, when I started the channel, the Finger Point Society said none of us are biased, we're all objective. And if anybody is biased, they should go and walk in Disneyland. Yet I'm quoting what he said ward by ward.

00;05;21;05 - 00;05;35;23
Unknown
So, Bias got a lot of attention. But it's not about bias. It's about human performance and understanding that in forensic science, a human is very important. Before I came in, it's like the human didn't exist, the elephant in the room.

00;05;35;23 - 00;05;50;11
Unknown
And I always say, no, the human is very important. The human is part of the process, and it was totally ignored. They talk about the evidence, and I remember in one of my first workshops, somebody came up and stood up quite angry and said, the finger planted doesn't lie.

00;05;50;11 - 00;06;08;24
Unknown
You know, that was quite clever. But after a few seconds they said, yes, the fingerprint doesn't lie, but the finger point doesn't speak, right. It's a human person that needs to interpret it. So that's kind of before I even answer your question or to answer your question about bias, is it a big problem or not?

00;06;08;24 - 00;06;31;09
Unknown
Well, depending on your criteria of what the big problem is, it depends on how you define a problem. You know, in some domains, they make many mistakes in the medical domain, you know, the medical errors. Many people died because of medical evil and people accept it and they don't think it's a big problem compared to aviation.

00;06;31;13 - 00;06;46;28
Unknown
If one person fly, they're an airplane. You know, they ground all the airplanes because the society has certain norms and values or I'm not sure what they mean to call it what their accepted acceptable deaths. What's normal? What's not normal.

00;06;47;07 - 00;07;02;14
Unknown
So, I don't know what criteria is it? Is it a problem? Well, it is a problem. And whether it's a small problem or a big problem, whatever is small or big is which we. Can discuss for days and not move forward.

00;07;02;22 - 00;07;19;22
Unknown
I think that forensic scientists do a very important job, and no matter if they have big problems or small minuscule problems, they should always think, are we doing a good job? What are we doing well? What are we not doing well and how we can improve?

00;07;19;29 - 00;07;33;15
Unknown
The only thing I would tell you that they don't accept anything, which I hope not many people still say. But when I can talk in forensic scientist, you know, ten, 15 years ago they said, we are perfect, we are infallible, we never make a mistake.

00;07;33;16 - 00;07;46;20
Unknown
And I remember when I submitted my first paper in the journal, one of the reviewers said, well, one plus one equals two. If someone says one and one equals three, does it mean that mathematics is wrong? The person is an idiot.

00;07;46;20 - 00;08;01;11
Unknown
So, any time a mistake is made in forensic science, it's not because of issues in forensic science that the examiner is definitely incompetent and needs more training because we are perfect, and the domain is infallible. That is not true.

00;08;01;19 - 00;08;14;11
Unknown
And then add one more thing. Sorry for going on and on and on. No, it's just. This is perfect. I walk in, in the medical domain. I within a conference giving a keynote in front of 3000 medical doctors.

00;08;14;11 - 00;08;27;12
Unknown
And they say, how many of you made a mistake? They are picking the hand up. I say, how many of you have killed the patient because you made a mistake? There are big standards in the forensic conferences. Ten years ago, I would say.

00;08;27;12 - 00;08;42;01
Unknown
How many of you made the mistake? Nobody picks a hand up. I say, look, you don't know you made the mistake. But if you go back to every mistake and every decision you've ever made, could you maybe you've made the mistake, and nobody picks the hand up.

00;08;42;01 - 00;08;53;09
Unknown
They said it's impossible. So, you know, I'm a bit and otherwise I'd go went down and started to shake people hand. They said, why are you shaking our hand? They said, I always wanted to meet God. Yeah. What does it mean?

00;08;53;09 - 00;09;11;29
Unknown
That it's nothing to. You haven't made a mistake, but it's impossible that you've ever made a mistake. This is unacceptable. Even now, the forensic domain is moved forward. In the last decade or so, we all know that people make mistakes, even cold walking, competent, dedicated forensic examiners.

00;09;12;05 - 00;09;33;04
Unknown
And whether it's a big problem or a small problem or medium problem, we need to look at it to understand it and try to minimize it to do the best we can as experts. Okay. Could you very briefly talk about the types or three types of cognitive bias, one, contextual confirmation and expected frequency?

00;09;33;04 - 00;09;53;07
Unknown
Could you just explain very briefly between the three? I don't know what you pick those three and there are many other ones and I'm not sure I would have picked obviously maybe some of them. The I think the important thing, first of all, I never give different if it's also when I teach at university because students learn

00;09;53;07 - 00;10;09;12
Unknown
definitions and they don't understand that some people really understand the concept but can give you a formal definition. So, let's not focus on the formal definitions of different biases. Then the viewers can Google it and find it in my papers when I'm forced to do that.

00;10;10;02 - 00;10;32;14
Unknown
Generally speaking about diffusing biases in general, our people go astray, but not randomly systematic. They go astray, not driven by the actual evidence. And that can happen. I would say eight different sources of bias fits in one of my papers that recently came out.

00;10;32;14 - 00;10;54;09
Unknown
I believe you have a link on eight sources of bias. I would mention two major ones. One is contextual information where you get excuse me, an expectation. Yeah, that's a paper and it's open access where anyone can freely accessible it, publicly accessible with the paper in the details.

00;10;54;19 - 00;11;13;16
Unknown
Some of those biases derive from an expectation that you get from irrelevant information. So, you expect to find something because, you know, the suspect confessed to the crime. You know, there's DNA convicting or identifying the person and so on and so forth.

00;11;13;24 - 00;11;33;10
Unknown
That kind of an expectation before you actually look at the evidence with this kind of contextual information? Well, you already your brain expects to find something, and the brain sees in a way what it wants to see and ignores things that it doesn't want to see or explain away what doesn't fit the expectation?

00;11;33;10 - 00;11;56;10
Unknown
The human mind is very dynamic, it's very flexible, so it can do that. So, this is a kind of a confirmation bias by contextual, irrelevant information. So, there's a kind of a group of phenomena. The other group that I would also mention is when people go backwards, you need to go from the evidence to the suspect and

00;11;56;10 - 00;12;12;02
Unknown
what happens in fingerprints and firearm and DNA. Sometimes they go from the suspect to the evidence, so they see the DNA profile of the suspect or the fingerprint of the suspects, and then they go and look for the suspect in the evidence.

00;12;12;03 - 00;12;39;03
Unknown
That's called backward reasoning or backward tracking. You need to start with the evidence. We call it lasu. Linear, sequential unmasking started to act. The evidence, the evidence needs to drive your decision making. And after you see the evidence without seeing the suspect, without the reference material, then you are due the comparison to the actual suspect.

00;12;39;14 - 00;12;55;25
Unknown
I also need to mention that this group of bias should have subgroups and there are other biases, and it gets a lot more complicated. But it's important to emphasize that I'm talking about implicit biases. I'm talking about hardworking, dedicated forensic examination.

00;12;55;25 - 00;13;13;15
Unknown
This many of them get very defensive, defensive, aggressive when they say you're biased because it's not people who are intentionally biased, they're not aware of it, are very hardworking, competent, and they get very annoyed at it because they think they're doing a great job and they're doing a great job, but they can do a better job is

00;13;13;15 - 00;13;34;07
Unknown
a bit open minded to consider the influences in their decision making, especially in forensic domains that have some discretion, judgments, objectivity. Then those biases are more powerful. And you often talk about something like the illusion of control, and you say that acknowledging bias can be very helpful.

00;13;35;14 - 00;13;49;19
Unknown
I'm curious about the ways to help fix the bias problem. But you know what? What do you have to say about first of all, first of all, acknowledging the problem like an alcoholic acknowledging they're an alcoholic. Right.

00;13;50;14 - 00;14;00;28
Unknown
That drinks are not bad. I use quite a lot in my training that, you know, the problem with many alcoholics that they say I'm not an alcoholic, I just enjoy drinking all the time and I can stop any time.

00;14;01;05 - 00;14;27;02
Unknown
The minute you acknowledge you're an alcoholic, you're halfway in moving to recovery and sort of your issue with alcohol. So, acknowledging that people and experts and you can be biased by irrelevant contextual information or any one of the different sources of bias is a great step forward of people to do that.

00;14;27;02 - 00;14;39;27
Unknown
If you don't acknowledge it, we're not going to be able to start dealing. We have what we call the buy the bias blind spot. It's quite easy. You see bias in other people, but harder to see your own bias.

00;14;40;13 - 00;14;53;21
Unknown
So acknowledging is necessary, but it's not sufficient. Many people say, oh, now I know I'm biased, I get it. They can be biased. And I try to control it by sheer willpower. By sheer willpower, you cannot control bias.

00;14;53;21 - 00;15;05;25
Unknown
We can control our thought processes. We all know that, you know, when your boyfriend or girlfriend dumps you, you don't want to think about this. Well, you can't stop thinking about it. In fact, we call it ironic obsessing.

00;15;06;02 - 00;15;22;02
Unknown
The more you try not to think about it, the stronger the bias. It's like a judge telling the jury, please ignore that the suspect had a criminal record for murder before. Remember, do not pay attention and do not think that he did the murder them only to ignore it.

00;15;22;09 - 00;15;35;12
Unknown
The more the brain takes it into account, subconscious and without awareness, it's like somebody. It's like somebody saying, don't think about a pink polka dotted elephant or something like that, and then you can't stop thinking about it. Absolutely.

00;15;35;12 - 00;15;48;14
Unknown
And you can't unsee what you've seen. So, when you're exposed to irrelevant information, okay, I'm not going to take it into account once you're exposed. There's no way back on that. So, you have to take actual steps to minimize bias.

00;15;48;14 - 00;16;10;03
Unknown
But if people don't start to acknowledge its existence, then it's already moving ahead. And then then you can start thinking about how do I minimize exposure to irrelevant information? How do I not go backwards? What they mentioned in a sequential unmasking, and I believe the issue of bias is going to be moving forward quite a bit because

00;16;10;03 - 00;16;30;01
Unknown
ISO accreditation, ISO 1720 and even ISO 1725, which is for the more objective domain now, require showing impartiality, freedom from bias. So, this is going to be a good pressure and you need a bit of cognitive background to understand the issue.

00;16;30;06 - 00;16;41;01
Unknown
So. Saffo And the examiners are getting into it and that's great, but really studying the mind in the brain, it's not something you're going to read a book or ten books or a year and be able to understand.

00;16;41;01 - 00;16;58;16
Unknown
So, if you want to really understand the wider picture and understand the underpinning of bias, you can really effectively deal with it. You need to get a bit more into it, and that's quite a bit more effort in getting some type of cognitive informed training on these issues.

00;16;58;24 - 00;17;14;15
Unknown
It's interesting here in Canada, on the civil side, they have something called a Form 53. And so, what that is, is a form that they send to the expert that you have to sign. It's a one-page form, and it basically says that your duty is to the court and not to the side that retained you.

00;17;14;23 - 00;17;27;29
Unknown
And I find it interesting because it's just a little piece of paper. You sign it, you send it back, but it has absolutely no bearing whatsoever on whether or not you are. So, there's different agencies. Some are going to be labs.

00;17;27;29 - 00;17;42;05
Unknown
They're going to be very big. They have a very big structure and a lot of people. So, they can do a lot more of the linear, sequential unmasking. And then there are, for example, smaller, even private companies or people like me that are very small.

00;17;42;23 - 00;17;58;04
Unknown
And I get calls sometimes and it and I do about 50, 50% prosecution and defense and it doesn't matter which side, but I've seen it before. Sometimes they want to tell you about the client, or they want to tell you about the suspect and how he's such a great person and everything else.

00;17;58;04 - 00;18;16;00
Unknown
So, the contextual by the start is really bad. What can I say personally? What can I do when I know that it's coming in? I tell you what I do. I don't appear like in court, but I've been involved in cases in Canada and in the US and UK and other country.

00;18;16;07 - 00;18;33;26
Unknown
For the defense and for the prosecution and the lawyers. I don't want to ascribe intention to them. Probably it's intentional, but regardless if it's intentional or not, the first thing that comes out of their mouth when they talk to me to let me believe feel about the case, let me give you the background.

00;18;34;03 - 00;18;45;02
Unknown
And that's why I tell them to shut up in a polite way as much as I can, which is not a lot of time. And I say, please don't tell me the background, but if you tell me the background, you're going to bias me.

00;18;45;09 - 00;19;04;22
Unknown
I want you to just give me the relevant information that you need me to look into. And it's very interesting because your data here about the case and I didn't know a lot of irrelevant information. Now, the one the case where they gave me the information and they said, well, don't take it into account, they said, I'm

00;19;04;22 - 00;19;25;01
Unknown
really sorry. You gave me this information. You burnt me. I can't delete it from my mind by mail willpower. And I said, I'm not taking your case because they provided me with irrelevant contextual information. But you can tell them not to say that if you want it back, they can, even though very academic and they live in

00;19;25;01 - 00;19;35;25
Unknown
the ivory tower, a very practical person, if you tell the lawyer to shut up or therefore the examiner through the police detective to shut up, they're going to get into trouble. What do you want to tell them is.

00;19;36;02 - 00;19;51;02
Unknown
Say, look, before you say anything to me, this issue of bias is very strong. Now that this crazy doctor, Dr. Dong, talking about it, and they're going to ask me in court when I testify in court. The other side is going to say, what did they believe?

00;19;51;07 - 00;20;06;24
Unknown
What did you know about the case? And they're going to make me look bad and bias. I want to help you and I can help you better if you don't tell me all the briefings that you know I don't need just to make me feel sympathy to the client or to the victim, whatever the case.

00;20;07;02 - 00;20;26;24
Unknown
So, if they know that you're going to testify, they're going to be cross-examining you on that. And in fact, even without cross-examining, I call for reports when you have to report everything you knew. Right. What irrelevant, contextual information you knew about the case and some laboratory they've adopted internally, quietly.

00;20;27;00 - 00;20;37;29
Unknown
So, you tell them before you say anything on the phone to me, before you email me, know that I'm going to disclose what you're telling me. Don't make me look bad like you told me. Like you won't believe some lawyers.

00;20;37;29 - 00;20;52;21
Unknown
What they say is, I'm not talking about information. They say, no, this is a scam of, oh, this may burn in hell, help me convict this person. And all the staff know that it's not between you and I anymore and under the table.

00;20;52;21 - 00;21;13;18
Unknown
So, this are different techniques to minimize that. But, you know, you guys, you and other people can come up with your techniques as good as I may be better than I. As long as you acknowledge that it can affect you subconsciously and you understand some of the cognitive mechanisms that underpin that, and you can develop new

00;21;13;18 - 00;21;35;29
Unknown
techniques and you can adapt them to big labs or small labs, and depending if you work for the police or you do work for the defense, it's all different manifestation of the same basic cognitive mechanism. Well, I probably have to thank you, because on two occasions I actually testified at trial, and one in particular, the defense lawyer

00;21;35;29 - 00;21;48;22
Unknown
asked me very specifically, are you aware of what cognitive biases? And I was very fairly well prepared, let's put it that way. And I said, yes, I do. There's been papers, there's Dr. Rachel Draw. He has this in this.

00;21;49;02 - 00;22;03;21
Unknown
And after I rang off a list of papers and people, he didn't ask me any more questions. But I think the point about it is a very good point is that if you acknowledge that it can happen and that there probably is some influence externally, you can try to mitigate it.

00;22;03;21 - 00;22;16;08
Unknown
I know here one of the things that we try to do is when we receive information from a case, sometimes we have I hand the data off to somebody else in my office, but I let them work independently without me.

00;22;16;19 - 00;22;32;01
Unknown
I'm sort of like the filter. I don't let them know everything that I know, and then we work independently and then we bring our numbers back together after and then we compare. So, I know that you have some recommendations for the way that the information is handed off to the people doing the work.

00;22;32;01 - 00;22;51;02
Unknown
So, if you're a latent print expert and maybe you can talk of something about this is how you the supervisor committee could limit or filter some of the information. So. Yes, there's so many things going on tend to be a bit organized.

00;22;51;11 - 00;23;05;11
Unknown
So, one of the things is who will filter out the information to decide what's relevant and irrelevant? You don't want the police detective to do that. So, depending on a lot of that stuff, the lack of many solutions to that.

00;23;05;19 - 00;23;22;05
Unknown
One of them is what we call a case manager. In a case manager. You have someone who gets all the information and decides what's relevant or not and gives it to another examiner. So, in the case manager, I get everything right and then I give you what is relevant.

00;23;22;05 - 00;23;43;07
Unknown
Now everybody wants to be a case manager so we can rotate in one instance. I'm a case manager and I give you the relevant information. In another case, you the case manager, you decide what's relevant or not. Now, remember that it's not only the case manager important at the beginning to filter out information, but often once you

00;23;43;08 - 00;24;04;12
Unknown
write a report and reach your faulty conclusion, the detective doesn't always know how it relates to the case. They need a bit of translation. What the forensic findings mean about the case. So, the case manager not only walks is the beginning in terms of filtering information, but later they meet with a detective and explain the findings and

00;24;04;12 - 00;24;22;13
Unknown
what it may mean to the activity level and to the case itself by the actual forensic examiner. Making the decision is not part of it. That's a case manager. The go between the detectives and all the information and the people actually making their decisions themselves.

00;24;23;03 - 00;24;40;10
Unknown
Okay. I wanted to ask you about other ways that maybe things could improve, like, for example, proficiency testing. What kind of things can people do in their proficiency tests? Because I've seen some proficiency tests and they're like some of them are going to be careful what I say.

00;24;40;14 - 00;24;55;23
Unknown
They're just let's put it this way, they're not challenging. They're extremely basic. They're not meant to challenge a person who is, you know, experience or has been doing the job for some time. But what can you include in a proficiency test that may assist?

00;24;56;20 - 00;25;19;27
Unknown
You know, what you're talking about is, again, a very big and important question, proficiency testing and anyway, studies that addressed the paper that came last week in Forensic Science International Center. Again, it's open access on our way. And I've been a paper in a for a regional area, which means that forensics come out of forensic science in

00;25;20;04 - 00;25;41;18
Unknown
called the flow in evaluate and that's a title of the paper. So, they have a huge amount of challenges and problems in proficiency testing and in a yeah that's are one and. Where? First of all. Do they mimic?

00;25;41;18 - 00;26;01;12
Unknown
Are they similar to casework in terms of the difficulty, in terms of including inconclusive evidence, in terms of what do you do with someone said inconclusive decisions in terms of people know they're being tested versus say that they've tested whether they think it's case work.

00;26;01;13 - 00;26;15;25
Unknown
I would give the example that I don't drive according to the speed limit all the time. But when I'm in a driving test, I know that a police officer with their police camera or whatever, I drive by the speed limit, you know, people behave very differently.

00;26;16;04 - 00;26;33;26
Unknown
And so, we have that. We have a whole range. You know, if you have bias during evaluate and proficiency testing, they just look at that every day. So not kind of with contextually relevant information, which happens in a lot of the case or so, the proficiency test, the studies are a good step.

00;26;34;04 - 00;26;51;16
Unknown
We need them, but they require and they sit in their British way. They require quite a lot of improvement in a lot of issues. Some of the issues are just very, very challenging to design them properly. So, it's very big, big question.

00;26;55;16 - 00;27;07;14
Unknown
Oh. Are you there? Yeah, I'm here. I'm here. Oh, my boy. You. You disappeared. No, no, not at all. In fact, as you were speaking, I wanted to bring up you. You've we've got some really good references for some of the people here.

00;27;07;14 - 00;27;22;14
Unknown
So, this one here by us is in forensic experts. It's only it's a very short article, but I think it's a good first read if people are interested in some information. The one that you had talked about before, cognitive and human factors and expert decision making.

00;27;22;14 - 00;27;38;27
Unknown
Six Fallacies Any thoughts about this is a really good paper. Open access, like you said, goes through a lot of detail. You know, some of the things you're actually speaking about. Now, let me ask you then. So, from you know, when you first started until now, well, let's not even go back that far.

00;27;38;28 - 00;27;56;10
Unknown
Let's just talk about the next or the past, let's say even five years or so or up until now, have you seen sort of an accelerated rate of acknowledgment and people trying to implement more devices in their systems to try and avoid this kind of thing?

00;27;56;10 - 00;28;16;29
Unknown
Or is there still some battles you, see? Do you still get resistance? What can you tell me about how people have been adopting and implementing some of these things? First of all, generally speaking, a new step forward and I compare it also to the medical and other domain, how long it takes them to take on board

00;28;17;06 - 00;28;39;13
Unknown
a certain thing. So, the military and so therefore in the community, the forensic domain, generally speaking, huge transformation. You don't recognize the domain, what it was only ten, 15 years ago, how to move forward, some of it by people being open minded and wanting to create some of it because people see, you know, the wind is changing

00;28;39;13 - 00;28;53;19
Unknown
. It comes up in court and they have no choice. And they're trying to put an old book in a new cover, but huge changes and a court of taking it on board in the UK, for example, where the forensic science regulator is taking it on board.

00;28;54;09 - 00;29;12;11
Unknown
So, you have huge change on the back. There are definitely there are still many, many battles in certain domains. There is still a huge amount of resistance to this, and people get very emotional and defensive, aggressive. So, there's still a long way to go.

00;29;12;22 - 00;29;33;26
Unknown
But the point the community, generally speaking, has really, really moved forward in a positive way. And I do get some back that, you know, I was in a conference and people get up and start screaming before they understand what is said and so on and so forth, because they get very defensive when they hear bias, they think

00;29;33;26 - 00;29;52;05
Unknown
it's like they see them as they see them and antisemitism. They don't understand what cognitive bias it happens to the best of us, and I always try to emphasize it and so on and so forth. So, I think that although I don't want to give a mark so great to the forensic community, but I would say that

00;29;52;05 - 00;30;10;17
Unknown
it just went leaps and bounds in taking it on board. And not only bias, but other cognitive factor, the human factor that affects performance, whether it's fatigue and many other influences that didn't exist, are really moving sort of for the community is really done well.

00;30;10;25 - 00;30;30;18
Unknown
But still there are you can call them dinosaurs, but there are some very powerful dinosaurs, you know, who still exist to fight date and so on and so forth. Yeah, well, I think bias is one of these things when someone says, well, you're biased right away, there's this guard as a shield that comes up because you don't

00;30;30;18 - 00;30;42;12
Unknown
want to be seen as especially when you're a forensic expert, you want to be seen as this person that sways one way or the other. So, I can see that. But I think more people are before that. Yes, to what you're saying.

00;30;42;12 - 00;30;58;19
Unknown
But add to other ingredients. Nobody likes to be called bias, but Foster or they don't like an outsider. The forensic science community is very neat, and they don't want some cognitive scientist like me is a weird guy to begin with, coming and telling them an outsider.

00;30;59;04 - 00;31;16;15
Unknown
And also, I have a lot of sympathy to the forensic examiner with both in Canada and the UK. In the US, in many countries it's an adversarial legal system. So, every time you acknowledge the weakness, which is good to involve the other side, he's going to eat you alive in court.

00;31;16;15 - 00;31;34;10
Unknown
In the cross-examination, you acknowledge that you make mistakes, you acknowledge that you are biased and use it against you. And the adversarial system is not only not scientific, it's basically anti-scientific. So, I think the forensic examination, a very difficult situation of wanting to move forward.

00;31;35;00 - 00;31;46;22
Unknown
You have to acknowledge your weaknesses. You have to sure they know when you are wrong. We all make mistakes, but the adversary system makes it very, very problematic to do that. So, all of what you said is correct.

00;31;46;22 - 00;32;01;10
Unknown
Add this to it and you understand why the foreign examiners have as stance they do a have that. If you. Okay. Are you able to talk about anything that you're currently working on or want to work on in the future?

00;32;01;11 - 00;32;16;29
Unknown
Like what kinds of things, you can maybe talk about, you know, where this where this area is going or where you're trying to go with this whole area. Yes, I would say a number of directions. There are many things going on.

00;32;17;13 - 00;32;42;09
Unknown
One is that bias doesn't only impact the interpretation of the results. Bias can impact what the results are because it affects how you sample your testing strategy when you stop looking. It not only can affect what the results are in addition to the interpretation of the results, but it can even affect what the data are.

00;32;42;20 - 00;32;59;28
Unknown
Because and I have a few examples in the paper on the sources of bias that I show the lengths of the bias in impacting not only the interpretation of the results. Also, we're looking not only on bias, but reliability, even without bias.

00;33;00;08 - 00;33;19;13
Unknown
How consistent are examining this race? One another regardless of bias, even without by how consistent is one examiner to another on the same evidence? This is what we call between either examiner defensive, but even the same examiners looking at the same data.

00;33;19;27 - 00;33;38;13
Unknown
Same evidence can reach different conclusions, even in fingerprinting or the study showing that 10% of the time they're seeing fingerprint examiners looking at the same plates all reach a different conclusion, even without bias. So that if reliability is even lower, then bias is on top of it.

00;33;38;13 - 00;33;56;16
Unknown
But even without context, even without the relevant context, different examiners reach different conclusion, and the same examiner can reach the same conclusion. Another domain that I'm working on is forensic pathology now very resistant to that, the medical doctor.

00;33;56;21 - 00;34;14;16
Unknown
They want to know everything. And we're having some very interesting studies on this right now that will be coming out soon that I'm sure you will hear about, because they show a certain biases that are problematic, to say the least.

00;34;14;24 - 00;34;33;04
Unknown
Well, well, what about experience? I mean, if the observer if the person themselves can change their conclusions based on some kind of bias or whatever, what about experience? I mean, I can imagine a newbie can probably perform better than the experienced person in some cases or vice versa or I don't know.

00;34;33;04 - 00;34;48;21
Unknown
But what do you have to say about experience generally experience expert domains and they walk again in many expert domains is very good. But many times, ignorance is bliss. Novices, we have a whole slate of articles when novice said so.

00;34;48;21 - 00;35;05;17
Unknown
Best experts we even have in the paper on the source of the bias ever found. The example? Well, novices surpassed experts in forensic decisions, so experts is generally good, but novices also bring a lot of things to the table.

00;35;05;28 - 00;35;22;10
Unknown
And also, experience can cause people to start. You know, you start low and experience, you go high, high, but then you don't continue to go up forever or you can even go down. So, experience some things called your what they call metacognitive issues over self-confidence.

00;35;22;18 - 00;35;39;24
Unknown
You get very confident initiative and overconfidence. So, experience is a mixed bag in against quite a complicated issue. And I'm sorry I began by saying, you know, this is complicated. This is complicated. Yes. But the human mind in the brain is a complicated thing, how people take and make decisions.

00;35;40;03 - 00;36;02;24
Unknown
And we haven't even talked about technology, how technology is getting more and more prevalent in medicine, in aviation, in forensic sciences. And it's not like the woman is out of the picture. We have what we call distributed cognition when the human offloads, some of the processing onto technology and toward all these collaborations and their work as partners

00;36;03;01 - 00;36;17;16
Unknown
, that introduces a lot of issues. And technology doesn't cause bias to go away. It hides a bias in it introduces new biases. So, this is another issue that we could spend another hour only on that and scratch yourself for us.

00;36;17;23 - 00;36;34;15
Unknown
So, there's a lot of work to do. It's very interesting. It's very important. And, you know, the for the examiners that are doing a very important piece in the criminal justice and in civil cases, you know, and that's very important for us and important for society.

00;36;34;23 - 00;36;52;21
Unknown
Well, you know, you said technology, and that actually was going to be one of my questions to you. And that is when people are using more technology in their work. But from your perspective, is there any technology that could assist with limiting bias, making decisions, artificial intelligence, these types of things?

00;36;52;21 - 00;37;10;03
Unknown
Do you see an opportunity there? Absolutely. Technology can help. But can the, you know, between can and does is a big difference. The technology can help. A technology may not help or not make it work and technology can make it more.

00;37;10;06 - 00;37;23;12
Unknown
Much, much worse. A basic bias in wars. And many people say when I use a computer, I used a I, then there is no bias when in fact there is bias in the technology, whether it's a face or face recognition.

00;37;23;22 - 00;37;41;01
Unknown
There's also a general, if you own technology, how it affects a human examiner. And I'm writing a paper now, not in forensic science, how technology is making us more stupid, how technology has decreased human intelligence. Because, like, again, I.

00;37;42;01 - 00;37;58;11
Unknown
Like a broken record. It's very complicated. But technology impacts how we think, how we communicate and doesn't eliminate bias. And it can be a very helpful tool, but it can also cause problems. So, it's not one way or the other.

00;37;58;11 - 00;38;13;15
Unknown
It's a tool and depends how it's used, how it's developed, and if it's built quickly to interact with the human examiners. Okay. I have I have this thing that I always say now. It happened in the last few years, but I call it the hierarchy of evidence.

00;38;13;24 - 00;38;27;15
Unknown
And that has to do with, you know, when you have a really pristine sample of something, you know, you have great information. And then what happens when that begins to deteriorate? So, and what you can see about it, the difficulties are involved.

00;38;27;23 - 00;38;43;12
Unknown
How does how does evidence play a role in this? A huge a huge role. There is what I call the biased danger zone. You started asking me how big is the problem of bias? Well, many times people ignore bias and get all defensive.

00;38;43;12 - 00;38;58;11
Unknown
It doesn't exist. And some people overreact. I think there's bias everywhere and they want to kill a fly with a cannon. So, what you need to do is to know when you in the bias danger zone. If you go back to my example of driving, I speed.

00;38;58;19 - 00;39;10;23
Unknown
But when I'm tired and I just landed in Canada, I'm not sure which side of the road to dove on. I had a glass of wine or whatever. I throw the snow ice on the road, then they slow down.

00;39;11;03 - 00;39;28;13
Unknown
And this is what the forensic examiners need to do once they get an understanding and training about cognitive bias, understand when they are in the bias danger zone, what I call the bias danger zone. And one of the critical elements in the bias danger zone is the evidence.

00;39;28;20 - 00;39;43;12
Unknown
When the evidence, as you said, is very clear, it's you know, there's good quantity and quality of information. You're not near the threshold. It's an easy decision. Then there's not a lot of leeway for bias to impact your decision.

00;39;43;21 - 00;40;03;06
Unknown
And the decision is more difficult. You have more judgment, more discretion. You'll near the borderline. It's a more difficult case. You know, you avoid the examiner know, sometimes it's easy, sometimes it's not easy, as it's more difficult, then the bias has a more power to influence you.

00;40;03;11 - 00;40;19;12
Unknown
That is an important ingredient in the bias danger zone. What about when let's say, for example, you have you done your analysis? What about the delivery of the evidence to the jury? So, the output and how it's transmitted?

00;40;20;12 - 00;40;32;23
Unknown
Do you ever see any cases where, for example, you could take I don't know, let's just say a piece of evidence, but it's pushed this way or it's pushed that way, depending on, you know, I mean, the situation.

00;40;33;09 - 00;40;56;00
Unknown
Absolutely. And unfortunately, the jurors and even professional judges are often convinced not by the scientific married, but how you present the evidence in court and what forensic examiners often do. They try to present the evidence as strong as possible rather than presenting the strength of the evidence.

00;40;56;00 - 00;41;10;26
Unknown
And they have training, you know, how to handle in court. And they train they look at the dual eyes and, you know, speak with confidence and so on and so forth. No, you shouldn't speak with confidence. You should speak in confidence depending on the strength of the evidence.

00;41;10;27 - 00;41;25;08
Unknown
I remember I appeared in court one and they asked me some questions. I was very confident. I said, yes, absolutely. And then they asked me a question and I knew the answer was yes, but it was not as powerful, and I could have just said yes, but intentionally.

00;41;25;08 - 00;41;39;13
Unknown
I wanted to convey to the jury that it's not too strong. So said that we think about it for minute. Yes. And I deliberately did it because the way I present it to the jurors needs to reflect the strength of the evidence.

00;41;39;13 - 00;42;01;02
Unknown
If we go back to your example a few minutes ago, some cases are difficult, some cases are easy. This needs to be transmitted. You shouldn't always in court be as persuasive as you can based on your training, how to appear in court, you need to present your confidence in accordance to the strength of the evidence.

00;42;01;08 - 00;42;16;22
Unknown
And this is very, very critical for the jurors who take it more by how you present it and not what you say as much. If you say, you know, I've been doing it for 20 years and I'm telling you this is an ID, they're going to believe you.

00;42;17;05 - 00;42;39;20
Unknown
And so, this presentation is a big issue, but it's a different if you want if you let let's get the forensic decision and accurate and reliable and impartial, any objective is possible. And then how do we convey effectively to the jurors in a way that they understand because they don't understand science.

00;42;39;20 - 00;42;54;27
Unknown
Most of the judges, they train. Or there should be a drug race in the state of Massachusetts. And when I started, I asked, how many your background in social sciences and humanities? Almost all of them. Not many have scientific background or so you have to remember.

00;42;54;28 - 00;43;11;07
Unknown
And again, I'm to the complexity. Science is complicated. The jewels and the court are not complicated because they want black and white. They need to say either, you know, the person accused of the crime is the suspect guilty or not guilty.

00;43;11;07 - 00;43;25;20
Unknown
It's a benign decision and the evidence is not always, yes, it's a match 100% that match. It cannot belong or it's definitely an exclusion. You know, there is a. And if you get, I don't want to get them in to talk about statistics and so on.

00;43;25;27 - 00;43;43;24
Unknown
So, science is just a bit more complicated than yes, no. But the jurors and the jury at the end of the day to make a decision, get they're not guilty. And again, how to communicate to the jurors the strength of the evidence and the science and the complexity is another huge issue.

00;43;44;00 - 00;43;59;04
Unknown
And that is, of course, with a lot of cognitive and psychological issues, how to convey my message today to the fore and the community, which is not a new message. When you appear in court, don't do the best you can because you can do a very, very, very convincing job.

00;43;59;12 - 00;44;15;18
Unknown
Do a convincing job when the evidence is convincing, when the evidence is strong, but not as convincing. Don't do the best you can in looking to either the jury and being confident. Do less to give the jurors the right understanding of the evidence.

00;44;15;18 - 00;44;25;23
Unknown
And a lot of it is not in the objective data that you give them, and you count, you know, they can give them number. It's how you speak and how you look at them, the tone of your voice.

00;44;25;24 - 00;44;42;16
Unknown
So, you need to think about it. They say, you know, it's complicated. It's not clearly cut. You know, it's not to simply match or not match or an identification and so on and so forth. You know, a lot of these areas that you're talking about are identification or they're very black and white.

00;44;42;16 - 00;44;58;27
Unknown
So, is it this person? Is it not? Is it this you know, this person's fingerprint, is it not? Is it in forensic anthropology? Is it a male or a female? Something like this in the world that I work, I'm often dealing with not so much an error rate, but just error and uncertainty.

00;44;58;27 - 00;45;11;06
Unknown
So, for example, I'm trying to measure where a bloodstain pattern, an impact pattern came from, and it's not black and white as if, you know, there's no right or wrong answer. It's a matter of how far you are from the ground truth.

00;45;11;17 - 00;45;24;15
Unknown
And, you know, unfortunately, when it's a crime scene, you never have the ground truth. You don't really know unless there's a video camera or something. But in that case, they would need me. So maybe this is a comment more than a question, but I've seen it before where we don't.

00;45;24;27 - 00;45;42;04
Unknown
And this might be an interesting research project where, for example, people are given a project to calculate something, but given some contextual information, does the error or the range or the uncertainty get smaller and wider depending on, you know, the contextual information?

00;45;42;13 - 00;45;59;05
Unknown
And I don't know if you've seen that. And there are two issues here. The contextual information will cause the examiners to reach more consensus, but it may be for the wrong reasons and the consent just doesn't mean they have the two.

00;45;59;05 - 00;46;12;18
Unknown
And so that's why we don't know the ground truth. We have to build databases where we know the ground doors and how you build a database and the complexity of the paper that you mentioned. They are all in Illinois in building those databases.

00;46;13;02 - 00;46;27;28
Unknown
But the fact that contextual information increases consensus, you know, you can flip a coin and we all agree it's one thing to do another, but it doesn't mean it's true. The question is not only the reliability, the consistency, but the validity.

00;46;27;28 - 00;46;45;24
Unknown
Do we try and answer? And to do that, among other things, we need to know the ground truth which we do not knowing case or so. We have to build a database in it with fingerprints. It's very easy with firearms, it's very easy with a blood pattern.

00;46;45;24 - 00;46;59;13
Unknown
And it's a bit more complicated because you can take a person and shoot one person from one angle and shoot another person from another angle and get, you know, your path and the blood splatter and know the ground and so on.

00;46;59;20 - 00;47;15;02
Unknown
So, it's a if you and it's as you say, what is important is to do those objects. And that forensic science existed for decades and decades and decades in code without any research. If you look at the research, a lot of forensic domain.

00;47;16;01 - 00;47;36;20
Unknown
Bias, reliability, but even the scientific basis forget about bias and reliability in the cognitive and the human factor. There's a basic scientific underpinning has never been done in many forensic domains, and they appeared in court. So now the catching up, which is very good, but of course, it's very problematic for people what we've been doing.

00;47;36;29 - 00;47;48;14
Unknown
Presenting in court and doing something for 20 years now they're going to do research. Is it valid or not? Is it reliable or not? That's kind of putting in the wagon in front of the whole with it should be the other way around.

00;47;48;22 - 00;48;05;16
Unknown
But if you look at the literature in many domains, most of the little things from, you know, from the last ten years or so. So, there's a lot a lot of projects, a lot a lot of data collection to do and a lot of understanding to improve forensic science, because it's a science for those that want to

00;48;05;16 - 00;48;18;02
Unknown
learn more, for those that want to be more aware of what's going on. You offer training or I think you do some workshops. I mean, you trouble with COVID now? I guess it's a little bit of problem, but what are your options for people that want to get training in this area?

00;48;18;10 - 00;48;37;04
Unknown
So, when I used to travel and will travel again, maybe some days, I limited myself to once a month because I'm a researcher. I'm not somebody who just does training. I could do training all the time. I get quite a lot from all over the world, and I could just go into to training.

00;48;37;04 - 00;48;58;15
Unknown
But I'm a researcher and just feel I need to communicate. I don't want to be in the ivory tower like many academics. And I came back to the real world now with the virus and everything we do online training just to access, advertise, you know that and A, B and C, National Accreditation Board and organized training.

00;48;58;15 - 00;49;13;11
Unknown
On October 20, just over a month, we're doing a five-hour training on forensic decision making. You can go to my Web page and look at events and they link to that. So, we do training and I do a lot of training for labs.

00;49;13;11 - 00;49;32;25
Unknown
Whole labs have done during the virus in the last six months. I've done quite a lot of online training from an hour to 5 hours to a day to todays on these issues. So, this is our information. And for the ID, if you go to the linked events, eh, there's quite a lot of events and I'm happy

00;49;32;25 - 00;49;47;21
Unknown
to add events and people can join events from events or flee if people want to. And these webinars that you mentioned need to find the quicker next week. Yes, I'm giving a free webinar. It's on the web page.

00;49;47;21 - 00;50;01;20
Unknown
You can click on the link. People can go to the free webinar, but also more training courses, you know, that are longer and more difficult. An hour or two of training is really just like, you know, an appetizer, just a taste.

00;50;01;20 - 00;50;17;10
Unknown
If you really want to get into it, you need more. And there are a and a be the National Accreditation Board 25 October I'm doing it and I do a lot of training. People are welcome to email me not about training, but also if you have any questions, you can find my email on the internet.

00;50;17;29 - 00;50;35;29
Unknown
You can go to my name. It's weird enough you to find me quickly and find my email and if you have any questions towards articles, I'll be happy to send it to you because I believe that dialog between academics and practitioner is the way forward in academics in the ivory tower and talk to one another.

00;50;35;29 - 00;50;54;04
Unknown
That's great, but they don't impact the real world when they're pretty finished with one another. And that's nice, but they need external academic input. And by collaborating, interacting, this is how I hope we'll be moving forward. There not only hope we've been doing you've been doing a great job for over a decade.

00;50;54;05 - 00;51;10;28
Unknown
We just need to continue next time. One last thing before we go. You have a paper that was just released last week, misuse of measurements in forensic science. Can you just say a couple of words on that? That's got to be a bit of a controversial paper, but not my first one.

00;51;11;05 - 00;51;31;25
Unknown
We've looked at all the alleyways, studied and found that all the animal studies are misleading in reflecting L because they don't include case work or for example, when they do an alleyway study, if the examiner said inconclusive, they don't count it then in there, or in fact, they counted to the correct response.

00;51;32;03 - 00;51;52;23
Unknown
So, if you go to an alleyway study and they tell you this is an always study, and if it's inconclusive, it's a correct response. Then people make a lot of inconclusive decisions. If you look at the paper, we find a study that 50% or 80 or 90% of the answered or inconclusive, and they count all of them

00;51;52;23 - 00;52;08;04
Unknown
as correct responses. They never count inconclusive as even potentially incorrect and so on and so forth. So, in this paper we highlight the weaknesses in the design of the study and how to solve it. It's very easy.

00;52;08;13 - 00;52;30;07
Unknown
We have a very practical designs on how you can have more accurate studies. But this new paper, again, it's open access in forensic science, international energy of misuse of scientific measurements in foreign design showing. And that will take us back to proficiency testing, how you can actually test human performance.

00;52;30;25 - 00;52;46;13
Unknown
And it's painful because if you find errors, you know it's going to be used against you in court. And that's a problem. And I'm not, you know, saying it's not a problem. But if you don't acknowledge it and you say, you know, we're not alcoholic or cigarette is good for you, we're not going to move forward.

00;52;46;13 - 00;53;02;28
Unknown
We have to acknowledge and do this research. And we find that in our late studies, again, people can read the paper. It's publicly, freely available to make their own mind up. But we found some very serious problems with all the study and how to fix it.

00;53;03;03 - 00;53;16;02
Unknown
But then we're going to get it always a bit higher than until now, which is not a comfortable reading for anyone, but we have to do that. Okay. Dr. Drew, I want to say thank you so much for being here today.

00;53;16;02 - 00;53;27;22
Unknown
I realize it's later in the UK and appreciate your time. You always have so much to say, and I wish we had more time. But thank you very much for inviting me in and giving me this opportunity to talk to you and to the community.

00;53;28;01 - 00;53;30;06
Unknown
I appreciate that. All right. Great. Thank you.