Building High-Quality, Ethical AI Systems

Transcript

(This transcript has been edited for brevity.)

DAVID CARTY: Bollywood is a familiar dance style for most people. It spread all over the world with its fusion of traditional Indian, folk, and even modern hip-hop influence. But it’s not the only type of Indian cultural dance. Rishu Gandhi practices a type called Kathak dance. And it’s an important part of her heritage.

RISHU GANDHI: Ever since I was a kid, I was actually some trained dancer in Bharatanatyam, which is another Indian classical dance form. So when I was little, I don’t remember how old I was. But I know as soon as I started walking. So my inspiration for Kathak, I would say comes from Bharatanatyam. They do differ in terms of very technicality. And Kathak has a lot of expression associated with it, too. So I think it’s beautiful, and it does touch on cultural aspect of Indian dance form. So Kathak, I would say, is the only Indian classical dance form that originated from North India. So it’s amazing. And I’ve been learning about it three to four years. And I love it. So I do give a lot of credit to my mom and my dad for introducing me at a young age. And I genuinely do appreciate my Indian culture, my roots. That’s where I come from. Through my dance journey as well, because that comes part of it like costumes, wearing certain clothes like saris, and dressing up for performances, and stuff. So it’s amazing.

CARTY: But it’s not just dance performance. The Kathak style includes a test element with it as well.

GANDHI: So I just gave my written exam. I have physical exam in like two months. So I’m transitioning from my third year to fourth year. So the first two years, you don’t do highly technical Kathak performances. So, in other words, it’s called nritta. Nritta that means there’s a lot of footwork. So you don’t do a lot of them because you’re still learning the fundamentals. It’s like learning alphabets. But then after you mastered all the alphabets, then you start making sentences. So that’s how it works in Kathak, too. So we learn the fundamentals in the first two years. Then we really start to perform. So then we do very classical dance piece. It takes years of experience to learn that. So I have a deep respect towards those people who has kept it alive through their dance forms and by performances.

CARTY: While there’s a cultural interpretation to this style of dance, Rishu appreciates that it has allowed her to connect with her community here in the United States.

GANDHI: There is a New York Kathak festival that happens every year. That’s another great way to promote dancers and promote the dance form itself. So there are amazing Kathak maestros from India. They travel here. And we do, it’s like a recital like a nice three days program. So it’s amazing. Yeah.

CARTY: This is the Ready, Test, Go. podcast brought to you by Applause. I’m David Carty. Today we are live at the Open Data Science Conference East here at Hynes Convention Center in Boston.

My first guest is Rishu Gandhi, Senior Data Engineer in Cybersecurity at Wells Fargo. Our conversation runs the gamut of everything AI, but especially focuses on responsible and ethical AI. Let’s jump right into it.

So let’s start with a basic question. What is responsible AI? And from the perspective of a data enthusiast, how should it be implemented across the business?

GANDHI: Yeah, so I’ll break it down very simply. So responsible AI is designing, developing, and utilizing artificial intelligence technologies in a manner that upholds societal values, respect human rights, and reduces potential harms to humans and communities. So it’s like if you’re building AI-driven solutions or product, make sure they’re ethically correct, and they’re not harming anyone. So from a business standpoint, how we would implement that. To be honest, we would implement it in every stage of data science journey. So that would be in the design phase, development phase, and after development which deployment phase. So we need to ensure that we practices in each phase of life. And it doesn’t end there. We also need to make sure once we release our product, we need to continuously monitor and see what it does to the society. If it’s not doing what it intended to do, then we need to redo the whole process. So it’s more like an iterative approach. So it should be implemented in each stage of data science product journey. Yeah.

CARTY: Right. That’s exactly what I was going to follow up with. So it’s not a one-and-done sort of thing. It’s about iterating on what you’ve already put in place just like you iterate on the product itself, and what you’ve built. And I imagine part of that is soliciting different perspectives as well. Because what you would consider responsible and ethical, maybe I don’t, or maybe somebody else doesn’t. So it’s about trying to gather those opinions and make sure that everybody is being served in a way that is helpful and not harmful.

GANDHI: Absolutely. And what you just described, that’s actually one of the core principles of responsibility, which is called inclusiveness. So you want to make sure the AI products that you build, I mean, it’s true. It won’t be focusing and considering everyone. Because let’s say if there’s a health-related product. Obviously, it’s going to benefit the health community more than someone from the financial community. But on a higher level, it’s important that we be mindful and be inclusive towards diverse backgrounds and communities, how our products can be accessible, and beneficial to the majority of the masses. So that’s where we would want to take human feedback, and then reiterate it, and make sure we’re constantly improving. I mean, improvement is the only constant in any case.

CARTY: Yeah, so how do you go about implementing those feedback loops like you’re talking about, and trying to identify maybe underrepresented, or misrepresented groups that maybe you missed in the initial phase of requirements gathering or however you identified that at the starting point?

GANDHI: So we would do it in the design phase. So let’s say like whichever objective that we’re trying to achieve, and our AI, let’s say we’re trying to build an AI application with a certain objective. The way we collect data, we can be mindful about that. So just to give an example, let’s say, I want to build an AI-driven app where it helps scan resumes of applicants, and at least helps in filtering down. Because you get millions of resumes, let’s say, in a month. How can you filter it down? So let’s say that’s what you’re doing. So when you do the data collection process, you want to make sure you’re actually taking the historical data of successful hires from different backgrounds. So it cannot be just from university X and Y. You want to make sure you’re including all the universities. You’re also considering whatever the key metrics that you want to collect, you’re including all demographics, too. And once you collect the data, then you can also do a little bit data exploration. So that’s like a one side of this whole data science journey. So in the data exploration side, you can learn about the data that you collect. So are there any biases? Is a certain demographic overly represented or underrepresented? If that’s the case, then how do you address that? Because let’s say if your algorithm learned that all of these successful hires are from University X, it’s going to know that University X, I’m going to hire this person. But that’s not the case. We’re not promoting diversity at the workplace. So that way, you can make sure how to tackle those issues, definitely in the design phase. Because once you concur that part, then it becomes much easier because garbage in, garbage out. So if you’re not putting the garbage in, you will get something useful out of it on the other end.

CARTY: Right. It all starts with high-quality data. That’s the foundation for everything that we’re talking about here at the show today and everything that you’re putting out there into the world. And as you talk about what that, you’re trying to identify biases in the data as best as you can. It’s probably difficult to get to the bottom of all of it, though. I mean, at some level, there’s always going to be some bias involved and systems are supposed to make calculated decisions based on this data. So how do you maybe go a level deeper in trying to identify some biases that might not be obvious right at the forefront there?

GANDHI: So let’s say you did not. And you’re right. Sometimes the algorithm, even though you did all your measures accurately, let’s say, you don’t see it in the design phase. You may not even see it in development phase, but, again, you can actually check responsibility in every aspect. So let’s say in the development phase, one way you could look for is the explainability of your algorithm. So that’s where your model needs to be transparent and explainable. What do I mean by that? So I think right now, there’s this hyper AI. Any magic that we see in terms of technology space, especially non-technical people, that’s. AI What is AI? So there’s a black box. Your data goes in. It does some magic and it comes out. But what does really happen in black box? So if we have transparency and explainability of what really happens in that algorithm, then we can see more insights. So let’s say your model in the development phase, it says that, OK, I made this prediction because of this, this, this. So we can learn what are those three points that produced its conclusion from. If we see, OK, I see maybe it’s picking University X all the time. Maybe y. So there’s a bias. Let’s go back. Let’s maybe add some more data, which is fairer data. Or maybe let’s fine-tune our design phase again. So make sure we’re not seeing that. So that’s in the development phase. We could do the similar thought process in the deployment phase, too. So once your model is built, whatever predictions it gives out if it deviates from what you intended to do, again, we have to go to the iterative approach. So that’s something I would say. Yeah.

CARTY: Right. And how important is that interpretability and transparency, especially when it comes to being accountable to regulators? I think that we all anticipate a little bit more regulation in this space moving forward. So how important is it to maintain those qualities?

GANDHI: I think it’s really important for multiple reasons. First and foremost, I would say hefty fines depending on where you’re located like or where your company is located, whether it’s local, or international rules, and regulations. If it falls under that umbrella, then you need to be compliant because I know there’s this European Union code of conduct, which I know they adopted it in March of this year. But it may go in action, I think, next month or in June. They have fines up to 35 euros million or 7% of the global revenue of that company operating in the EU area. So that’s one motivator why it really needs to be implemented. And I think accountability from a technical perspective, it also helps to address issues that are there and how we can solve them. So we can hold people responsible for whatever the consequences that come from our AI-driven solution. So we need to have some accountability, especially when we go to the audit phase, audit and review. So we know OK, why certain decision was made, who made it. So if, let’s say an item not pinpointing a specific a group. But let’s say I’ll just pick my example. Me, as a data engineer, let’s say, I made a certain decision. It’s there. It’s somewhere there, that Rishu Gandhi did this. Let’s say we do an audit phase. It’s like why does she do it? Maybe let’s teach her not to do this or let’s take another approach. So how do we know that? Because we’re being accountable for our own actions. And that’s where I think it becomes very important to have that accountability. And also transparency aspect of it.

CARTY: How difficult is it to try to enforce change? I think everybody is very enthusiastic about what all of these different AI capabilities are moving forward. But trying to slow things down to incorporate more of this visibility, transparency, logging, whatever the case may be, it can be a little bit difficult I imagine. Everybody’s trying to move faster, trying to realize more efficiency gains, more profitability. How can you enforce change to arrive at that level of accountability that you need to be at?

GANDHI: That’s a good question. And you’re right. I agree with you. I think all of us were moving really fast, especially after the OpenAI, the ChatGPT that came out two years ago, I think it’s everyone’s just we’re in this race. But I think first, we can recognize that OK, we’re not in a race against it. We’re for it. AI could be our friend. So certain things we really need to be mindful to really address these issues. And to be honest, there is no one solution that fits all. It’s also just by experience, we’re going to come across multiple issues. But while that happens, in the meantime, we can stay educated on the latest trends, what is happening in the AI space, what are the rules and regulations that are occurring, and understand why they’re even occurring. Because maybe something happened. That’s where it came from. So just be staying educated about those things. I think that helps a lot. And as we explore more, I think what we can do as a community is to be self-aware, stay upon it, spread the awareness around our community, and also even among the non-technical community, too. So once the problem arises, we were aware of the problem. So we can fix it sooner rather than just laying it out there, and then it turns out, oh, my God. This happened. What do we do now? So all that fiasco. So I think we should take it upon ourselves to be educated about it.

CARTY: Yeah, so be proactive.

GANDHI: Exactly.

CARTY: Spread that education. Don’t try to bury mistakes or anything like that. Understand that it’s a learning process.

GANDHI: It is a learning process. Sure.

CARTY: Yeah, exactly. Hold yourself accountable. And the team accountable for that.

GANDHI: Absolutely.

CARTY: How valuable can the human perspective be in validating all of this? I mean, we talked about this in the beginning of there are these different cultural perspectives or different human or customer needs that can vary quite a bit one way or the other. So how valuable is that human perspective in making sure you have an ethical responsible system?

GANDHI: Yeah, I think it’s very important. And to be honest, we could never not have humans monitor AI systems. I mean, that would be a very scary world, to be honest. So I mean if you think about it, I mean, I’m going to talk I guess more on the philosophical term. But humans, we’ve been here for millions of years. So we know what feels right and what doesn’t feel right. Does AI know that? Maybe not. It’s almost like, let’s say, you have a kid. The kid is like a blank slate. Whatever you teach your kid, that’s how they become. So, similarly, machines, we can keep teaching them. But there won’t be a point where we’re like you know? What I’ve taught you everything. Now, you go do everything. No, because again, that’s the machine versus humans. For humans after a certain age, you become self-aware of your own. You learn it on your own. But the machines, they don’t have ethical aspect of it. They can learn that, but we have to teach it. So I think human feedback is definitely necessary. And it’s also important to having human feedback in your building your application machines, it builds trust among the communities. Because we know that the AI systems that we’re making, we can trust them. And it will actually help a broader adoption and acceptance of the solutions because we know the human feedback was implemented. So I think having human feedback is definitely important. And let’s say if anything goes out, we need humans to fix that. Otherwise, if machine were to fix its own self, I mean, I personally would not trust it. I would want human feedback to be implemented. And sometimes, we have this preconceived notions that my product will do this in real world. It turns out it does not do that. Or I thought that maybe that’s how my consumers would take it as, but they don’t. So having that human feedback in the design phase helps a lot. Because at the end of the day, who are they going to be the consumers? Will be the humans. So let’s ask them their feedback. What do they like to see? How do they intend to use this app? So I think that way, human feedback is very important. Yeah.

CARTY: Yeah, and what you’re getting at is there’s a real business value to doing that. I think thinking about a conceptually trying to roll humans into this process. I think from a business standpoint, when you’re talking about going so fast, and you have this black box that seems like magic sometimes, and what it’s capable of, you might think, well, if you’re slowing down to accommodate and account for more human perspective, that can take away from the value. But in reality like you’re saying, it actually helps with adoption, it helps with confidence in the product in the long run.

GANDHI: Absolutely.

CARTY: From a software quality standpoint, digital quality standpoint, what’s your biggest concern about AI-infused systems moving forward? And what are some things that organizations can do to address those concerns?

GANDHI: I think my biggest concern, at least something that I came across, in my research and in general, I think it’s bias, potential bias really. And I think it is a scary. I think it’s a very scary face. So I think potentially, it is risky. And we need to know how to tackle that. I’ll give you one example. And I actually talked about this in my presentation. So one of the example was let’s say you have an AI-driven medicine helper help app, which basically its intention to help doctors prescribe medicines based on a patient’s lab work. And maybe a little bit of their past visits history. So let’s say you have an AI-driven app. If it doesn’t practice responsibly, what do I mean by that? If it doesn’t reduce his bias, or it doesn’t have a data quality checks that addresses bias, what could happen? I’ll give you an example of what would happen. So imagine, it would actually lead to inaccurate medications. Why? Because the data that collected and that was trained on, maybe that your data quality, maybe let’s say it was either overrepresented or underrepresented some of the demographics. And it doesn’t take into consideration of every patient is different, their genetic history is different. Each race will have their respective genetic history which affects a lot of things. So if the bias is not reduced and tackled, you’re like person like me, let’s say, I’m found with diabetes, versus another person. It’s not necessarily that the medicine would work on him or her would, work on me as well because I have my own respective genetic history. So I think that’s where bias is very risky. So we need to make sure that the data that we collect, it fairly represents all the demographics. So this is just one example. I could use the same example that I used about the resume checker-like biases. Let’s say if it’s biased towards certain universities, it’s disadvantages to very success, very educated talent and candidates who just got rejected based on because they were from university A instead of University X. So, again, there is bias issues that are happening. So a lot of work we need to do in how we collect the data and what measures we take to fix the biases, I think, it’s very important. So that’s something I’m scared. But at the same time, I’m glad that we’re talking about it. Because the more we talk about it, the more we can find ways to fix it. That’s good.

CARTY: The scope of the problem is potentially massive when you think about all the different industries involved. I mean, you talk about lending decisions, or college acceptance, whatever example, you want to bring up from any industry. You’re relying on decades worth of data that could be problematic. It’s hard to root out some of those challenges with the data. But it’s ultimately the task that we have to prep ourselves for moving forward. Because ultimately, you can achieve great things if you’re able to identify some of those challenges.

GANDHI: Yeah, for sure.

CARTY: Big broad question. But when you think about the future, and I think when you’re at a conference like this, it’s easy to think about, wow, there’s so many intriguing possibilities of where we could go, where do you hope that we will be in terms of AI-infused products and implementation, and how do we get to that ideal future?

GANDHI: That’s an amazing question. And so I’m a very optimistic person in general. And I’m glad because a lot of talks at the conference, I’ve been noticing a lot of them either directly or indirectly talks about responsible AI. It has other synonyms responsible AI, ethical AI, considerable AI. These are all just synonyms. But they all come back to the same statement that, hey, let’s be responsible and wise about how we use AI in our day-to-day lives. So what I envision is when I say I envision, it’s my personal idea of how we would be, I think, our lives will definitely be AI-dependent. I mean, AI has become integral part of our lives. If anything, it’s going to be even more and more. So we cannot run away from it. It’s about time we recognize that AI is here to stay. It’s our friend. So one thing as a consumer is let’s try to use it in our advantage. But at the same time, let’s be mindful. So one example, last year, I visited Italy and I did like a little Europe trip with my parents. We planned an entire trip using ChatGPT.

CARTY: Oh, really?

GANDHI: Yeah. For example, since I’m the owner of my own data, so I would make sure I would not input hey, my name is Rishu Gandhi, I want to go here. Obviously not. But if you use it smartly, it’s actually very helpful. And it helps you like plan a lot of things that you might take you more time. So I think that will become the more norm. We will use it in our day-to-day lives. And another thing that I do see what would happen is, so right now, definitely the governance side is definitely there. They’re talking about AI. Why? Because there needs to be proper restrictions onto how we use it and all the governance and risk and compliance side of it. But what I do see is in future, we will have just more experts from various backgrounds collaborating together. And let’s everyone sit at the table and talk about this is the AI product. What is your opinion? So one expert might be from a psychology domain. One person from an economic domain. Because it does affect literally all other domains. So all of the experts from different backgrounds sitting at the same table, discussing how we can ensure. If me comes let’s say if I come from a financial background, OK, I do see this, this, this. But I also see these problems. So let’s be mindful. Someone from economical background, they might have a different approaches to the same problem. So it’s important that we do that. So the idea is that you know we really demonstrate our commitment to using data and technology for the betterment of humanity. So I think that’s something that I hope to see you in future that we’ll be just doing it more.

CARTY: And most importantly, as a travel planner, ChatGPT gets a thumbs up. Did you have a great trip?

GANDHI: I loved it. Yeah, absolutely. And there were some times, let’s say, I’m waiting at this local train station in London. And I’m like, my train is late for an hour. Tell me five things to do around this location. There it is. It’s like you can do this. You can do that. I’m like, OK, good. So

CARTY: Like having a personal assistant in your pocket.

GANDHI: It is. Yeah, absolutely. Yeah.

CARTY: OK, Rishu, lightning round questions for you here. First, what is your definition of digital quality?

GANDHI: My definition would be any digital product that is made with ethical considerations in mind. So we realize that human feedback is necessary. So that’s also something we consider. And also I think another aspect would be inclusiveness. So let’s be mindful when we create our data. It’s accessible and beneficial to majority of communities and society. So that would be a great digital quality product for me.

CARTY: Great. How will AI evolve in the next five years?

GANDHI: It will definitely be, like I mentioned, it’s going to be in our day-to-day lives even more from, I don’t know, laundry pods from buying this to that. We’ll constantly be talking about it. And another thing that I do see it happening, especially in my household. So my parents, they come from a generation where obviously there was not that many digital products. The fact that they’re starting to learn and use it, so that’s something that I’m really excited about, too. How they can use it to their advantage? So just little things. It’s like we could all be smart people just by using it responsibly and getting the information that we need to do whatever we want. So I think it’s an amazing tool.

CARTY: Yeah, you can check yourself on an assumption that you’re making on the fly, or you can deepen your knowledge on something on the fly, or on your car ride from work to home. I mean, there are so many possibilities for learning, I think. And this is something about our day-to-day lives today. I mean, we have more possibilities to learn it feels like every day. And you just hope people take advantage of that. And this is another avenue for that.

GANDHI: Yeah, for sure.

CARTY: What is your favorite app to use in your downtime?

GANDHI: I would say for educational content, I would say YouTube. And I also do watch a lot of podcasts. So I think I think YouTube is one place for all. And for entertainment purpose, the Netflix is a good one. Amazon Prime, there are some good shows that they have. So for entertainment purpose, I would say that’s where I get the entertainment and fun for my life.

CARTY: Great. And what is something that you are hopeful for?

GANDHI: I’m hopeful to be alive, to be doing this today. No, I’m hopeful just, I guess, to be learning new things. On learning different things, too. Because sometimes, we have preconceived notions about whether it’s people, or just things in life in general. And so sometimes we need to unlearn that, too, so that we can learn different perspectives. So just here to learn. Talk to people. Yeah.

CARTY: Yeah, it’s an ongoing job.

GANDHI: Absolutely. Yeah.

CARTY: Great. Well, Rishu, thank you so much for joining us. We appreciate it.

GANDHI: Thank you so much. This was really fun. And absolutely, you have been a great host. Thank you so much.

CARTY: Thank you. Enjoy the rest of the show.

GANDHI: Thank you.