A computer can keep you in jail, or deny you a job, a loan, insurance coverage, or housing, and yet you cannot face your accuser. The predictive models generated by Machine Learning to drive these weighty decisions are generally kept locked up as a secret, unavailable for audit, inspection, or interrogation. In this video, we'll cover explainable machine learning, and the loudly advocated machine learning standards, transparency, and the right to explanation. I'll discuss why these standards generally are not met, and I'll overview the policy hurdles, and technical challenges that are holding us back. Now, there's truly a movement among machine learning practitioners for transparency. This trend seems to convey a widely agreed intention, to make models accessible, inspectable, and understandable. But in practice, the most consequential decisions driven by machine learning are not explained. The most life-changing predictive models, operate from behind a locked door, inaccessible to any auditing. The practitioners in such areas often don't espouse transparency at all, and in fact, they're not nearly as vocal. We don't hear from them. But their control in machine learning deployment, that impacts many lives. This includes the compass crime risk model we've covered in the last few videos. The behavior of the model is public, by a way of sampled data, such as that analyzed by the ProPublica journalists, who sounded the alarm on bias. But the inner workings are not. You only get to see each score it outputs, not the way that score was derived based on the factors known about the defendant input to the model. But without being authorized to audit, to look inside a model, and view it in full detail, we cannot vet the model for whether it violates any and all agreed upon fairness criteria. For example, in 2015, Amazon scrapped a model they generated to score job applicant resumes, when they noticed it preferred male applicants over female. The model penalized resumes that included the word, "Women's," as in women's chess club captain, and it downgraded graduates of two all women's colleges, according to Reuters. This model was discriminatory as we defined it in the first course. It based its decisions in part on a protected class, gender. It's apparent, that the developers didn't intend for this to happen. They presumably withheld gender from the model, not including it as an independent variable. But the model found a way to access the gender for some resumes, and it went ahead, and used it. This would have been much more difficult, or impossible to determine by only observing the model's scores by only examining its behavior, rather than its inner workings. Also, in peer-to-peer lending, one of the single highest indicators that you're not going to pay back a loan is, if you use the word "god" in your loan application. This is a little bit eerie and suggests a potentially, dark future. It means that a lender would be wise to not give a loan to anybody who mentions "god". They may be punished without even realizing why, according to Seth Stephens-Davidowitz, the author of, "Everybody Lies." So if not audited, a financial risk model could in this way, penalize loan applicants based on their religious status. As I mentioned in the last course, one study showed that Google searches for "black-sounding" names, were 25 percent more likely to show an ad, suggesting that the person had an arrest record, even if the advertiser had nobody with that name in their database of arrest records. This is evidence that the model infers race, and drives ad targeting decisions based in part on race. But we'd of course, have a lot more insight if we could actually audit the model itself. I'll mention one more example. Research efforts in China and the US, to predict criminality from a photograph of a person's face. It looks at the photo and asks, do they look like a criminal? Has been met with a great deal of public push-back. One problem with this work, is the potential that race, as determined from the photograph itself, would turn out to be a factor in the models predictions. Now, even though fairness cannot be fully vetted without full access, we're not going to see full model transparency applied sweepingly across the board. The predictive models deployed at large in the world will not all be publicly disclosed anytime soon, especially in the private sector. Just as a company is not obliged to publish every meeting and every office that leads to every corporate decision, it's also not obliged to show the world it's predictive models. After all, a predictive model is a proprietary trade secret. For example, a model used to target marketing is an asset that provides the company a competitive edge. While it's true that models can target marketing in unethical ways and often deserve auditing, their mandatory transparencies unlikely to emerge in general. Safety and security also mandates that certain models remain a secret. Systems that operate equipment or vehicles in real time, such as for self-driving cars are vulnerable to what are called adversarial attacks. It turns out that deep neural networks can be tricked and blinded by finding just the right odd pattern of, say, stickers to place on a traffic sign without making the sign appear to humans as if it's been attacked in any way. It's possible to design a text like these that are much more potent if you have full access to the model, so model access must be protected. Models that discover leads for criminal investigations and fraud transactions as potentially fraudulent, only provide value if they're undisclosed, since otherwise they can be gamed. If you want to avoid detection and you have unfettered access to the model, you'll have an easier time determining how to do so. The detection models developed by government agencies help them identify persons of interest. These models may capture arcane patterns involving travel, communication, and financial transactions. A criminal who finds out exactly what those patterns are, can readily avoid triggering a high model score, so can a fraudster or even a cheating student. I actually use the model to detect when my students copied programming assignments when I was a professor. But if you've already been picked up by the cops and a models actually informing how long you're going to stay in jail, there needs to be due process and accountability. Crime risk models are trusted advisors that influence incarceration by informing bail, sentencing, and parole decisions. The model is like an expert witness, and so just as a witness can be cross-examined, so too must a model be available for questioning. But crime risk models such as COMPAS, are sealed tight in a secretive black box, unavailable for audit or inspection. The way COMPAS calculates risk scores is unknown to law enforcement, the defendant, and the public. In fact, the models creators publicly disclosed that it only incorporates six of 137 factors collected about each defendant, but exactly which six remains a secret. A common counterargument to transparency is that some models, such as deep neural networks are too complex for a human to understand. They're an opaque black box, very challenging to interpret. But as we've discussed, there are techniques to uncover a good amount about what any given model is actually doing. This includes experimenting on the model by changing input variables to see how much they affect the output, or tracking what happens inside the model. Like we previously looked at this example, neural network for self-driving cars, where you can see how a hidden layer helps detect when the car should steer towards the left. These kinds of model inspection methods are known as Explainable machine learning or Explainable AI, techniques to help humans understand how a predictive model works. Those opposed to allowing access to inspect models, also frequently add onto this line of argument, by pointing out that non-technical users of models, be they a defendant, a lawyer, or an officer of the court, would be prone to misinterpretations and confusion. They don't know how to understand it even if it's a less complex model. But certainly, these parties could and should enlist a machine-learning expert in order to conduct the audit. Of course, that audit won't be possible if access to the model itself isn't granted in the first place. Now, to clarify, in this context model transparency really just means model access. When you're talking on the technical level, transparency can also refer to whether a model is intrinsically simple enough to be more easily understood, such as decision trees. That's in contrast to neural networks and ensemble models, which are more opaque. But in the context of social justice and civil rights, transparency means that the overall decisioning process can be seen, which requires that you can actually access and look at the predictive model itself. There's a movement to make models transparent in the name of accountability and due process. This movement has seen legislative successes and failures. It's an ongoing legal battle. To be clear, there are two levels where full access that allows a detailed audit of the model bears fruit. One, the overall decision making process, that is, whether the model violates any standards or principals in general across defendants and two, for individual cases. On this level, the standard that's called for is known as the right to explanation. For example, a defendant would be told which variables contributed to their risk score. For what aspects of his or her background, circumstances, or past behavior was the defendant penalized, then the defendant can respond accordingly, providing context, explanations, or perspective on those factors. Now, there's a lot of interest and a lot of technical work developing explainable machine learning. Data scientists hold it as a high priority. This survey from 2018 shows that 85 percent of data scientists say that models should either frequently or always be understandable by humans. But unfortunately, explainable machine learning is rarely actually applied for the purpose of transparency. Industry research published in 2020 showed that the majority of explainable machine-learning deployments are not for end users affected by the model, but rather for machine-learning engineers who use explainability to debug the model itself. There is thus a gap between explainability in practice and the goal of transparency since explanations primarily serve internal stakeholders rather than external ones. Science and engineering shouldn't be secretive when it affects the public good, when it determines who's granted access to rights and resources. You can look under a bridge and see how the girders and beams work to ensure its integrity, so too should we be able to see how a machine that informs who stays in jail works. If not, it's like implementing a public policy the details of which are confidential. It's like having an expert witness without allowing the defense to cross-examine. A crime risk model isn't just a tool used under the control of an officer of the court. The officer doesn't actually control what it does other than deciding when to push the go button. Rather, the model is a means to actually perform judgment. It's operations belong squarely within the public sector. It cannot be protected by the trade secrecy generally granted to commercial enterprises. In what manner besides inspecting the mechanics can we hold the creators of decision-making machines accountable? Just the same as we hold human decision-makers and those who engineer and design bridges accountable. The fairness, accountability, and transparency in Machine Learning conference puts it plainly on the homepage of their website, "There's increasing alarm that the complexity of machine learning may reduce the justification for consequential decisions to the algorithm made me do it." When I interviewed law Professor Andrew Ferguson, the author of "The Rise of Big Data Policing", he told me that predictive analytics is clearly the future of law enforcement. The problem is that the forecast for transparency and accountability is less than clear.
