Today, we're going to be talking about challenging some measuring mortality as a you're kind of primary impact measure of your evaluation. So what we'd like to do is walk through some of the challenges with measuring mortality. And then later on, we'll get in more details about how you try to, work around those challenges to be able to measure, mortality is an impact. So the first thing is that you've got to think about when you're measuring, when you're thinking about mortality as your measure of impact for your evaluation, is deaths a rare events. So here's some data from a few years back. That says, maternal mortality rate ratio, which is kind of SDG and MDG indicator is 216 on average. That means out of 100,000 live births, under-5 mortality Is 41 out of 1000 live births, neonatal mortality 19. And the key is, all of these are becoming more and more rare. So, in terms of you thinking about what you might do this year or five years or 10 years later, one of the keys is, mortality and death. It's good news that they're becoming a rare events, but it also means it's much harder for evaluation to actually measure those and use them as kind of the gold standard of impact, their thing. So, the key is for you to be able to do this. You've got to think about, how many, if you're doing surveys or something like that and we'll discuss that later. How many households are you going to have to find, a number of children, given the mortality rate that will produce a big enough sample. That would actually allow you to measure a difference. Challenge 2 is that, death can be socially and culturally sensitive. So the key is usually the way people get measures you'll see, in low and middle income countries, you get measures of mortality is through surveys. So you're asking people to report. So one of the big issues is that sometimes they're stigma or people don't want to talk about it. And this can cause a problem when you're trying to collect data, on one mortality, and also if you're interested in cause of mortality, it even gets more complex. Another issue with displaying socially and culturally sensitive. One of the places where comes a big role, is when you're trying to distinguish between stillbirths and early neonatal deaths. And that often, there's different cultural aspects of what counts as a live birth versus a stillbirth and that this can include two can lead to much under reporting and kind of incorrect measurement. And eras between stillbirths and neonatal, which we'll talk about later, in this course. So the third challenges who can remember what, the idea is that in surveys or in most of these things, looking at measuring mortality, you're asking people, to remember what happened in the past. This can cause several problems. A mission of events, especially with sibling histories, that we use a lot from maternal mortality. People can under report deaths because they don't know or they forget, because it happened quite a long period of time, and the sister of the sibling no longer lives close to them. You can also have misclassification of events, that people call it a neonatal death of stillbirth, which really you can't distinguish. It depends on that person's interpretation. And finally, there is the issue of dating arrows, that when people are asked about mortality. And you ask them to say, how old was the child, there's a tendency to lump them or heat them according to one month, one week, one day, one year. And you don't get specific information about the date of mortality. Here's another example of kind of an idea. If you want to show about heaping, here's a classic example where they're looking at reports of mortality by age, in months from mothers. And what you see is, you get a little higher in the first month of life and then it goes down down, and then suddenly at 12 months you get this big spike. Well, no one really believes that. In fact, all the kids died in the 12th month life or at the end of the first year. Whether the idea is kids that were died in the 10th month of the 9th month or the 13th or 14th month got heaped into the one year. So with recall, there's a tendency to, heap data, and therefore you're going to have these problems. If you're trying to say, how many kids died at age one, you're going to get a very skewed report based one, not them doing anything wrong. But it's been a while, what they remember. Well, he was about a year. Another challenge for you is, really in some ways we've been talking about surveys and recall. But, the key is mortality area are all flawed but in different ways, we're going to talk about this as we go through the course. But population censuses are one way that people often get mortality data. They're very large, very expensive. You'll never use that for an evaluation. But you may, if a census of being run in the country use that as data In your evaluation. The key is they're usually done about every 10 years. They do some rebirth histories, which again we'll discuss later, and it produced an estimate of mortality in the past. The key is, you still have all the problems in terms of recall, issues with these population censuses. The second major approach is large scale sample surveys, bi national or sub national. Here they use different methodology, but it's, you're doing a sample, so not a census. So it's much cheaper, they have standard techniques, for measuring it. It's often done kind of the two biggest programs that do this or the demographic and household surveys. Which includes direct and indirect methods of mortality, and the MICS survey, which primarily used direct, other ways. You can look at mortality data, civil registration and vital statistics systems. This is kind of the ideal situation that in fact, you say, well we've got vital statistics and civil registration. Every country has a vital registration system, and some kind of civil registration. But the issue is that in most low and even middle income countries, they're incomplete. They don't have accurate data. The things change over time, so you can't even look at trends. And that, truthfully in the current world we're in, this will never be used as a measure of impact, if you're doing an evaluation and low income country. Another data source, possible data source from mortality of sample registration systems. Simple registration systems are like a vital statistics system, in that they do intense work and try to ensure completeness of reporting on births and deaths. However, rather than cover all people in the country, they instead sample and have different clusters that represent often states and the country. And do intensive work there to produce information on births and deaths. This is a very good system in terms of a, the idea is you think of vital registration low and middle income countries. They exist but they're not reliable. SRS the way you can scale those up. So without covering everything in the country, you do cover, selected clusters that are to be representative of the country, and allows you to get a countrywide estimate of births and deaths. That's very reliable, unfortunately it's only done in two countries right now, China and India both do, have SRS system. There's also some initial work, COMSA. The comprehensive mortality surveillance for action project is doing work on developing sample registration systems in Mozambique and sierra Leone. The final thing I'll talk about here, is demographic and surveillance systems. They're similar to the sample registration, except usually it's just one side. Often you'll see RCTs or run within in demographic and surveillance systems because, just like a civil registration they track, births and deaths and it's fairly intensive. Does a good job, very consistent data. But the key is, if you can find a DSS system that overlaps and works within where your evaluation is, that's great. It then becomes a source of mortality data, at post baseline and in line. But in fact if you don't overlap completely, these systems are not very helpful. The last challenge and I don't know if it's a challenge, but maybe it's a challenge for you to decide, is the idea of do you measure or model mortality? You're going to see as we go through, given the technical challenges and cost in mortality measurement. Often, it kind of swamps your whole evaluation budget. If you're really going to try to go out and measure mortality, and modeling can be an important alternative. And in some cases it's the only choice, if you're trying to look at things like, very rare events like maternal deaths or cause specific death, if you've got long time lags in terms of nutritional impact. If you want to build counterfactual is using other comparisons to other countries or other regions, or to assess contribution of individual interventions in reducing deaths. Models can be very effective, for kind of expanding what you can do with impact. But again, you're not measuring mortality, you're measuring other indicators, and then you're trying to make an estimate. So it's a trade off that evaluations have to decide how they're going to handle impact. One of the things kind of a follow up to this measuring modeling, one of the keys is that the nice thing I like to think since this is what I've done for the last 20 years of my academic career is modeling. There's still a lot of resistance to modeling that. I just have this little quote here of some group that said, yeah, we love your model, but we're never going to use it, to actually make decisions, it's got to be an RCT. One of the problems with modeling is that, you have to do a lot of work to convince people that the model is going to produce accurate and replicable results. And especially for the funders of your evaluation, they want to have a sense that they can go out and say, well, we have good evidence that our program worked and modeling makes that less likely. It's a little harder argument to sail about models. So that's something you're going to have to measure, you're going to have to weigh in terms of do you want to measure? Which often cost much more money. And in fact in some cases, like we say, with very rare events, it's going to cost massive amounts of money to collect data and there's still issues there. Are you willing to use a modeling approach that might, allow you to estimate the mortality impact in your program. So let's just go through the summaries here, that we kind of talked about child mortality deaths. That's a rare event, means you have to have large sample size. If you're going to measure them, they are often stigmatized, can affect how data are reflected in their accuracy. Recalling reporting of deaths are subject to bias from emissions, and that can lead to all kinds of errors about misclassification and dating, sources of mortality. All the sources of mortality data have important limitations. And often, one of the problems with these various sources, is they don't exist in the areas or fit with the area where you're running your evaluation. And finally, modeling mortality is an important alternative measurement, but requires complex skills, careful assessment of model equality. And the ability of your funder, to agree that this is the approach you want to take, now going forward. And of course, these challenges vary for maternal deaths, under-5 deaths and deaths around the time of birth. So what we're going to do is, we're going to consider each of those separately, because they have different issues, and we'll be moving on to maternal mortality in the next lesson.
