Performance Evaluation: the Challenge of Noisy Data

Loading...

From the course by University of Pennsylvania

People Analytics

оценки

University of Pennsylvania

People Analytics

оценки

Course 3 of 5 in the Specialization Business Analytics

People analytics is a data-driven approach to managing people at work. For the first time in history, business leaders can make decisions about their people based on deep analysis of data rather than the traditional methods of personal relationships, decision making based on experience, and risk avoidance. In this brand new course, three of Wharton’s top professors, all pioneers in the field of people analytics, will explore the state-of-the-art techniques used to recruit and retain great people, and demonstrate how these techniques are used at cutting-edge companies. They’ll explain how data and sophisticated analysis is brought to bear on people-related issues, such as recruiting, performance evaluation, leadership, hiring and promotion, job design, compensation, and collaboration. This course is an introduction to the theory of people analytics, and is not intended to prepare learners to perform complex talent management data analysis. By the end of this course, you’ll understand how and when hard data is used to make soft-skill decisions about hiring and talent development, so that you can position yourself as a strategic partner in your company’s talent management decisions. This course is intended to introduced you to Organizations flourish when the people who work in them flourish. Analytics can help make both happen. This course in People Analytics is designed to help you flourish in your career, too.

From the lesson

Introduction to People Analytics, and Performance Evaluation

In this module, you'll meet Professors Massey, Bidwell, and Haas, cover the structore and scope of the course, and dive into the first topic: Performance Evaluation. Performance evaluation plays an influential role in our work lives, whether it is used to reward or punish and/or to gather feedback. Yet its fundamental challenge is that the measures we used to evaluate performance are imperfect: we can't infer how hard or smart an employee is working based solely on outcomes. In this module, you’ll learn the four key issues in measuring performance: regression to the mean, sample size, signal independence, and process vs. outcome, and see them at work in current companies, including an extended example from the NFL. By the end of this module, you’ll understand how to separate skill from luck and learn to read noisy performance measures, so that you can go into your next performance evaluation sensitive to the role of chance, knowing your environment, and aware of the four most common biases, so that you can make more informed data-driven decisions about your company's most valuable asset: its employees.

Introduction to People Analytics8:45

Goals for the Course1:40

Course Outline and Overview3:50

People Analytics in Practice4:55

Performance Evaluation: the Challenge of Noisy Data6:46

Chance vs. Skill: the NFL Draft22:08

Finding Persistence: Regression to the Mean11:56

Extrapolating from Small Samples5:53

The Wisdom of Crowds: Signal Independence5:56

Process vs. Outcome7:47

Summary of Performance Evaluation3:36

Meet the Instructors

Cade Massey
Practice Professor
The Wharton School
Martine Haas
Associate Professor of Management
The Wharton School
Matthew Bidwell
Associate Professor of Management
The Wharton School

So in this module we wanna talk about performance evaluation, and as most of you

know from having worked in organizations there are a couple of purposes.

One is rewards and punishment.

Mostly firms tie rewards, whether it's a raise or

a bonus or sometimes a promotion to some kind of performance evaluation.

But also firms do this for feedback.

They want employees to get better at their jobs.

This is a means of giving that kinds of feedback.

The kind of feedback is necessary to get better at your job.

But this is performance evaluation, and we're gonna focus on performance

evaluation as opposed to talent management.

It's really tough to compare employees, if they're not in identical situations.

So we'll talk elsewhere in the people analytics course about talent management

and comparing employees.

Right now, we really want to focus on evaluating the performance management of

a given employee.

For this starting place for this lecture and a little bit for

life you might think about beginning by assuming all employees are equal ability.

That's a rough decent starting place.

For what we're gonna do here.

Begin by assuming all employees are equal ability.

And now, we're trying to figure out how hard are they working and

how can we improve them at their job.

So the fundamental challenge in performance evaluation

is that performance measures are noisy.

Whatever means we wanna use for evaluating how our employees performing is

imperfectly related to what their actually doing with how hard they're working or

how smart they're working.

And that slippage is essentially noise.

This is how a statistician would think about it,

this is how an economist would think about it.

They're noise in the performance measures.

And this is gonna be a problem for us if we're trying to infer from these

performance measures something about the employee.

You can think about it as a normal distribution.

So for any given level of effort,

you might see the performance measure line up perfectly with that level of effort or

you might see it come in a little high or a little low.

So a range of outcomes can occur due to factors outside the employees control.

So no matter how hard they work or how smart they work they have competitors,

they have team members, they have bosses,

they have an economy that they're interacting with.

All of these things are outside the employees control.

So there's noise, there's slippage between what the employee tries to do,

how hard they work, how smart they work and

the performance measure that results as a consequence of that.

So this is our challenge, separating skill from luck.

And for a person like me, this is one of the great challenges in life.

You're forever trying to separate from what we see how much is luck and

how much is skill?

There's an economist at the University of Chicago, John Huizinga.

Who said recently in an interview, this distinction between ascertaining skill and

luck shows up all the time.

Who do you give your money to for investing?

How much do you pay a certain employee?

It's everywhere.

It isn't just about sports.

It's about life.

So that's really what we are trying to do here in this performance

evaluation module.

We are trying to separate skill from luck.

You might think of it as signal verses noise.

We wanna understand the true signal,

even though we only get noisy performance measures.

So in southern environments this isn't so hard.

For example, in this picture I've shown two normal distributions and

you can think of this as what might result if an employee gives high effort or

if the employee gives low effort.

And we're gonna use for our simple model here, we're gonna use effort, but

that's just a simplification.

You can think of this as working smart.

You can think of this, you can even though we wanna avoid for

the moment thinking of this as being a high talent versus low talent.

So for the moment let's just think of it as high effort.

Are they working hard or are they not working hard?

In the world I've described in this picture,

the outcome measures that result from high effort or low effort are quite different.

So you see that the high effort distribution is quite a bit higher,

quite a bit closer to the good outcomes then is the low effort distribution.

So that when you get a performance measure that's pretty high on the good outcomes

scale, you can be pretty sure that came from high efforts.

So you can infer this employee did something well.

This employee did something right.

You can give them positive feedback.

Conversely if you see something in this world that is very

low on the outcome scale, close to the bad outcomes end

then you can be quite confident that that came from low effort.

That's because these signals are quite separate.

So in this environment,

it's not very hard to tell whether your employee is working hard or not.

You get pretty clean signals.

Now it's not perfect.

If you get an average performance measure, it can be ambiguous and

that's gonna be the case often.

But for a broad range of outcomes,

you can be pretty confident about what kind of effort the employee gave.

But in other environments, it's brutal to determine whether the employee gave high

effort or low effort because of how overlapping the distributions are.

So for example in this situation,

the good outcome measure more likely came from high effort, but

there's still a very reasonable probability that it came from low effort.

That maybe someone didn't put in much work, but the economy broke just right for

them or their boss covered them or

the team they worked with happened to be really on that month.

All of those things are outside the employee's control, but

can lead to a low effort performance resulting in a positive consequence.

And vice versa, you might get a bad outcome and

think on average that came from low effort but there's still a very reasonable

chance, because of the noise in the measure that it came from a high effort.

Again, it could be that they put in 80 hours that week, but

their teammates let them down or their boss made a bad decision or

their competitor made an unexpected move.

So in these environments, it's very difficult to ascertain

the level of effort from a noisy performance measure.

This in essence is the challenge with performance evaluation.

It's not always quite this difficult, there's not always this much overlap but

there's always some level of noise and it makes it difficult to infer,

it makes it difficult to give the right kind of feedback.

We want to give you some help sorting this environment because all of you are being

judged in this way to some extent, and all of you are judging employees in this way.

So this is gonna motivate everything that comes next essentially.

We're gonna give you some examples, some techniques and importantly

some biases that people have whenever they try to make these inferences.

So the merger plan, first is to give you an extended example and

then drop in to four specific issues and unpack those.

Because there are biases associated with each of those issues and

that will lead to some prescriptions which we provide as we go on and

then we summarize in the last stage

Ознакомьтесь с нашим каталогом

Присоединяйтесь бесплатно и получайте персонализированные рекомендации, обновления и предложения.

Coursera делает лучшее в мире образование доступным каждому, предлагая онлайн-курсы от ведущих университетов и организаций.

© Coursera Inc., 2018 Все права защищены.

Загрузить из App Store

Загрузить в Google Play