0:01
Welcome to the course on Evaluation and Metrics of Recommender Systems.
In the previous two courses, we've looked at a number of ways of doing
recommendation using basic popularity, segmented statistics, content of items.
Then we've looked at collaborative filtering,
how to use user item interactions in order to produce recommendations.
But one thing we've not yet talked about is, how do we know which of these
approaches is going to work better in a particular situation?
Or if they're going to work at all or we should just pick some random movies for
people to watch.
In this course,
we're going to be learning how to match recommenders to their desired outcomes and
measure their ability to meet the needs in these different kinds of use cases.
0:45
And also how to optimize the recommenders behavior.
We've seen a number of parameters such as neighborhood sizes,
how do you pick which one you need?
The tools that we give you in this course will answer that question.
0:57
>> The key concepts in this course start with metrics.
We'll take you through some basic metrics for accessing the accuracy of
recommenders, prediction, its ability to help in making correct decisions.
We'll look at rank metrics that evaluate the quality of a top analyst and
then we'll move in to more advance metrics looking at concept such as diversity and
serendipity.
1:23
We'll look at the whole concept of evaluation without users
understanding what you can do and the framework for dealing with offline data.
And covering up or
hiding parts of your data to evaluate whether you can predict what you'd hidden.
And we'll look at evaluation with users.
Looking at lab experiments and field experiments,
including the online A/B tests.
As well as some work on surveying users and
log analysis to understand user behavior.
As we put all of that together, we'll get a holistic concept of how we can
evaluate our recommender system and use that evaluation to feed forward and
to tune the system and redesigning it to work better.
>> This course is structured with the first week focusing on basic metrics for
evaluating the accuracy of predictions and recommendations.
For this class, it will be an assignment and a spreadsheet and
there'll be a quiz on the topics.
We'll then spend two weeks on advanced metrics and
offline evaluation structures with programming assignment where you'll
actually implement and run a recommender experiment in LensKit.
If you're taking the honors track we recommend getting started on this
assignment particularly early.
Because, it's fairly computationally expensive to evaluate several algorithms
over several data sets.
So, you'll need to leave some time for the program to run.
We'll also have a quiz again over the topics and
then we will have a week on online evaluation.
Throughout this, we have a running assignment, weeks 2 through 4,
where you design an experiment to evaluate a recommenders' system.
These are going to be peer-graded, you're going to be looking at each other's
experimental designs to assess their effectiveness and appropriateness.
And you can treat this as a warmup for the Capstone project at
the end of this specialization if you're planning to take the entire package.
Michael D. EkstrandAssistant Professor
Joseph A KonstanDistinguished McKnight Professor and Distinguished University Teaching Professor
