This course will introduce the learner to the basics of the python programming environment, including fundamental python programming techniques such as lambdas, reading and manipulating csv files, and the numpy library. The course will introduce data manipulation and cleaning techniques using the popular python pandas data science library and introduce the abstraction of the Series and DataFrame as the central data structures for data analysis, along with tutorials on how to use functions such as groupby, merge, and pivot tables effectively. By the end of this course, students will be able to take tabular data, clean it, manipulate it, and run basic inferential statistical analyses. This course should be taken before any of the other Applied Data Science with Python courses: Applied Plotting, Charting & Data Representation in Python, Applied Machine Learning in Python, Applied Text Mining in Python, Applied Social Network Analysis in Python.
Group by
Loading...
From the course by University of Michigan
Introduction to Data Science in Python
6591 оценки
University of Michigan
6591 оценки
Course 1 of 5 in the Specialization Applied Data Science with Python
From the lesson
Week 3
In this week you'll deepen your understanding of the python pandas library by learning how to merge DataFrames, generate summary tables, group data into logical pieces, and manipulate dates. We'll also refresh your understanding of scales of data, and discuss issues with creating metrics for analysis. The week ends with a more significant programming assignment.
Meet the Instructors
Christopher Brooks
We've seen that even though PANDAS allows us to iterate over every row in a data
frame this is generally a slow way to accomplish a given task and
it's not very pandorable.
For instance, if we wanted to write some code to iterate over all the of
the states and generate a list of the average census population numbers.
We could do so using a loop in the unique function.
Another option is to use the dataframe group by function.
This function takes some column name or names and splits the dataframe up into
chunks based on those names, it returns a dataframe group by object.
Which can be iterated upon, and
then returns a tuple where the first item is the group condition,
and the second item is the data frame reduced by that grouping.
Since it's made up of two values, you can unpack this, and
project just the column that you're interested in, to calculate the average.
Here's some examples of both methods in action.
Let's first load the census data, then exclude the state level summarizations,
which had a sum level value of 40.
Here's some examples of both methods in action.
Let's first load the census data, then exclude the state level
summarizations which had a sum level of 40.
In the first we used the census date.
We get a list of the unique states.
Then for each state we reduce the data frame and calculate the average.
If we time this we see it takes a while.
I've set the number of loops here the time it should take to ten
because I'm live loading.
Here's the same approach with a group by object.
We tell pandas we're interested in group and with a state name and then we
calculate the average using just one column and all of the data in that column.
When we time it we see a huge difference in the speed
Now, 99% of the time, you'll use group by on one or more columns.
But you can actually provide a function to group by as well and
use that to segment your data.
This is a bit of a fabricated example but
lets say that you have a big batch job with lots of processing and
you want to work on only a third or so of the states at a given time.
We could create some function which returns a number between zero and
two based on the first character of the state name.
Then we can tell group by to use this function to split up our data frame.
It's important to note that in order to do this you need to set the index of
the data frame to be the column that you want to group by first.
Here's an example.
We'll create some new function called fun and
if the first letter of the parameter is a capital M we'll return a 0.
If it's a capital Q we'll return a 1 and otherwise we'll return a 2.
Then we'll pass this function to the data frame reply, and
see that the data frame is segmented by the calculated group number.
This kind of technique, which is sort of a light weight hashing,
is commonly used to distribute tasks across multiple workers.
Whether they are cores in a processor, nodes in a supercomputer, or
disks in a database.
A common work flow with group bias that you split your data,
you apply some function, then you combine the results.
This is called split apply combine pattern.
And we've seen the splitting method, but what about apply?
Certainly iterative methods as we've seen can do this, but
the groupby object also has a method called agg which is short for aggregate.
This method applies a function to the column or
columns of data in the group, and returns the results.
With agg, you simply pass in a dictionary of the column names that you're interested
in, and the function that you want to apply.
For instance to build a summary data frame for the average populations per state,
we could just give agg a dictionary with the Census 2010 pop key and
the numb/pie average function.
Now, I want to flag a potential issue and
using the aggregate method of group by objects.
You see, when you pass in a dictionary it can be used to either to identify
the columns to apply a function on or
to name an output column if there's multiple functions to be run.
The difference depends on the keys that you pass in from the dictionary and
how they're named.
In short, while much of the documentation and examples will talk about a single
groupby object, there's really two different objects.
The data frame groupby and the series groupby.
And these objects behave a little bit differently with aggregate.
For instance, we take our census data and convert it into a series with the state
names as the index and only columns as the census 2010 population.
And then we can group this by index using the level parameter.
Then we call the agg method where the dictionary that has both the numpie
average and the numpie sum functions.
PANDAS applies those functions to the series object and, since there's only one
column of data It apples both functions to that column and prints out the output.
We can do the same thing with a data frame instead of a series.
We set the index to be the state name, we group by the index, and
we project two columns.
The population estimate in 2010, the population estimate in 2011.
When we call aggregate with two parameters,
it builds a nice hierarchical column space and all of our functions are applied.
Where the confusion can come in is when we change the labels of the dictionary we
passed to aggregate, to correspond to the labels in our group data frame.
In this case, pandas recognizes that they're the same and maps the functions
directly to columns instead of creating a hierarchically labeled column.
From my perspective this is very odd behavior,
not what I would expect given the labeling change.
So just be aware of this when using the aggregate function.
So that's the group by function.
I use the group by function regularly in my work, and it's very handy for
segmenting a data frame, working on small pieces of the data frame,
and then creating bigger data frames later.
Ознакомьтесь с нашим каталогом
Присоединяйтесь бесплатно и получайте персонализированные рекомендации, обновления и предложения.
Coursera делает лучшее в мире образование доступным каждому, предлагая онлайн-курсы от ведущих университетов и организаций.
© Coursera Inc., 2018 Все права защищены.
