WEBVTT 00:02.220 --> 00:07.830 In this video we learn how to use summary statistics and accumulations to further analyze solid data 00:07.830 --> 00:10.890 frames and in particular data frames. 00:10.890 --> 00:13.030 This numeric values. 00:13.320 --> 00:20.820 So first of all let's import pandas and we are again working with our Titanic's dataset and we check 00:20.820 --> 00:25.390 the first five rows here and we already know the year. 00:25.390 --> 00:27.370 Very helpful describe method 00:30.510 --> 00:33.070 so the described method returns us. 00:33.120 --> 00:40.020 Many summary statistics for our numerical columns so we FBI one American column survived PE class H 00:40.270 --> 00:42.340 S P parts and fair. 00:42.480 --> 00:48.690 And first of all we have count we are pandas actually counts on none missing values. 00:48.690 --> 00:57.400 So here in the H column we have some missing values then we are further mean value over all values in 00:57.580 --> 00:59.800 the particular column. 00:59.800 --> 01:05.470 So for example the average fare ticket price was thirty two dollars or pounds. 01:05.470 --> 01:12.830 Then we have the standard deviation the minimum 21st percentile the median the seventy first percentile 01:12.870 --> 01:14.560 and the maximum. 01:14.560 --> 01:22.110 So the oldest passengers 80 years old and for most of these summary statistics here on the left side 01:22.770 --> 01:25.110 there also exists a method. 01:25.110 --> 01:32.360 So for example that is the count method that we can apply for the whole data frame and that's press 01:32.360 --> 01:33.460 shift type here. 01:34.460 --> 01:39.650 So the count method counts nominal missing values for each column or row. 01:40.280 --> 01:45.590 So now let's run here and we get you for each column. 01:45.590 --> 01:52.790 The amount of num missing values and we can define here the X's where we want to count actually the 01:52.790 --> 01:56.560 number of non missing values and by default it set to zero. 01:56.750 --> 02:01.670 And alternatively we can also pass here in quotation marks index 02:05.030 --> 02:11.120 and with a zero index we get actually for each column the amount of non missing values. 02:11.240 --> 02:16.580 And in my opinion this is a bit confusing because what we get here for each column the amount. 02:16.850 --> 02:21.070 But we set X's to index and that column. 02:21.200 --> 02:29.990 And the reason is that X's equals index means that we go here along the rows and along the axis here 02:30.020 --> 02:37.680 for each column so in my opinion this is a bit confusing and that might be further room for improvement 02:37.890 --> 02:42.600 for the next version here of pandas and let's try. 02:42.600 --> 02:45.950 Also columns here columns are 1 02:51.050 --> 02:58.420 and here we get the amount of num missing values for each row and you could say that okay we are going 02:58.420 --> 03:05.250 here along the columns and therefore I have to use the parameter columns. 03:05.310 --> 03:10.440 All right so either you use them columns or X equals slums so that's the same here 03:14.730 --> 03:19.620 and then you can also take the mean and the free set axes to zero. 03:19.620 --> 03:27.080 We get the mean for our columns or along the rows so here again we ask for the. 03:27.090 --> 03:33.870 The average ticket price of thirty two dollars or pounds and we could also try to calculate the mean 03:33.870 --> 03:43.550 for each row and we still get your values so actually if you look here at the row we have the numerical 03:43.550 --> 03:50.210 values in each row but also and string values so still the mean method the returns for each row here 03:50.210 --> 03:55.690 a value and let's check here why this is the case. 03:55.700 --> 04:03.030 So let's go here inside and we actually have the parameter numeric only. 04:03.440 --> 04:04.640 And let's check this 04:08.200 --> 04:09.750 include only float integer. 04:09.780 --> 04:11.130 Are we in columns. 04:11.140 --> 04:12.690 And if they're set to None. 04:12.700 --> 04:21.540 And by default numeric only is set to num then pandas will attempt to use everything and then use only 04:21.540 --> 04:22.560 numeric data. 04:23.340 --> 04:24.880 So pandas is quite smart here. 04:24.900 --> 04:32.240 Tries to calculate the mean over all values and if it finds some numerical values and modern American 04:32.240 --> 04:40.220 values or string values it then excludes the string values from the mean calculation and pin. 04:40.330 --> 04:41.830 Does this automatically. 04:41.830 --> 04:47.350 So here you again can see that pandas is quite powerful and quite smart. 04:47.440 --> 04:47.890 All right. 04:47.890 --> 04:49.720 So there's also the method. 04:49.770 --> 04:57.930 Some and we can take this sum for each columns so your father survive column B F 342. 04:58.160 --> 05:03.430 That means that we have three hundred forty two pattern passengers that survived. 05:03.620 --> 05:12.470 So we have the value one for survival and 0 4 not survived and depending on the column actually the 05:12.470 --> 05:18.650 sum makes sense does not make sense here for the sixth column we have just made of human female a female 05:18.680 --> 05:19.580 concatenated 05:23.350 --> 05:31.590 So for example if you want to take you the sum for all rows by passing here one we again get the sum 05:31.710 --> 05:40.260 of on American values so also here pandas tries to take the sum of all elements and once it finds some 05:40.260 --> 05:47.170 string values that are just ignores so the string values and takes only the numerical values. 05:47.520 --> 05:53.600 Right so let's have a look again about what data frame and the title of our video is here. 05:53.590 --> 05:59.980 Not only salary statistics but also accumulations and one example for a combination this comes from 06:01.080 --> 06:04.590 so let's try the come some method on the fare column 06:08.310 --> 06:12.590 so come some stands for cumulative sum and we can see here. 06:12.730 --> 06:22.480 So we're starting with the seven point to five and then for each row come some sums up all rows so that 06:22.480 --> 06:29.710 came before a slow here in the second row we saw turf here seventy eight and then seventy eight plus 06:29.740 --> 06:36.520 the seven point nine gets eighty six and then eighty six plus fifty three guess one hundred to thirty 06:36.520 --> 06:37.600 nine and so on. 06:38.590 --> 06:47.710 All right let's go on and that is the method the top car and the top car actually calculates the correlation 06:47.710 --> 06:54.240 between numerical columns and the correlation is actually an indication of that a relationship of two 06:54.240 --> 07:03.330 random variables so in other words if and how two random variables move together and the correlation 07:03.330 --> 07:10.230 can take values between minus one and my one and minus one is an indication for strong and negative 07:10.230 --> 07:11.080 correlation. 07:11.160 --> 07:15.930 One variable goes up and the other goes down and vice versa. 07:15.930 --> 07:22.920 And the value of zero so zero correlation stands for or as an indication for no relationship between 07:22.920 --> 07:31.900 those two variables and the correlation of one is an indication for strong positive correlation so when 07:31.900 --> 07:39.650 one variable moves up the other moves up as well and we can never look here at our Titanic dataset and 07:39.650 --> 07:40.780 we can see here on the left. 07:40.780 --> 07:42.770 So we have to this advice column. 07:42.770 --> 07:51.350 And of course the survive column is a hundred percent positive correlated to itself and we see here 07:51.380 --> 07:58.190 that that is quite high and negative correlation to the peasants are class so the lower the peasants 07:58.190 --> 07:58.610 are class. 07:58.620 --> 08:05.140 So we have peasants us in class 1 the higher is the likelihood that the peasant just survived. 08:05.790 --> 08:13.320 So he has some indication that actually in money the roots of the vote not only in business but also 08:13.410 --> 08:16.830 on board of the Titanic and TFR the H. 08:16.830 --> 08:22.720 We have weak negative correlation and also he has the best P in pots. 08:22.740 --> 08:30.760 It s only a small correlation here and for the fact column we have a fairly strong positive correlation. 08:31.480 --> 08:37.870 So the more a passenger paid for his ticket the more likely it was then that the passenger survived 08:40.000 --> 08:47.500 and we could also have a look here at the fact column and the ticket price is actually positive correlated 08:47.530 --> 08:49.290 to the survive. 08:49.300 --> 08:56.630 So we already have seen this and also ticket prices are quite strong negatively correlated to peasants 08:56.680 --> 08:58.340 class. 08:58.460 --> 09:00.120 So this is a no brainer. 09:00.320 --> 09:07.890 So a passenger in class one paid more for his ticket than a passenger in class 3 and we can also see 09:07.890 --> 09:13.010 here a positive correlation to the amount of parents and children and the ticket price. 09:14.010 --> 09:21.120 So obviously the passengers that traveled with parents and children paid more for that ticket than passengers 09:21.120 --> 09:23.100 who travel alone. 09:23.100 --> 09:29.780 So it might be the case that in the first class the passengers travelled with their families in a suite 09:29.840 --> 09:37.470 and in the third class there were many young men who traveled alone trying to find their happiness in 09:37.470 --> 09:38.210 America. 09:39.260 --> 09:39.920 All right. 09:39.920 --> 09:47.180 And there's also the car method for Penn State aframe and there we can return the correlation between 09:47.210 --> 09:48.950 only a two column. 09:48.950 --> 09:56.690 So you selected here the survival column and then with the call method we can calculate the correlation 09:56.690 --> 09:59.360 between the survival column and the peak plus column. 09:59.360 --> 10:05.910 So you have it in the brackets have to pass the peak class column as a pen that's serious. 10:06.140 --> 10:09.030 And here we get minus 0 point 3 3. 10:09.050 --> 10:17.680 So the same as here so if you were a passenger on the third class it was less likely that you survived 10:17.890 --> 10:19.360 the Titanic disaster. 10:21.700 --> 10:26.080 All right now we are finished services video and I hope to see in the next one by.