WEBVTT 00:00.480 --> 00:06.870 In this video we learn how to handle missing or and a values and this process actually includes two 00:06.870 --> 00:10.310 major steps so identifying missing values. 00:10.320 --> 00:17.660 And once we have identified them then we have to decide what to do with these missing values and the 00:17.710 --> 00:18.370 principle. 00:18.390 --> 00:19.680 Three options. 00:19.710 --> 00:26.730 So the first one is that we do nothing and leave them as they are and actually penned us is the best 00:26.730 --> 00:28.790 prepared to handle missing values. 00:28.890 --> 00:34.550 Since then many methods we can decide whether to include or to exclude and a values. 00:34.680 --> 00:40.710 And in most cases the default setting is that pandas simply ignores missing values. 00:40.710 --> 00:47.580 Then the second option is to delete missing values or more specific earlier observations or rows with 00:47.760 --> 00:49.260 missing values. 00:49.260 --> 00:53.420 And we can also delete entire columns with the missing values. 00:53.580 --> 00:59.740 And then the third alternative is to replace the missing values by either the actual values. 00:59.760 --> 01:04.440 If you can manage to find them or to replace by approximate values. 01:04.560 --> 01:10.380 So this is in principle a very complex topic not coding wise but in the decision making process what 01:10.380 --> 01:16.560 to do when as it always depends on the specific case and that's actually no black and white. 01:16.560 --> 01:20.640 And in this video and in this course we will focus on the coding part. 01:20.730 --> 01:24.350 However in my general append costs I would go much more into detail. 01:24.360 --> 01:32.870 Also on the decision making part so let's import pandas and again we are working with the Titanic dataset 01:33.030 --> 01:40.710 where we have many missing values and let's have a look and here in the info method we can already see 01:40.710 --> 01:42.290 our missing values. 01:42.390 --> 01:47.080 So we have here and the H column 177 missing values. 01:47.230 --> 01:50.450 Or to put it the other way around 714. 01:50.520 --> 01:53.140 None all or none missing values. 01:53.310 --> 01:58.410 And also on the impact column we have two missing values and we have many missing values in the tech 01:58.410 --> 01:59.610 column. 01:59.610 --> 02:06.140 And instead of identifying and a values with the info method we can do the same with the IS and a method 02:07.450 --> 02:13.570 and here pandas checks for each and every element whether the element is an a value or not. 02:13.600 --> 02:19.250 So here we can see we get many forces because here we have a non missing values. 02:19.630 --> 02:22.370 But here in the deck column we can see many truth. 02:22.390 --> 02:24.210 So here we have many missing values. 02:24.310 --> 02:31.240 And then we can also sum up the missing values for example per column to get the total amount of missing 02:31.240 --> 02:32.170 values. 02:32.230 --> 02:37.380 So as a short repetition a truth stands for one and a false for zero. 02:37.540 --> 02:39.760 Therefore we can change the sum method 02:42.620 --> 02:50.350 and then we get to your one hundred seventy seven missing values in the H column for example and actually 02:50.350 --> 02:56.430 the opposite of b is and a method is not and a method. 02:56.680 --> 02:58.220 Let's have a look here. 02:58.380 --> 03:00.680 So also here for each element. 03:00.700 --> 03:08.590 We get actually a true if the element is not missing and we get a false if uh the element is missing. 03:08.740 --> 03:12.900 And again we can also take the sum and uh. 03:12.920 --> 03:17.500 This is actually the same U S that we have here in the info method. 03:17.700 --> 03:27.280 So 714 man now I use in the H column and then we can also filter our data frame for example for all 03:27.300 --> 03:32.780 rows of all passengers where we have an A values in the impact column. 03:32.790 --> 03:40.610 So these are only two and therefore we you see the is and a method on the impact column and we filter 03:40.610 --> 03:42.340 our data frame with the log. 03:42.360 --> 03:49.000 Operator here we get two passengers where we do not know the port of embarkation. 03:49.340 --> 03:54.350 And uh now let's have a look at the shape of our data frame. 03:54.350 --> 03:58.460 So we have eight hundred ninety one rows and nine columns. 03:58.460 --> 04:03.920 And as I said before we have three options how to deal with missing values. 04:04.010 --> 04:07.520 And here in the impact column this is a very good example. 04:07.600 --> 04:15.740 We uh we actually do nothing because we only have two passengers with any NBA use and uh also the impact 04:15.740 --> 04:19.510 column is uh not the most important column in our data frame. 04:19.520 --> 04:22.740 So we just uh leave it as it is here. 04:22.790 --> 04:25.260 So we leave here and a values. 04:25.270 --> 04:33.760 And the second alternative would be to drop and a values and here we can either drop a rose with a value 04:33.770 --> 04:41.890 so we can drop columns with an A values and to do this Panda's provides us with the drop and a method 04:42.520 --> 04:49.160 and uh let's simply run here the Sarah and here we have the data frame where we apply it the drop and 04:49.160 --> 04:55.850 a method and it's quite hard to to identify and see anything and uh therefore we're also changing the 04:55.940 --> 05:04.040 attribute uh shape and we can see starting from eight hundred ninety one passengers and nine columns 05:04.310 --> 05:11.230 by just using the drop and may method the results us in a data frame with only one hundred eighty two 05:11.230 --> 05:18.600 rows and nine columns and the drop and a method actually by default drops all rows that we have at least 05:18.600 --> 05:26.510 the one missing value and as we have over 600 missing values only in the column then penned US deletes 05:26.520 --> 05:32.110 the actually all passengers where we have at least the one missing value and we only have left here 05:32.130 --> 05:40.310 one hundred eighty two however we can change this behavior and this the parameter however and the default 05:40.310 --> 05:41.610 setting is any. 05:41.960 --> 05:47.100 So by default Penn US drops all the rows where we have at least the one missing value. 05:47.420 --> 05:56.470 And let's again check it here and alternatively we can also pass all to the whole perimeter and by passing 05:56.560 --> 06:04.570 all to the whole perimeter then Penn drops only rows where all values are actually missing values and 06:04.660 --> 06:06.480 let's try this out here. 06:06.520 --> 06:09.950 So here we have all eight hundred ninety one passengers. 06:10.060 --> 06:16.990 So there's no passenger where we have only missing values and in the drop dropping a method that we 06:16.990 --> 06:21.610 have also the excess power meter and the default setting is zero. 06:21.970 --> 06:29.680 So append us the drops throws but we can also decide that pandas should drop columns and therefore we 06:29.680 --> 06:32.580 pass one to the axis perimeter. 06:32.920 --> 06:36.970 And again we can use any or all for the whole perimeter. 06:37.270 --> 06:46.880 So let's try out the any parameter and he will get data frame with all passengers but only six columns. 06:46.940 --> 06:51.750 So we dropped all columns where we have at least one missing value. 06:51.770 --> 06:56.450 And these are actually the columns H and parked and take 06:59.540 --> 07:02.870 and with the whole perimeter we are actually quite limited. 07:02.870 --> 07:04.780 So either we can define that. 07:04.840 --> 07:12.200 We want to drop all rows are columns where at least one value is missing of our all values are missing 07:12.200 --> 07:12.730 and. 07:12.770 --> 07:20.330 This is actually quite inflexible and therefore there's that the additional parameter Thresh and t we 07:20.330 --> 07:23.290 can define how many non missing values. 07:23.330 --> 07:31.130 We at least want to have in a row or in a column to leave the row or column in the data frame and let's 07:31.130 --> 07:32.800 focus here on the columns. 07:32.930 --> 07:39.720 So we pass one to the access parameter and we can say we want to drop all the columns where we have 07:39.980 --> 07:46.160 less than 500 non missing values or to put it the other way around. 07:46.220 --> 07:53.940 You want to drop all columns where we have more than 300 or 91 missing values. 07:54.230 --> 07:56.900 And let's check this out here. 07:57.380 --> 08:05.570 And we lost one column and obviously this should be the tech column because here in the column we only 08:05.570 --> 08:13.470 have two hundred three none missing values and this might be actually a good strategy in our case. 08:13.550 --> 08:20.870 So the attack column is actually of little use for us because the information itself is not too useful. 08:21.230 --> 08:29.210 And also we have only 200 known missing values and therefore we decide to drop all columns with less 08:29.420 --> 08:31.620 than 500 known missing values. 08:31.730 --> 08:39.230 And also here we have to pass the true to the in-place parameter and let's have a look. 08:39.950 --> 08:44.090 So here we have successfully dropped the tech column. 08:44.090 --> 08:46.550 So as a short recap we decided to leave. 08:46.610 --> 08:50.620 The two are missing values and the impact column be deleted. 08:50.750 --> 08:53.920 The complete attack column and uh. 08:53.990 --> 09:02.280 Now let's fill it out that our frame for file peasant says that we do not know the H So these are all 09:02.280 --> 09:08.850 peasant just where we have missing values and the H column and having 177 peasants as we do not know 09:08.850 --> 09:09.830 the H. 09:10.020 --> 09:17.760 It might not be the best position to just delete all 177 peasant just so it might be a better idea to 09:17.760 --> 09:25.890 replace the missing values and uh therefore we can either try to find the RE values or we can replace 09:25.940 --> 09:33.990 the missing values by approximations and here in this case one good approximation could be to use simply 09:33.990 --> 09:37.410 the mean age over all peasant just as a value. 09:37.920 --> 09:45.910 So first of all we calculate the mean age of all passengers and uh this is twenty nine point six nine. 09:46.080 --> 09:52.680 And then we can fill each and every missing value in the H column with uh the mean age and we can do 09:52.680 --> 09:55.470 this with the feel and a method. 09:55.470 --> 10:02.340 So first of all we select the H column and then there's a parameter value and here we can pass the full 10:02.340 --> 10:04.080 value that we want to have here. 10:04.080 --> 10:09.990 So here in this example it's the the mean age and also here we have to pass the truth with the in-place 10:09.990 --> 10:10.830 power meter. 10:11.460 --> 10:16.280 So let's do this and let's have a look at our H column. 10:16.390 --> 10:23.500 So before we could see here that the passenger index position five we have here a missing value. 10:23.500 --> 10:24.960 And now we have here. 10:24.970 --> 10:28.230 The mean age twenty nine point six nine. 10:28.240 --> 10:36.510 And also finally we can inspect the our data frame again with the Inform method and here we can see 10:36.510 --> 10:44.070 that we have no missing values now in the H column so we have successfully replaced all missing values 10:44.070 --> 10:46.010 here by the mean age. 10:46.230 --> 10:50.880 And with this we are finishing with this video and I hope to see also in the next one by.