WEBVTT 00:01.060 --> 00:05.850 Welcome to the introduction video and a values are missing values. 00:05.850 --> 00:12.720 And before we are going to detect and handle missing values in our Titanic and Olympic data set we should 00:12.720 --> 00:19.920 first of all understand what and values are and how and why and a values are generated and actually 00:19.920 --> 00:25.630 and a values are missing values are also a very important topic because then real data. 00:25.650 --> 00:30.240 We always have missing values and we always need to handle missing values. 00:30.300 --> 00:36.360 And so far throughout the costs we already have seen missing values and we have worked with these missing 00:36.360 --> 00:40.350 values and now it's time to dig a bit deeper into this topic. 00:40.350 --> 00:45.020 And we have already imported pandas and we also import here not supply. 00:46.230 --> 00:52.260 And we are working with a saved state asset which is a start in the UCSB five states that see us we 00:52.260 --> 00:53.520 use its import this 00:56.350 --> 00:57.830 and several look here. 00:57.850 --> 01:01.730 So here on the left hand side is the index of our data frame. 01:01.780 --> 01:08.260 We see the phosphates man Steven Mike Andy and Paul and as columns we have feared the big days Monday 01:08.260 --> 01:15.190 Tuesday Wednesday Thursday and Friday and the values and our data frame are actually sales data. 01:15.190 --> 01:22.710 So for example my cat 45 say it's on Monday and we can see here that for Steven on Thursday we fear 01:22.760 --> 01:29.030 a and a and value so and a and stands for Not a number and pen does. 01:29.050 --> 01:33.960 This is a strong indication that we have fear missing values in our data. 01:34.060 --> 01:38.110 So it might be a good idea to also open the original CSP file. 01:38.110 --> 01:40.660 So let's do this here. 01:40.670 --> 01:41.620 So this is here. 01:41.660 --> 01:48.800 See as we fire and we can see here the column labels Monday Tuesday Wednesday Thursday Friday and on 01:48.800 --> 01:55.280 the left hand side here the arrow index labeled Steven Mike Andy and Paul and we have seen that for 01:55.510 --> 01:59.910 and we have a any end value on Thursday. 01:59.930 --> 02:08.070 So let's start here with Monday and on Monday we have sets of 34 then on Tuesday 27 on Wednesday 15. 02:08.480 --> 02:13.640 And then again we have a comma and after the comma we have actually no value here at all. 02:13.640 --> 02:18.500 So no information so not even though a wide space on empty space. 02:18.500 --> 02:23.850 And then again we have a comma and then we have the value for Friday. 02:23.900 --> 02:29.300 So he and our CSB five and never we have no value or no information at all. 02:29.330 --> 02:33.590 Then we get a and they end value in our data frame. 02:33.620 --> 02:42.450 So now let's go back to our Jupiter notebook and we can also call the info method here so here we have 02:42.510 --> 02:49.760 the columns Monday Tuesday Wednesday Thursday and Friday and we can see here that in the first column 02:50.180 --> 02:53.010 we have only three nominal values. 02:53.150 --> 02:58.920 So obviously here we have a missing value and we can also select the missing value. 02:58.940 --> 03:06.740 So we you see at the loke operator and then we are going into the row with the row label Steven. 03:07.800 --> 03:11.220 And add to the column that the column labeled Thursday. 03:11.250 --> 03:12.480 So let's do this here 03:16.430 --> 03:18.290 and here we can see we get an A. 03:18.290 --> 03:19.730 And so not a number. 03:19.730 --> 03:23.520 And this is a strong sign that we have here a missing value. 03:23.870 --> 03:31.010 And we can also produce some more missing value so let's go back here to our CSB fire and for example 03:31.040 --> 03:35.240 let's delete here also the sales information for Friday 03:38.020 --> 03:46.540 so let's say here our CSP file and let us go back to our Jupiter note book and let's read in part here 03:46.570 --> 03:53.530 our fire sale starts CSP and now we can see here that we have two missing value. 03:53.540 --> 04:00.970 So I'm Thursday and on Friday and we can also produce missing values with Python. 04:01.040 --> 04:07.460 So for example let's assume that we want to f a missing value at the index position 1 1. 04:07.460 --> 04:12.850 So we take the row with the index position one here it's Mike and then the column at index position 04:12.850 --> 04:13.240 1. 04:13.250 --> 04:15.000 It's here to stay. 04:15.020 --> 04:19.200 So we want to replace here 9 with a missing value. 04:19.400 --> 04:23.360 And what we can actually do we can assign the keyword none. 04:23.540 --> 04:32.470 So let's do this here and let's again check our sales data frame and now we can see here that we have 04:32.530 --> 04:38.510 a n and or missing value also here for the Tuesday sets of Mike. 04:38.560 --> 04:45.610 However the preferred and more reliable way to produce missing values is by assigning NPD up name. 04:45.640 --> 04:49.700 So now we are selecting the element that next position to 2. 04:49.750 --> 04:56.740 So it's actually here 54 and we are assigning the value end picked up non sense actually a name PI for 04:56.740 --> 05:00.310 missing value or not a number. 05:00.310 --> 05:08.150 And let's again check our size that frame and now we have also here missing values for the events they 05:08.150 --> 05:09.650 say it's off Andy. 05:09.990 --> 05:12.660 And once again we can call the info method 05:16.040 --> 05:20.170 and now we can see that on Monday we have financial values. 05:20.180 --> 05:23.580 On Tuesday we have only three nominal values. 05:23.690 --> 05:28.060 And also on Wednesday Thursday and Friday we have only three and unknown values. 05:28.070 --> 05:34.820 So we have one missing value in each column here and the three a day that sometimes problematic too 05:34.830 --> 05:41.370 in identifying missing values because instead of having no value or no information at all there might 05:41.370 --> 05:46.500 be some other values that should indicate that there is no value. 05:46.500 --> 05:54.790 So let me demonstrate this and we again go to our CSB fire and we have seen here that on the first day 05:54.790 --> 06:01.630 we have no information or know where you at all for Stephen however might be the case that somebody 06:01.630 --> 06:08.410 instead of uh including you know a value or no information at all includes here a white space or an 06:08.410 --> 06:10.000 empty space. 06:10.000 --> 06:17.080 So let's do this here and let's say fear our sea as we fire and let's go back here to our Jupiter note 06:17.080 --> 06:27.110 book and let's read in part to our CSB fire. 06:27.160 --> 06:28.200 So now let's have a look. 06:28.200 --> 06:33.980 Yes first day and we can see here that we have here and our Panda's data frame no information or no 06:33.980 --> 06:42.250 value at all because he and our CSB fire is simply have a whitespace space or an empty space and also 06:42.250 --> 06:48.340 here we have a white space or an empty space and we can also select here the element with the local 06:48.360 --> 06:56.260 operator again and instead of getting a non bee simply getting a string object which only consists of 06:56.260 --> 06:58.950 a white space on empty space. 06:59.320 --> 07:08.140 And if you call now the info method then we can see here on Thursday that we do not see any missing 07:08.140 --> 07:08.560 value. 07:08.590 --> 07:14.510 So we have formed the Moral values because here we have simply a string object with the white space 07:15.220 --> 07:16.660 and with real data. 07:16.660 --> 07:20.950 This is a real problem because at a first glance we do not really see that. 07:21.100 --> 07:24.490 On Thursday here we have a missing value. 07:24.520 --> 07:24.900 All right. 07:24.910 --> 07:31.200 So now let's have another example the SO and working with real data. 07:31.200 --> 07:39.120 It is very often the case that people if there is no data people assign just uh a value like no data 07:39.180 --> 07:40.140 or missing data. 07:40.170 --> 07:44.390 So for example here for Paul on Thursday we have no value. 07:44.520 --> 07:46.160 And then Paul insert it here. 07:46.170 --> 07:46.950 No data 07:51.860 --> 07:55.980 so let's go back to our Jupiter notebook and that's very important. 07:59.950 --> 08:04.790 And now we can see here on Thursday for Paul that we have here the string no data. 08:05.020 --> 08:12.340 And if we call you at the info method and if we have a look here on the Thursday column we still do 08:12.340 --> 08:14.320 not identify any missing value. 08:14.330 --> 08:19.090 So still we have fun on no on none missing values even in reality. 08:19.090 --> 08:24.910 We have two missing value so we have here a missing value for Steven and we have a missing value for 08:24.910 --> 08:31.860 Paul and we can only detect these two missing values if we further analyze them. 08:31.870 --> 08:39.120 The column CSO for example is the first column with methods like and unique value accounts unique on 08:39.130 --> 08:40.350 so on. 08:40.360 --> 08:46.330 All right so this was the introduction on any values are missing values and in the next videos you learn 08:46.330 --> 08:50.190 how to identify and handle missing values and our two data sets. 08:50.260 --> 08:51.970 So up to see you there by.