WEBVTT 00:01.230 --> 00:06.030 Welcome to the first session on emerging joining and concatenate data frames. 00:06.130 --> 00:11.750 And in this video we will add additional Rose observations to a data frame on other matters. 00:11.740 --> 00:14.760 He will combine two data frames vertically. 00:15.220 --> 00:18.940 So this sounds complicated but let's have a look at an example. 00:18.940 --> 00:23.440 And first of all we are importing pandas. 00:23.440 --> 00:25.300 And then we have an example. 00:25.870 --> 00:27.520 So we have two data frames. 00:27.520 --> 00:36.370 We have the data frame 2008 where we have our columns where the athlete Michael Phelps won a medal in 00:36.370 --> 00:39.380 the 2008 edition in Beijing. 00:39.910 --> 00:49.360 So in total we are we have eight gold medals and we have a second data frame 2012 with all medals from 00:49.360 --> 00:53.200 Michael Phelps and the addition 2012 from London. 00:53.200 --> 00:59.050 So here we have four gold medals and two silver medals and a quite common operation is now to actually 00:59.050 --> 01:05.110 combine those two data frames and actually add the rose after the two data frames. 01:05.260 --> 01:12.310 So to combine those two data frames vertically and by doing so we are getting a combined data frame 01:12.340 --> 01:18.170 with all rows of the first data frame and all rows of the second data frame. 01:18.190 --> 01:20.340 So this is a quite easy operation. 01:20.350 --> 01:25.600 And also on pandas if you know how to do it that's quite straightforward. 01:25.600 --> 01:28.720 So now for our coding example we have two data frames. 01:28.780 --> 01:36.880 So we have the data frame man 2004 and the data frame man two thousand eight and let's import both and 01:37.060 --> 01:40.250 let's have a look here. 01:40.470 --> 01:49.650 So here we have all male athletes that won in the addition 2004 at least one medal in a swimming event. 01:49.680 --> 01:56.430 So we have here the first column with the athlete names and here the second column with the total number 01:56.430 --> 02:04.760 of medals and the addition 2004 and obviously here the data frame is sorted by the medals column from 02:04.760 --> 02:05.690 high to low. 02:06.290 --> 02:09.020 So no surprise here from the top Michael Phelps 02:12.600 --> 02:17.310 and in total we have four year fifty nine rows and fifty nine athletes. 02:17.310 --> 02:20.540 So starting from the next level is zero. 02:20.910 --> 02:26.540 And then we have a second data frame called men two thousand and eight. 02:26.550 --> 02:30.330 And that's also impart this one here. 02:30.330 --> 02:37.560 And no surprise that these are the male athletes and the addition two eight in Beijing that at least 02:37.560 --> 02:40.490 won one medal in a swimming event. 02:40.650 --> 02:48.240 And also here the data frame assaulted by the medals column and we have Michael Phelps and in total 02:48.240 --> 02:53.120 we have sixty two athletes are sixty two rows. 02:53.140 --> 02:58.780 And let's assume that actually you want to combine both data frames and actually want to add all rows 02:58.800 --> 03:08.350 of the 2008 Dara frame to the rows of the 2004 data frame and we can do this with the pen method. 03:08.370 --> 03:13.720 So here we have our data from 2004 and then we are using the apparent method and let's have a look here 03:13.720 --> 03:22.040 with the shift tab so the appellant method appends pencil rows of other to the end of this frame. 03:22.040 --> 03:30.100 So this rain miss the man 2004 frame and the other frame is here the man 2008 frame. 03:30.200 --> 03:34.830 So we have here the power meter other very passive man two thousand eight. 03:34.940 --> 03:42.050 And by definition all rows of the 2004 frame are at the top of the new data frame and all rows of the 03:42.080 --> 03:47.980 two thousand and eight frame are at the end of the new data frame. 03:48.050 --> 03:54.450 So actually by using the append method we are returning a new object on new data frame. 03:54.620 --> 04:00.440 So let's try this out and we also have here apparently to ignore index and by default it's set to form. 04:00.450 --> 04:02.960 So we have a look here later. 04:03.310 --> 04:11.610 And here we can see on the top all the rows of the 2004 data frame and here at the end we have the rows 04:11.610 --> 04:14.750 of the two thousand and eight data frame. 04:14.910 --> 04:21.510 And in total we have one on the 21 athletes all rows and we can see here on the left hand side that 04:21.510 --> 04:25.680 we still have the original index labels from the original data frames. 04:25.680 --> 04:28.970 So for example here we have Lin. 04:29.430 --> 04:38.160 This that the index labeled 61 and also in the original 2008 data frame you had the 61 as index later. 04:38.430 --> 04:44.430 However we can change this behaviour by setting the ignore index parameter to true 04:49.120 --> 04:56.720 and by doing so the original index labels are obviously ignored and penance creates a brand new Range 04:56.770 --> 05:00.120 Index from zero to one hundred and twenty. 05:00.160 --> 05:01.300 So let's have a look here. 05:02.230 --> 05:12.960 So starting here from zero to one hundred and twenty. 05:13.090 --> 05:16.990 So now we have successfully combined both two data friends vertically. 05:17.260 --> 05:24.460 However one pitfall this year that it's quite hard for each row to identify that the row is originally 05:24.460 --> 05:25.930 coming from. 05:25.930 --> 05:31.450 So if we again want to split here the combined data frame and the original data frames that could be 05:31.450 --> 05:33.300 quite hard actually. 05:33.490 --> 05:37.730 And that comes actually at the conquered method and to play. 05:37.750 --> 05:40.300 So this is actually the second alternative. 05:40.300 --> 05:43.790 To combine two data frames so vertically. 05:43.930 --> 05:46.270 So with the PD dot com cap method. 05:46.270 --> 05:49.000 So this is a direct pan mass method. 05:49.630 --> 05:56.840 So let's check here and the concrete method concatenate pen as objects along a particular axis. 05:56.860 --> 06:04.720 So here we can define whether we want to combine the two data frames vertically or horizontally and 06:04.720 --> 06:06.460 we can define an axis. 06:06.460 --> 06:07.960 So let's check this. 06:07.990 --> 06:15.670 So if you pass through the Axis parameter 0 or index we combine both two data frames along the index 06:15.670 --> 06:17.190 or vertically. 06:17.200 --> 06:23.290 And if you pass one our columns you actually combine the two data frames horizontally. 06:23.290 --> 06:32.080 But actually the default setting is zero so vertically and that's obviously the objects power meter 06:32.450 --> 06:38.970 a. have to pass a list with the data frames that we want to concatenate. 06:39.080 --> 06:43.400 And in this case it's man 2004 and man 2008. 06:43.880 --> 06:48.210 But we could also concatenate here more data frames than just two. 06:48.440 --> 06:50.840 So five 5 10 or 100. 06:50.870 --> 06:55.220 And also here we have to pair me to ignore in next and the default setting is false. 06:55.220 --> 07:02.530 So let's try this out here and again here we are starting with the 2004 rows. 07:02.630 --> 07:10.490 So by definition the most left in data frame that we pass it to the objects pair meter is on the top 07:12.520 --> 07:19.340 and then we are having the two thousand and eight rows and we have again 121 rows. 07:19.630 --> 07:25.210 And again here we can see on the left hand side that that we have for the original index this 07:28.060 --> 07:31.170 and bypassing true to the ignore index pair meet. 07:31.180 --> 07:37.600 We can change this and we can create a brand new Range Index. 07:37.600 --> 07:42.010 So this is the same as service with the append method. 07:42.030 --> 07:45.530 So now we have fair Range Index from zero to one hundred and twenty 07:48.440 --> 07:55.460 and before this the append method we had the problem that exit was quite hard to identify for each row 07:55.670 --> 07:56.200 from it. 07:56.210 --> 07:57.410 So it's a data frame. 07:57.410 --> 08:02.450 The row was coming from and you have this is the PD dot com cat method. 08:02.450 --> 08:03.700 We can change this. 08:03.710 --> 08:06.430 And here we have the apparently keys. 08:06.830 --> 08:10.910 And we can actually create a mighty index C on the left hand side. 08:11.060 --> 08:13.850 And actually we are adding an outer and next level. 08:14.300 --> 08:20.470 We can define index labels for our 2004 rows and for our 2008 rows. 08:20.540 --> 08:28.130 So here we are having to date our friends and for the keys barometer we can pass a list with two labels 08:28.280 --> 08:34.340 that we want to f for the rows of 2004 and for the rows of 2008. 08:34.340 --> 08:44.010 So let's create here a list and apparently you want to have the labels 2004 and 2008. 08:44.020 --> 08:45.490 So let's run this out. 08:45.640 --> 08:52.760 But before it sets here the ignore index to forwards again so we don't need this here and now you may 08:52.770 --> 08:55.690 have on the left hand side our multi index. 08:55.690 --> 09:04.660 And as of the order index level we are having near 2004 and 2008 and we can see all rows of firm 2004 09:06.210 --> 09:16.340 and we are starting with our rows of 2008 and a total of course we have 121 rows so we have actually 09:16.340 --> 09:21.970 two alternatives to add rows to a combined two data frames vertically. 09:22.130 --> 09:29.330 So we have the append method and the PD up concrete method and actually two differences which makes 09:29.330 --> 09:32.680 actually the com cat method more powerful. 09:33.260 --> 09:39.680 So first of all we can define the key is where after we have combined the data frames that can still 09:39.680 --> 09:43.190 identify where each row was coming from. 09:43.190 --> 09:50.440 So from that data frame and that's actually a second advantage of the con cap method. 09:50.440 --> 09:57.370 So if you want to concatenate many data frames for example 10 50 or 100 so let's assume we have a data 09:57.400 --> 10:00.970 frame for each citizen from eighteen hundred to 96. 10:01.000 --> 10:07.630 So I think in total it's a lot funnier than the concrete method this and why appropriate service to 10:07.630 --> 10:09.070 the concrete method. 10:09.070 --> 10:15.130 Here we can pass a list with all data frames and then the conquered method automatically appends all 10:15.130 --> 10:25.110 data frames and with the pen method here we can also pass a list of more than one data frame so many 10:25.110 --> 10:26.400 data frames. 10:26.400 --> 10:29.370 But first before we have to select the one data frame. 10:29.370 --> 10:38.100 So here for example the 2048 frame and this data frame we have to apply the append method and then we 10:38.100 --> 10:42.480 have to pass a list of all other data frames that you want to combine. 10:42.480 --> 10:49.440 But excluding the very first data frame because we already have here the very first data frame and this 10:49.440 --> 10:54.500 is a bit error sensitive and you wouldn't do this if you have many data frames. 10:55.140 --> 11:02.570 So let's go back here to our com CAD to methods and there's actually a third advantage. 11:02.570 --> 11:09.830 So with the Concorde method we not only can combine data frames vertically but also horizontally in 11:09.860 --> 11:12.790 with the perimeter axis. 11:13.040 --> 11:14.210 So here. 11:14.210 --> 11:20.470 So as a summary the Concorde method is actually the more general method with more functionality. 11:21.110 --> 11:25.550 And actually I always use only the Concorde method. 11:25.550 --> 11:30.110 All right and that's actually one another perimeter. 11:30.140 --> 11:37.070 So before we create it here the mighty index service the outer index level where we have here for each 11:37.070 --> 11:38.680 row the addition. 11:38.690 --> 11:48.000 So 2004 or 2008 and we can also give you a name to the levels so we can give you names for the auto 11:48.000 --> 11:50.310 index level and the next level. 11:51.020 --> 11:53.840 And that's the parameter names. 11:53.840 --> 12:00.740 So here we have the parameter names and next year we want to name the outer index. 12:00.780 --> 12:01.280 Yeah. 12:01.310 --> 12:07.910 So we can do this by passing a year within a list. 12:07.910 --> 12:15.380 So let's run the cell and all we have here the name for our outer index the year and once we are happy 12:15.380 --> 12:22.010 here with our concatenated data frame we can also save our data frame for example in the variable man 12:22.040 --> 12:23.420 0 4 0 8 12:29.830 --> 12:37.920 and of course we can re inspect our safe data frame and actually it might be the case that we are not 12:37.920 --> 12:41.280 really happy here with the commodity index. 12:41.280 --> 12:51.240 So we can also reset the index with the reset index method so here we have again our data frame and 12:51.240 --> 12:58.950 with the reset index method we can transform our mighty index into columns and create a brand new Range 12:58.980 --> 13:06.950 Index so let's do this here. 13:06.960 --> 13:12.450 So now here we have the columns here and the former level one mighty index. 13:12.450 --> 13:18.930 And here we have our new Range Index and then we can also drop here the column level 1 13:22.080 --> 13:24.970 and this could now be our final data frame. 13:25.530 --> 13:33.660 So we have here on athletes are rows from the 2004 edition and all matters and then here in the end 13:33.690 --> 13:40.760 we have all rows from the two thousand and eight of this and or 2008 data frame. 13:41.880 --> 13:48.460 And in total one hundred twenty one are right we are finished now with this video and I hope to see 13:48.460 --> 13:50.520 you also in the next one by.