WEBVTT 00:00.820 --> 00:06.820 In this video we learn how to convert time series from one frequency to another frequency with the free 00:06.830 --> 00:08.470 sample method. 00:08.470 --> 00:14.740 So when our temperature data set we have hourly data and in that time span of three years we are having 00:14.740 --> 00:20.670 a lot of timestamps and data points and we have also seen that high frequency data looks quite messy. 00:20.680 --> 00:26.470 And we tried to plot and visualize it and therefore I'd a normal workflow to aggregate data from higher 00:26.470 --> 00:28.630 frequency to lower frequency. 00:29.320 --> 00:35.650 So in our example we could convert our temperature data from our lead to a daily or monthly and converting 00:35.650 --> 00:40.810 time serious data from high frequency to low frequency is also called down sampling. 00:41.440 --> 00:46.690 All right so let's first of all import pandas and we are also importing met plot lip and we use of the 00:46.690 --> 00:47.930 Seabourn style. 00:48.490 --> 00:51.500 And then we also import our temperature data set. 00:51.520 --> 00:55.670 And of course we set a date time index. 00:55.960 --> 00:57.680 So let's have a look here. 00:57.850 --> 01:01.130 The first five rows and the info method. 01:01.300 --> 01:05.310 So give you f on the left hand side our daytime index. 01:05.540 --> 01:12.260 And as we can see here we have our lead temperature data and we can convert the hourly data flight sample 01:12.290 --> 01:14.020 to daily frequency. 01:14.420 --> 01:18.610 And we can do this service at the very power for the re sampling method. 01:18.770 --> 01:23.830 And actually the recycling method works in a quite similar way then the group by method. 01:23.930 --> 01:29.330 So first of all we apply the re sampling method on our temperature data frame and then let's have a 01:29.330 --> 01:31.250 look here inside the brackets. 01:31.250 --> 01:33.680 So here we have a pyramid I call it rule. 01:34.190 --> 01:42.290 So let's have a look here and here we have to pass the offset string or object representing target conversion. 01:42.290 --> 01:50.160 So here we can pass the frequencies like daily weekly annual or whatever so we can use our frequencies 01:50.270 --> 01:56.900 that we learned also in the last videos and TV on to transform or convert our time series to daily so 01:56.900 --> 01:58.430 we pass the idea. 01:58.850 --> 02:05.180 And if you now understand we are getting actually an object called daytime index a re sampler. 02:05.420 --> 02:12.350 So this is quite similar to a group by up checked and but there is sample method actually DOES IT groups 02:12.520 --> 02:16.640 or timestamps on our data frame in two respective groups. 02:16.700 --> 02:22.640 And in this case here in two daily groups so far each single day here in our data from the Reid sample 02:22.640 --> 02:30.940 method creates one group and the group contains all timestamps of the respective day and same s with 02:30.940 --> 02:38.140 a group by object that can also convert here the day time index every sample object into a list by using 02:38.140 --> 02:39.760 here the list function 02:52.730 --> 02:58.930 and here we are getting a list with one data frame for each day in our data frame so our data frame 02:58.930 --> 03:01.340 consists of three years of data. 03:01.510 --> 03:06.070 So approximately the amount of days is 1400. 03:06.340 --> 03:14.640 So actually here our lists would contain about 1400 data frames so let's check this also with the length. 03:14.630 --> 03:15.270 Function 03:21.490 --> 03:25.690 so here we have one thousand four hundred sixty one elements on our list. 03:26.020 --> 03:30.620 And uh let's have a look at the one specific element here. 03:30.640 --> 03:38.920 So for example for the very first element so here we can see the very first element in our list and 03:38.950 --> 03:42.730 each element is actually a top layer consisting of two elements. 03:42.730 --> 03:47.200 So the first eleven this year the information about the day. 03:47.200 --> 03:55.000 So here we have the first of January 2000 and 13 and the second element is actually a data frame and 03:55.000 --> 03:58.660 this data frame contains all timestamps for their respective days. 03:58.660 --> 04:03.320 So in our example the first of January 2013. 04:03.370 --> 04:09.320 So in total we have here in this data frame twenty four timestamps for 24 hours. 04:09.430 --> 04:19.190 So the very last timestamp is 11:00 p.m. So this is our very first element in our list and this is a 04:19.190 --> 04:23.830 topic that we can also select actually the data frame and the top US. 04:23.840 --> 04:34.190 So this is the second element here at an exposition one and TBA actually the data frame for the first 04:34.190 --> 04:38.510 of January two thousand and 13 with twenty four timestamps 04:44.600 --> 04:51.300 and you can also slice our list for example for the second group so far the second day. 04:52.010 --> 04:55.390 So yeah it's the 2nd of January 2013. 04:55.400 --> 05:06.490 Also give us the 24 timestamps and actually we can do this for 1400 something days so this was actually 05:06.490 --> 05:10.380 the first part or the first step of down sampling data. 05:10.720 --> 05:15.680 So we are grouping our data by specific rules so for example daily. 05:16.180 --> 05:20.740 And now actually in a second step We have to define how to aggregate the data. 05:21.070 --> 05:28.000 So for each day we have 24 timestamps and 24 data points and we want to convert these 24 data points 05:28.000 --> 05:34.800 into a one data point so folks and we can take the very first data point at midnight or the last data 05:34.830 --> 05:42.420 point at 11:00 in the evening while we can take the mean or the median or whatever and we can aggregate 05:42.450 --> 05:49.470 our data quite easily by changing an aggregation method or function to the recent low object. 05:49.500 --> 05:55.470 So here we are creating a reasonable object and then we have to apply and that creation function or 05:55.470 --> 05:56.860 method on it. 05:57.030 --> 06:01.190 And here in this example we want to have for each day the very first times. 06:01.320 --> 06:03.270 At midnight so we can do this here 06:07.230 --> 06:13.830 and we can see now our risk armpit or down sample data frame and on the left hand side we can see that 06:13.890 --> 06:18.370 now we have a daily frequency starting from January the 1st. 06:18.390 --> 06:24.690 Then we have January the 2nd and so on and the temperature data here for each day is actually the very 06:24.690 --> 06:31.140 first timestamp at midnight because we apply to the first method. 06:31.140 --> 06:34.800 So let's again have a look at our original temperature data frame 06:37.540 --> 06:47.010 and maybe let's have a look at the first 25 rows and we can see on January the 1st to the very first 06:47.010 --> 06:54.510 time stamp at midnight that we have eleven point seven degrees and a minus one point one in New York 06:55.350 --> 07:01.230 and exactly this information or this data we have here now and our reason but data frame. 07:01.230 --> 07:07.720 So the very first time stamp on January the first year at midnight and actually here we have many options 07:07.720 --> 07:13.480 to aggregate so we can also select here the very last timestamp of each day. 07:15.250 --> 07:22.810 So let's have a look at the very last time stamp here we have 11:00 p.m. And here we have four 14 degrees 07:22.940 --> 07:26.360 in a and two point seven in New York. 07:26.470 --> 07:35.690 And here we have it and also a very common aggregation method all functions of the mean. 07:35.770 --> 07:41.250 So you're paired us calculates that the mean over our twenty four time stamps in a day. 07:43.210 --> 07:46.400 Oh we could also calculate the median 07:49.330 --> 07:50.030 and in. 07:50.030 --> 07:55.240 We could also take here the sum of all or 24 timestamps for values. 07:57.700 --> 08:03.730 So in this case here with our temperature data this does not really make sense but in other cases this 08:03.730 --> 08:05.270 might be helpful. 08:05.290 --> 08:09.300 All right. 08:09.370 --> 08:16.020 So now let's go through some other examples of some different frequencies and you can also convert our 08:16.050 --> 08:24.130 hourly data into increments of two hours and we can take the very first time stamp of each increment 08:24.130 --> 08:25.260 here. 08:25.270 --> 08:32.130 So here we have the period from midnight to 2:00 and you take the values at midnight then give you have 08:32.140 --> 08:39.760 the period from 2:00 to 4:00 give us the values at 2:00 and so on and of course we can also transform 08:39.760 --> 08:47.680 our data to a weekly data by passing here W and we are taking get the mean value over all values 08:52.940 --> 08:59.630 and by default we can see here on the left hand side that the last day of each respect the Feig and 08:59.690 --> 09:02.220 the last day is a Sunday. 09:02.270 --> 09:06.860 So we have here the week until the 6th of January it's a Sunday. 09:07.310 --> 09:11.330 And the average temperature in L.A. was ten point two one. 09:11.600 --> 09:15.970 And then we have the week until the 13th of January. 09:16.010 --> 09:19.050 And the average temperature was ten point four nine. 09:19.070 --> 09:25.410 And so on and the last day of a particular week is here by default always Sunday. 09:25.680 --> 09:32.960 But we can also customize this and we can say that we want to have a week from first territory Wednesday 09:33.270 --> 09:35.690 and therefore we can pass yet w Wednesdays. 09:35.720 --> 09:38.420 So let's try this out here. 09:38.460 --> 09:42.220 So now we can see here on the left hand side to the end of each week. 09:42.450 --> 09:47.910 And I'll be at the end of each week as the Wednesday because now we have grouped our data to a weekly 09:47.910 --> 09:51.620 date with Greeks from Thursday to Wednesday. 09:51.630 --> 09:54.550 So here the second to say Wednesday and also the 9th. 09:54.570 --> 09:57.090 So Wednesday and so on. 09:57.090 --> 10:02.390 So let's move on to a monthly frequency and therefore we pass here hmm 10:05.300 --> 10:08.140 and by default we can see here the month end. 10:08.180 --> 10:16.460 So for example here the 30 first of January and the twenty eight of February and the value ECI actually 10:16.460 --> 10:21.050 the aggregated mean values for the respective month. 10:21.050 --> 10:26.440 So here we have the mean temperature so for January for February and so on. 10:26.630 --> 10:33.260 And instead of having here the very last day of the month of the label you can also change this to the 10:33.320 --> 10:35.980 very first day by passing year. 10:36.050 --> 10:37.220 As for start 10:42.310 --> 10:45.340 and actually the value you see in our data frame do not change. 10:45.550 --> 10:52.630 So these are the aggregated and mean data for January and also here we have the same data. 10:52.630 --> 10:56.740 The only thing that is different this year on the left hand side to the labels. 10:57.130 --> 11:00.390 So here we have as the label it's the very last day. 11:00.400 --> 11:01.870 And yes the very first day. 11:01.870 --> 11:08.710 So first of January first of temporary and so on and we can actually further customize them the label. 11:08.950 --> 11:11.810 And they said the perimeter offset. 11:11.890 --> 11:13.270 So let's have a look here 11:18.220 --> 11:25.390 and here with our offset we can actually adjust the reasonable time labels and let's assume that for 11:25.390 --> 11:33.500 each month we want to f the 15 s uh the label here then we can positive for example 14 days. 11:33.520 --> 11:40.390 So we are starting from the very first day off a month and we are adding 14 days and then we are ending 11:40.390 --> 11:42.050 up at the 50s. 11:42.070 --> 11:43.930 So let's try this out here 11:48.400 --> 11:52.180 and still here the value is in our data frame are the same as here above. 11:52.180 --> 11:53.980 So this does not change. 11:54.100 --> 11:57.490 The only thing that does change actually here the label is. 11:57.520 --> 12:06.110 So here we had the very first day of each month you we had the very last day and here we have the 15th. 12:06.190 --> 12:10.690 All right so now let's go also to quarterly data and we can pass a Q 12:14.670 --> 12:15.860 and only by default. 12:15.870 --> 12:18.300 We are getting the very last day of a quarter. 12:18.300 --> 12:25.770 So for example here the thirty first of March then the 30th of June then the last day of September and 12:25.770 --> 12:28.200 the last day of December. 12:28.920 --> 12:37.560 So this is the default setting and having quarters from January to March April 2 to in July to September 12:37.590 --> 12:39.260 and October to December. 12:39.690 --> 12:47.250 But we can also change this and we can define for example that the very last month of a quarter should 12:47.250 --> 12:53.730 be February. 12:53.800 --> 13:00.700 So now you'll be half the quarter and the twenty eighth of February then we have the quarter and thirty 13:00.700 --> 13:01.990 first of May. 13:01.990 --> 13:03.390 And so on. 13:03.430 --> 13:10.810 So essentially we have four quarters from March to May and then from June to August and so on. 13:11.140 --> 13:20.080 So this is we're flexible actually and we can also respond to our data frame to a yearly frequency by 13:20.080 --> 13:22.960 passing year y or alternatively a 13:27.900 --> 13:33.630 and by default we have fear in the index the very last day of the period. 13:33.630 --> 13:40.410 So for example here we have the aggregated data for the year 2013 and we have here the very last day. 13:40.410 --> 13:48.360 So the thirty first of December but we can also customize see of the labor by passing y s fourth stop 13:48.360 --> 13:48.780 date 13:53.220 --> 13:58.770 and here we have actually the same values again in our data frame but we simply changed t on the left 13:58.770 --> 14:04.790 hand side of the labels from the very last day of the year to the very first day of the year. 14:04.830 --> 14:10.280 So these were some examples how to read sample a data frame and there are even more options. 14:10.290 --> 14:16.950 And feel free to have a look at the documentation to see the full options but uh for the time being 14:16.950 --> 14:17.720 we are not finished. 14:17.730 --> 14:20.340 And I hope to see you also in the next video by.