Time-series forecasting is one of the most widely dealt with machine learning problems ever. Time series forecasting finds crucial applications in various fields including signal communication, climate, space science, healthcare, financial and marketing industries. Deep learning models outshine in time series analysis nowadays with great performance in various public datasets.
The key idea of deep learning models is to learn the inter-sample relationships to predict the future. However, intra-temporal relationships among different features within a sample are hardly dealt with. Extracting intra-temporal relationships yields better pattern understanding in the time series representation learning even with fewer data samples. Without an intra-temporal relationship extraction system, a deep learning model becomes data-hungry and requires a lot of manually-annotated tabular data for supervised training.
Present self-supervised representation learning approaches let models train with unannotated as well as fewer data. Self-supervised learning approaches yield huge success in different computer vision tasks including image inpainting, video inpainting, object tracking, rotation prediction and pace prediction. However, these approaches fail to capture the complete temporal-relationships in the case of high-dimensional time series structured data.
Haoyi Fan and Fengbin Zhang of Harbin University of Science and Technology, China and Yue Gao of Tsinghua University, China introduced SelfTime, the Self-supervised Time Series Representation Framework. This framework explores inter-sample relationships among subsequent samples and intra-temporal relationships within each sample to capture the underlying spatial and temporal patterns in an unannotated structured time series dataset.
How does SelfTime work?
A sample or a piece of a sample taken for exploring the pattern is called an anchor sample or an anchor piece respectively. In the case of inter-sample relation reasoning, a sample is taken as an anchor and is transformed using the standard augmentation method to develop a new transformed-time-series sample that is denoted the positive sample. Another real sample is taken from some nearer time scale and is denoted the negative sample. Inter-sample relation reasoning is captured between these positive and negative samples.
On the other hand, a sample of interest is broken into pieces to learn intra-temporal relationships. One of the broken pieces is taken as an anchor and the rest are called references. Intra-temporal relation reasoning is performed between the anchor and each of the references individually. The number of references for each anchor can be varied based on the problem. If three references are chosen for analysis, it is termed 3-scale temporal relations and the references are called short-term, middle-term and long-term relations based on the temporal distance between the anchor and references.
A shared representation learning backbone is developed with two relation-reasoning-heads based on the inter-sample relations and the intra-sample relations extracted separately. Finally, the time series representations are extracted from unlabeled data by supervising the two different relation-reasoning-heads. Since SelfTime explores in-depth relationships among different data samples and different time pieces, it exhibits extraordinary performance in representation learning of time series data, especially high-dimensional data.
Python implementation of SelfTime
The desired requirements for SelfTime architecture building are Python 3.6 or 3.7, Pytorch 1.4.0 and CUDA GPU runtime. The following command downloads the source code to the local machine.
!git clone https://github.com/haoyfan/SelfTime.git
Output:
Verify proper download by exploring the SelfTime directory using the command,
!ls SelfTime
Output:
Change directory to run the containing files using the following line-magic command
%cd SelfTime/
The following command pre-trains the inter-sample relation head-on in-built CricketX dataset
!python train_ssl.py --dataset_name CricketX --model_name InterSample
The following command pre-trains the intra-temporal relation head-on in-built CricketX dataset
!python train_ssl.py --dataset_name CricketX --model_name IntraTemporal
The following command pre-trains SelfTime model on in-built CricketX dataset
!python train_ssl.py --dataset_name CricketX --model_name SelfTime
Output:
Once trained the framework parts separately, the three parts of the framework can be evaluated on the test data. The following commands evaluate the three networks on the in-built CricketX data’s test set.
!python test_linear.py --dataset_name CricketX --model_name InterSample !python test_linear.py --dataset_name CricketX --model_name IntraTemporal !python test_linear.py --dataset_name CricketX --model_name SelfTime
Finally, supervised training of the whole framework based on the pre-trained individual networks can be performed using the following command. This training is governed by early stopping based on on-time evaluation performance.
!python train_test_supervised.py --dataset_name CricketX --model_name SupCE
A portion of the output:
The results of the final supervised training are stored in a new directory called ‘results’. Model performance can be explored by visualizing the losses and other metrics over iterations.
Performance of SelfTime
SelfTime has been evaluated and compared with other well-acclaimed models using public time-series datasets such as CricketX, UWaveGestureLibraryAll (UGLA), DodgerLoopDay (DLD), and InsectWingbeatSound (IWS) from the UCR Time Series Archive, and the bearing datasets XJTU2 and MFPT.
Transformed data samples are generated by augmenting the magnitude domain and the time domain such as jittering, scaling, cutout, magnitude warping, time warping, window warping and window slicing. These transformed samples are used as positive samples to extract inter-sample relationships. All the recent state-of-the-art models have undergone identical augmentations to justify comparisons.
SelfTime massively outperforms recent state-of-the-arts in time-series domain namely, Triplet Loss model (2019), Deep InfoMax (2019), Forecast (2020), Transformation (2020), SimCLR (2020) and Relation (2020).
