path: root/models/failure_prediction/jnotebooks/Attention_LSTM.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "l43uL109ZssB"
   },
   "source": [
    "Contributors: **Rohit Singh Rathaur, Girish L.** \n",
    "\n",
    "Copyright [2021](2021) [*Rohit Singh Rathaur, BIT Mesra and Girish L., CIT GUBBI, Karnataka*]\n",
    "\n",
    "Licensed under the Apache License, Version 2.0 (the \"License\");\n",
    "you may not use this file except in compliance with the License.\n",
    "You may obtain a copy of the License at\n",
    "\n",
    "    http://www.apache.org/licenses/LICENSE-2.0\n",
    "\n",
    "Unless required by applicable law or agreed to in writing, software\n",
    "distributed under the License is distributed on an \"AS IS\" BASIS,\n",
    "WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n",
    "See the License for the specific language governing permissions and\n",
    "limitations under the License."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "YQ6lT1e2hrx4",
    "outputId": "14ee2b60-05d7-4163-d420-58970bc4e4b0"
   },
   "outputs": [],
   "source": [
    "from google.colab import drive\n",
    "drive.mount('/content/drive')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "tLhroy5BnMnC"
   },
   "outputs": [],
   "source": [
    "# Importing libraries\n",
    "import tensorflow as tf\n",
    "import matplotlib.pyplot as plt\n",
    "import matplotlib as mpl\n",
    "import pandas as pd\n",
    "import numpy as np\n",
    "import os"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 521
    },
    "id": "2-UpMVsSnfCI",
    "outputId": "87e645f2-f2f9-4e8e-e0b9-fd807224892a"
   },
   "outputs": [],
   "source": [
    "df_Ellis  = pd.read_csv(\"/content/drive/MyDrive/LFN Anuket/Analysis/data/Final/Ellis_FinalTwoConditionwithOR.csv\")\n",
    "df_Ellis"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 293
    },
    "id": "92xBt43BnjAo",
    "outputId": "747e6ee9-2da1-43af-c751-1261f54d6c2c"
   },
   "outputs": [],
   "source": [
    "df_Ellis.plot()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "46U0FXYB6AbD"
   },
   "source": [
    "We showed here the histograms of Ellis data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 879
    },
    "id": "RSo-aa-SIoBR",
    "outputId": "eeead93b-89b4-4798-976c-1b065bbbc4a3"
   },
   "outputs": [],
   "source": [
    "# we show here the hist\n",
    "df_Ellis.hist(bins=100,figsize=(20,15))\n",
    "#save_fig(\"attribute_histogram_plots\")\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 634
    },
    "id": "gggaMJ_2LtFs",
    "outputId": "c3ac6c2e-c7f3-48ae-c562-23bb7b60c5b4"
   },
   "outputs": [],
   "source": [
    "cpu_system_perc = df_Ellis[['ellis-cpu.system_perc']] \n",
    "cpu_system_perc.rolling(12).mean().plot(figsize=(20,10), linewidth=5, fontsize=20) \n",
    "plt.xlabel('Timestamp', fontsize=30);"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 634
    },
    "id": "R_ctvXcQL1Xf",
    "outputId": "d553c7b1-3a00-40dd-ad5f-cc695d3ecbae"
   },
   "outputs": [],
   "source": [
    "load_avg_1_min = df_Ellis[['ellis-load.avg_1_min']] \n",
    "load_avg_1_min.rolling(12).mean().plot(figsize=(20,10), linewidth=5, fontsize=20) \n",
    "plt.xlabel('Timestamp', fontsize=30);"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "9jTOMzOqEQEZ"
   },
   "source": [
    "## Identifying trends in Time Series data\n",
    "There are several ways to think about identifying trends in time series. One popular way is by taking a rolling average, which means that, for each time point, we take the average of the points on either side of it. Note that the number of points is specified by a window size, which we need to choose.\n",
    "\n",
    "What happens then because we take the average is it tends to smooth out noise and seasonality. We will see that below right now. Check out this rolling average of `'ellis-cpu.wait_perc'` using the built-in `pandas` methods.\n",
    "\n",
    "When it comes to determining the window size, here, it makes sense to first try out one of twelve months, as we're talking about yearly seasonality.\n",
    "\n",
    "Note that in the code chunk above we used two sets of squared brackets to extract the `'ellis-cpu.wait_perc'` column as a DataFrame; If we would have used one set, like `df_Ellis['ellis-cpu.wait_perc']`, we would have created a pandas Series.\n",
    "\n",
    "In the code chunk above, you also chained methods: you called methods on an object one after another. Method chaining is pretty popular and pandas is one of the packages that really allows you to use that style of programming to the max!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 634
    },
    "id": "Gkd5ecCmL6Bw",
    "outputId": "c8f8cd9a-8477-46b9-e350-9b501f8bf55b"
   },
   "outputs": [],
   "source": [
    "cpu_wait_perc = df_Ellis[['ellis-cpu.wait_perc']] \n",
    "cpu_wait_perc.rolling(12).mean().plot(figsize=(20,10), linewidth=5, fontsize=20) \n",
    "plt.xlabel('Year', fontsize=30);"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "JKNmf5TeFvQt"
   },
   "source": [
    "We have successfully removed the seasonality and we saw an upward trend for `ellis-cpu.wait_perc`! But how do these two search terms compare?\n",
    "\n",
    "We can figure this out by plotting the trends of `'ellis-cpu.wait_perc'`, `cpu_system_perc` and `'load_avg_1_min'` on a single figure:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 624
    },
    "id": "EycZrQU0MBSX",
    "outputId": "02683682-38ad-4745-c7bb-99a9058bb755"
   },
   "outputs": [],
   "source": [
    "df_dg = pd.concat([cpu_system_perc.rolling(12).mean(), load_avg_1_min.rolling(12).mean(),cpu_wait_perc.rolling(12).mean()], axis=1) \n",
    "df_dg.plot(figsize=(20,10), linewidth=5, fontsize=20) \n",
    "plt.xlabel('Year', fontsize=20); "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "PTkbQPa_6MNb"
   },
   "source": [
    "We established the correlation matrix for Ellis data.\n",
    "Seaborn has five built-in themes to style its plots: `darkgrid`, `whitegrid`, `dark`, `white`, and `ticks`. Seaborn defaults to using the darkgrid theme for its plots, but we can change this styling to better suit our presentation needs.\n",
    "\n",
    "To use any of the preset themes pass the name of it to `sns.set_style()`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 710
    },
    "id": "Pi8UMMitMa3Q",
    "outputId": "6e2c3c3c-5a69-42ef-87a8-7dfdd65e7ceb"
   },
   "outputs": [],
   "source": [
    "# we establish the corrmartrice\n",
    "import seaborn as sns\n",
    "color = sns.color_palette()\n",
    "sns.set_style('darkgrid')\n",
    "\n",
    "correaltionMatrice = df_Ellis.corr()\n",
    "f, ax = plt.subplots(figsize=(20, 10))\n",
    "sns.heatmap(correaltionMatrice, cbar=True, vmin=0, vmax=1, square=True, annot=True);\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "tdg8lD0OHEtH"
   },
   "source": [
    "Correlation between rows or columns of two DataFrame objectsCompute pairwise"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "rkYwyKtXMvpy",
    "outputId": "ab6efc64-5329-4ad2-e5a8-acecf7c79eff"
   },
   "outputs": [],
   "source": [
    "df_Ellis.corrwith(df_Ellis['ellis-load.avg_1_min'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 235
    },
    "id": "5oQK-ddinvCM",
    "outputId": "3e1fc354-aefb-496b-9e08-d05a458f9c23"
   },
   "outputs": [],
   "source": [
    "## ## using multivariate feature \n",
    "\n",
    "features_3 = ['ellis-cpu.wait_perc', 'ellis-load.avg_1_min', 'ellis-net.in_bytes_sec', 'Label']\n",
    "\n",
    "features = df_Ellis[features_3]\n",
    "features.index = df_Ellis['Timestamp']\n",
    "features.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 386
    },
    "id": "qbqn755fo81g",
    "outputId": "a8c2999d-5fd1-4de3-f8fd-05cd6ff14622"
   },
   "outputs": [],
   "source": [
    "features.plot(subplots=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "jJQD1x9psWCH"
   },
   "outputs": [],
   "source": [
    "features = features.values"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "Q23ry_WpH6tt"
   },
   "source": [
    "train test split for simple time series moving window average"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "xf8WCiykpUzN",
    "outputId": "6375471f-a7ec-4416-95e0-4f9a82872168"
   },
   "outputs": [],
   "source": [
    "### standardize data\n",
    "train_split = 141600\n",
    "tf.random.set_seed(13)\n",
    "\n",
    "### standardize data\n",
    "features_mean = features[:train_split].mean()\n",
    "features_std = features[:train_split].std()\n",
    "features  = (features - features_mean)/ features_std\n",
    "\n",
    "print(type(features))\n",
    "print(features.shape)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "22Y4ztlZIpFG"
   },
   "source": [
    "Defined a `multivariate_data` function to generate multivariate data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "1a0hNDmppnLB"
   },
   "outputs": [],
   "source": [
    "### create mutlivariate data\n",
    "\n",
    "def mutlivariate_data(features , target , start_idx , end_idx , history_size , target_size,\n",
    "                      step ,  single_step = False):\n",
    "  data = []\n",
    "  labels = []\n",
    "  start_idx = start_idx + history_size\n",
    "  if end_idx is None:\n",
    "    end_idx = len(features)- target_size\n",
    "  for i in range(start_idx , end_idx ):\n",
    "    idxs = range(i-history_size, i, step) ### using step\n",
    "    data.append(features[idxs])\n",
    "    if single_step:\n",
    "      labels.append(target[i+target_size])\n",
    "    else:\n",
    "      labels.append(target[i:i+target_size])\n",
    "\n",
    "  return np.array(data) , np.array(labels)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "MxS23z--ZtQt"
   },
   "source": [
    "Generate multivariate data using a defined function `multivariate_data`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "Z0CivgkitfgE",
    "outputId": "3f2a841e-eec2-4cf9-e370-4ab5a8bcc725"
   },
   "outputs": [],
   "source": [
    "### generate multivariate data\n",
    "\n",
    "history = 720\n",
    "future_target = 72\n",
    "STEP = 6\n",
    "\n",
    "x_train_ss , y_train_ss = mutlivariate_data(features , features[:, 1], 0, train_split, history,\n",
    "                                            future_target, STEP , single_step = True)\n",
    "\n",
    "x_val_ss , y_val_ss = mutlivariate_data(features , features[:,1] , train_split , None , history ,\n",
    "                                        future_target, STEP, single_step = True)\n",
    "\n",
    "print(x_train_ss.shape , y_train_ss.shape)\n",
    "print(x_val_ss.shape , y_val_ss.shape)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "DQ8txbggaJ2-"
   },
   "source": [
    "The `tf.data.Dataset` API supports writing descriptive and efficient input pipelines. Dataset usage following a common pattern:\n",
    "- Creating a source dataset from our input data.\n",
    "- Applied dataset transformations to preprocess the data.\n",
    "- Iterate over the dataset and process the elements.\n",
    "Note: Iteration happens in a streaming fashion, so the full dataset does not need to fit into memory.\n",
    "Once we have a dataset, we can apply transformations to prepare the data for our model:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "VBdr2epGu3aq",
    "outputId": "719a1cea-6548-4597-fc05-b183cc485387"
   },
   "outputs": [],
   "source": [
    "## tensorflow dataset\n",
    "batch_size = 256\n",
    "buffer_size = 10000\n",
    "\n",
    "train_ss = tf.data.Dataset.from_tensor_slices((x_train_ss, y_train_ss))\n",
    "train_ss = train_ss.cache().shuffle(buffer_size).batch(batch_size).repeat()\n",
    "\n",
    "val_ss = tf.data.Dataset.from_tensor_slices((x_val_ss, y_val_ss))\n",
    "val_ss = val_ss.cache().shuffle(buffer_size).batch(batch_size).repeat()\n",
    "\n",
    "print(train_ss)\n",
    "print(val_ss)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "IN6IQykaEzcF",
    "outputId": "f7877fba-a745-4f3e-f654-c8cdeed5181e"
   },
   "outputs": [],
   "source": [
    "x_train_ss.shape[-2:]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "SIc3631raYA_"
   },
   "source": [
    "We used a custom loss function to evaluate the model:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "9eQpwUyGglu_"
   },
   "outputs": [],
   "source": [
    "def root_mean_squared_error(y_true, y_pred):\n",
    "        return K.sqrt(K.mean(K.square(y_pred - y_true))) "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "1cKtTAzqyiyL",
    "outputId": "004c1521-5b4c-4304-fa58-76bd01a9dcca"
   },
   "outputs": [],
   "source": [
    "from keras.layers import Activation, Dense, Dropout\n",
    "from keras.utils.vis_utils import plot_model\n",
    "### Modelling using LSTM\n",
    "steps = 50\n",
    "\n",
    "EPOCHS =20\n",
    "\n",
    "single_step_model = tf.keras.models.Sequential()\n",
    "\n",
    "single_step_model.add(tf.keras.layers.Bidirectional(tf.keras.layers.LSTM(32, return_sequences=True, input_shape = x_train_ss.shape[-2:])))\n",
    "#single_step_model.add(tf.keras.layers.Dropout(0.3))\n",
    "single_step_model.add(tf.keras.layers.LSTM(units=100,return_sequences=False))\n",
    "#single_step_model.add(tf.keras.layers.Dropout(0.2))\n",
    "#model.add(Dense(units=1, activation='relu'))\n",
    "single_step_model.add(tf.keras.layers.Activation(\"relu\"))\n",
    "single_step_model.add(tf.keras.layers.Dense(1))\n",
    "single_step_model.compile(optimizer = tf.keras.optimizers.Adam(), loss = 'mae',metrics=[tf.keras.metrics.RootMeanSquaredError(name='rmse')])\n",
    "#single_step_model.compile(loss='mse', optimizer='rmsprop')\n",
    "\n",
    "single_step_model_history = single_step_model.fit(train_ss, epochs = EPOCHS , \n",
    "                                                  steps_per_epoch =steps, validation_data = val_ss,\n",
    "                                                  validation_steps = 50)\n",
    "\n",
    "plot_model(single_step_model, to_file='/content/drive/MyDrive/LFN Anuket/Analysis/data/Final/Bi-LSTM.png', show_shapes=True, show_layer_names=True)\n",
    "single_step_model.summary()\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "rePzISUuFmlM"
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 281
    },
    "id": "Pgev0dgzzBVx",
    "outputId": "90d76f79-4228-4f67-ef58-eb0c6f9e9390"
   },
   "outputs": [],
   "source": [
    "## plot train test loss \n",
    "\n",
    "def plot_loss(history , title):\n",
    "  loss = history.history['loss']\n",
    "  val_loss = history.history['val_loss']\n",
    "\n",
    "  epochs = range(len(loss))\n",
    "  plt.figure()\n",
    "  plt.plot(epochs, loss , 'b' , label = 'Train Loss')\n",
    "  plt.plot(epochs, val_loss , 'r' , label = 'Validation Loss')\n",
    "  plt.title(title)\n",
    "  plt.legend()\n",
    "  plt.grid()\n",
    "  plt.show()\n",
    "\n",
    "plot_loss(single_step_model_history , 'Single Step Training and validation loss')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 281
    },
    "id": "EnYf6j4okEoC",
    "outputId": "16fab4ff-15d9-424a-a599-6f03e1ad0e8b"
   },
   "outputs": [],
   "source": [
    "## plot train test loss \n",
    "\n",
    "def plot_loss(history , title):\n",
    "  loss = history.history['rmse']\n",
    "  val_loss = history.history['val_rmse']\n",
    "\n",
    "  epochs = range(len(loss))\n",
    "  plt.figure()\n",
    "  plt.plot(epochs, loss , 'b' , label = 'Train RMSE')\n",
    "  plt.plot(epochs, val_loss , 'r' , label = 'Validation RMSE')\n",
    "  plt.title(title)\n",
    "  plt.legend()\n",
    "  plt.grid()\n",
    "  plt.show()\n",
    "\n",
    "plot_loss(single_step_model_history , 'Single Step Training and validation loss')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "WMegV8mNAwe_"
   },
   "outputs": [],
   "source": [
    "### fucntion to create time steps\n",
    "def create_time_steps(length):\n",
    "  return list(range(-length,0))\n",
    "\n",
    "### function to plot time series data\n",
    "\n",
    "def plot_time_series(plot_data, delta , title):\n",
    "  labels = [\"History\" , 'True Future' , 'Model Predcited']\n",
    "  marker = ['.-' , 'rx' , 'go']\n",
    "  time_steps = create_time_steps(plot_data[0].shape[0])\n",
    "\n",
    "  if delta:\n",
    "    future = delta\n",
    "  else:\n",
    "    future = 0\n",
    "  plt.title(title)\n",
    "  for i , x in enumerate(plot_data):\n",
    "    if i :\n",
    "      plt.plot(future , plot_data[i] , marker[i], markersize = 10 , label = labels[i])\n",
    "    else:\n",
    "      plt.plot(time_steps, plot_data[i].flatten(), marker[i], label = labels[i])\n",
    "  plt.legend()\n",
    "  plt.xlim([time_steps[0], (future+5) *2])\n",
    "\n",
    "  plt.xlabel('Time_Step')\n",
    "  return plt"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "q99i2c-9XKF3"
   },
   "outputs": [],
   "source": [
    "### Moving window average\n",
    "\n",
    "def MWA(history):\n",
    "  return np.mean(history)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 1000
    },
    "id": "xFJT1rZDAUVL",
    "outputId": "d0dc9e7a-bc96-4f19-815f-da9ed8a82408"
   },
   "outputs": [],
   "source": [
    "# plot time series and predicted values\n",
    "\n",
    "for x, y in val_ss.take(5):\n",
    "  plot = plot_time_series([x[0][:, 1].numpy(), y[0].numpy(),\n",
    "                    single_step_model.predict(x)[0]], 12,\n",
    "                   'Single Step Prediction')\n",
    "  plot.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "_KXWQVmyCSix"
   },
   "source": [
    "# **MultiStep Forcasting**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "Lu7m2Rr4AbMK",
    "outputId": "e0d07c10-c644-4a91-8723-5fc94e35c3a5"
   },
   "outputs": [],
   "source": [
    "future_target = 72 # 72 future values\n",
    "x_train_multi, y_train_multi = mutlivariate_data(features, features[:, 1], 0,\n",
    "                                                 train_split, history,\n",
    "                                                 future_target, STEP)\n",
    "x_val_multi, y_val_multi = mutlivariate_data(features, features[:, 1],\n",
    "                                             train_split, None, history,\n",
    "                                             future_target, STEP)\n",
    "\n",
    "print(x_train_multi.shape)\n",
    "print(y_train_multi.shape)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "GLRv5D16CrHj"
   },
   "outputs": [],
   "source": [
    "#  TF DATASET\n",
    "\n",
    "train_data_multi = tf.data.Dataset.from_tensor_slices((x_train_multi, y_train_multi))\n",
    "train_data_multi = train_data_multi.cache().shuffle(buffer_size).batch(batch_size).repeat()\n",
    "\n",
    "val_data_multi = tf.data.Dataset.from_tensor_slices((x_val_multi, y_val_multi))\n",
    "val_data_multi = val_data_multi.batch(batch_size).repeat()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "fjXexah9C8yg",
    "outputId": "8e521731-9e70-48b3-b630-44adc32f93fb"
   },
   "outputs": [],
   "source": [
    "print(train_data_multi)\n",
    "print(val_data_multi)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 385
    },
    "id": "7mtLZ6S-DPU-",
    "outputId": "c6ae5f81-b61d-47e6-cd05-f96a5658ae2d"
   },
   "outputs": [],
   "source": [
    "#plotting function\n",
    "def multi_step_plot(history, true_future, prediction):\n",
    "  plt.figure(figsize=(12, 6))\n",
    "  num_in = create_time_steps(len(history))\n",
    "  num_out = len(true_future)\n",
    "  plt.grid()\n",
    "  plt.plot(num_in, np.array(history[:, 1]), label='History')\n",
    "  plt.plot(np.arange(num_out)/STEP, np.array(true_future), 'bo',\n",
    "           label='True Future')\n",
    "  if prediction.any():\n",
    "    plt.plot(np.arange(num_out)/STEP, np.array(prediction), 'ro',\n",
    "             label='Predicted Future')\n",
    "  plt.legend(loc='upper left')\n",
    "  plt.show()\n",
    "  \n",
    "\n",
    "\n",
    "for x, y in train_data_multi.take(1):\n",
    "  multi_step_plot(x[0], y[0], np.array([0]))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "eaIjVnzCJwCS"
   },
   "source": [
    "Bi-directional LSTM:\n",
    "On some sequence prediction problems, it can be beneficial to allow the LSTM model to learn the input sequence both forward and backwards and concatenate both interpretations. This is known as bidirectional.\n",
    "\n",
    "Here, `tf.keras.layers.Bidirectional` is a bidirectional wrapper for RNNs which inherits from `Wrapper`, `Layer`, and `module`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "snN_Flr5DWQN",
    "outputId": "be27d60a-53b9-4e83-f1e3-45bf9b751196"
   },
   "outputs": [],
   "source": [
    "multi_step_model = tf.keras.models.Sequential()\n",
    "multi_step_model.add(tf.keras.layers.Bidirectional(tf.keras.layers.LSTM(32,\n",
    "                                          return_sequences=True,\n",
    "                                          input_shape=x_train_multi.shape[-2:])))\n",
    "multi_step_model.add(tf.keras.layers.Dropout(0.2))\n",
    "multi_step_model.add(tf.keras.layers.LSTM(units=100,return_sequences=False))\n",
    "multi_step_model.add(tf.keras.layers.Dropout(0.2))\n",
    "#model.add(Dense(units=1, activation='relu'))\n",
    "multi_step_model.add(tf.keras.layers.Activation(\"relu\"))\n",
    "#aDD dropout layer (0.3)\n",
    "multi_step_model.add(tf.keras.layers.Dense(72)) # for 72 outputs\n",
    "\n",
    "multi_step_model.compile(optimizer=tf.keras.optimizers.RMSprop(clipvalue=1.0), loss='mae',metrics=[tf.keras.metrics.RootMeanSquaredError(name='rmse')])\n",
    "\n",
    "multi_step_history = multi_step_model.fit(train_data_multi, epochs=EPOCHS,\n",
    "                                          steps_per_epoch=steps,\n",
    "                                          validation_data=val_data_multi,\n",
    "                                          validation_steps=50)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 281
    },
    "id": "Ay5m27doDsTt",
    "outputId": "74bc7df8-2315-401b-8cd8-d7632c14abb7"
   },
   "outputs": [],
   "source": [
    "plot_loss(multi_step_history, 'Multi-Step Training and validation loss')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 1000
    },
    "id": "6ZFP49W4D2wp",
    "outputId": "cdd29cbc-c26e-46d0-aaaf-0d2a8a4cb047"
   },
   "outputs": [],
   "source": [
    "for x, y in val_data_multi.take(5):\n",
    "  multi_step_plot(x[0], y[0], multi_step_model.predict(x)[0])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "DNKMjVoAVqZP",
    "outputId": "befefa8a-e8e1-4147-d23f-db668c092471"
   },
   "outputs": [],
   "source": [
    "scores = multi_step_model.evaluate(x_train_multi, y_train_multi, verbose=1, batch_size=200)\n",
    "print('MAE: {}'.format(scores[1]))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "YXcsNZ8yu9Ay"
   },
   "outputs": [],
   "source": [
    "scores_test = multi_step_model.evaluate(x_val_multi, y_val_multi, verbose=1, batch_size=200)\n",
    "print('MAE: {}'.format(scores[1]))\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "uCFgbZEOvZ9A"
   },
   "outputs": [],
   "source": [
    "y_pred_test = multi_step_model.predict(x_val_multi, verbose=0)\n",
    "\n",
    "plt.figure(figsize=(10,5))\n",
    "plt.plot(y_pred_test)\n",
    "plt.plot(y_val_multi)\n",
    "plt.ylabel(\"Value\")\n",
    "plt.xlabel(\"Timestap\")\n",
    "plt.legend(loc='upper left')\n",
    "plt.show()"
   ]
  }
 ],
 "metadata": {
  "colab": {
   "collapsed_sections": [],
   "name": "Attention_LSTM.ipynb",
   "provenance": []
  },
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.7"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}