DL_Course_SamU/lab_1-2/assignment1.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Лабораторная работа 1"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "1) Классификация данных методом k ближайших соседей ( kNN)\n",
    "\n",
    "2) Классификация данных методом опорных векторов (SVM)\n",
    "\n",
    "3) Построение softmax-классификатора\n",
    "\n",
    "Вариант 1: задания 1 и 2 на наборе данных CIFAR-10\n",
    "\n",
    "Вариант 2: задания 1 и 2 на наборе данных MNIST\n",
    "\n",
    "Вариант 3: задания 1 и 3 на наборе данных CIFAR-10\n",
    "\n",
    "Вариант 4: задания 1 и 3 на наборе данных MNIST"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Лабораторные работы можно выполнять с использованием сервиса Google Colaboratory (https://medium.com/deep-learning-turkey/google-colab-free-gpu-tutorial-e113627b9f5d) или на локальном компьютере. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 1. Классификация данных методом k ближайших соседей ( kNN)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import random\n",
    "import numpy as np\n",
    "import matplotlib.pyplot as plt\n",
    "from scripts.data_utils import load_CIFAR10\n",
    "\n",
    "\n",
    "%matplotlib inline\n",
    "plt.rcParams['figure.figsize'] = (10.0, 8.0) \n",
    "plt.rcParams['image.interpolation'] = 'nearest'\n",
    "plt.rcParams['image.cmap'] = 'gray'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "1.1 Скачайте данные в соответсвии с заданием.\n",
    "\n",
    "CIFAR-10 по ссылке https://www.cs.toronto.edu/~kriz/cifar.html\n",
    "или используйте  команду !bash get_datasets.sh (google colab, local ubuntu)\n",
    "\n",
    "MNIST \n",
    "sklearn.datasets import load_digits\n",
    "digits = load_digits()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Clear previously loaded data.\n",
      "Training data shape:  (50000, 32, 32, 3)\n",
      "Training labels shape:  (50000,)\n",
      "Test data shape:  (10000, 32, 32, 3)\n",
      "Test labels shape:  (10000,)\n"
     ]
    }
   ],
   "source": [
    "cifar10_dir = 'scripts/datasets/cifar-10-batches-py'\n",
    "\n",
    "try:\n",
    "   del X_train, y_train\n",
    "   del X_test, y_test\n",
    "   print('Clear previously loaded data.')\n",
    "except:\n",
    "   pass\n",
    "\n",
    "X_train, y_train, X_test, y_test = load_CIFAR10(cifar10_dir)\n",
    "\n",
    "print('Training data shape: ', X_train.shape)\n",
    "print('Training labels shape: ', y_train.shape)\n",
    "print('Test data shape: ', X_test.shape)\n",
    "print('Test labels shape: ', y_test.shape)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "1.2 Выведите несколько примеров изображений из обучающей выборки для каждого класса.\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAWEAAAD3CAYAAADMt+U9AAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjEsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy8QZhcZAAAgAElEQVR4nOy9eZxlx1Xn+T0Rd3v7y70yK6uy9tJWJcmyZMmSLNmWEDYGb9B0z4CB6Wagm6VpNg/+MDPQ0GO3gTYNtKEHaDOAjfFgYcnIxpYty1iSJWspqVSLal9y39/+3l1j/riv5FS5KrNKEhZm8vf55Cffe3HvjRMnIn5x4sSJuGKMYR3rWMc61vHaQL3WAqxjHetYx/+fsU7C61jHOtbxGmKdhNexjnWs4zXEOgmvYx3rWMdriHUSXsc61rGO1xDrJLyOdaxjHa8hXjYJi8idIjLxagrznQoROS0id13g99tF5MhlPuvPROQ3Xz3pXlv8UyzPt1smEdktIvtEpC4iP/ttzPeC7fKfKkTk10TkL1dJPygid34bRbosiIgRkR2Xe9+6JfyPCGPM14wxu19rOS4V32md9jsIvww8bIwpGGN+77UW5jsVxpirjTEPv5Jn/FNs4+sk3IWIWP+c8/vnhO9A3Y0BBy+UICL62yzLZeE7UNcvG69VWdck4e7I8SsickhElkXkYyLiXeC6/01ETnSnXIdE5N0r0n5URB4Rkd/uPuOUiLxtRXpJRP5URKZFZFJEfvNyG6eIbBKRe0VkXkQWReQPRGS7iDzU/b4gIh8XkfJ5ZXu/iOwHmq+wEm48X0fnu2wulJ+IXC8iz3T19tfAt+j25eBy9SEifwFsBj4rIg0R+eWXme9FyyMi7xCRZ0WkIiKPicjeFWkjIvLprrynVk7bu9PUvxGRvxSRGvCjr6JMPy4ix0VkSUTuF5GRFWnfJSJHRKQqIh8Vka+KyL+5zLwfAt4M/EFXr58QkT8Ukc+JSBN4c7f9/3m37GdE5FdFRHXv1yLyO936OiUiPy3ptPdS2+p1IrK/W4a/Ptd31yi3EZGfEpFjwDFJ8RERmes+Z7+IXNO91pW0X58VkVkR+SMRyVyCXt4vaV+vd3X81m6S09VFXVL3w+tX3POiFbuiTfx199pnROTaNfL8ljbeLeu/FpGzwENyATfreflqEfmAfJPrnhaRTRfI6zYRGReRN6+lC4wxq/4Bp4EDwCagF3gU+E3gTmBixXU/AIyQEvsPAk1guJv2o0AI/DiggX8LTAHSTf8M8N+BHDAIfAP4ibVkW5G3Bp4DPtJ9hgfcBuwA7gZcYAD4B+B3zyvbs92yZS41v1ego5fkBzjAGeA/ADbw/V09/ebLleVV0MddryDfi5YHeB0wB7yhK9+PdPNzu23maeD/6D5jG3ASuKf73F/rPudd3Wsvua7WkOktwEJXNhf4feAfuvf1AzXgPYAF/Pvuff/mZejl4XP3AX8GVIFbu2XxgD8H7gMKwBbgKPCvu9f/JHAIGAV6gC8BBrAusV1+g7Rf9gKHu8+7aLm79xngwe49GeCebv2UAQGu5Jt9+3eB+7vXFoDPAh9cQ67dwDgw0v2+BdjerecO8PZuG/kg8PiF2ueKNvH93Xr9ReAUYF+CTu5aka/p6j/XLeudrOizF7jnl4Dnu2UQ4Fqgb4XednT1NQ7cdEnt4xIr8idXfH87cOJCwp5337PAO7uffxQ4viIt2xV4AzAE+KzoWMC/Ar5yGY38FmB+rYZJ2on3nVe2/+Xlks7l6uj8/IA3sWIw6v72GK+chF+JPl4JCV+0PMAfAr9x3vVHgDtIifnseWm/Anys+/nXWEESr6JMfwp8eMXvedKOvQV4H/D1FWnS7VivBgn/+Yo03W3/V6347SdIfcgAD7HCIAHu4vJI+IdWfP8w8Eerlbv73QBvWZH+FtKB4WZAnaeTJrD9vLZ3ag25dpAOyHexgjS79fylFd+vAtoXap/da1cStAKmgdsvQSfnk/C2Fel3sjoJH6HLaxd4tum22zPAnkttH5c6pRlf8fkM6cj6EojI+4Cf7xYM0ortX3HJzLkPxpiWiJy7ppd0JJvu/nZOoSvzXAubgDPGmOg8mQaB3wNuJx2lFbB83r2Xk89qWFNHF7huBJg03Rpcce8rxSvRxyvBauUZA35ERH5mRZrTvScGRkSksiJNA19b8f3l1tNqMo0Az5z70RjTEJFFYGM3bXxFmjl/mvoKsLIs/XzTWl8p38YVMq68/nL1MLPic6v7vD4uXu7T5+djjHlIRP4A+G/AZhH5W1LL0yM1qJ5e0XeFtO4uCmPMcRH5OVIivVpEvkDKHReS1xMR6/y2fAEZk279XKzfrYbL5ZoTq6T/HOkg+/ylPvBSF+ZW+jw2k1oWL0JExoA/Bn6a1DQvk07PhbUxTmoJ9Btjyt2/ojHm6kuU7dwzNl/AT/ZB0tFprzGmCPzQBWR6tY6RW1VHF8lvGtgoK1pw995Xiperj1eqi9XKMw78pxV1XDbGZI0xf9VNO3VeWsEY8/ZXQbbVZJoiHRwAEJEcKUFNdu8bXZEmK7+/QqwsywKpFTq24rfNXRnOyb8y32/xP74MrFbuC8mIMeb3jDE3AFcDu0in5QtAG7h6Rb2VjDH5tQQwxnzCGHNbVw4D/OeXUY4XddH1oY9y8X73YtZr/NYkHVjOPVeTuu7OYZzUdXIx/ADwru4gc0m4VBL+KREZFZFe4APAX5+XniMtyDyAiPwYcM2lPNgYMw18EfgdESmKiJJ0AemOS5QNUr/XNPAhEclJuih2K6m11wAqIrKRtOH8Y2EtHV0IXwci4GclXaR7D3DTqyDLy9XHLKk/9uVitfL8MfCTIvKG7kJPTkS+R0QKXXlr3cWaTHfx4xoRufEVyHIpMn0C+DERuU5EXOD/Ap4wxpwGHgD2iMi7uoPZT5G6z15VGGNi4FPAfxKRQteg+XngXLzsp4B/LyIbJV1Eff+rkO1q5f4WiMiN3XqzSUmqA8TGmIS0Xj/SnWXRlfOe1TKXNG76Ld28O6REHr+MctwgIu/p1s/PkRpzj69xz1pt/Cip9f093fL+Kqnf/Bz+BPgNEdnZbcd7RaRvRfoU8FbS9vbvLqUQl0rCnyAlypPdv5cEuhtjDgG/Q9rgZ4E9pItTl4r3kU7JDpFOj/8GGL7Um7sN+XtJfU1ngQnSxcFfJ118qJJ2qnsvQ6bLxao6uhCMMQHpws+Pkpb7B3kVZHwF+vgg8KuSRi/84svI96LlMcY8Rbow+wfdtOPd61bKex3p4soCaWMvXa4MlynTl4H/Hfg06aC1HfiX3bQFUqvmw8AiqX/yKdKO/mrjZ0jJ7STwCGlb+h/dtD8mbVf7gX3A50gHlZdDWsDq5b4Iil05lkldJYvAb3fT3k9al49LGrnyJdJFq9XgAh8irecZ0sX4D7yMotxHWp/LwA8D7zHGhGvc82IbJ13UewmMMVXg35G2v0nSelnphvovpAPjF0kXbv+UdEFv5TPOkhLx++USomnORSdc/AKR06SLCl9a62HrWMc/V3SnuxPA/2yM+cprKMfbgD8yxoytefE/Y4jIrwE7jDE/9FrL8kqxvlljHeu4CETkHhEpd6fNHyD1n6813X21ZciIyNu7rpSNwP8J/O23U4Z1/ONinYTXsY6L4xbSlfAFUnfJu4wx7W+zDELqRlomdUccJo2nXsc/E6zpjljHOtaxjnX842HdEl7HOtaxjtcQl3VWwgf/7AmTJOmirFLf5O8EQ7Ii1E66H40xGMAICLJyZ8kFkKYlxoAx/Mb/eudFY4zjODZxHPPS0M8VT+rmcbH0S4WIYFnWRR8yW5k0h84cZKG5QDaXJYo1mXweL+cglk21kTDoltk2PIatHapBkydOPUc+IxSVhSMWlWaDJGhjwphly8O4WYIwod/L86bNu8hbGVRajovK8ejDnzd+7NDTv4Fnnn2KgYEhyuUePDeLbds4rovWNtrSaKXRWmHrkM5SnXatih+HbNi8jcSxiIMI32+wNDOFJzbHjj7F0NgmvEwWjHDTG+++qBxhEBq/2WL2zCm
      "text/plain": [
       "<Figure size 432x288 with 70 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "1.3 Разделите данные на обучающу и тестовую выборки (X_train, y_train, X_test, y_test). Преобразуйте каждое изображение в одномерный массив. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "1.4 Напишите реализацию классификатора в скрипте /classifiers/k_nearest_neighbor.py и обучите его на сформированной выборке. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "from scripts.classifiers import KNearestNeighbor\n",
    "\n",
    "classifier = KNearestNeighbor()\n",
    "classifier.train(X_train, y_train)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "1.5 Выполните классификацию на тестовой выборке"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "1.6 Визуализируйте матрицу расстояний для каждого изображения из тестовой выборки до изображений из обучающей выборки. \n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "1.7 Посчитайте долю правильно классифицированных изображений из тестовой выборки.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "1.8 Постройте график зависимости доли правильно классифицированных изображений от числа соседей, используемых при классификации."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "1.9 Выберите лучшее значение параметра k на основе кросс-валидации.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "1.10 Переобучите и протестируйте классификатор с использованием выбранного значения k.\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "1.11 Сделайте выводы по результатам 1 части задания."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 2.  Классификация данных методом опорных векторов (SVM)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "2.1 Разделите данные на обучающую, тестовую и валидационную выборки. Преобразуйте каждое изображение в одномерный массив. Выведите размеры выборок."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "2.2 Проведите предварительную обработку данных, путем вычитания среднего изображения, рассчитанного  по обучающей выборке.\n",
    "\n",
    "2.3 Чтобы далее не учитывать смещение (свободный член b), добавьте дополнитульную размерность к массиву дынных и заполните ее 1."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "mean_image = np.mean(X_train, axis=0)\n",
    "print(mean_image[:10]) \n",
    "plt.figure(figsize=(4,4))\n",
    "plt.imshow(mean_image.reshape((32,32,3)).astype('uint8')) \n",
    "plt.show()\n",
    "\n",
    "\n",
    "X_train -= mean_image\n",
    "X_val -= mean_image\n",
    "X_test -= mean_image\n",
    "\n",
    "\n",
    "\n",
    "X_train = np.hstack([X_train, np.ones((X_train.shape[0], 1))])\n",
    "X_val = np.hstack([X_val, np.ones((X_val.shape[0], 1))])\n",
    "X_test = np.hstack([X_test, np.ones((X_test.shape[0], 1))])\n",
    "\n",
    "\n",
    "print(X_train.shape, X_val.shape, X_test.shape)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "2.4 Реализуйте loss-функции в scripts/classifiers/linear_svm.py\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "\n",
    "from scripts.classifiers.linear_svm import svm_loss_naive\n",
    "import time\n",
    "\n",
    "\n",
    "W = np.random.randn(3073, 10) * 0.0001 \n",
    "\n",
    "loss, grad = svm_loss_naive(W, X_dev, y_dev, 0.000005)\n",
    "print('loss: %f' % (loss, ))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "2.5 Убедитесь, что вы верно реализовали расчет градиента, сравнив с реализацией численными методами (код приведен ниже)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "\n",
    "loss, grad = svm_loss_naive(W, X_dev, y_dev, 0.0)\n",
    "\n",
    "from scripts.gradient_check import grad_check_sparse\n",
    "f = lambda w: svm_loss_naive(w, X_dev, y_dev, 0.0)[0]\n",
    "grad_numerical = grad_check_sparse(f, W, grad)\n",
    "\n",
    "\n",
    "loss, grad = svm_loss_naive(W, X_dev, y_dev, 5e1)\n",
    "f = lambda w: svm_loss_naive(w, X_dev, y_dev, 5e1)[0]\n",
    "grad_numerical = grad_check_sparse(f, W, grad)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "2.6 Сравните svm_loss_naive и svm_loss_vectorized реализации"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "tic = time.time()\n",
    "_, grad_naive = svm_loss_naive(W, X_dev, y_dev, 0.000005)\n",
    "toc = time.time()\n",
    "print('Naive loss and gradient: computed in %fs' % (toc - tic))\n",
    "\n",
    "tic = time.time()\n",
    "_, grad_vectorized = svm_loss_vectorized(W, X_dev, y_dev, 0.000005)\n",
    "toc = time.time()\n",
    "print('Vectorized loss and gradient: computed in %fs' % (toc - tic))\n",
    "\n",
    "difference = np.linalg.norm(grad_naive - grad_vectorized, ord='fro')\n",
    "print('difference: %f' % difference)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "2.7 Реализуйте стохастический градиентный спуск в /classifiers/linear_classifier.py . Реализуйте методы train() и predict() и запустите следующий код"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from scripts.classifiers import LinearSVM\n",
    "svm = LinearSVM()\n",
    "tic = time.time()\n",
    "loss_hist = svm.train(X_train, y_train, learning_rate=1e-7, reg=2.5e4,\n",
    "                      num_iters=1500, verbose=True)\n",
    "toc = time.time()\n",
    "print('That took %fs' % (toc - tic))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "y_train_pred = svm.predict(X_train)\n",
    "print('training accuracy: %f' % (np.mean(y_train == y_train_pred), ))\n",
    "y_val_pred = svm.predict(X_val)\n",
    "print('validation accuracy: %f' % (np.mean(y_val == y_val_pred), ))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "2.8 С помощью кросс-валидации выберите значения параметров скорости обучения и регуляризации. В кросс-валидации используйте обучающую и валидационную выборки. Оцените accuracy на тестовой выборке."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [],
   "source": [
    "learning_rates = [1e-7, 5e-5]\n",
    "regularization_strengths = [2.5e4, 5e4]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "2.9 Сделайте выводы по второй части задания"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 3.  Построение softmax-классификатора"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "3.1 Разделите данные на обучающую, тестовую и валидационную выборки. Преобразуйте каждое изображение в одномерный массив. Выведите размеры выборок."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "3.2 Проведите предварительную обработку данных, путем вычитания среднего изображения, рассчитанного  по обучающей выборке.\n",
    "\n",
    "3.3 Чтобы далее не учитывать смещение (свободный член b), добавьте дополнитульную размерность к массиву данных и заполните ее единицами."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "mean_image = np.mean(X_train, axis=0)\n",
    "print(mean_image[:10]) \n",
    "plt.figure(figsize=(4,4))\n",
    "plt.imshow(mean_image.reshape((32,32,3)).astype('uint8')) \n",
    "plt.show()\n",
    "\n",
    "\n",
    "X_train -= mean_image\n",
    "X_val -= mean_image\n",
    "X_test -= mean_image\n",
    "\n",
    "\n",
    "\n",
    "X_train = np.hstack([X_train, np.ones((X_train.shape[0], 1))])\n",
    "X_val = np.hstack([X_val, np.ones((X_val.shape[0], 1))])\n",
    "X_test = np.hstack([X_test, np.ones((X_test.shape[0], 1))])\n",
    "\n",
    "\n",
    "print(X_train.shape, X_val.shape, X_test.shape)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "3.4 Реализуйте функции в classifiers/softmax.py\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from scripts.classifiers.softmax import softmax_loss_naive\n",
    "import time\n",
    "\n",
    "# Generate a random softmax weight matrix and use it to compute the loss.\n",
    "W = np.random.randn(3073, 10) * 0.0001\n",
    "loss, grad = softmax_loss_naive(W, X_dev, y_dev, 0.0)\n",
    "\n",
    "# As a rough sanity check, our loss should be something close to -log(0.1).\n",
    "print('loss: %f' % loss)\n",
    "print('sanity check: %f' % (-np.log(0.1)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "3.5 Убедитесь, что вы верно реализовали расчет градиента, сравнив с реализацией численными методами (код приведен ниже)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "loss, grad = softmax_loss_naive(W, X_dev, y_dev, 0.0)\n",
    "\n",
    "\n",
    "from scripts.gradient_check import grad_check_sparse\n",
    "f = lambda w: softmax_loss_naive(w, X_dev, y_dev, 0.0)[0]\n",
    "grad_numerical = grad_check_sparse(f, W, grad, 10)\n",
    "\n",
    "\n",
    "loss, grad = softmax_loss_naive(W, X_dev, y_dev, 5e1)\n",
    "f = lambda w: softmax_loss_naive(w, X_dev, y_dev, 5e1)[0]\n",
    "grad_numerical = grad_check_sparse(f, W, grad, 10)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "3.6 Сравните softmax_loss_naive и softmax_loss_vectorized реализации"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "tic = time.time()\n",
    "loss_naive, grad_naive = softmax_loss_naive(W, X_dev, y_dev, 0.000005)\n",
    "toc = time.time()\n",
    "print('naive loss: %e computed in %fs' % (loss_naive, toc - tic))\n",
    "\n",
    "from scripts.classifiers.softmax import softmax_loss_vectorized\n",
    "tic = time.time()\n",
    "loss_vectorized, grad_vectorized = softmax_loss_vectorized(W, X_dev, y_dev, 0.000005)\n",
    "toc = time.time()\n",
    "print('vectorized loss: %e computed in %fs' % (loss_vectorized, toc - tic))\n",
    "\n",
    "\n",
    "grad_difference = np.linalg.norm(grad_naive - grad_vectorized, ord='fro')\n",
    "print('Loss difference: %f' % np.abs(loss_naive - loss_vectorized))\n",
    "print('Gradient difference: %f' % grad_difference)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "3.7 Реализуйте стохастический градиентный спуск в /classifiers/linear_classifier.py . Реализуйте методы train() и predict() и запустите следующий код\n",
    "\n",
    "3.8 Обучите Softmax-классификатор и оцените accuracy на тестовой выборке."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "3.9 С помощью кросс-валидации выберите значения параметров скорости обучения и регуляризации. В кросс-валидации используйте обучающую и валидационную выборки. Оцените accuracy на тестовой выборке."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [],
   "source": [
    "learning_rates = [1e-7, 5e-5]\n",
    "regularization_strengths = [2.5e4, 5e4]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "3.10 Сделайте выводы по третьей части задания"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}