{
"cells": [
{
"cell_type": "markdown",
"metadata": {
"id": "9DZOlfTodng_"
},
"source": [
"# Introduction to programming artificial neural networks\n",
"#### Tutorial for Methods In Neuroscience at Dartmouth ([MIND](http://mindsummerschool.org/)) 2023\n",
"By [Mark A. Thornton](http://markallenthornton.com/)\n",
"\n",
"This tutorial offers an introduction to programming your own customized artificial neural network (ANN) for the first time. It is based on the popular ANN programming framework [PyTorch](https://pytorch.org/). You will build up an ANN to perform regression, starting from a very simple network and working up step-by-step to a more complex one.\n",
"\n",
"This notebook focuses on the implementation of ANNs. If you're interested in a complementary conceptual introduction to ANNs, their potential uses in social neuroscience, and their limitations, please consider [my preprint](https://psyarxiv.com/fr4cb) with [Beau Sievers](http://beausievers.com/).\n",
"\n",
"The figure below, created by [Lindsey Tepfer](https://pbs.dartmouth.edu/people/lindsey-j-tepfer) for the aforementioned preprint, illustrates the (A) general structure of ANNs, (B) the internal structure of individual units, which approximate generalized linear models, and (C) the training process using stochastic gradient descent via backpropagation. The terminology in this figure will reappear throughout the tutorial.\n",
"\n",
"![](https://mysocialbrain.org/misc/data/ann_tutorial/Fig1_DS_hires_top.png)\n"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "jWqksu8YQCXj"
},
"source": [
"## Setup\n",
"This section includes the import statements for the packages/functions we'll need here, detection of the available hardware for ANN fitting, and code to simulate the artificial data we'll be using."
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "JM8eq9V-QG4a"
},
"source": [
"### Import packages"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "o_XidOKtfJUu"
},
"outputs": [],
"source": [
"import pandas as pd\n",
"import numpy as np\n",
"import matplotlib.pyplot as plt\n",
"from tqdm import tqdm\n",
"import torch\n",
"from torch import nn\n",
"from torch.utils.data import DataLoader, TensorDataset\n",
"import torch.nn.functional as F\n",
"import statsmodels.api as sm\n",
"from sklearn.metrics import r2_score, mean_squared_error\n",
"%matplotlib inline"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "XUJmvHi8QQ1m"
},
"source": [
"### Detect and set hardware device\n",
"Depending on where you run your notebook, you may be able to take advantage of different hardware. If a cuda-enable graphics card is available, this will be preferred. Mac chipsets (MPS) and traditional processors (CPU) are the fallback options. On Colab, you may want to change your runtime type to take advantage of the GPU runtimes they offer."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/"
},
"id": "Y6GMlwnOG6jc",
"outputId": "ae61ad36-0115-41f8-fef9-afdc95e1a826"
},
"outputs": [
{
"output_type": "stream",
"name": "stdout",
"text": [
"Using cuda device\n"
]
}
],
"source": [
"device = (\n",
" \"cuda\"\n",
" if torch.cuda.is_available()\n",
" else \"mps\"\n",
" if torch.backends.mps.is_available()\n",
" else \"cpu\"\n",
")\n",
"print(f\"Using {device} device\")"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "kH3b-GAhRj2W"
},
"source": [
"### Simulate data"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "D5NfwioWR_hM"
},
"outputs": [],
"source": [
"# set random seeds\n",
"torch.manual_seed(0) # pytorch's random seed\n",
"np.random.seed(0) # numpy's random seed"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "z1xOFx7CQZGW"
},
"outputs": [],
"source": [
"# linear relationship\n",
"\n",
"# set sample sizes\n",
"ntrain = 100000 # training\n",
"ntest = 1000 # testing\n",
"nval = 10000 # validation\n",
"\n",
"# the x-variable is drawn from a standard normal distribution\n",
"x_train = np.random.normal(0,1,ntrain)\n",
"x_test = np.random.normal(0,1,ntest)\n",
"x_val = np.random.normal(0,1,nval)\n",
"\n",
"# this y-variable is a linear function of the x-variable\n",
"# a bit a Gaussian noise is added, as well as a constant slope and intercept\n",
"y_lin_train = x_train*1.5 + np.random.normal(0,.1,ntrain) + 1\n",
"y_lin_test = x_test*1.5 + np.random.normal(0,.1,ntest) + 1"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "QLrB7DrHRn2r"
},
"outputs": [],
"source": [
"# nonlinear relationship\n",
"# the second y-variable is a quadratic function of the x-variable\n",
"y_non_train = np.power(x_train,2) + np.random.normal(0,.1,ntrain)\n",
"y_non_test = np.power(x_test,2) + np.random.normal(0,.1,ntest)\n",
"y_non_val = np.power(x_val,2) + np.random.normal(0,.1,nval)"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "h3Abtp0zoATl"
},
"outputs": [],
"source": [
"# reshape so as to avoid confusing pytorch with 1d data\n",
"x_train.shape = (ntrain, 1)\n",
"y_lin_train.shape = (ntrain, 1)\n",
"y_non_train.shape = (ntrain, 1)\n",
"x_test.shape = (ntest, 1)\n",
"y_lin_test.shape = (ntest, 1)\n",
"y_non_test.shape = (ntest, 1)\n",
"x_val.shape = (nval, 1)\n",
"y_non_val.shape = (nval, 1)"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/",
"height": 449
},
"id": "rhoZb8csRqf6",
"outputId": "d7b2235e-f73f-4008-a1eb-beeb5a246593"
},
"outputs": [
{
"output_type": "display_data",
"data": {
"text/plain": [
""
],
"image/png": "iVBORw0KGgoAAAANSUhEUgAAAjUAAAGwCAYAAABRgJRuAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjcuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/bCgiHAAAACXBIWXMAAA9hAAAPYQGoP6dpAABKm0lEQVR4nO3deXhU9d3+8fdMksm+ELIRCGGRJYKArCqI0iKggqViVWotWrdasI+ibcVfq1VbsY9WfVzq2roWRVFQcQUUcAFkEWQL+xIICVkgyUyWySTz++OYQCAJScjkZM7cr+uaJnPOmZmPTZi5811tXq/Xi4iIiIifs5tdgIiIiEhrUKgRERERS1CoEREREUtQqBERERFLUKgRERERS1CoEREREUtQqBERERFLCDa7gLZUXV1NdnY20dHR2Gw2s8sRERGRJvB6vZSUlJCamord3nB7TECFmuzsbNLS0swuQ0RERFogKyuLLl26NHg+oEJNdHQ0YPyfEhMTY3I1IiIi0hTFxcWkpaXVfo43JKBCTU2XU0xMjEKNiIiInznV0BENFBYRERFLUKgRERERS1CoEREREUtQqBERERFLUKgRERERS1CoEREREUtQqBERERFLUKgRERERS1CoEREREUtQqBERERFLUKgRERERS1CoEREREUtQqBERERFLUKhpDW4XbJhrfBURERFTKNS0hq0LobQAMj8yuxIREZGApVDTGjImQmQC9L3U7EpEREQCVrDZBViCIxIGXGl2FSIiIgFNLTUiIiJiCQo1IiIiYgkKNSIiImIJCjUiIiJiCQo1IiIiYgkKNSIiImIJCjUiIiJiCQo1IiIiYgkKNSIiImIJCjUiIiJiCQo1IiIiYgkKNSIiImIJCjUiIiJiCQo1IiIiYgkKNSIiImIJCjUiIiJiCQo1bcntgg1zja8iIiLSqhRq2tLWhVBaAJkfmV2JiIiI5SjUtKWMiRCZAH0vNbsSERERywk2u4CA4oiEAVeaXYWIiIglqaVGRERELEGhRkRERCxBoUZEREQsQaFGRERELEGhRkRERCxBoUZEREQsQaFGRERELEGhRkRERCxBoUZEREQsQaFGRERELEGhRkRERCxBoUZEREQsQaFGRERELKFdhJrZs2czbNgwoqOjSUpKYvLkyWzbtq3ONeXl5UyfPp2OHTsSFRXFlClTyM3NNaliERERaW/aRahZtmwZ06dPZ+XKlSxatIjKykrGjRuHy+WqveaOO+7gww8/5J133mHZsmVkZ2dz+eWXm1i1iIiItCc2r9frNbuIE+Xl5ZGUlMSyZcsYPXo0RUVFJCYmMmfOHK644goAMjMzycjIYMWKFZxzzjn1Pk9FRQUVFRW194uLi0lLS6OoqIiYmJg2+W8RERGR01NcXExsbOwpP7/bRUvNiYqKigCIj48HYO3atVRWVjJ27Njaa/r27UvXrl1ZsWJFg88ze/ZsYmNja29paWm+LVxERERM0+5CTXV1NbfffjsjR46kf//+AOTk5OBwOIiLi6tzbXJyMjk5OQ0+16xZsygqKqq9ZWVl+bJ0ERERMVGw2QWcaPr06WzatImvv/76tJ8rNDSU0NDQVqiqjbldsHUhZEwER6TZ1YiIiPiFdtVSM2PGDBYuXMiXX35Jly5dao+npKTgdrs5evRonetzc3NJSUlp4yrbwNaFUFoAmR+ZXYmIiIjfaBehxuv1MmPGDObPn88XX3xB9+7d65wfMmQIISEhLFmypPbYtm3b2L9/P+eee25bl+t7GRMhMgH6Xmp2JSIiIn6jXXQ/TZ8+nTlz5vD+++8THR1dO04mNjaW8PBwYmNjueGGG5g5cybx8fHExMRw2223ce655zY488mvOSJhwJVmVyEiIuJX2sWUbpvNVu/xl19+meuuuw4wFt+78847efPNN6moqGD8+PH861//alb3U1OnhImIiEj70dTP73YRatqKQo2IiIj/8et1akRERESaS6FGRERELEGhRkRERCxBoUZEREQsQaFGRERELEGhRkRERCxBoUZEREQsQaFGRERELEGhRkRERCxBoUZEREQsQaFGRERELEGhRkRERCxBoUZEREQsQaFGRERELEGhRkRERE6f2wUb5hpfTaJQIyIiIqdv60IoLYDMj0wrQaFGRERETl/GRIhMgL6XmlZCsGmvLCIiItbhiIQBV5paglpqRERExBIUakRERMQSFGpERETEEhRqRERExBIUakRERMQSFGpERETEEhRqRERExBIUakRERMQSFGpERETEEhRqRERExBIUakRERMQSFGpERETEEhRqRERExBIUakRERMQSFGpERETEEhRqREREpH5uF2yYa3z1Awo1IiIiUr+tC6G0ADI/MruSJlGoERERkfplTITIBOh7qdmVNEmw2QWIiIhIO+WIhAFXml1Fk6mlRkRERCxBoSbQ+NmgLxERkaZSqAk0m96FHYth03tmVyIiItKqFGoCjRewmV2EiIhI69NA4UBz1hQICfObkewiIiJNpVATaPxsJLuIiEhTqftJRERELEGhRkRERCyh3YSa5cuXM2nSJFJTU7HZbCxYsKDO+euuuw6bzVbnNmHCBHOKFRERkXan3YQal8vFwIEDeeaZZxq8ZsKECRw6dKj29uabb7ZhhRaldWtERMQi2s1A4YsvvpiLL7640WtCQ0NJSUlpo4oCxPGblWkAsYiI+LF201LTFEuXLiUpKYk+ffpw6623UlBQ0Oj1FRUVFBcX17nJCfxsszIREWlFFmut95tQM2HCBF577TWWLFnCP/7xD5YtW8bFF19MVVVVg4+ZPXs2sbGxtbe0tLQ2rNhP1EzxdkSaXYmIiLS141vrLcDm9Xq9ZhdxIpvNxvz585k8eXKD1+zevZuePXuyePFifvrTn9Z7TUVFBRUVFbX3i4uLSUtLo6ioiJiYmNYuW0RExL+4XUag6Xtpu/7jtri4mNjY2FN+fvtNS82JevToQUJCAjt37mzwmtDQUGJiYurcREREBHDmwWd/hh5j2nWgaQ6/DTUHDhygoKCATp06mV2KiIiI//nyIXDmwtLZZlfSatpNqHE6naxfv57169cDsGfPHtavX8/+/ftxOp384Q9/YOXKlezdu5clS5bws5/9jDPOOIPx48ebW7iIiIg/GnMPRKfAhbPMrqTVtJsxNUuXLmXMmDEnHZ82bRrPPvsskydP5vvvv+fo0aOkpqYybtw4HnzwQZKTk5v8Gk3tkxMREbEst8sYIJwx0W+6nZr6+d1uQk1bUKgREZGAt2GuMeMpMsFv1iez/EBhaQMWW79ARESAnj+Bgp3GAGGLUaiRhlls/QIRkYDndhkDhGO7wu4vza6m1SnUSMO02rCIiLVsXQhxXaFovyXf2xVqpGGOSKN58rM/G+sZiIiIf8uYCLGdYdyDfjNIuDkUaqRxFlzHQEQkoBw/PtLiW+Mo1EjjLLiOgYhIQAmg8ZHBZhcg7VxUIkx8zOwqRESkJZx5sOtLsAEXPWh2NT6nlhoRERGr+vIhyNsKBbstOdvpRAo10nRuF6x9Dda9prVrRET8wZh7oNNAYxyNBWc7nUjdT9J0WxfC3q+M74PD/GYlShGRgOWIgPSRfrUlwulQqJGmy5gIVW7j+wBI/CIifsvtgo3vwoHVEN/DGCQcAH+IKtRI0zkiYfC1x+774aZoIiKWVvO+XFEMP7wD0clQlAUjbja7sjahMTXSMm6XsShfcXZATBMUEfELNdO3D/1gLLIX3sGyC+3VR6FGms/tgo//BCU5sHOxJTdFExHxSz1/Ysx2SuhjfD9hdsAEGlCokZbYutAINvmZENHRWG1Ys6FERMxVs1llhRNyNkBwaEAFGlCokZbImAhn/ATOmQ4hEVBaCJveM7sqEZHAVTMkILoThEZD99EBOaFDA4Wl+eoMGPbCxnfAU2FqSSIiAa1m9+3C3ZA2DPpfHnCtNKCWGjldQaEQl240c4qIiDlqdt9OG2Z0PwXoBA611MjpOWsKhIQZg4XXvQbeH48F4F8IIiJt7vilNQZcadzP/Cggu55ALTVyumq2sd/+Cax73dhbJED/QhARaXMn7sBd854coH9YKtRI66isgCoPFB/SFG8RkbbgdhnjGUOjA7Zl5kQKNdI6QkLBboeqCljygKZ4i4j4Us1sp9LCgJy63RCFGmkd/adA0pmAHdyBO0hNRKRN1Mx2KtqvVprjKNRI63BEwsUPw6CpEBoFXUbAhrlqsRERaU1uF6x9zdjbKTQaUgeZXVG7olAjrccRafwj69ADPrxN+0KJiLS2rQth15fG+mCHNgT09O36KNRI68qYaOwImz7K+KpmURGR1pMx0WgNj+kMnQZCZILeZ4+jUCOtyxFp7Agb3y2gdoYVEWkTjkj46b1GsPF6jUCj99laWnxPWl/NOgkiItI63C5YPwcO/QCdBoC7HA58B+Gxer89jkKN+Mbxq1zqrwgRkdOz8V349hmo9hi3HqON4+p6qkOhRnyjZpXL9W9C7mYYcw9EJZpdlYiI/3G74MBqYyKGuwQ6DzluU2E5nsbUiG9kTDQGsGWvB2cuLJ1tdkUiIv5p60KI7wGdz4bz7zSWzpB6KdSIb9SMqxl7H0R0hNgusGAGOPPMrkxExL+knWPsqzfqTqOFRl36DVKoEd+KSoRuo2DrB7DtY1h0r9kViYj4D7cL3v+dsR3C1/80u5p2T6FGfC9jIgSFQXCIMVpfrTUiIqdWs79TaAxUlkLKALMravcUasT3HJFw1esQFgdRqfDqRAUbEZGGuF3GNjOb3jX2dwqLhfSRcObPzK6s3VOokbYRlQhT34b8bVBWpG4oEZGGbF0IR/bB9/+FvG3QZQgkZRjjaqRRCjXSdrJWQnQnqHJDlUcbXoqI1CdjIuxaAvnbjT8Eg8O0HUITaZ0aaTsZE6GiBA6uhcoyWPWCcX/4jWZXJiLSPtSsHOw8DMGhRqDpf7lmPDWRWmqk7TgijQDT40I4uheO7IYf5sK619RiIyICx1YO9pQbi+1d+ZoCTTMo1Ejby5gICRnGXyCHNsDK52HTe2ZXJSJiPhsQ3sFopRl+i1ZibyaFGml7jkiY+ChUVxl7mBzZa3ZFIiLmc+bB3m+h4xlw7gytHNwCCjViDkckDP0NhIRDh+5GU6u6oEQkkH32Z9jxOeTvgLAYdTu1gEKNmCeyI0Qlg7cKlv4vrHnF7IpERMzhzIODq41uJ0eYZjq1kEKNmCc4FDoNhNI8qKqAtf9Ra42IBI6aRfYK98HLF4MtCBxRcOXraqVpIYUaMU//KdDzJ5A+CuxB4KmA9W+aXZWISNuoWWTvP+OM5S3KjsAIDQ4+He0m1CxfvpxJkyaRmpqKzWZjwYIFdc57vV7uvfdeOnXqRHh4OGPHjmXHjh3mFCutwxEJQQ7oNAgiEoz9TTbN0xRvEbE+twsqio1lLbzVUF0JZ4zV4ODT1G5CjcvlYuDAgTzzzDP1nv/f//1fnnzySZ577jlWrVpFZGQk48ePp7y8vI0rlVaVMRFiU421GKqrICTCGF/z3GijSVZExGrcLvjkbvjupR9nfwZBr/Ew8Z/qdjpNNq/X6zW7iBPZbDbmz5/P5MmTAaOVJjU1lTvvvJO77roLgKKiIpKTk3nllVe4+uqrm/S8xcXFxMbGUlRURExMjK/Kl5bYMBeKD8H3r0PRfmNX785nw7QPzK5MRKR1bZgLG+fBzs+BIAiNhDszFWga0dTP73bTUtOYPXv2kJOTw9ixY2uPxcbGMmLECFasWNHg4yoqKiguLq5zk3aqpsVm6G+gQzfwlMG5/2N2VSIirS/tHMhaBWEdwG6Ha99XoGklfhFqcnJyAEhOTq5zPDk5ufZcfWbPnk1sbGztLS0tzad1ymlwRMKAK2HodVBeZKw2/NndZlclItK6CvfBi2OgwmUEmhlroctgs6uyDL8INS01a9YsioqKam9ZWVlmlySn4oiE4bcaA4gdkbDqRQ0aFhH/53bB2tfgjZ9DWQHgMaZwx6ebXZml+EWoSUlJASA3N7fO8dzc3Npz9QkNDSUmJqbOTfzAOTdDSn+oKIJP74Z5NxpvBgo3IuKvti6EvV8Zy1eAMSli6lxza7Igvwg13bt3JyUlhSVLltQeKy4uZtWqVZx77rkmViY+4YiESU/BkSzwemD7p7DkAVjzqtmViYg0n9sFxQch6ztjXZrgSIjpom4nH2g3ocbpdLJ+/XrWr18PGIOD169fz/79+7HZbNx+++387W9/44MPPmDjxo38+te/JjU1tXaGlFhM1kpI6PnjnWooLYBN75pakohIi2xdCKv/A0f3QZUH7MHGMhbS6oLNLqDGmjVrGDNmTO39mTNnAjBt2jReeeUV/vjHP+Jyubj55ps5evQoo0aN4tNPPyUsLMysksWXMibCrqXgzDdW2aQK8rbCt88Yg4k1U0BE/IHbBeXFxoxOABvQZzwkZ5hallW1y3VqfEXr1PgZt8vY5PKrf0JZIdgdENcZLpxlzJQSEWnvlj8Gyx+B6moIjYC4NDj71zD8RrMr8yuWWqdGApQjEiITIP08iOkMwRFQchgOZ2rQsIi0f24XrPwXeEqhuhzsITDsJm2F4EPtpvtJpF4ZE6HKDeEd4Ie3oaocvn4MHNHGYn0ZE9UVJSLt0/o5UFUJBBmzngZeBcGhes/yIbXUSPvmiITB18LFDxvNtwB4Ydls2LIANr1nZnUiIvU7sA4++zO4SyEkDC78f5ByFvS91OzKLE2hRvyDIxJG/h5jlB1QVWEMJPZUmFmViEhdzjz48A54Y4rRsux1G93oUR2NsYBqpfEphRrxH6NnwiWPgs1h3PeUwro34DutOiwi7cSiv8D2z6D8qHHfHg7n/R76X25qWYFCoUb8hyPSmDGQdNxUyJwN8O1T8MndCjYiYi63C7I3QslB474tBEbfYbxvqYWmTSjUiP+56nWwh/54p9rYGO7wFo2vERFzrXkZ8rb8eKcaks+EmFRTSwo0zQ4106ZNY/ny5b6oRaRp4tOh7yXH7pflQ/5O2LlErTUi0vbcLmMrl8//DPw4oSGmM6QOUrdTG2t2qCkqKmLs2LH06tWLhx56iIMHD/qiLpHGXfIIhMUdu19xFPavhMyPzKpIRALV+jnw1WPAj2vZhnWA7qPhJ39Rt1Mba3aoWbBgAQcPHuTWW29l7ty5dOvWjYsvvph58+ZRWVnpixpFThaVCKP/YPRZ13DmwNEDaq0Rkba18wtqAw0Yf3ClDIDdX5pVUcBq0ZiaxMREZs6cyYYNG1i1ahVnnHEG1157Lampqdxxxx3s2LGjtesUOdnQ641F+Wp5jS0V1r9pWkkiEkDcLlj3mrG8RI3oVBg8zZjGrTVp2txpDRQ+dOgQixYtYtGiRQQFBXHJJZewceNGzjzzTB5//PHWqlGkfo5ImPYBOKKOHassM1YeVmuNiPja6pfh0/8HVaXG/eSz4KL74ZybtSaNSZodaiorK3n33XeZOHEi6enpvPPOO9x+++1kZ2fz6quvsnjxYt5++20eeOABX9QrUldyBvx+PQTV7NZeBQfXwLdPw4a5Cjci4hvOPPj2/8BdfOzYwKkKMyZr9t5PnTp1orq6mqlTp/Ldd98xaNCgk64ZM2YMcXFxrVCeSBNEJUKviyDzQ+O+twqWPmQ0A7vy4bzp5tYnItbhdhkDg799BsqOCzRBETD0OtPKEkOzQ83jjz/OL37xC8LCwhq8Ji4ujj179pxWYSLNMvFxqHTBri+OHSvJhg1vK9SISOvZuhC+/y8c3Q9UHTs+6na10LQDze5+uvbaaxsNNCKmiEqEq96AC2ZR59f68EbI3WpaWSJiIc482Poh5G6mTqBJ6g8jZ5hWlhyjFYXFOrYuhMJdYDuuAdJbBc+ea4yx0fgaETkdSx6AzI+h2n3sWFg8/HqBWmnaiWZ3P4m0WxkTocoNFU7Y/gnH1o3wwrJHISwWBl9rZoUi4o/cLmOm045POdZCY4f4HvCr94yWYmkX1FIj1uGINELLFS9B52F1z1UcMRbIUmuNiDTX6pdh0b3gPHzsWNKZ8NvlxrYt0m4o1Ij1OCJh6hxwxNY9vnMRvHGl0S8uItIUzjz46hHqjKEhCDoNUJdTO6RQI9YUlQi9xwG2Y8fcJbB/Bcy5Wi02InJqhfvg6WFQfrTu8bOmwEVai+1EpW4P878/QKnbY1oNCjViXRNmQ+chJxysgkMbYOGdCjYi0rg3r4byI3WPXfYsTHlR42jq8dnmHAqdbj7fnGtaDQo1Yl1RicY2Cmf+vO5xbyX88BasfMGcukSkfXO7YPEDkLel7vEzxsHgX5pTkx8Y3y+FjlGhjOuXbFoNCjVibY5ImPwM9PjJCSd+3PxSrTUiUsPtMrZX2fiusQxEHUHQe7wpZfmLCEcwk8/uTITDvInVCjVifY5IuPoNiOlS93hlCTw3WgOHRcSw6V3YvAC+fAiqK+qe63spDJpqSlnSdAo1EhgckXDdxxAcXvd44U747y/UYiMixtJWWavAeaju8VF3weXPabaTH1CokcARnw69J3DSr/2h72Hli6aUJCLthDMPvnsJygrqHk/MgKQ+CjR+QqFGAku3kdAhHQipe/yL+7VHlEgge+8WyN1wwkE7DLvR6HoSv6BQI4Fl0C/h/DuhU/8TTlTDs+fBgXWmlCUiJtrzDexecvLxXhfB8Bst3UpzqrVl2sPaM82hUCOBpWYrhbOvqedkNbw0RgOHRQKF2wXfPgOvXnLyueBI6DWu7WvysVK3h7dW7+et1fvIKnRx46tr2F9Q2uDaMu1h7ZnmUKiRwDTol3DhPXV39K7x/AXw3YsaPCxidd88DZ/fc/LxqGQ48zJLzHY6PsSUuj18tjmHr3bk8fbqA/zhnQ2EBdv5bk9hg2vLtIe1Z5pDu3RLYHJEwoV/ggFXw5ODgOpj50oOwsrnjV29B1xpVoUi4ituF3zzFCybffK5M8YZgab/5X7X7VTq9vDBhoOAjZE9O/Lcst10jgvn7TVZZHSKJiw4mPH9Uli5u4DU2DDO7tqBPfkubr+oV4Nry9SsPeMvFGoksMWnG4MAMz+se7xwB2xfYpzzszc2ETmFTe/C8kdOPh4SZQSawde2fU2nqdTt4e8fbSWnqIwdh508v2wX3RMiWbm7gLBgO9tznTw4uSMRjmDum9SPzzfnMq5fsqkL5fmCup9EJj4OcV1PPr7pLdj0XtvXIyK+4XYZXcvfPAXeEwa+2oKhzyVGC40fOHEA72ebc+jSIZy9BS48VdVEOoIprahiyuDOxEWE8LNBqXy9w5iu3h5W/vUVm9fr9ZpdRFspLi4mNjaWoqIiYmJizC5H2hNnHvxngrEY3/GCI+GmJZCcYU5dItI63C5YMB22fABUnXx+3EMw9Lp21zJbMw5mfL+U2hCSVejiN6+sYUL/ZLp0iMARbOf8Xgl8vaOAIelxPPPlTvqlxhISbMdV7iEqLJjQ4CC/bplp6ue3Qo1IjXWvwQe31XMiGLoMhctfMLqrRMT/LH8cvvhr/efiusLtG9u0nFPJd5bzyGfb8FRV07VDJNnFZfRLNT63Xv12H9XVXlwVHtI7RjKqVwLpHSNPGvtS6vZYpptJoaYeCjXSKLcL3voV7P6i/vMRCfC7lcbu3yLiP5x58Fi/k/dzAsAOt37b7lpj//TuBr7ankeQ3U7nuDDiIh1sOlhMmdtD/9RYMnNLSIgMxYuXDhEOXpw21O+DS2Oa+vmtMTUiNRyR0O/nkHxW/edL8+GtX2qqt4i/+fgPDQQa4EbzupdrxsXkO8trv761ej+vrdhLuduD3W4nLtzBBb2T2JFbgqeqCkeQnQpPNR/9fhRXD09jQJdY/m/qIEsHmubQ/wsixztrChxYDR43FGw7+fyBNbDmFThvepuXJiLN5HbB8sdgy/z6z1/w/6DL4LatCSPMzFt7gA/XHyQ6PISnv9jJRRlJPLlkB5WeajxeL0HYSI4J4+eDO7M+6yihwUHEhTvolRzFXeP7kBAVxrXndmvz2ts7hRqR4zki4eKHjVlPP7wDe5edcEE1fH4v9PxJu2uuFpHjOPNgzlWQvbb+8z1/CrGd2rYmjEDzh7fXs2jrYUKCbXg8XmLDQ5jz3X7CQ+wUlnqwAUF2Gx2jHUSHhZDRKZq1+44wZXAXbji/R5vX7E8UakROVLOVQv/LYd4NsP2TEy7wwCsXwx2b291MCRHBaKF5ZSLkZ9Z//sYvjZmObbBRpdEqk8XavUcIDrLRMzGKxVsP467yUlnlJSzYhrvSQ5f4SErdlSQFBVFV7SXYbqd3UhTj+iXzwYaDDEqLIypMH9mnov+HRBriiIQr/g0vjoW8LXXPlR2Bf/SA6z8xpflaRBrgdhk7bjcUaKZ9bPyb9dG/2+NX9b1sYCpzVu3n8cXb8XiqqKoGL+AIMq512MERHERcRAjpHSOIcAQzKC3OOBdsZ9LAVCIcwVw2sDNhwcF+s1WBmTT7SeRUnHnwaC+Mt6N63LpSXVEi7cWXD9e//QHAkJtg0qOt+nL5znIe+XQb/TrHcMWQND7bnMPizbnsKyylqLySvKIyyutZFicqNIggm43gIBs3nt+Dg0fK6BwXTqe4cL/alqCtNPXzWy01IqcSlQij7oSvG3gzfO58uG2t1rARMdsP7zYcaEJjIblvq7/k44t2sPFgEUu3H+bhT7YREQIut5fyyuqG/gzCBoSF2BnZM4EqL8RHOph2XrfaNWWk5RRqRJpi9ExwRNW/eJe3Ep4cYDRrdx/Z5qWJBLzCfTDv+oYHBTuiofuFEBza7Kc+cUXfmkXxeiVF4aqoYsnWHHKL3bUBptR98nME2SAyNIiI0CCCbUFg83L9ed2ZOqJrncXx1EJz+tT9JNIcjTVtY4eLHoRh12sAsUhbcbvg2ZFwZE/950Oj4ZZv4MCqFm1QO//7AxQ63USFBeMItrNyVwHf7z/CngIXlfV0K9UIwuiwPq9nPGP6JuMItuMItvPTjCS+3lFgiVV+25Llup/++te/cv/999c51qdPHzIzGxgMJuILI2+DoDD44r56TlbD0ochPNYvd/kV8UurX2440ITFwc3Lja7hZnQP17TODE3vwOLNOWw8WERocDD7CpxUe6HK2+AIO8BombnuvHTO6tKh3tYXtcj4jt+EGoB+/fqxePHi2vvBwX5VvliBIxJG3w49RsNLY04+X1kCH98Nznw452a12Ij4UuE+WPpQ/efsDpi5pcF/g8YspWzAy2UDjZAxb+0B1u4tIOtIGUE2G3fM3dDkUiIcNmLDHIQ5gpg4oBNHXJUaH2MCv0oFwcHBpKSkNPn6iooKKiqOLY1dXFzsi7IkEHUZDFfOgbd/efI5jxOWPQJxnWHAlW1fm0ggcLvgmfOhqoFtS65d0OgfFfPWZvHSV3uIiwjBVV7F/O8PsDm7pNEWmIZkdIomNiyYhOgwRnSPJzoshHEXqnvJDH6199OOHTtITU2lR48eXHPNNezfv7/R62fPnk1sbGztLS0trY0qlYBw5qXQd1L956pc8N5NsPhB7RUl0tq2fAQPpUJVUf3n79pZ76D9rEIXv3xxJdtzipm35gAHCsv44UAxD360lU3NDDTD0jvQNT6MjE7R9EuJ5ulrBjM2I5kpQ7ow+ezOCjQm8ZuBwp988glOp5M+ffpw6NAh7r//fg4ePMimTZuIjo6u9zH1tdSkpaVpoLC0HmcePH8+lBxq+Jrel8AVL6krSuR0Fe6Dt66BwxsbvqbPJTD1TeDYar4b9h+la8dInlu2i7LK6tMqIdgGsy7J4Ibze1Dq9tSZvSS+09SBwn4Tak509OhR0tPTeeyxx7jhhhua9BjNfhKfcObBfyYYy643ZOhNMLF1F/0SCTj/GgmHN510uCamVPWZRMikxyl1dGDe2izmrzvI7jwnRfWtftdEdozp2CN6dMRZ4eGijGSmjuiqENPGLDf76URxcXH07t2bnTsb+SARaQtRifDb5TBnaj0bYP5ozYtw3m1aoE+kOdwu2LrQ2ED2mycbDTTbEsbyvPcOdr+cyb4CJ66Kajyn8Se7ww4pceFcfnZnvtqRzwW9E7Urth/w21DjdDrZtWsX116rqbPSDjgiYcAVcHCdMQOqPk8OMLZUyPkBMiaqO0rkVLYuhOJD8MYVkLO+zqnjO5Ey6cF1B6Zw+EBOi1/KDiTEhHBej0SGpHeo3XsJoFtClGYy+Qm/CTV33XUXkyZNIj09nezsbO677z6CgoKYOnWq2aWJGPpPgfIS+Pyehq959hwY8TuockOQQ+FGpDFp58B/xoPTGLN20miYatjiTeUXlX+mjLAWv0yfxEh+dV63OptIHk/ryvgPvwk1Bw4cYOrUqRQUFJCYmMioUaNYuXIliYmJZpcmYnBEwnnTjabyZ0cCDfTjr/qXsZdUZAJkfqRp3yIncrtg+WN19lurrv0fgxfY7O3KdZWzWhRoohwwqlcyceEh3DWhDwlRLQ9F0n747UDhltBAYWkzhfvq/IVZr0HT4JLZaqkROU6ps4icOTNIz/4A2/EnTmimud9zDW9X/7RZgSYsGCb0T+Xcnh0B6m2VkfbJ8gOFRdq1+HT4/VqYdyNs/7j+a9a/amypMOZuBRsJSCduFlnq9vDl83cyoeiEQHOC7d4UjhLb5EDjsMOZqTGckRTF3ZdkqFXGwtRSI+JLbhf8sz9UFDZ8TUJ/uPlzBRsJOPO/P8DuPCcfb8zhsp7B9F33Vy5idaOPOUA8z3kuY3716EZDTceIIFJiIghzBHHJWZ00DdvPqaVGpD1wRMItS+GpIeCtrP+a/E3w9AgY8Vvt8C2WdfwmkU9/sZPE6FDmrT1ATrGxQGqfI49ykX1do8+R6U3iV5X3U0hsveeDgM7x4bz066FsOVSiRfECkH7aIr4Wnw63rYUnhwANBJviLPjy79rhWywp31nOTa+u4XBJBZUeD4edntpz4ZRzvf0jxp0i0GzxduHVqotPCjSpMQ4Gp8cTHGRnSHoHpgzpQoQjmN4pao0PRAo1Im0hPh3G/RU+/ysNBhtPKSy6HzoP1Vo24ve25xRz63/XcnaXGOZ9X//6MeGU80zwP7nQvrnR59ro7cYbVRexsPrc2mNdOoRx/Xnd1a0kdeg3QaStDL0egsNg8f3gbmDH+LI8ePZcuHCWpnuLX6rZb2n2x5mUVVazK6+03ut6kMV7jj8T01DI/9EX1WfxR8/vKLHH4gixMfaMRC7onVjbIiNyPP1GiLQVRyQMvxHOuAhengAl2Q1c6IWlD0FYByjYAyNnqMVG2p1St4cPNhzEWV7F1kNFpHWI4O21+8g+2nhIAUjlMB857ia0kX2xvcCTnsl8nPAbRneKZki3eAUZOSX9doi0tfh0uG0NLJgOW+Y3fF35EVj2EER2NMKQiMnyneU8vmgHd1zUi8VbD/PGin1sy3VSWdX0SbQ9yOITx58IaeSaEkL42n4O8eP+yPvn9VGQkSbTb4qIGRyRMPkZOLAaig80fu3n94Kn3Oi+UouNmKTU7WH6f9dxxOVm/ONZFLiatxpIPEXcG/wKl9pXEdTQRXaojOzMkg7XUtB9IomxcQo00iz6bRExiyMSbl4KL1zYeLDxuGDJ343xOKHRGkAsbSqr0MUtr61hV56TqirwnPohJ0nlMAsddxNH+ckn7cd/H07orcuY4OjA55tztYmkNJsW3xMxmzMP5lwF2WtPfW3auZDYCyY8rGAjrapmgO/mg8XM+MkZfLghm5e/2U2esyUx5ph4ivjCMYOYE/dCs9f5AgTDuAeM/dNETtDUz2+FGpH2wO2CVy6D7DWnvjahP3QbAeMeVLCRVpHvLOfW19eSmVNCSUUDG7G2QA+y+NBxD+E1gcZe93zt3ZgucM7vYOh1+p2WemlFYRF/4oiEX74FL18CBdsbvzZ/E+RvBedhuPx5fQhIs9W0yqzdW0ihq5Kvdha0+mv0IIvPwv5UO37G3tCFI26Fix9u9deXwKRQI9JeRCXC9R/D65dD7g+nuLgKMj+EBSEw+WkFG2lUzRYF5/dKYPHWXL7ans+izTm4q0/92OaKp4jnwx5nCEY4bzDMAMSmw0//0vpFSMBSqBFpT6IS4YZPYf2bsHQ2lOY3fv2W9+DgGrjpC+OxIscpdXt4YdkuXli+B7vNi9frxVXZ+iMO7ECn2DB+Hr+P2w/d0XiQqZHYD6a9r0AurUpjakTaK2cevHYZHN7StOtH3QWjZ+pDQih1e5izah8vfbWndsNIXwixwy3npDCqaiUDBg0n4tWxTXtg+ii45m39rkqTaaBwPRRqxO+4XTDvBtj+SdOuP/NydUcFqJogM3/dQbYfKjnF5gMtlxwdysC0WJKiw7jnoq5EfPZH2Ps1lJxivaUavS+By55Uy6I0iwYKi1iBIxKu+Dcsfwy+fvTU1295D3I3QkRH+PkLxurFYjnHr+xb5q7iljfWsjW7pJFNB05PrAPGnJnKneN68+0uY1DxpJ4hRLx2KRze2LQnCQqFm5dBcoaPqhRRS42I/1j+GHzxINDE0Z0xaXDBH+GsKWq5sZB8ZzlT/vUtOUVltOLs63r1Sori6mFpx3bCdrtg60JIGQD/HtfwxqwnSjsPrnpNrTPSYmqpEbGaqAToOwn2fw2lTZiCW5wFH/8B9iyHy/5PwcbP5TvL+Z83v+ebXYU+fZ2oEPjvzecxMK3DySc3vQuZn8L8m5v+hP2vhCtebL0CRRqhUCPiL/pPMbZKmPhP+PZJ43YqVeWw6R04egDO/qVabfxITRfTZQM78bvX11BQ5rtmmZQYB9PO7c60kd0a32spbwds/6iJz2qDPhfDhIdapUaRplD3k4i/+tdIOLypGQ+ww6g7YeyffVaStEyp28MHG7IBLyN7JvC3hVtYknkYjw/WkakRGgRnJMfw3K8GkxbfSNB1u2D9HNi0wGglbIrkgXDDJwrQ0mrU/SRidVfPgbd/DTkb4cR9depVDV8/AmVHIXWAWm3aiVK3h78s2MhX2/MocFVS5eM/My/olcDPB3dh8tmdT32x2wXzboTtHzf9BULj4dp39bslplCoEfFX8enw22XGejbv3Qy7v2ja49a+CGttcGQfjNVqrm2tZur1p5tyyOgUzX9XZjV16HeLxThspCdEccmAVOIjHafe/drtgi8fhhVN6OKs4YiFpD5w+UsaECymUagR8XdRifDr+bD8cfjir018kNeYIl7lNnb97q9WG185fouCBd8f5KWvdpFTbKwis2bfUZ+97gW9Ehh7ZjKOYDuTBqY2PlamhjMPPp0FWz6D6ibObAJI6Auj74QBV7a8YJFWoFAjYhXn3GzMkPpgRtMfs+JJo7tg3wq49FEFm1ZW6vbw94+2EmSDmXM3+GwdGYAgoF/nGB79xUC2HCphXL/kpgWZGs48eO58cB5q3gsnnwUjboG+lzbvcSI+oIHCIlZzYJ2x23dVWfMe54iBUXfAObco3Jym7TnFXPfyd2QX+W6LghoOG9x8YU9+N+aM5oWYGs48+OA22P4pNCt22aHPBJik1YHF97RNQj0UaiRguF2wYIaxwnBzJWTAdR/qg6qZjBlMBzlQWMbTX+7y6Ws5bDC6bxIPTzmLhKiwlj9R4T54ejhUlzfvcT3HGovpKfxKG9HsJ5FA5og09oDqMgyWPNC8Vpv8rfDPDBh9F8SmarxNI2paZPJKKqj08Whfhw2m/6QXN13Qo2UtMjXcLljzCnz3Ahzd2/zH970MLn9OvxPSLinUiFiVIxLO+50x1uHfY8F1uOmP9VbCstnG94X7ILE3ZEwM+A+yUreHeWsP8M2OfFbvyaOwzLdJpmdiBGWVVQzv1pGHLj+r5WHGmQdfPgR9JsI7v4ZKZ/OfIzgaxv0VBk0N+N8Dab/U/SQSCGr+Ov9yNlSWtOw5+lwCU14KyA+0rEIXt7y2hswcp8+nXwOclRrD3N+eC8Dnm3ObP+j3RB/eAdnr4dC6FjzYDmdOhkv+V12SYhqNqamHQo0EPGce/OdiKNzRssc7OkCPUeCIgnEPWvpDrmY9mbdXZ7H9sMvnrxcRYqdbfCS9O0Xz54kZLR8r43YZezR5ge4XwKd3N2/xvOOFxsEty7Xbu5hOY2pE5GRRicaCfStfhFXPNK9LCsB9BDI/NL7f+xXcvNRSwSbfWc6DH25mzb4jHDzq25lLwTZIjAmjX2o0ceEO7r6k7+kN+q2xdSFs+wT2fgMVRS18Egf0nQATH7PUz1esTy01IoFs+ePwxYM0bZuFethCYeRtsO9ryJgEQ6/3y+6pfGc5d8/7gcWZeT5/rRiHjfN6JfG3n/dvnRADRuvMxneh+BB89X9QfRotS0NvgomPtk5dIq1ELTUicmo1C/YVZ8PyJ6C6tHmP91YYKxMDZG+AQ+shbQQEhbbbvaWMwb5ZrNpVwPbDLnYcbsGg2WZKjAyhc4cInvrl2Y1vHtlSa142BgJXnkaYcUTDebfBec1YvFGknVFLjYgc89lfmrffT0PCOsDoP8B500//uVpJvrOc2R9vZWt2MVtyfBtkbMDNo7vTIzGq6VsUNEfNdgaHNkDB9tN/vl6XwDVvnv7ziPiIWmpEpPnG3A1FWbD1E/A2c0G245Ufgc/vgS0fQsducFHbDyqu2XMpPiKEm19bQ0ULe9ia44zECKad150pQ7q0fpCpUbgPnhsJ7hbOYjueLRjO+Z3xcxexALXUiEhdbhdkfgQEwXu/aZ3nDI2BhD4w5d8+n0lT6vbwzBc7eGH5bp8viAfQKdrB1cPTSYkLa/1WGbfLGPgbEgXzroNqd+s8b1i8sbji0OvaZRehyInUUiMiLeOIPLbb8tF98M3//TiL5jT+/qkohoOr4ckBQBB0GwVnToJBvzTOb114Wov75TvLeeTTbcRGhPDC8j0tr7MZxmYk8eTUs33XIuN2wYLpsGUhUNk6zxl/Bgy9AYZOU5gRS1JLjYg0rKbVpscYWPU8fPVIK7+ADRL7QMfe0HscDL620atrupTG90sB4IMN2TjLK3n2yx0UlPq2f8kOvHz9UHKKjanerdoqU7O2THE2bHgLbA4o3NY6zw0QkQjTPoTkjNZ7TpE2pMX36qFQI3IaNsyFI/tgzzI4+D14WnlBuqBwcETAla9D8YGTWm5K3R7+/tFWohxBfLo5h4KScpyVvn37CgKmjUyn1F3FXeP7tN4UbDAG+37+F6j2QFkh7FrSes9dwxYKt63W4nni9xRq6qFQI3Iaalpt+l4KR/bDS+OgsthnL1cdEs2XAx+h/87niUnpzoOV1/D53mryXa3UFdOIjhF2BqcnMPbMZK4a1vX0nqxmXEzaObD4PmPae68JxoaSLV0f6FTCEiChB1z+kgKNWIJCTT0UakRaUU3IiT8D/j0evK00iBXq7q/0450KbBwmnrmeC8knng+rjb2RJtlXAPBh9bmU0bKWlLM6RXHl8PTaWUulbk/L9lyq6UbyVBi37PWQtw0Kd7dsE8nmCu8A132ibiaxHIWaeijUiPiIMw9enwK5G07raapP+qZhHuAwsUAQlQTxkmci71Wfz3j7Gj6rHlon4IRTftLxIKBTh1D+M204vVNijrWo1Ddg+fhzcPJ1bpexAN7KZ6Ek19jlvC2lj4Z+P9MO2mJZCjX1UKgR8SG3Cza9B658WD/nlIvCnZRbWnn69T5vPPG2Yjw4OOBNxOkN52z7Do7aowmKTCE+vR8h582ARX+Bnj+FI3sgNg2cOXDhLNi8ALLXQuoQKMmBVc9Bh27gKYfCnT++Sijg2z2iGmQPhS5D4MrXtD+TWJ5lQ80zzzzDI488Qk5ODgMHDuSpp55i+PDhTXqsQo1IG9kwF0oLjAHFOz41pnQfpzktMq3CXu+3/ikoFHqPN3bgVsuMBAhLrlMzd+5cZs6cyXPPPceIESN44oknGD9+PNu2bSMpKcns8kSkRsZEY7zNhIegx2hjqvK6N6gsziWopmXD14HG3uhdP2OHPhMgMhF+8he1zIg0wK9aakaMGMGwYcN4+umnAaiuriYtLY3bbruNu+8+9TLfaqkRaTs1a8oMTe/Ao59tY93+QrKOVBBPEY8GP8WF9i2+eWF7nS/+KygMhlxvLHzYeYhaZSSgWa6lxu12s3btWmbNmlV7zG63M3bsWFasWFHvYyoqKqioONbfXVzsu+mnInJMvrOcX72wim2HnSetQ1xILL/x/Jl4ing65EniKaSnLZeglr6Y7cebFQRHQO8JMPlpBRiRFvCbUJOfn09VVRXJycl1jicnJ5OZmVnvY2bPns3999/fFuWJBLysQhf/89b35JaUcfDIqad3FxLLDZV/YOKPU7LXVp/B0yGP0deWSxU0HnKsNEYmeSCk9IML7oYDq4x1gBRoRFrEb0JNS8yaNYuZM2fW3i8uLiYtLc3EikSspdTtYd7aLJZm5vHFtrxmP76MMN6pHlN7/5LKx0nlMI+GPM9XVWcyIWgtGba9xhuVVYKMowMk9Yar/3vy2BgtlCdyWvwm1CQkJBAUFERubm6d47m5uaSkpNT7mNDQUEJDQ9uiPJGAUOr28MGGbAqdFbyxYh/Zxa0/nTmbJH5Z+RcAXqm+lGnh33B2SBYDrphFpx+ehV2LjW0F/EF0JwiLBVcBjJppbCRZsyqzWmNEWp3fhBqHw8GQIUNYsmQJkydPBoyBwkuWLGHGjBnmFidicTVh5oPvs/l2d4FPX+vsLjFMHNiZqLDgHzeNnHLsZO8XjYX+vngQEvpAaaGx6J3XAxUujCX52pjNAcEOqCyF9FEQ2RFCwuGiB4y9rDa9Z1zX//K6O6CLSKvzq9lPc+fOZdq0aTz//PMMHz6cJ554grfffpvMzMyTxtrUR7OfRE7t+J2wDxSWcv0r35F7tMKncaFDqI0ZY/tyw/k9Wv4kbhesfxMOrIEuQ+HMn8GW92H7Z5C1EvpeBpvfA++Pc8mrKgEb2ILAHgxVP27QGRQJVWXG911HQnkhFO6B8FiISoaeY2HLAkg+U2vFiLQRyy6+9/TTT9cuvjdo0CCefPJJRowY0aTHKtSINMxojTnIl5mHWbGzgOIKH222+CM70C81hn/9ajBr9x1t/j5LzXX8hpxwcgtKfdc15biI+JxlQ83pUKgROVmp28OcVft4dcVe8orKKfdhlgkGeneKZtKAVOKjHD92L/lNL7iImMRy69SISOs4cVG8FbvzOVzi2w0YQ+wwY0wv/uei3j59HREJbAo1IgEkq9DFr/69ktKKKvKdlSctjNeaftInkTF9je1LHMF2Jg1M9eGriYgo1IhYmtG1tJ+Pf8imorKKzTlOn71WiA1Cgu0M7tqBLvHh3Dupn7qWRKRN6R1HxGJK3R5e+WYP/121j5Iyj88H/N4yujs9EqNqW2I+35zr+0G/IiL10LuOiEXUrO47f102Gw8cxeOjvqXQILhmRDqllVXcNb4PCVFhdc5PPruzb15YROQUFGpE/NTxq/su3pJLvtPN/iNlPnmtuHA7YzM68cDk/mqBEZF2S+9OIn7m+CnYh4+WU1Htm9dJjArhVyO6ER/lqB3oq0AjIu2Z3qFE/EDNwni5RRW89NUuSnyUZGJDIMQRwr+vG8bAtA4+eQ0REV9RqBFpx/Kd5Tz44RZW7i4gt8Td6s9vA85MjuLxqWez5VCJBviKiF/Tu5dIO1KzMF6PhEhmzPme3OIy3D6YvDSsWxw9E6K497Jj0657p2iVbRHxbwo1Iu1EvrOc6/+zmu05xT4ZJ5MU7eC8HvHcOb5v2+y1JCLSxvSOJmKSfGc5j3y6jfSOEXy9M59vdxW2+muckRjJ1cPT2HSwmHN7duSqYV0BSIvXhowiYj0KNSJtKN9ZzuyPtnK0zMOKXXmUVrb+YjJd48OZOqxr7YaRcGxBPBERK1OoEWkjWYUuLnv6G46Utv7mkUPS4rhyeFqD0661IJ6IBAKFGhEfMFb3PcCqXXlsySlhT37rL4o3skc8D18xQONjRER+pHdBkVZUs+/S88t2UVTeutOWgoHEmFBGntGRByafVRtiND5GRMSgUCNymrIKXfzp3Y1MHdaZu+b9QIWndZ8/NMjG9SO7cbSskrO7dtDKviIiDdA7o8gp1KwdM75fSp0wUbOB5GOfb+domYdvdxW0yutFOeykd4igU3wExWVu/jb5LC2MJyLSBHqHFDmFzzbnkFNUzkMfbeWWC3rw9Jc76RQTxpzv9nG4pPUG/abGObj1gl5MGdLlpPCihfFERE5NoUbkFMb3S+EvCzay5WAxF63LoryVpmGnxISSGOUgyG6jW8dI/jzpTBKiwlrluUVEApFCjUg9arqchqZ34K8fbObLzDxaY5HfGAcM75lEZGgQVVVezu+dWLsgnoiInB6FGpHj1MxeevmbvSRHh3HHoQ2n9Xw2IC48iJtHn8G0kd1qu5VK3R4tiCci0soUakQwQsYLy3bz4vKduH7sXspztnxX7JjQIEb3TmREj471jpGJcARrQTwRkVamUCMBJd9ZzuOLdjDt3HSeX76Lo6Ueth4qoqS8kpLT2EWyU1wIFW4Y3q0DcREO7prQR+NjRETamEKNBIxSt4ffv7ked2UV45/4itMZ7msDYsODSI2NYMqQLkSFGf+UtIaMiIh59O4rlnT82jIFzgpmzl1PcWklewtctLRBJtgGvRIjiQgL4YmrB9ExKrR2XIyCjIiI+fROLJZSE2YqPFXsyXMx++OtBNts5DorqGpmmAm2gx3oEh8BXnj1huEnbUmgcTEiIu2HQo1YQr6znEc+3UZ5ZRXOyipyj5Sx+VBJi7uYIkLgJ31T+OvP+vH1jgK1xoiI+AG9S4tfqmmROb9XAgu+P8jzy3Zhs9koc1firPA2O8wkRASBPYiRPTvSMymKw8UV3HNphmYpiYj4EYUa8UsfbDjI19vzeeXrPWzJKaaymRti24GkmFA8VdVcclYKK3YfYdLATiTHhBEaHMSN5/dQy4yIiJ/Ru7b4DWMDyQN8syOftfsKiQoNYW9habOeo2t8KMO7dWRY945MGpgKwOebc/n92N7qZhIR8XN695Z26/gupq925LMv38Wzy3bh9hidS/muxjeTtAPJMaHYbRASHESvpCg6Rjm4d1K/OsGlpntJ3UwiIv5NoUbanVK3hw82ZLNiZz47Dzt59PNteKuhwFlORRO6mVKiHXiqvfxsUCoudxVnd+3ATzOS1BIjImJxeneXduH4dWXmrc3iP1/vxe2pIt/pprKq8YG/dsARbMNugx6J0Vx7bjqTBqaetIaMWmJERKxNoUZMk1Xo4q53NnDRmclEhYVwxOXmLws2smbvEcoqq6is8hIWYqOyyog0NsALRIZAaaXxvSPIRu/kKOw242vNWBnNWhIRCTwKNdKmjm+RueudH8g8VMLBo2UM7BzH6n1HiAsPJqe4nAqPl4QoB67yakKCoNoLDjuM7JVETHgwhS43W7JLeOHXQ8jMKQG0RYGISKDTJ4C0qQ82ZLNiVwE5ReXkFJUR7gjCBizdcZiIkGAOFZVTXe3FBhxxubHZIMRmIzosmMFdO5AcE8btF/Xi6x0FPHON0bU0MK2D2f9ZIiLSDijUiE8dP4Pp4405vLsmi8JSN+v2F1JWUUVJhYewIBtGtIE3bhzB/R9uIae4jPjwUI6WuxnUpQMjesTjCLara0lERBqkUCM+9dnmHAqdbh75bBvf7iogv6QCR7CN2HAHZZVV2Lxeyqu8hAQF8dsLejIwrQNv3DiCzzfnMqpXR81YEhGRJrObXYD4t1K3h/nfH6DU7an9Pt9ZXntsfL8UOkaF0i81hg7hIcRHhtAxMpTLBqbSJyWGhOgwkqPDOb9XAlFhRnCpaYlJiApj8tmdFWhERKRJ9GkhLWKsJXOQ1XuOUO6uwu2pxhFsZ39+KVO/XMX4M5N5aO9Wbr+oF+UeD87yKhzBQfxmZA9+dnYqX+8oICkmlHX7jjIkvQPRYSGM65ds9n+WiIj4MYUaabZSt4e/f7SVojI3+wtKKS730DslmmXbD7O/wEXv5Bg+2ZxD304xPPLZNsrd1Ww4cJSo0GD25LtqW2BK3R5iwhzqXhIRkVahTxJpts8255AUFcrOw066J0ZSVeXlgw3Z5JWUY7fZyHO6mTq8K5sOFtE/NRZHsJ0BXWLZkVvC7Rf1qn0eDfgVEZHWpFAjzTa+XwoPfbSVkWd05ODRMgb37MDvf9qLPy/YRGpsKH/+cW+lE1f0FRER8SWb1+ttbAV6SykuLiY2NpaioiJiYmLMLqddOH4xvPrCR76znMcX7eCOi3oR4Qjmgw0HARtjM5J4YtEOOseF0ykuXC0uIiLiM039/Naf0AGuZsr155tz6w0mjy/aweHicp5YtIMh3TqwYmchNhuEBQdxz6UZta0xIiIiZvObKd3dunXDZrPVuT388MNml+X3aqZc1xdMSt0e+qRE0zHKwe0X9WJ8vxTOO6Mj5/bsWNutpCnXIiLSXvjVp9EDDzzATTfdVHs/OjraxGr804ndTY0N1v1scw6VnmrO6ZFAQlQYAFcN69qW5YqIiDSZX4Wa6OhoUlJSmnx9RUUFFRUVtfeLi4t9UZZfOVV30/HG90tR95KIiPgNv+l+Anj44Yfp2LEjZ599No888ggej6fR62fPnk1sbGztLS0trY0qbb/O75XAnnwXo3p1POW16l4SERF/4jeznx577DEGDx5MfHw83377LbNmzeL666/nsccea/Ax9bXUpKWlBfTsp/nfH6DQ6aZjVKhmLImIiF9o6uwnU0PN3XffzT/+8Y9Gr9m6dSt9+/Y96fh//vMfbrnlFpxOJ6GhoU16PU3pNsbUaP0YERHxJ34RavLy8igoKGj0mh49euBwOE46vnnzZvr3709mZiZ9+vRp0usp1IiIiPgfv1inJjExkcTExBY9dv369djtdpKSklq5KhEREfFHftH/sGLFClatWsWYMWOIjo5mxYoV3HHHHfzqV7+iQ4cOZpfX6k61yq+IiIiczC9mP4WGhvLWW29xwQUX0K9fP/7+979zxx138MILL5hdmk8cP+1aREREmsZvZj+1Bn8ZU6PBvCIiIsf4xZgaqV9jq/yKiIhI/fyi+0lERETkVBRqRERExBIUakRERMQSFGraUKnbw/zvD1DqbnzPKhEREWk+hZo2pKnaIiIivqNQ0wqa2gIzvl8KHaNCGdcvuY0qExERCRwKNa2gqS0wNVO1tfaMiIhI61OoaQVqgRERETGfmgxagRbLExERMZ9aakRERMQSFGpERETEEhRqRERExBIUakRERMQSFGpERETEEhRqRERExBIUakRERMQSFGpERETEEhRqRERExBIUakRERMQSFGpERETEEhRqRERExBIUakRERMQSAmqXbq/XC0BxcbHJlYiIiEhT1Xxu13yONySgQk1JSQkAaWlpJlciIiIizVVSUkJsbGyD523eU8UeC6muriY7O5vo6GhsNpvZ5bS54uJi0tLSyMrKIiYmxuxyAp5+Hu2Hfhbth34W7Ud7+ll4vV5KSkpITU3Fbm945ExAtdTY7Xa6dOlidhmmi4mJMf0XVI7Rz6P90M+i/dDPov1oLz+LxlpoamigsIiIiFiCQo2IiIhYgkJNAAkNDeW+++4jNDTU7FIE/TzaE/0s2g/9LNoPf/xZBNRAYREREbEutdSIiIiIJSjUiIiIiCUo1IiIiIglKNSIiIiIJSjUCBUVFQwaNAibzcb69evNLifg7N27lxtuuIHu3bsTHh5Oz549ue+++3C73WaXFhCeeeYZunXrRlhYGCNGjOC7774zu6SANHv2bIYNG0Z0dDRJSUlMnjyZbdu2mV1WwHv44Yex2WzcfvvtZpfSJAo1wh//+EdSU1PNLiNgZWZmUl1dzfPPP8/mzZt5/PHHee6557jnnnvMLs3y5s6dy8yZM7nvvvtYt24dAwcOZPz48Rw+fNjs0gLOsmXLmD59OitXrmTRokVUVlYybtw4XC6X2aUFrNWrV/P8888zYMAAs0tpMk3pDnCffPIJM2fO5N1336Vfv358//33DBo0yOyyAt4jjzzCs88+y+7du80uxdJGjBjBsGHDePrppwFjf7i0tDRuu+027r77bpOrC2x5eXkkJSWxbNkyRo8ebXY5AcfpdDJ48GD+9a9/8be//Y1BgwbxxBNPmF3WKamlJoDl5uZy00038frrrxMREWF2OXKcoqIi4uPjzS7D0txuN2vXrmXs2LG1x+x2O2PHjmXFihUmViZg/BsA9O/AJNOnT+fSSy+t8+/DHwTUhpZyjNfr5brrruO3v/0tQ4cOZe/evWaXJD/auXMnTz31FI8++qjZpVhafn4+VVVVJCcn1zmenJxMZmamSVUJGC1mt99+OyNHjqR///5mlxNw3nrrLdatW8fq1avNLqXZ1FJjMXfffTc2m63RW2ZmJk899RQlJSXMmjXL7JItq6k/i+MdPHiQCRMm8Itf/IKbbrrJpMpFzDV9+nQ2bdrEW2+9ZXYpAScrK4v/+Z//4b///S9hYWFml9NsGlNjMXl5eRQUFDR6TY8ePbjyyiv58MMPsdlstcerqqoICgrimmuu4dVXX/V1qZbX1J+Fw+EAIDs7mwsvvJBzzjmHV155Bbtdf3P4ktvtJiIignnz5jF58uTa49OmTePo0aO8//775hUXwGbMmMH777/P8uXL6d69u9nlBJwFCxbw85//nKCgoNpjVVVV2Gw27HY7FRUVdc61Nwo1AWr//v0UFxfX3s/Ozmb8+PHMmzePESNG0KVLFxOrCzwHDx5kzJgxDBkyhDfeeKNdv2lYyYgRIxg+fDhPPfUUYHR7dO3alRkzZmigcBvzer3cdtttzJ8/n6VLl9KrVy+zSwpIJSUl7Nu3r86x66+/nr59+/KnP/2p3XcHakxNgOratWud+1FRUQD07NlTgaaNHTx4kAsvvJD09HQeffRR8vLyas+lpKSYWJn1zZw5k2nTpjF06FCGDx/OE088gcvl4vrrrze7tIAzffp05syZw/vvv090dDQ5OTkAxMbGEh4ebnJ1gSM6Ovqk4BIZGUnHjh3bfaABhRoR0y1atIidO3eyc+fOkwKlGlJ966qrriIvL497772XnJwcBg0axKeffnrS4GHxvWeffRaACy+8sM7xl19+meuuu67tCxK/pO4nERERsQSNRBQRERFLUKgRERERS1CoEREREUtQqBERERFLUKgRERERS1CoEREREUtQqBERERFLUKgRERERS1CoEREREUtQqBERERFLUKgRERERS1CoERG/lZeXR0pKCg899FDtsW+//RaHw8GSJUtMrExEzKANLUXEr3388cdMnjyZb7/9lj59+jBo0CB+9rOf8dhjj5ldmoi0MYUaEfF706dPZ/HixQwdOpSNGzeyevVqQkNDzS5LRNqYQo2I+L2ysjL69+9PVlYWa9eu5ayzzjK7JBExgcbUiIjf27VrF9nZ2VRXV7N3716zyxERk6ilRkT8mtvtZvjw4QwaNIg+ffrwxBNPsHHjRpKSkswuTUTamEKNiPi1P/zhD8ybN48NGzYQFRXFBRdcQGxsLAsXLjS7NBFpY+p+EhG/tXTpUp544glef/11YmJisNvtvP7663z11Vc8++yzZpcnIm1MLTUiIiJiCWqpEREREUtQqBERERFLUKgRERERS1CoEREREUtQqBERERFLUKgRERERS1CoEREREUtQqBERERFLUKgRERERS1CoEREREUtQqBERERFL+P88iYh2g1F4DQAAAABJRU5ErkJggg==\n"
},
"metadata": {}
}
],
"source": [
"# plot simulated (training) data\n",
"plt.scatter(x_train,y_lin_train,s=.1)\n",
"plt.scatter(x_train,y_non_train,s=.1)\n",
"plt.xlabel(\"x\");\n",
"plt.ylabel(\"y\");"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "2fLBHHB4gmWm"
},
"source": [
"## Simple linear regression\n",
"We'll begin by showing how ANNs can approximate simple (OLS) regression. In fact, ANNs can approximate nearly all of the modeling techniques in the typical psych/neuro toolkit, ranging from this simple case up through things like factor analysis and beyond."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "viV5cWrggrLA"
},
"outputs": [],
"source": [
"# define model with a single linear unit which takes one input\n",
"model1 = nn.Linear(1, 1)"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "bOb0agjZhB_6"
},
"outputs": [],
"source": [
"# define loss (error) function as mean square error\n",
"# this is similar to ordinary least squares regression\n",
"loss = nn.MSELoss()"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "lZeIGQUBhV3D"
},
"outputs": [],
"source": [
"# define optimizers as stochastic gradient descent (SGD)\n",
"# feed in parameters of the model to be optimized and the learning rate\n",
"# (in most cases, we would probably use a much lower learning rate)\n",
"optimizer = torch.optim.SGD(model1.parameters(),lr=.5)"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "5PtHdXjUoIIx"
},
"source": [
"Now let's train the model:"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/"
},
"id": "pAp7eHfAhuaW",
"outputId": "c9b7f222-5b84-4b90-8bd7-d742d5575a0b"
},
"outputs": [
{
"output_type": "stream",
"name": "stdout",
"text": [
"Epoch [1/10], Loss: 2.4881\n",
"Epoch [2/10], Loss: 0.0100\n",
"Epoch [3/10], Loss: 0.0100\n",
"Epoch [4/10], Loss: 0.0100\n",
"Epoch [5/10], Loss: 0.0100\n",
"Epoch [6/10], Loss: 0.0100\n",
"Epoch [7/10], Loss: 0.0100\n",
"Epoch [8/10], Loss: 0.0100\n",
"Epoch [9/10], Loss: 0.0100\n",
"Epoch [10/10], Loss: 0.0100\n"
]
}
],
"source": [
"nepoch = 10 # epochs = how many times the model sees the dataset\n",
"# note that we're not actually using the GPU yet - see the next example for that\n",
"for epoch in range(nepoch):\n",
" # convert inputs and targets to torch tensors\n",
" inputs = torch.from_numpy(np.float32(x_train))\n",
" targets = torch.from_numpy(np.float32(y_lin_train))\n",
"\n",
" # propagate activity forward through network to make prediction\n",
" outputs = model1(inputs)\n",
"\n",
" # compute loss (error) of predictions\n",
" curloss = loss(outputs, targets)\n",
"\n",
" # backpropagate errors to change weights and biases via SGD\n",
" optimizer.zero_grad()\n",
" curloss.backward()\n",
" optimizer.step()\n",
"\n",
" # print\n",
" if (epoch+1) % 1 == 0:\n",
" print ('Epoch [{}/{}], Loss: {:.4f}'.format(epoch+1, nepoch, curloss.item()))\n"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "8QC_9JNbnjIQ"
},
"source": [
"Now that we've trained the model, let's take a look at its out-of-sample performance on our simulated test set. To do this, we'll make predictions for the test set, and compare them to the actual test values using R^{2} and RMSE, as well as plotting them against one another. As you'll see, the R^{2} is near perfect, and the RMSE is just about equal to the SD of the noise we injected when simulating the data. In other words, the model is performing as well as it could."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/",
"height": 484
},
"id": "LQWGAHURmE9P",
"outputId": "94b4694a-9475-49b9-a6f8-be49c95f98f7"
},
"outputs": [
{
"output_type": "stream",
"name": "stdout",
"text": [
"0.9957496739688491\n",
"0.09758071680706164\n"
]
},
{
"output_type": "display_data",
"data": {
"text/plain": [
""
],
"image/png": "iVBORw0KGgoAAAANSUhEUgAAAjUAAAGwCAYAAABRgJRuAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjcuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/bCgiHAAAACXBIWXMAAA9hAAAPYQGoP6dpAABPeUlEQVR4nO39eXzU5b3//z9mMplkkskeshL2CCFgWAVRKpTNykGtVvmdn63Y02OtBxek7VHsp3r0HJdaW6nHHlFb7aZiW4uN2hqoLZuirEaMgGELWci+LzOTWb5/xJkmbCYwyWQmz/vtlpvJe2beeSU3YZ5c1+u6LoPH4/EgIiIiEuSMgS5ARERExB8UakRERCQkKNSIiIhISFCoERERkZCgUCMiIiIhQaFGREREQoJCjYiIiIQEU6ALGEhut5uKigpiYmIwGAyBLkdERER6wePx0NLSQkZGBkbj2cdjhlSoqaioICsrK9BliIiIyHkoLS1l+PDhZ318SIWamJgYoOuXEhsbG+BqREREpDeam5vJysryvY+fzZAKNd4pp9jYWIUaERGRIPNFrSNqFBYREZGQoFAjIiIiIUGhRkREREKCQo2IiIiEBIUaERERCQkKNSIiIhISFGpEREQkJCjUiIiISEhQqBEREZGQoFAjIiIiIUGhRkREREKCQo2IiIiEBIUaERERCQkKNSIiInLB2h1ONuwro93hDFgNCjUiIiJywQqKKqlvdbCxqCpgNSjUiIiIyAVbkptGkjWCxbmpAavBFLDvLCIiIiEjymzi2qmZAa1BIzUiIiISEhRqREREJCQEVagpLy/n61//OklJSVgsFiZPnszu3bsDXZaIiIgMAkHTU9PQ0MBll13G/Pnz+etf/8qwYcMoLi4mISEh0KWJiIjIIBA0oeZHP/oRWVlZvPTSS75ro0ePPudr7HY7drvd93Vzc3O/1SciIiKBFTTTT/n5+cyYMYMbbriBlJQUpk6dygsvvHDO1zz22GPExcX5PrKysgaoWhERERloBo/H4wl0Eb0RGRkJwOrVq7nhhhvYtWsXd999N+vWrWPFihVnfM2ZRmqysrJoamoiNjZ2QOoWERGRC9Pc3ExcXNwXvn8HTagxm83MmDGD999/33ftrrvuYteuXezYsaNX9+jtL0VERGSoaXc4KSiqZEluGlHmwdWd0tv376CZfkpPT2fixIk9ruXk5HDixIkAVSQiIhLcup/XNBiOObhQQRNqLrvsMg4dOtTj2meffcbIkSMDVJGIiEhw6x5kBsMxBxcqaELNPffcwwcffMCjjz7K4cOHeeWVV3j++edZuXJloEsTEREJSt2DjPeYg8E29dQXQdNTA/DWW2+xZs0aiouLGT16NKtXr+bWW2/t9evVUyMiIhJ8Qq5R2B8UakRERIJPyDUKi4iIiJyLQo2IiIiEBIUaERERCQkKNSIiIhISFGpEREQkJCjUiIiISEhQqBEREQkR3Y89GIoUakREREJEKJzfdCEUakREREJEKJzfdCGC94AHERER6cF7ftNQpZEaERERCQkKNSIiIhISFGpEREQkJCjUiIiISEhQqBERERnkhvr+M72lUCMiIjLIDfX9Z3pLoUZERGSQG+r7z/SW9qkREREZ5Ib6/jO9pZEaERERCQkKNSIiIgGiBmD/UqgREREJEDUA+5dCjYiIyADyjs7UttqwO91YI01qAPYThRoREZEB5B2dWbupmDabkwhTGFFmrdvxB4UaERGRAeRdnr1qUbaWafuZoqGIiMgA6r48W8u0/UsjNSIiIhISFGpEREQkJCjUiIiISEhQqBEREZGQoFAjIiIiIUGhRkREREKCQo2IiIifdN8tWGc6DTyFGhERET9odzh55O0DVDbaWLupWGc6BYBCjYiIiB8UFFUyPMFCeWOHdgsOEO0oLCIi4gdLctPYWFTFijmjeuwaLANHoUZERMQPFGQCT9NPIiIiEhIUakRERCQkKNSIiIj4kXdZt5ZzDzyFGhERET8qKKrUcu4AUagRERHxoyW5aVrOHSBa/SQiIuJHWgUVOBqpERERkZCgUCMiItILagAe/BRqREREekENwIOfQo2IiAxZfRl9UQPw4KdQIyIiQ1ZfRl+8DcBRZq2xGawUakREZMhakpuGNdKEzelUr0wIUKgREZEhK8pswmwy0mZzqVcmBCjUiIjIkKZemdChUCMiIkHvQpZbq1cmdCjUiIhI0OtNw6/2mQl9CjUiIhL0ejOFpH1mQp9CjYiIBL3eTCGpdyb0aQJRRESGBB00Gfo0UiMiIkFP/TICCjUiIhKkugeZM/XLKOgMPQo1IiISlLoHmTP1y+QXlrP5YA1vFlac9loFntCkUCMiIkHp1CDjwXNKWDFgMJz5tVoJFZoMHo/HE+giBkpzczNxcXE0NTURGxsb6HJERMRPNuwro77VwbHaNkYnR/vCzsaiKhbnpp62Kqrd4TzrYzL49Pb9O2hHah5//HEMBgOrVq0KdCkiItLPvmi6yDtqs2pRti/QnGuZt3YRDk1BGWp27drFc889x8UXXxzoUkREZAB80XSRN6QkWyMVVoawoAs1ra2t3HTTTbzwwgskJCSc87l2u53m5uYeHyIiMvidOjIzNzuZ4qoWmm0ONffKWQVdqFm5ciVLly5l4cKFX/jcxx57jLi4ON9HVlbWAFQoIiIX6tSRmW3FtXR0utlb0qjmXjmroBqfW79+PXv37mXXrl29ev6aNWtYvXq17+vm5mYFGxGRILAkN83XyOv92uF0A+iYAzmroAk1paWl3H333WzatInIyMhevSYiIoKIiIh+rkxERPwtymxicW4qBUWVLMlNI8psYvnMEYEuSwa5oJl+2rNnD9XV1UybNg2TyYTJZGLLli08/fTTmEwmXC5XoEsUERE/0l4y0ldBM1KzYMEC9u/f3+PaN7/5TSZMmMC9995LWFhYgCoTEZFz8R5j4B1x6a1Tp6BEvkjQhJqYmBgmTZrU41p0dDRJSUmnXRcRkcGj+4hL91Oyvyjs6FRt6augmX4SEZHg5N0Y7/LsJH6z4xj3vl5IbauNgqJKKhttPJxfxPpdJ7RUWy5Y0IzUnMnmzZsDXYKIiPSCBw9/O1DNG/sqcDjdrN1UzP1Lc3g4v4iDVS202p1EmsI0MiMXRCM1IiLiV903zmt3OHnk7QNUNtowAF+dmsmEtBguSosBYMrIBIYnRGGNNKl3Ri5YUI/UiIjI4NO9h8aDh+EJFsobOlhx2SiizCaskV2HT75ZWAF4mJudzLK8DB1tIBdMIzUiIuJX3Xto7E4X5jAjnS63r2fG+7gHD202FxGmMAUa8QuFGhER8SvvqqVtxbW02Vxs+rSSoopmniw41OPxq/MyfSdqi/iDQo2IiPQL74jMwompRJiM5GbEAf/suQF0orb4lf5PEhGRfuEdkWl3OEm2RrI4N9XXOJwaG8Gjbx/g/qU5CjXiNxqpERGRfuUNN1FmEwVFlQyPt/Dh0Xoy4y06AkH8SqFGREQGzJLcNNLjLfzsX6eQHm9RP434lcb8RETEr9odTvILy3E4PZhNRq7utly7+9EH2mhP/E0jNSIickG6b7YHXfvU7Dhcz5/3lfPBkTpNMcmAUagREZGzOjWwnEn3zfaga4ppzrgkrpqcTkS4kcuzkwaqXBniFGpEROSMuh9xcOpoS22rjR9s2E9tq4252ckcq23zhZcos4nlM0eQaDVzUUoM24vrAlG+DEEKNSIi4tN9ZKagqLLriIPGDt9y7A37yqhttXHXqx9xsrGDtZuK2VZcy+jkaF948T5vbnayNteTAaVQIyIiPt2nkpbkppEeZ/HtJVNQVEllY1egmT4ynvZOFxelxZwWXrz32F5c52sG/qIpLBF/UKgREREf7y7Ai3NTe6xU8o68lDd2MHtMIjUtdkYmRdFmc/rCi3eF05LcNKyRJmxOl2/Ep3vPjUh/UagRERGf7hvleXUfebl/aQ4jEqOZMiKeUUnRvqmpU+9hNhlpszl9Iz6ahpKBoFAjIiLndOrojTecJEabfVNTp66SOtOIj45DkP6mUCMiMsSdadm2d3VTaX0b+YXlNNs6yS+s8E0ntdlcRJjCfEHl1CkmBRkJBIUaEZEh7kw9L09tKqa62cZ9r+8/bSO9M00naYpJBgNFaBGRIW5Jbhobi6p8gaTd4WR8mhWXy80Dy8bx/pE6HM4EzCbjaQ3EXme6JjLQFGpERIYQ7/TRktw0Xy9MQVElc7OTPz+vyU1RRTOjkqKZPTaZrMRolidGB7pskV5RqBERGUK6TzVdOzWT/MIKdhyp44Ojddgcbkrq2/B4wACsmDMq0OWK9IlCjYjIENJ9qqnd4WRfSQP2ThfhBpgxKgFTmAF7p4upIxLU5CtBR43CIiJDSPdVSQVFlYxKjqahvRObyw1ATnosNW0O5oztOsepNwdaigwWCjUiIiHu1GBS22rj3j8WUtdqJzHazNKL0zAZjXxS0cRf9p+kzebk538/zIZ9ZeQXlms3YAkaCjUiIiGuoKiSyiYbj759gHaHk6c2FbO/rInffXACh9ON2WTk0rFJ3DF/HKOTo5iQFsPEzFjqWx0YMGiptgQNhRoRkRDW7nBid7opqW1jWEwEj759gO9cMYZYi4nsVCtvfVxBQ2snEaYwdpc0MDE9jsuzh/G16VkkWSNYlpehTfQkaCjUiIiEIO+U0x/3lLLjSB25mXHUtNjJjLfw/pE6rp02nMRoM5eNS/ad3zQ3O5ljtW1cnp2kHYElKCnUiIiEIO/S7U/Lm3G5PBSVN7FqUTbp8RY8QEObA4MHUmMjfec3bSuuZXRyNNuL6wJdvsh5UagREQkR3RuCvccWfO/K8cRHhTPy87By7dRMFuaksK24lvQ4S4/zm3TUgQQ7hRoRkRDxxz1l/HZHCa/vKfOdpr2tuLZrhCbO4gsr24prmZudTE2rvUeA0ZSTBDuFGhGRIOEdialttflGZLynade22iiqaMLhdFNY2vj5cuwK6lsdvhGa7iMy6XEW37STSKgweDweT6CLGCjNzc3ExcXR1NREbGxsoMsREemTDfvKqG91cKy2jcwEC+UNHXS63dS3OkiNjeS2K8Zw3+v7mT9hGAYMmE1GDlW2sGpRNsnWyECXL3Leevv+rZEaEZEg4e15WbUom/KGDjLjLUzKiCMp2sxFaVb+caiaZGsEEaYwrJEmPqloIiU2gh8XHGL9rhPaFVhCnsYdRUSChLdPpqCoklWLsnn3QDUeYMrIeBraOtn8WQ3DrBGYTUY8eGjqcLDzWD3RZhNtdheRpjCunZoZ6B9DpN8o1IiIBJH8wnJ2HK7H4XTjAbZ9VkNkeBjgIT7ShDUijGV5GeQXVlDT7MAaaSIMA9aIMK1qkpDX51DT0dGBx+MhKioKgJKSEjZs2MDEiRNZvHix3wsUERmq2h1OCooqWZKb1q2h14DB4H2Gh8omG+nxFqIjwrCYTb7Tta/Oy8AAvmMQluVlqClYQl6f/w+/5ppruO666/jOd75DY2Mjs2bNIjw8nNraWn76059y++2390edIiJDhjfM2J1u2mxONhZVce3UzM97YjxcOjaJZXkZABjoSjgLclLYXlznG42JMptYPnNEoH4EkYDoc6Pw3r17mTt3LgB//OMfSU1NpaSkhN/85jc8/fTTfi9QRGSo8e4GbADfZni1rTb+/de7KW/oYO+JBvILywFYPnMEy/Iy2FZcy+LcVI3GyJDW5//729vbiYmJAWDjxo1cd911GI1GZs+eTUlJid8LFBEZapbkprGxqKpHSHnk7QOEGw2s31mK2WTgQEUznU4P1kjTaSM6IkNVn0dqxo0bxxtvvEFpaSkFBQW+Pprq6mrt/SIi4gfenX0B32Z749NiaHc4CQ8zUN/moKyhnY9KG08b0REZyvocah544AG+973vMWrUKC655BIuvfRSoGvUZurUqX4vUEQkVHU/q+lMn+cXllPZaGPly/t4v7gWDwbio8wYDQZSYywAWCNNLMhJwcOQ2UdV5KzOa0fhyspKTp48SV5eHkZjVy7auXMnsbGxTJgwwe9F+ot2FBaRwcS7Q3CSNQIPntM+t0aa2F/WRFWLjX0ljUwdEU+LrZOEaDP17Z3Mu2gY6XGWHq/V9JOEon7dUTgtLY2YmBg2bdpER0cHADNnzhzUgUZEZLDx7hB8eXYS9s+XXtucLuZmJ5NkjWBZXgb3L80hNtJERkIkcRYTz9w0jSW56fz8/z/Vd0ilTtcW6dLnkZq6ujpuvPFG/vGPf2AwGCguLmbMmDH827/9GwkJCfzkJz/pr1ovmEZqRCRQ2h1O8gsrAA9X5/U8Cbv7mU6jk6OxRpowm4zMzU5mW3EtLbZO9pY0cunYJC3TliGp30Zq7rnnHsLDwzlx4oRvAz6A5cuX884775xftSIiIa6gqJIdR+r44Eg9bxZW+PpnoOeZTl3TT1Df6mDtpmLqWx1EmMKYNz7FtzeNiJxZn5d0b9y4kYKCAoYPH97jenZ2tpZ0i4icxdzsZD44WsekjDjsThfvH67D4XSzfOaIHqudvJvsvVlYwUVpMURoN2CRXuvzSE1bW1uPERqv+vp6IiIi/FKUiEio2VZcy0UpMcREhmM2hWEwdB1h4F2y7f3v+l0nyC8sxwO02ZzsL2sKdOkiQaPPoWbu3Ln85je/8X1tMBhwu9088cQTzJ8/36/FiYiEiu7NvFfnZXDp2CQ+qWiistHmm2b6ccEhfr+rlO2f1WEAyhs7yEywsLGoKtDliwSFPo9nPvHEEyxYsIDdu3fjcDj4z//8T4qKiqivr+e9997rjxpFRILKmQ6ijDKbWJybSkFRJXOzk/m4rIkESzh/2F1KTnoM49NiyM2Io83uIjLMgAdYtSi7x3lOInJufQ41kyZN4rPPPuOZZ54hJiaG1tZWrrvuOlauXEl6enp/1CgiEjTaHU4eerOI1g6nr2cGoLbVxsqX95IQbeaDo3VkxFr43YfHabE7qWuzE2MJ54FlucRGhmNzumizOdleXKd9Z0T64Lw6z+Li4vjBD37g71pERIJeQVElrXYXlc0237V2h5O7Xv2IhjYHR2rauP2KsRRXtWA2hYHNSZjRSG5GHAA2pwuH04U10qQRGpE+6nOo2bp16zkf/9KXvnTexYiIBLsluWk4nG4A3xLsgqJKpo2I5/e7S7k4M47iqhYmZsbSZOukrtXBsrwMrp8+3Lfs2wDMG5+iFU8ifdTnPzHz5s077ZrBYPB97nK5LqggEZFg5e2l6R5mluSmsSQ3jUePH+CmS0ay83g9I5OiMWAgKTqC/752ElFmk6/XxhuINEoj0nd93lG4qann8sLOzk727dvHD3/4Qx555BEWLFjg1wL9STsKi0h/Wr/rBP84WEVNs50wo4HLxw0jLT4SD+BwuogwhbEgJ4XtxXW+vplTz31SD43I6Xr7/t3nkZq4uLjTri1atAiz2czq1avZs2dPX28pIhLUvCM0DqeLQ5WtNLY7iDKb2Hm8nmVT0tlxpB57p4tmm5MFOSm+DfY2FlX5RmS6fy4i5+e8DrQ8k9TUVA4dOuSv24mIBI2CokrfcQbfmD2SscOiuWRUAj/71ykszEklMtxIbauDyHAjazcVA/h2EY4ym3p8LiLnr89/gj7++OMeX3s8Hk6ePMnjjz/OlClT/FWXiMigdKY9aOZmJ7N2UzGrFmWzrbiWqyZnkGSNIMps6lrebXeyYEIK2w7XctsVYwL8E4iErj6HmilTpmAwGDi1FWf27Nm8+OKLfitMRGQwKiiqpLLJxqNvH/CFmBZbJy12J+8eqGZZXgZvFlZgc7rILyyn1eakvMFGXYudK8an8P6ROswlDT1CkYj4R5//RB07dqzH10ajkWHDhhEZGem3okREBpt2h/PzkOJi86Fq5oxJ5sl3DtFqd3KivoPGdgcXZ8YRZTZhNhmpb3VgjTTxpYuGsa+kgfR4C+UNHSREm6lvdbCxqEpNwSJ+1udQM3LkyP6o4ws99thj/OlPf+LgwYNYLBbmzJnDj370I8aPHx+QekRkaMkvLGf9zlLq2xzkZsRysrmDTrebssYOGtrteDxw4GQz63ed8G2e5z1de1leBhuLqrj1S11TT2oKFukfvQo1Tz/9dK9veNddd513MeeyZcsWVq5cycyZM3E6ndx///0sXryYTz/9lOjo6H75niIytJypX+afuvbjireEE2cJZ2JGLB8eqWfcMCvLLs6guLqFiRmxZ9w8z9sI7KURGpH+0at9akaPHt27mxkMHD169IKL6o2amhpSUlLYsmXLWXcxttvt2O1239fNzc1kZWVpnxoROaMN+8p67BdT22rjqU3FfOeKMRQUVVFQdJIvj08lOjKMwtJG7E4Ps0YnEm4yAh4W5qTy1/2VFJU38b0rx5Ns1bS8iD/4dZ+aU/toBgPvJoCJiYlnfc5jjz3GQw89NFAliUiQW5KbxsaiKi7PTmLDvjI+OFpHfauD7/3hY042deBwuvjj3jImpMfi6HRxvK6diemxbPy0ErfHw0cljUzMjKOj0827B6p9h1mKyMDw2z41A8ntdrNq1Souu+wyJk2adNbnrVmzhqamJt9HaWnpAFYpIoNdu8PJhn1ltDucQNc00eLcVB77ywF+9d5xRiVGkxobyRUXDcPldtPU0Uldqw2DBxraHZjCDPz9UDVgoLG9k1a7i6KKJrqdHCMiA+i81hOWlZWRn5/PiRMncDgcPR776U9/6pfCzmXlypV88sknbN++/ZzPi4iIICIiot/rEZHg5N00783CCswmI0ty0ygoqqS4qpWyhg6O1LTy8LWTeCi/iPhoMzXNdpy4OVDZzL9eksVf91fyldw0rJEmHE43ZpPRdwyCGoFFBl6fQ827777L1VdfzZgxYzh48CCTJk3i+PHjeDwepk2b1h819nDHHXfw1ltvsXXrVoYPH97v309EQpd3uqnZ5uD9w3U4nG7mjE1i7d8+w2wy0ulyk19YzqikaErq2rg8exifVDTzlUlpxESG841LR7E4N/W0pmI1AosERp9DzZo1a/je977HQw89RExMDK+//jopKSncdNNNXHnllf1RI9C1c/Gdd97Jhg0b2Lx5c6+bl0VETuXdc8Y7ugL4poyeLDhEVaONTreHulYHr+8uZ2SihR/fkMf7R+qYN2EYEaYw33JtERk8+txTc+DAAW6++WYATCYTHR0dWK1WHn74YX70ox/5vUCvlStX8rvf/Y5XXnmFmJgYKisrqayspKOjo9++p4iEpoKiSnYcrufP+yr44Eg9EaYw5o1PYVleBifqO7C7PLg8sKukns+qmvmsppX3j9TxcVkTbfau07YVaEQGnz7/qYyOjvb10aSnp3PkyBFyc3MBqK2t9W913Tz77LMAzJs3r8f1l156iVtuuaXfvq+IhJ4luWm02pyAhzCjgQU5XXvK5BdWkBEXwbGaMBxuMBk8dLo8hBsMeIDh8RZK6tpIiA6n3eFUsBEZZPr8J3L27Nls376dnJwcrrrqKr773e+yf/9+/vSnPzF79uz+qBHgtLOmRET6wrux3tzsZLYV1+IBShs6SI2NYO2mYiYNj2XzoRo+KWtm7DArwxMtHK5tp6ndwahh0Sz8vAE4wWqmzebUMQcig1Cfp59++tOfMmvWLAAeeughFixYwGuvvcaoUaP45S9/6fcCRUT66tSl2vDPlU5rNxVT3+qgqKKJjLhI6ts7yYy3YMBAXaudVnsndpeHSLOJ/9+MLC4bl8y4lJhuK5o8WCNNWt0kMgj1OdSMGTOGiy++GOiailq3bh0ff/wxr7/+esDOhRIR6c4bYDYWVfmuzc1O5lhtG7ddMYYkawR3zB9HfJSZJ792MenxFhbkpDAiMYqshCjMJgOtHU7MJiMPLsslPc7C4txUCooqabOpp0ZksOpzqPn3f/93Nm/e3A+liIj4x5LcNJKsET1GU/52oIoWm5PNh2rw4OEfh6ppsTt5/0gd107N5G8HqrA73eRmxHLt1OEYDLD3RAPQtUQ7ymw6431FZPDoc6ipqanhyiuvJCsri+9///sUFhb2R10iIufNe4CkdzSl3eFk17EGSurb+Ki0kfpWB5+WN9Nz418DJqORqSMT+Nr04cRHmRmVFN1jtOfU+4rI4NLnUPPnP/+ZkydP8sMf/pBdu3Yxbdo0cnNzefTRRzl+/Hg/lCgi0jtn6qWBrukoW6cLR6ebrZ/V0GpzMi7VCm43O4/VUVrfBni4dGwSy/IyAJg0PJbEaLNGZUSCSK9O6T6XsrIyXn31VV588UWKi4txOp1f/KIA6e0pnyISnNbvKmF7cS0RJiMzRydydV7X6qT8wnI6nR7+9++HcbrdON0e5mYPo6i8CWuEiehIE2mxkcwZm8TymSNOO61bRAKrt+/fF3SgZWdnJ7t37+bDDz/k+PHjpKbqXzQiMnBOHZlxON18Ut7MwZMt/GFXGa9+eIJ///Vuyuo7KCpv4qtTM+hwOLkqN5WYiDC+MXskkzJjWZSTgsvlZu+JBtodTvXOiASp85oY/sc//sErr7zC66+/jtvt5rrrruOtt97iy1/+sr/rExE5q4KiSiobbTz69gFWLcqmqLyZnPQYalvspMZZ2HigisgwI7/fXUZqbARN7Z2MTrZS2mDjhRUzevTcHK9tJzPB4tt/RiM0IsGnzyM1mZmZXHXVVdTW1vL8889TVVXFiy++yIIFCzAYDF98AxERP1mSm0Z5YweZCRZ+XHCIxg4H9e2dPPLVycRbwvnvq3OxO12kxJgxGODi4XFEm03MnzCsx32izCbuX5rjW7otIsGpzz01L7zwAjfccAPx8fH9VFL/UU+NSGjw7g68JDeNdoeTtZuKyYi38PLOEozArFGJ5GTEYTYZeevjk8RZTCRGmZkyMp7txbWcbLRx48wsls8cEegfRUR6obfv332efrr11lsvqDARkQvR7nDyyNsHSImJ4NG3D3BRWgwNbQ4OVbZg8IDT7eFoXTumMCOdbg+R4UaO1bbz6LcnE2U28VFJI+lxlkD/GCLSDy6oUVhEZCB4G4JrW208lF9EdbON/MIKhlkjKCxtYH9FE612BymxkaTFW7A5XRyobGFieixOl4d/mZzO9uI6oswmHrg6l4UTU31Lt0UkdCjUiMig1u5w8lB+Ee9+WsVjfznAp5XN7D3RwMjEKHYer2fssBgAmjtc1LbayYyLpKWjk7pWB2aTkRdWzGBkcrSvV6b7Bnpn29dGRIKTQo2IDGoFRZW0OZyUNXRwrLadhjYH6XEW2h0uLh+XzIm6Ni5KjaHJ1om900VpQwcGg4GEaDNmk/GcuwCf6YwoEQleCjUiMqgtyU1jbvYwxqZYcThd4DEwblg0V12cRqLVzMovj6Okrp20uEhcHhiVFMXFmfGMT43Gg+ecozDaj0YktPRq9VN+fn6vb3j11VdfUEH9SaufRIJDu8NJfmEF4OHqvK5RlvW7Sli/sxQ8MDbFitPtYcbIBD6paKK2zU5LeyfL8jL5yuQ0thfXYXM6abO5tCuwSAjw6+qna6+9tlff1GAw4HK5evVcEZGzKSiqZMeROgyAAQNmk5GFOakYMOBwuvmotBGXy80nFU202p3UNTsYnx7D9dOH+6ab2h1ONhZVaRRGZAjp1fST2+3u1YcCjYicy5kac7tf834+Y2QCkeFGpo9MwO50s/lQDX/dX4nZZMTudHG4phWT0UB2ipVIUxhjU6w6UVtEzu+YBBGR89G9Mdc7JdT9mgcP9a0OnttylItSYjCbjOw8WkdJXRvhBsjJiKPgk0pqW+xEGA14MGAwwKVjk4gwhWlURmSIO69Q09bWxpYtWzhx4gQOh6PHY3fddZdfChOR0LMkN+20KaHu1+pa7dz7+n7+a9lEPj3ZQm2rjZ3H6vEYYNnFGVgjTaTERFDVYuey7GGU1rXR6fHQYuvEExnAH0xEBoU+h5p9+/Zx1VVX0d7eTltbG4mJidTW1hIVFUVKSopCjYiclXdK6NRri3NTyS+sYMPeMiJNYTzw5yKunZbJgZMtON0eHC4PZpMRD1BS34HbA1sPVeP0gMEApfUdjEiMItIUpqZgkSGsz0u677nnHpYtW0ZDQwMWi4UPPviAkpISpk+fzpNPPtkfNYpICOreS1NQVMm24lraHS7KGjuIjzKxvbgW8JAaG4El3IjD6cbhdNHU4aDN7sTucmMAxg2z8tWpmVw6NknTTyJDXJ9Haj766COee+45jEYjYWFh2O12xowZwxNPPMGKFSu47rrr+qNOEQkx3XtpluSmsflQNQ3tDqYMj2fqiATe+eQkSTFmwsPCiLWEs/FAFUsnd+0rEx8F41OtzBydyLK8DDUDiwhwHqEmPDwco7FrgCclJYUTJ06Qk5NDXFwcpaWlfi9QREKTt5fm8uwkCooqMRkN4IGyxg7CjAaO1rTRanNyUVoMFU0dxEea6HR6GJ8Ww6SMON/ybRERrz5PP02dOpVdu3YBcMUVV/DAAw/w8ssvs2rVKiZNmuT3AkUktHinnQAW56by1KZiDp5sZndJI7GR4Xg8sKekAYfTRW2bgwnpsYwZZsViNvFZdYtvVVRBUaXObBKRHnq1o3B3u3fvpqWlhfnz51NdXc3NN9/M+++/T3Z2Ni+++CJ5eXn9VesF047CIoHlPZyyzeFk1uikrt2AW+zsPFYHBgNut4eEKDNut4dmu5NbLx/DyORoLs9OYntxne+/NqeLNptTuwWLDBG9ff/uc6gJZgo1Iv3P2/i7JDfttOmh9btKePXDUjweDxHhRmydburbHLTaOzEajMwanYg1IoxOtwe7002cJZwHl+Wedp/uuwVrCkok9Pn1mAQRkd460wZ70BVEdh2rx+V202xz4mpzExMZTlO7naiIcEYlRfHT5VOIMptodzh59O0DZMZbTrsPnHlpuIhIn0PN6NGjMRgMZ3386NGjF1SQiAS37pvpdR+1KSiqxN7pptXuwul202Zz0dTeSWq8hca2TrISo3o8//6lOTq7SUT6pM/TTz/72c96fN3Z2cm+fft45513+P73v899993n1wL9SdNPIgOn3eHkkbcPMDzBQnqchcuzk3jynUNkp8aw70QD7x+pY1hMBHGWcLISLYxLiaG8oYPRydHqlRGRHvpt+unuu+8+4/Wf//zn7N69u6+3E5EQ4R1lmZudzLbiWuxOFynWCLYcquaqi9P5y/5KOjrdWCNNXH5RMmWNHbhcHkYPi+b7S8azvbiOG2YMZ3txnUZnROS8+K1R+OjRo0yZMoXm5mZ/3K5faKRGpP9s2FdGfauDY7VtZMZbOF7fhs3hZF9pE2NTrJyoa+Oi1BjmjE0m3GSg0+nhk4omRiVFkx5n0ciMiJxVb9+/+7xPzdn88Y9/JDEx0V+3E5EgsyS3a7ffVYuyKW/sIDPewicVzUSajOw53kB6rIXmDifhJgNtNhcxkV0rm9LjLBqZERG/6PP009SpU3s0Cns8HiorK6mpqeH//u///FqciAw+3Zt5gR7Lt72jLfcvzWHN64VdZzQ5PWTERmB3unn2G9PocLj43h8/ZlFOV5DRCI2I+EufQ80111zTI9QYjUaGDRvGvHnzmDBhgl+LE5HBJ7+wnB2H63E43ZhNxh7Lt7sHnopGG00dnRgMRprsTjo/n+le+7fP+KyyhXa7i2Q1BIuIH/U51PzXf/1XP5QhIsHDgMEADqcbD2A2GWm2OVi/qwSH08Oe4w04nG7mT0jlaG0bU0ckcKy2jeHxFtZuKgYMWMKNWMINmnYSEb/qc6gJCwvj5MmTpKSk9LheV1dHSkoKLpfLb8WJyOBzdV4GBmDnsTpsTg/WiDBsnW4MQES4EZfbzd4TDWSnWBmVFM0VFw3jwWUTuff1/dx2xRgs5jDWbipm1aJs7QYsIn7V50bhsy2WstvtmM3mCy5IRALHe9jkmQ6K7H4QpdlkxOZwU97QgcvpZkJqDOUN7aTHWqhqttPY4eDAyWbS4y0UlTfx3pE6FkxIYU9JI8nWSP7nq5NJtkYO9I8nIiGu1/9MevrppwEwGAz84he/wGq1+h5zuVxs3bpVPTUiQcy7WV5KTASPvn2A+5fm9BhJ8R5/8GZhBXanCwNumtsdJFkT+NO+ckxhBt74qJzYSBOHTtr5xuyRvuXdBiDJGqHpJhHpV70ONU899RTQNVKzbt06wsLCfI+ZzWZGjRrFunXr/F+hiAyIgqJKhidY2PxZDcnRZt4srGD5zBG+x3PTY1n5yj6WTk7jbweqOVzdAgZ4fW8Z108fzr4TjVyUEg0GAy6XB2ukqcdRB5pqEpH+1uu/ZY4dOwbA/Pnz+dOf/kRCQkK/FSUiA897ZtOinBT+sr8Sh9MN/HMJ9/qdJ/B4PLy68wThJiMut4dONxgNLupa7Vw3PZM2mwtrZNdfK96Jaq1uEpGB0ueemn/84x8KNCIhyoMHs8nIiMQozKauvx68006LJ6ZhMBj46rRMZoxIICUuAnOYAas5DFOYkavzMkmyRrAsLwOzyUibzcnGoqoA/0QiMpT0OdRcf/31/OhHPzrt+hNPPMENN9zgl6JEZOC0O5ys31XCQ28WUdlow4ABa4SJBTldKxznZidzrLaNxbmpfGvuaEYnW5k9LpmpWQlMSIthTEoM31sy3rf5XpTZ5NtdWD00IjKQ+nz207Bhw/j73//O5MmTe1zfv38/CxcupKpq8P7LTGc/iZxuw74yNh+sweXxEG8JZ9LwONpsTqyRJswmI3Wtdt4sPEmYEdLjLMwek+Q7u8lsMrIsLwOgx2GW3h2GRUT8od9O6W5tbT3j0u3w8PBBfZiliJzZktw0HE43DqebcJOBy8Ym8cw/DmOzO7G7PRw42UxTu5OocCM1rQ4+qWhiQloMX56Q6msk9h5muXZTMaOTo307DIuIDKQ+Tz9NnjyZ11577bTr69evZ+LEiX4pSkT6V22rjR9s2E9tq40os4nlM0dgjTTR0NrJva/vp7Hdwb7SJvadaOBkow1zGNjdHtzurvBzqLK1x/26H2apaScRCZQ+j9T88Ic/5LrrruPIkSN8+ctfBuDdd9/l1Vdf5Q9/+IPfCxQR//txwSGKypt5suAQj1+fR7vDSYutk78frCY+KpyaFgcWcxhlVR0YDVDb1sno5ChiIsIxmYx8JTfNN+0E9DjMUiM0IhIofQ41y5Yt44033uDRRx/lj3/8IxaLhYsvvpi//e1vXHHFFf1Ro4j4WXZKDB8erSc7JQbo6od5r7iWw9XNYDAyZ0wiHx5rxxJuoL3TQ5TZQGO7kylZ8UxMjyPJGqGeGREZdM7rb6WlS5eydOnS065/8sknTJo06YKLEpH+4d1zBjzER4Wzv7SB9btOsDAnhfU7T+DyGOiwd1LeaONbl4/m74eqSYwKp6zJhiU8jFULL2JPSaOml0RkUOpzT82pWlpaeP7557nkkkvIy8vzR00i0k+8e84cONlCXauDrYfr2PhJJXe9+hH/fc0kkq3hRJlNON1uyhs7+PW/XcIj100mLjKcL2Uns6ek0bdsW0RksDnvULN161Zuvvlm0tPTefLJJ/nyl7/MBx984M/aROQCnXpApbehNy8rDoMBwowe3jtcy5HqFp7feoTff2cOX5mUxvi0WDLjLWwsqmJbcS1zxyVT3WzXCI2IDGp9+udWZWUlv/rVr/jlL39Jc3MzN954I3a7nTfeeEMrn0QGEe80k93pos3m4s3CCswmI3Ozk/Hg4arJ6Rgw8Kv3jwFOGto7OVrbTpTZxONf62ocfrOwApvTycKcVLYX17HislEaoRGRQa3Xm+8tW7aMrVu3snTpUm666SauvPJKwsLCCA8Pp7CwMChCjTbfk1DnDTMttk72HG9kxqgEYiLDsTmdtNlcfFbdQl2bgyPVrdwwfThmUxh/2X+SyiYbSdFmbpyZRbjJABgAD202F0nWCK1oEpGA8vvme3/961+56667uP3228nOzvZLkSLiX96emc+qWzAYwGwysjg3lVc+PMGmTyu54qIU/vZpFa02J8/8vZhEawTTRyQwOjkKu9PDJxVN2BxuDAaYPjKBY7VtXDM144u/sYjIINDrnprt27fT0tLC9OnTmTVrFs888wy1tbX9WZuI9JG3Z+b7S8Yzb3wKy/IyKCiq5M8flXOosoWCokqGRUfgdHtwu6G5o5N9pY1MyUpgYU4q318ynjnjkrh0bNdRCKOTo9leXBfoH0tEpFd6HWpmz57NCy+8wMmTJ7nttttYv349GRkZuN1uNm3aREtLS3/WKSJn4W0GLq1v45G3D3B5dhLJ1kgW56aSX1hOXaudpnYH9k4X1c0dHKtrw2QEo9FAYrSZiemxmE1Grp2aSbI1kuUzR7B85gjfqdtqDhaRYNHnAy27O3ToEL/85S/57W9/S2NjI4sWLSI/P9+f9fmVemok2Hl7ZrofGOk9d+ndg9VEm8NIijYzZWQCu47WUVzVSqOtk6jwMMoaOoiNCsPpgna7i1FJUXx12nBfcFETsIgMVr19/76gfWrGjx/PE088QVlZGa+++uqF3EpEeqGgqJLKRhuPvn3At0x7bnYyx2rbuPfK8bTYnLTYnbzyQQkfHmugrLEdq9lETZsda2QYKVYLWQlRLMhJYVxqDNZIk/adEZGQcUEjNcFGIzUS7NodTh59+wCZCRbS4yxcOzXTN1JzrLaNhjYHheWNtNg6cTjdjEqOJi7STLTFxJGqVr4xeyRJ1gguz05ie3GdRmhEJCgMyEiNiPSvUzfPizKbuH9pDulxFi7PTmLDvjJmfL5K6YYZw/n0ZDNut4cwjMRZzDicHn58w8Uszknl9nljuWZqBh48vgMoFWhEJJQEXaj5+c9/zqhRo4iMjGTWrFns3Lkz0CWJ9BvvEu03CytYv+sE63eVAF0nYW8rrqWy0ca9r+8nM97Cj945hCU8DFOYkS9dlMSopGi+MimN57YcZVleBstnjmBbcS31rQ42FlUF+CcTEfG/oAo1r732GqtXr+bBBx9k79695OXlsWTJEqqrqwNdmki/8C7R9gA7jtTxwZF6NhZVUdtq44OjdRysbCLcaOCl949z94JxXJwVx61zx/DdJRMYmRxFaX07Na02HnqziHaH03c/rWgSkVAUVD01s2bNYubMmTzzzDMAuN1usrKyuPPOO7nvvvu+8PXqqZFg5T22AGDO2CRueWknpjADsZFmjtW2EWYwMDzBginMwKKJqRRXt1Lf6qDF5qTT5SY93sLCnFTtDCwiQcnvOwoHmsPhYM+ePaxZs8Z3zWg0snDhQnbs2HHG19jtdux2u+/r5ubmfq9TpD9EmU2+jfSe+fthTEYjda0Obpw+AoCCT05S1WLH1unmdx+c4FuXj+ZQZQsPLBvD+0e6Ns/T6IyIhLqgCTW1tbW4XC5SU3v+xZyamsrBgwfP+JrHHnuMhx56aCDKE/GbdoeT/MJywMDVeV1HFBQUVXKy0cbvd5fy1amZGAyQmxHHVyansa24lutmZFHe0M7bhSeZPLzr+jcuHQXA8sTowP0wIiIDKGhCzflYs2YNq1ev9n3d3NxMVlZWACsS+WIFRZXsOFyPy+Phk7ImLkqzsvNYPR8cqcPh8rC9uJY/3D6HdoeTR94+wPB4C4lWM2OHxfDteRbabC62F9dpqklEhpygCTXJycmEhYVRVdVz1UZVVRVpaWlnfE1ERAQREREDUZ6I3yzJTcPhdLP3RAOZ8RYKS5v4uKyJ8DAjHtxcOanr//eCokqGJ1gob+hgxWWjiDKbaHc42VhUpakmERmSgmb1k9lsZvr06bz77ru+a263m3fffZdLL700gJWJ+Fe7w8nekgayU6wkWs3kZcURbwknLMzA/PEp/Ousrj6aJblpJEabmTT8n01z2n9GRIayoPqbb/Xq1axYsYIZM2ZwySWXsHbtWtra2vjmN78Z6NJELkj3Ppp9JQ0UVTRzuLqVqyans/HTSkYkRDIiKZq8rPgeZz+ZTUbfvjOabhKRoS6oQs3y5cupqanhgQceoLKykilTpvDOO++c1jwsEmwKiirZXlzLyUYbV01OxwBkp8bwys4SDAYDHg+MHmbmo9JGxqVYfSFmSW6apptERD4XVPvUXCjtUyODVbvDycP5RTR2dBJvCeeBq3MpKKrkYEUzbxRWsOLSURyqasHlcmONNDFlRDxX52maSUSGhpDbp0YkWLU7nL4pI8A3zbQwJ4W/HajC4XQDBsalWvnLxyepaOrg9T1lXD99OHuON3DLnFEkRpuZY00CwIOHNptLU04iIqdQqBHpZ97zmzYWVWFzuvj9rjIy4i18UtZEi81JWUM7HsDt9lBS347H46GwtJHrpw/norQYiiqaWDAhhV/vKOGeRdlEmU2achIROYOgWf0kEky6n67d87wlDxlxkVgjwli1KJs545JYODEVD10jMOkxEcRazL6G4J1H6zhc1coP/1xEdbONtZuKtcJJROQs9LeiSD/oPjpz7dRMFuemUlBUycKcVAwY8NC1/Hr5zBH8YMN+7J0uWu1Opo1IINJk5CuTu1Y3fXikjvR4C1Oz4jlW28aqRdmB/tFERAYtNQqL9APvJniXZyexrbgWu9NNm835+YnbHupbHSRZI7h2aia1rTZu/+1e7E4XF6VYGZsSQ3ljB/cvzQHwTTVpZEZEhqrevn9r+kmkH3iniP52oJrNh2rodLp9U1A9p6Mg2RrJ9TOGM2aYlbwRCZQ3dpCZYGFjUZWmmkRE+kChRqRfeXC5PHxU2oDN6fRtsmdzuno8a2FOCtYIE/PGD2PS8DgSo81qBBYR6SP980+kH12dl8nuo/XsPN5AfVsnL2w9RnREGCMSozHQ1RzM55+NTo7muS1Hyfz8PKdln5/QLSIivaORGhE/677yKcpsIsxkJMocxq7j9dS02CitayPSZMQD7Djcdfq2AQNJ1ghWLcqmvKGDzPiu6ScREek9jdSI+Fn3lU+Lc1PJzYijpLYdoweO1bdhCjeRkx4LeJgxKgGzyciyvAxf38z9S3O0D42IyHlQqBHxo3aHE7vThTXS5FvG3el089VpmTicbvILK0iymvmsuoWLUmKwRnYdStmdtzlYRET6RqFG5Dx1Nf1WAB4W5qR+vnTbRZvNRZI1giizibnZyazdVMyqRdldq5ymD/ct9d5eXIfN6dQp2yIifqJ9akTO0/pdJ/j9rlIy4i1EmIwcq21j6eR0rJEmWmydFFe3kp1iZX9ZM3PGJbF85ojT7uHdz0b70IiInJ32qRHpdx4y4ixYzWE4XW6qm+0cONmE3eli3Zaj7DvRwKs7T+A6x78btA+NiIj/KNSI9EL3FU3er8HA3IuS+d6V4zGFGUi2msnLSqCovJnEqHAqm2xckT2M+jYHC3JSAvsDiIgMAZp+EumFDfvKqG91+Bp7vb0z1kgTe0800NjhIMFi5oFlubQ7nNz16kdMGxHP3hONzB6TSHWznfuX5mhERkTkPGj6SeQ8nToqA/iONvAAlY029p1oxBppwgM0djg4dLKV3Iw4oswmkq2R/GLFDMalxPD0v06hutmufWdERAaAQo3IKbrvM1PbauMHG/ZT12rH5nThcLo4Xt/GqKRoIkxhXJ2XQbzFzKTMuB5Ls729MsnWSO5fmkN6vEX7zoiI9DOFGpFTdD9w8qlNxZxs7OCWl3byygcl/GF3GU6XG2uECZuzayTn+0vGE28JP2ffTNdxCCIi0p8UakROEWX+58Z537liDDanm5GJ0ZQ3dtDQ3ond6eFQZQttNhcbi6rYVlzL6ORothfXnfF+3Ud+RESk/yjUiJxBfmE5mw/W8P6ROn6xYgbDYiOYOSqR2aMSmJudzKpF2b7RnO4jO2fyRY+LiIh/aCmGDHntDicFRZUsyU0DukZWHE43BkPX41FmE99fMr7HzsBAjx2Az7UbsI49EBEZGBqpkSGv+/SQ9/MIUxjzxqewLC8DgG3FtWQmWFi7qZjaVttpq6NERCTwFGpkyOs+PeT9fFlehm90Zf2uE7TYOjle10ZmfFewUY+MiMjgo1AjQ1736aH8wnKabZ3kF1b4pqW2Fdfw+p5ynC4PiVZzj34aEREZPNRTI/K5gqJKNh+sZndJIzNHJWAA7M6uM508eGh3ONlf1tRjFEdERAYPjdTIkHSmXYPnZidTXN1GYlQ4da129pY08OHROlJjIpiYFkusJVw7A4uIDGIaqZEhpbbVxlObihmfFkOn083GoirfqMu24lquvjiDncfrWTIplTa7i5K6NqaOSPA1DG8sqtK0k4jIIKVQI0PKU5uKqW624XK7mT0mmcW5qb7embnZyWynjluvGAN0BZgVc0YB+JZ8a9pJRGTw0vSTDCnfuWIM7Q4XK+eP49qpmUSZTb5l3NuL63zXvM3D3R/XtJOIyOCmUCNDRrvDybotR7l8XDJ7Shp917UjsIhIaND0k4S82lYbP37nEE63m5FJXWc4rbhslO/xL9rxVzsCi4gEB4UaCVntDif5heVs2FtOS4cTs8mIJdzE/UtzaHc4eejNInIzYvna9CyizPqjICIS7PQ3uYQcb+Ov3elmx+F6EqPNGDCw9OJ0rp8+nCiziUfePkBReTOHq1qJjTSfNhLT/TwoBR4RkeCgv60l5Hgbe62RJuaMSwJgWV5Gj3Byz6Jsniw4RG5G3Bl7Zbo3B2vqSUQkOKhRWELO3OxkjtW2MWdsEmaT0bfHTPfN9pKtkTx+fR7fuHTUGUdi1BwsIhJ8FGokaJ1pV2Do2kRvdHI0z205etrp295l2Wd7rVf3Jd0iIhIcFGokaJ1t/5gZIxN492A1N1868rTTt70jL9p7RkQk9CjUSNDyBpXpI+P5wYb91LbaAFi35SjR5jB+s6PkjJvpdX+tppdEREKHQo0ELW9QWbflKNXNNtZuKga6moCTos2MSo5i/a4TZ5xi0vSSiEjoUaiRoHNqP8w9i7JJjY3k5ktH8oMN+wGYNTaJ/WXNfHCkTlNMIiJDhEKNBJ3u/TDtDifbimu5f2kOv95R4huxWZKbxpxxSUwfmYDN6TxrQ7CIiIQOhRoJCt1HZ7r3wxQUVVLZaOPRtw/wnSvGkBobyapF2USZTSyfOQJrpIk2m0ujNSIiQ4BCjQSFgqJKKpu6wgvgCzRzs5Mpb+wgM8HCnpJG7l+aw7biWt/IjBqCRUSGDoPH4/EEuoiB0tzcTFxcHE1NTcTGxga6HOmDdoeTR98+QGa8hfR4Cx481Lc6fIFlY1GVL+h4r2snYBGR0NDb92+N1Mig0JvN8O5fmkN6vOWM+8546MrmGpkRERm6FGpkQJ0tvOQXlrP5YA1vFlac9bXdl2F3/7x747CWaouIDF0KNTKgzr6TrwGD4Z9f1bbaemyody4anREREVCokQHmDSCXZyf1GLG5Oi+DeeNTfIdPPrWpuMeGeucKORqdERERUKiRAeYNINuKa3uM2JwaTLwb6q1alA2cHnJEREROpVAjAfFFU0bJ1kj+56uTSbZGAqeHHBERkVNpSbeIiIgMalrSLSIiIkOKQo0MiC/ah6a3zxERETkbhRoZEGdfyt3zOd5znBRsRESkrxRqpN+c7RDKs43ILMlN853jpAMoRUSkrxRqpN+cutOv92ym/MKKM47a+I5CiLNoIz0REekzhRrpN6cu2/aGHAOcdTm3NtITEZHzpVAjftd9esl70CT8M+Qsy8tQcBEREb/Tu4r4XX5hOTsO1/PeZzV0usHhdLN85gjfKIyIiEh/CIqRmuPHj/Otb32L0aNHY7FYGDt2LA8++CAOhyPQpckp2h1O9p1oxO5yUVLfgesseztq+baIiPhbUISagwcP4na7ee655ygqKuKpp55i3bp13H///YEubcj5ojBSUFTJqKRoWjqcXD4umXhLuO+QylOf90VLvEVERPoiaI9J+PGPf8yzzz7L0aNHe/0aHZNw4TbsK6O+1UGSNYJrp2bS7nBSUFTJktw0oswm2h1ONhZVcXl2EtuL61icm3rG3hnv8872uIiIiFfIH5PQ1NREYmLiOZ9jt9tpbm7u8SEX5mwrmk49bTvZGnnOZmCtchIREX8LylBz+PBh/vd//5fbbrvtnM977LHHiIuL831kZWUNUIWh69QwMjc7mWO1bVyenRTgykREZKgLaKi57777MBgM5/w4ePBgj9eUl5dz5ZVXcsMNN3Drrbee8/5r1qyhqanJ91FaWtqfP05I6t5DU9tq4wcb9lPbavM9vq24ltHJ0WwvrgtglSIiIgHuqampqaGu7txvhmPGjMFsNgNQUVHBvHnzmD17Nr/61a8wGvuWydRT03fde2h2Ha+nutlGamwk//PVyYB6Y0REpP/19v07aBqFy8vLmT9/PtOnT+d3v/sdYWFhfb6HQk3fdQ8t7Q4nazcVs2pRNsnWyECXJiIiQ0RIhZry8nLmzZvHyJEj+fWvf90j0KSlpfX6Pgo1IiIiwae3799BMV+wadMmDh8+zOHDhxk+fHiPx4Igk4mIiMgACIrVT7fccgsej+eMHyIiIiIQJKFGBt6ZVjqJiIgMZgo1ckZPbSqmutnG2k3FgS5FRESkVxRq5IzuWZRNamwkqxZlB7oUERGRXgmKRmEZeMnWf+5FIyIiEgw0UiMiIiIhQaFGREREQoJCjYiIiIQEhRoREREJCQo1IiIiEhIUakRERCQkKNSIiIhISFCoERERkZCgUBMC2h1ONuwro93hDHQpIiIiAaNQEwIKiiqpb3WwsajKd6170FHoERGRoUChJgQsyU0jyRrB4txU37XuQedMoUdERCTUKNSEgCiziWunZhJl/udRXt2DzplCj4iISKjRgZYhyht0vLp/LiIiEoo0UhMi1DcjIiJDnUJNiFDfjIiIDHUKNYNUX0de1DcjIiJDnULNINXXkZczNQuLiIgMJQo1g5RGXkRERPpG/6wfpE5dvSQiIiLnppGaQUwrmkRERHpPoWYQyy8sZ/PBGt4srAh0KSIiIoOepp8GoXaHk4KiShxODwZDoKsREREJDhqpCZBzTS15Vz5FmIzMG5/CsryMAFQoIiISXBRqAuRcS7a9K5+W5WVombaIiEgvKdQEyLmWbGvPGRERkb7Tu2aAaMm2iIiIf2mkRkREREKCQo2IiIiEBIUaERERCQkKNSIiIhISFGpEREQkJCjU+IHOaBIREQk8hRo/ONdGeiIiIjIwFGr84Fwb6YmIiMjA0OZ7fqCN9ERERAJPIzUiIiISEhRqREREJCQo1IiIiEhIUKgRERGRkKBQIyIiIiFBoUZERERCgkKNiIiIhASFGhEREQkJCjUiIiISEhRqREREJCQo1IiIiEhIUKgRERGRkKBQIyIiIiFhSJ3S7fF4AGhubg5wJSIiItJb3vdt7/v42QypUNPS0gJAVlZWgCsRERGRvmppaSEuLu6sjxs8XxR7Qojb7aaiooKYmBgMBkOgywkZzc3NZGVlUVpaSmxsbKDLGTL0ew8M/d4DQ7/3wBgsv3ePx0NLSwsZGRkYjWfvnBlSIzVGo5Hhw4cHuoyQFRsbq79sAkC/98DQ7z0w9HsPjMHwez/XCI2XGoVFREQkJCjUiIiISEhQqJELFhERwYMPPkhERESgSxlS9HsPDP3eA0O/98AItt/7kGoUFhERkdClkRoREREJCQo1IiIiEhIUakRERCQkKNSIiIhISFCoEb85fvw43/rWtxg9ejQWi4WxY8fy4IMP4nA4Al1ayPn5z3/OqFGjiIyMZNasWezcuTPQJYW8xx57jJkzZxITE0NKSgrXXnsthw4dCnRZQ8rjjz+OwWBg1apVgS5lSCgvL+frX/86SUlJWCwWJk+ezO7duwNd1jkp1IjfHDx4ELfbzXPPPUdRURFPPfUU69at4/777w90aSHltddeY/Xq1Tz44IPs3buXvLw8lixZQnV1daBLC2lbtmxh5cqVfPDBB2zatInOzk4WL15MW1tboEsbEnbt2sVzzz3HxRdfHOhShoSGhgYuu+wywsPD+etf/8qnn37KT37yExISEgJd2jlpSbf0qx//+Mc8++yzHD16NNClhIxZs2Yxc+ZMnnnmGaDrTLOsrCzuvPNO7rvvvgBXN3TU1NSQkpLCli1b+NKXvhTockJaa2sr06ZN4//+7//4n//5H6ZMmcLatWsDXVZIu++++3jvvffYtm1boEvpE43USL9qamoiMTEx0GWEDIfDwZ49e1i4cKHvmtFoZOHChezYsSOAlQ09TU1NAPr/ewCsXLmSpUuX9vj/XvpXfn4+M2bM4IYbbiAlJYWpU6fywgsvBLqsL6RQI/3m8OHD/O///i+33XZboEsJGbW1tbhcLlJTU3tcT01NpbKyMkBVDT1ut5tVq1Zx2WWXMWnSpECXE9LWr1/P3r17eeyxxwJdypBy9OhRnn32WbKzsykoKOD222/nrrvu4te//nWgSzsnhRr5Qvfddx8Gg+GcHwcPHuzxmvLycq688kpuuOEGbr311gBVLtI/Vq5cySeffML69esDXUpIKy0t5e677+bll18mMjIy0OUMKW63m2nTpvHoo48ydepUvv3tb3Prrbeybt26QJd2TqZAFyCD33e/+11uueWWcz5nzJgxvs8rKiqYP38+c+bM4fnnn+/n6oaW5ORkwsLCqKqq6nG9qqqKtLS0AFU1tNxxxx289dZbbN26leHDhwe6nJC2Z88eqqurmTZtmu+ay+Vi69atPPPMM9jtdsLCwgJYYehKT09n4sSJPa7l5OTw+uuvB6ii3lGokS80bNgwhg0b1qvnlpeXM3/+fKZPn85LL72E0ajBQH8ym81Mnz6dd999l2uvvRbo+hfVu+++yx133BHY4kKcx+PhzjvvZMOGDWzevJnRo0cHuqSQt2DBAvbv39/j2je/+U0mTJjAvffeq0DTjy677LLTtiz47LPPGDlyZIAq6h2FGvGb8vJy5s2bx8iRI3nyySepqanxPaZRBP9ZvXo1K1asYMaMGVxyySWsXbuWtrY2vvnNbwa6tJC2cuVKXnnlFf785z8TExPj62GKi4vDYrEEuLrQFBMTc1rPUnR0NElJSepl6mf33HMPc+bM4dFHH+XGG29k586dPP/884N+9F2hRvxm06ZNHD58mMOHD582LK+dA/xn+fLl1NTU8MADD1BZWcmUKVN45513TmseFv969tlnAZg3b16P6y+99NIXTs+KBJuZM2eyYcMG1qxZw8MPP8zo0aNZu3YtN910U6BLOyftUyMiIiIhQQ0PIiIiEhIUakRERCQkKNSIiIhISFCoERERkZCgUCMiIiIhQaFGREREQoJCjYiIiIQEhRoREREJCQo1IjKo3XLLLb5zrqBrR99Vq1Zd0D39cQ8RGXwUakTkvNxyyy0YDAYMBgNms5lx48bx8MMP43Q6+/X7/ulPf+K///u/e/XczZs3YzAYaGxsPO97iEjw0NlPInLerrzySl566SXsdjt/+ctfWLlyJeHh4axZs6bH8xwOB2az2S/fMzExcVDcQ0QGH43UiMh5i4iIIC0tjZEjR3L77bezcOFC8vPzfVNGjzzyCBkZGYwfPx6A0tJSbrzxRuLj40lMTOSaa67h+PHjvvu5XC5Wr15NfHw8SUlJ/Od//udph6GeOnVkt9u59957ycrKIiIignHjxvHLX/6S48ePM3/+fAASEhIwGAy+gydPvUdDQwM333wzCQkJREVF8ZWvfIXi4mLf47/61a+Ij4+noKCAnJwcrFYrV155JSdPnvQ9Z/PmzVxyySVER0cTHx/PZZddRklJiZ9+0yLSGwo1IuI3FosFh8MBwLvvvsuhQ4fYtGkTb731Fp2dnSxZsoSYmBi2bdvGe++95wsH3tf85Cc/4Ve/+hUvvvgi27dvp76+ng0bNpzze9588828+uqrPP300xw4cIDnnnsOq9VKVlYWr7/+OgCHDh3i5MmT/OxnPzvjPW655RZ2795Nfn4+O3bswOPxcNVVV9HZ2el7Tnt7O08++SS//e1v2bp1KydOnOB73/seAE6nk2uvvZYrrriCjz/+mB07dvDtb38bg8Fwwb9TEek9TT+JyAXzeDy8++67FBQUcOedd1JTU0N0dDS/+MUvfNNOv/vd73C73fziF7/wvdm/9NJLxMfHs3nzZhYvXszatWtZs2YN1113HQDr1q2joKDgrN/3s88+4/e//z2bNm1i4cKFAIwZM8b3uHeaKSUlhfj4+DPeo7i4mPz8fN577z3mzJkDwMsvv0xWVhZvvPEGN9xwAwCdnZ2sW7eOsWPHAnDHHXfw8MMPA9Dc3ExTUxP/8i//4ns8Jyen779IEbkgGqkRkfP21ltvYbVaiYyM5Ctf+QrLly/nv/7rvwCYPHlyjz6awsJCDh8+TExMDFarFavVSmJiIjabjSNHjtDU1MTJkyeZNWuW7zUmk4kZM2ac9ft/9NFHhIWFccUVV5z3z3DgwAFMJlOP75uUlMT48eM5cOCA71pUVJQvsACkp6dTXV0NdIWnW265hSVLlrBs2TJ+9rOf9ZiaEpGBoZEaETlv8+fP59lnn8VsNpORkYHJ9M+/UqKjo3s8t7W1lenTp/Pyyy+fdp9hw4ad1/e3WCzn9brzER4e3uNrg8HQo9/npZde4q677uKdd97htdde4//9v//Hpk2bmD179oDVKDLUaaRGRM5bdHQ048aNY8SIET0CzZlMmzaN4uJiUlJSGDduXI+PuLg44uLiSE9P58MPP/S9xul0smfPnrPec/LkybjdbrZs2XLGx70jRS6X66z3yMnJwel09vi+dXV1HDp0iIkTJ57zZzrV1KlTWbNmDe+//z6TJk3ilVde6dPrReTCKNSIyIC46aabSE5O5pprrmHbtm0cO3aMzZs3c9ddd1FWVgbA3XffzeOPP84bb7zBwYMH+Y//+I/T9pjpbtSoUaxYsYJ/+7d/44033vDd8/e//z0AI0eOxGAw8NZbb1FTU0Nra+tp98jOzuaaa67h1ltvZfv27RQWFvL1r3+dzMxMrrnmml79bMeOHWPNmjXs2LGDkpISNm7cSHFxsfpqRAaYQo2IDIioqCi2bt3KiBEjuO6668jJyeFb3/oWNpuN2NhYAL773e/yjW98gxUrVnDppZcSExPDV7/61XPe99lnn+VrX/sa//Ef/8GECRO49dZbaWtrAyAzM5OHHnqI++67j9TUVO64444z3uOll15i+vTp/Mu//AuXXnopHo+Hv/zlL6dNOZ3rZzt48CDXX389F110Ed/+9rdZuXIlt912Wx9+QyJyoQyeUzeBEBEREQlCGqkRERGRkKBQIyIiIiFBoUZERERCgkKNiIiIhASFGhEREQkJCjUiIiISEhRqREREJCQo1IiIiEhIUKgRERGRkKBQIyIiIiFBoUZERERCwv8H4Pg8DD4vxm8AAAAASUVORK5CYII=\n"
},
"metadata": {}
}
],
"source": [
"# plot the relationship between predicted and actual test values\n",
"ypred = model1(torch.from_numpy(np.float32(x_test))).detach().numpy()\n",
"plt.scatter(ypred,y_lin_test,s=.1)\n",
"plt.xlabel(\"Predictions\");\n",
"plt.ylabel(\"Actual values\");\n",
"\n",
"# print out performance metrics\n",
"print(r2_score(y_lin_test,ypred))\n",
"print(np.sqrt(mean_squared_error(y_lin_test,ypred)))"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "OFrpjeWzodA3"
},
"source": [
"Now let's compare the results to an ordinary least squares (OLS) regression - the more traditional way to model this sort of data in psychology and neuroscience. First, we'll print out the weight and bias of the sole unit in the neural network, and then fit and print the equivalent parameters (slope and intercept) of the OLS regression, using the statsmodels package."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/"
},
"id": "_n_TKrIcivpL",
"outputId": "024612e2-9d6a-4346-96bc-ec69653803ed"
},
"outputs": [
{
"output_type": "stream",
"name": "stdout",
"text": [
"weight tensor([[1.5001]])\n",
"bias tensor([1.0004])\n"
]
}
],
"source": [
"# print out weight and bias parameters\n",
"for name, param in model1.named_parameters():\n",
" if param.requires_grad:\n",
" print(name, param.data)"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/"
},
"id": "IT5vuyDPm_Ls",
"outputId": "7e3f3d88-364a-4fed-da39-b6e8e744fa45"
},
"outputs": [
{
"output_type": "stream",
"name": "stdout",
"text": [
" OLS Regression Results \n",
"==============================================================================\n",
"Dep. Variable: y R-squared: 0.996\n",
"Model: OLS Adj. R-squared: 0.996\n",
"Method: Least Squares F-statistic: 2.242e+07\n",
"Date: Fri, 04 Aug 2023 Prob (F-statistic): 0.00\n",
"Time: 03:48:42 Log-Likelihood: 88450.\n",
"No. Observations: 100000 AIC: -1.769e+05\n",
"Df Residuals: 99998 BIC: -1.769e+05\n",
"Df Model: 1 \n",
"Covariance Type: nonrobust \n",
"==============================================================================\n",
" coef std err t P>|t| [0.025 0.975]\n",
"------------------------------------------------------------------------------\n",
"const 1.0004 0.000 3166.125 0.000 1.000 1.001\n",
"x1 1.5001 0.000 4735.113 0.000 1.499 1.501\n",
"==============================================================================\n",
"Omnibus: 0.830 Durbin-Watson: 2.004\n",
"Prob(Omnibus): 0.660 Jarque-Bera (JB): 0.840\n",
"Skew: 0.006 Prob(JB): 0.657\n",
"Kurtosis: 2.991 Cond. No. 1.00\n",
"==============================================================================\n",
"\n",
"Notes:\n",
"[1] Standard Errors assume that the covariance matrix of the errors is correctly specified.\n"
]
}
],
"source": [
"# observe that a traditional OLS regression learns the same parameter values\n",
"ols = sm.OLS(y_lin_train,sm.add_constant(x_train))\n",
"res = ols.fit()\n",
"print(res.summary())"
]
},
{
"cell_type": "markdown",
"source": [
"As you can see, despite the different implementation, an ANN can yield effectively identical results as familiar OLS regression, under idealized circumstances."
],
"metadata": {
"id": "mhrZZ0EjQFWQ"
}
},
{
"cell_type": "markdown",
"metadata": {
"id": "6I6mIhdCo7l2"
},
"source": [
"## Nonlinear regression\n",
"In the previous section, we saw how an ANN can be used to approximate a simple linear regression. However, one of the main advantages of ANNs is that they can model highly nonlinear relationships. In this section, we'll start to see how that can be achieved."
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "1nppLlORpfAR"
},
"source": [
"### Trying the linear model on nonlinear data\n",
"First, we will try to fit our simulated nonlinear (bivariate quadratic) data using the same model architecture we tried in this last section. This time we'll code up that model in a different way (i.e., a more object-oriented approach, instead of the more imperative approach above)."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/"
},
"id": "o9xuB5FbjloO",
"outputId": "ebefcaf1-3216-4d12-8250-14e0223822ae"
},
"outputs": [
{
"output_type": "stream",
"name": "stdout",
"text": [
"SimpleLinearANN(\n",
" (layer_stack): Sequential(\n",
" (0): Linear(in_features=1, out_features=1, bias=True)\n",
" )\n",
")\n"
]
}
],
"source": [
"# define model architecture\n",
"class SimpleLinearANN(nn.Module):\n",
" def __init__(self):\n",
" super().__init__()\n",
" # the \"Sequential\" class offers a convenient way to fit many simple ANNs\n",
" self.layer_stack = nn.Sequential(\n",
" nn.Linear(1,1)\n",
" )\n",
"\n",
" def forward(self, x):\n",
" pred = self.layer_stack(x)\n",
" return pred\n",
"\n",
"model2 = SimpleLinearANN().to(device)\n",
"print(model2)"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "Y1Yf_fu5rTpm"
},
"source": [
"We'll also define our training and testing more functionally now, so that we can reuse these functions later."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "TUuIhd1IrRzK"
},
"outputs": [],
"source": [
"# define training function\n",
"def train(dataloader, model, loss_fn, optimizer):\n",
" size = len(dataloader.dataset)\n",
" model.train()\n",
" for batch, (X, y) in enumerate(dataloader):\n",
" # note that we're actually using the GPU here\n",
" X, y = X.to(device), y.to(device)\n",
"\n",
" # Compute prediction error\n",
" pred = model(X)\n",
" loss = loss_fn(pred, y)\n",
"\n",
" # Backpropagation\n",
" loss.backward()\n",
" optimizer.step()\n",
" optimizer.zero_grad()\n",
"\n",
" if batch % 100 == 0:\n",
" loss, current = loss.item(), (batch + 1) * len(X)\n",
" print(f\"loss: {loss:>7f} [{current:>5d}/{size:>5d}]\")"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "cbPGbq6KsTeo"
},
"outputs": [],
"source": [
"# define test function\n",
"def test(dataloader, model, loss_fn):\n",
" size = len(dataloader.dataset)\n",
" num_batches = len(dataloader)\n",
" model.eval()\n",
" test_loss, r2 = 0, 0 # evaluate loss (MSE, and R^2)\n",
" with torch.no_grad():\n",
" for X, y in dataloader:\n",
" X, y = X.to(device), y.to(device)\n",
" pred = model(X)\n",
" test_loss += loss_fn(pred, y).item()\n",
" r2 += 1 - torch.sum((y-pred)**2) / torch.sum(y **2)\n",
" test_loss /= num_batches\n",
" r2 /= num_batches\n",
" print(f\"Test performance: \\n R^2: {r2:>8f}, RMSE: {np.sqrt(test_loss):>8f} \\n\")\n",
" return r2.cpu().numpy(), test_loss"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "vGQhIDY74e2n"
},
"source": [
"Next, we'll put our simulated data into pytorch's dataloader - an iterable class that supports batching (breaking up the overall dataset into smaller batches), and shuffling (randomizing the order of that data), among other features. Generally it's a good idea to choose a batch size that is a power of 2 (for memory efficiency), and to shuffle your data to prevent catastrophic forgetting that can come from showing the model a bunch of similar examples in a row."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "nKdn5XrU4dQI"
},
"outputs": [],
"source": [
"# turn our numpy variables into pytorch datasets\n",
"xtrain_tensor = torch.from_numpy(np.float32(x_train))\n",
"ytrain_tensor = torch.from_numpy(np.float32(y_non_train))\n",
"nonlinear_training_data = TensorDataset(xtrain_tensor, ytrain_tensor)\n",
"\n",
"xtest_tensor = torch.from_numpy(np.float32(x_test))\n",
"ytest_tensor = torch.from_numpy(np.float32(y_non_test))\n",
"nonlinear_testing_data = TensorDataset(xtest_tensor, ytest_tensor)\n",
"\n",
"xval_tensor = torch.from_numpy(np.float32(x_val))\n",
"yval_tensor = torch.from_numpy(np.float32(y_non_val))\n",
"val_data = TensorDataset(xval_tensor, yval_tensor)\n",
"\n",
"# put the datasets into the dataloader\n",
"train_dataloader = DataLoader(nonlinear_training_data, batch_size=64, shuffle=True)\n",
"test_dataloader = DataLoader(nonlinear_testing_data, batch_size=64, shuffle=True)\n",
"val_dataloader = DataLoader(val_data, batch_size=64, shuffle=True)"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/"
},
"id": "5U4j0CF94BFe",
"outputId": "5dc3e7e6-17d1-413f-aac8-1ffff049dd47"
},
"outputs": [
{
"output_type": "stream",
"name": "stdout",
"text": [
"Epoch 1\n",
"-------------------------------\n",
"loss: 3.297604 [ 64/100000]\n",
"loss: 2.270447 [ 6464/100000]\n",
"loss: 2.475495 [12864/100000]\n",
"loss: 2.216384 [19264/100000]\n",
"loss: 2.483925 [25664/100000]\n",
"loss: 3.111267 [32064/100000]\n",
"loss: 2.170219 [38464/100000]\n",
"loss: 3.388379 [44864/100000]\n",
"loss: 4.701067 [51264/100000]\n",
"loss: 1.805016 [57664/100000]\n",
"loss: 1.798295 [64064/100000]\n",
"loss: 2.880607 [70464/100000]\n",
"loss: 1.875021 [76864/100000]\n",
"loss: 1.566702 [83264/100000]\n",
"loss: 1.444066 [89664/100000]\n",
"loss: 3.873803 [96064/100000]\n",
"Epoch 2\n",
"-------------------------------\n",
"loss: 1.891131 [ 64/100000]\n",
"loss: 1.695512 [ 6464/100000]\n",
"loss: 1.589921 [12864/100000]\n",
"loss: 1.268939 [19264/100000]\n",
"loss: 1.167627 [25664/100000]\n",
"loss: 1.327818 [32064/100000]\n",
"loss: 1.701994 [38464/100000]\n",
"loss: 0.960532 [44864/100000]\n",
"loss: 1.701208 [51264/100000]\n",
"loss: 5.352643 [57664/100000]\n",
"loss: 1.198715 [64064/100000]\n",
"loss: 2.584759 [70464/100000]\n",
"loss: 2.101575 [76864/100000]\n",
"loss: 2.096920 [83264/100000]\n",
"loss: 3.844053 [89664/100000]\n",
"loss: 2.158092 [96064/100000]\n",
"Epoch 3\n",
"-------------------------------\n",
"loss: 2.072096 [ 64/100000]\n",
"loss: 2.084110 [ 6464/100000]\n",
"loss: 2.557325 [12864/100000]\n",
"loss: 1.547599 [19264/100000]\n",
"loss: 1.849058 [25664/100000]\n",
"loss: 2.230387 [32064/100000]\n",
"loss: 0.976333 [38464/100000]\n",
"loss: 0.971154 [44864/100000]\n",
"loss: 2.253049 [51264/100000]\n",
"loss: 1.980433 [57664/100000]\n",
"loss: 1.413823 [64064/100000]\n",
"loss: 3.303815 [70464/100000]\n",
"loss: 3.310146 [76864/100000]\n",
"loss: 1.572646 [83264/100000]\n",
"loss: 2.585217 [89664/100000]\n",
"loss: 2.679968 [96064/100000]\n",
"Done!\n"
]
}
],
"source": [
"# new we'll train the model (note that loss does not seem steadily to improve)\n",
"loss = nn.MSELoss()\n",
"optimizer = torch.optim.SGD(model2.parameters(),lr=.5)\n",
"epochs = 3\n",
"for t in range(epochs):\n",
" print(f\"Epoch {t+1}\\n-------------------------------\")\n",
" train(train_dataloader, model2, loss, optimizer)\n",
"print(\"Done!\")"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "KcUmrZitoiTx"
},
"source": [
"As we can see below, the performance of the linear model is poor, and it is not capturing the nonlinear relationship in the data.\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/"
},
"id": "XvlFx0v924KF",
"outputId": "0449644b-0027-48a1-fb69-ac9b3f9b8ced"
},
"outputs": [
{
"output_type": "stream",
"name": "stdout",
"text": [
"Test performance: \n",
" R^2: 0.300736, RMSE: 1.491297 \n",
"\n"
]
}
],
"source": [
"# performance metrics\n",
"perf = test(test_dataloader, model2, loss)"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/",
"height": 449
},
"id": "3yeGKp3YnHVL",
"outputId": "b299dd42-346e-430e-9ef3-b659fbfc9cb2"
},
"outputs": [
{
"output_type": "display_data",
"data": {
"text/plain": [
""
],
"image/png": "iVBORw0KGgoAAAANSUhEUgAAAjIAAAGwCAYAAACzXI8XAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjcuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/bCgiHAAAACXBIWXMAAA9hAAAPYQGoP6dpAABQWUlEQVR4nO3deXzT9f0H8FfupEmbXvRID84CpWBB5FIUUbAeKyCKzHngfnPuwCk63cRbN0GdUzyYeDudghewio7CmAgoKkepUEspUHrQi/RO0hzf5Pv7oyZSKNCUpMm3fT0fjzwe9Juj729p833lc8pEURRBREREJEHyUBdARERE1FMMMkRERCRZDDJEREQkWQwyREREJFkMMkRERCRZDDJEREQkWQwyREREJFnKUBcQbB6PB9XV1YiMjIRMJgt1OURERNQNoiiira0NJpMJcvmp2136fJCprq5GWlpaqMsgIiKiHqisrERqauop7+/zQSYyMhJAxw8iKioqxNUQERFRd7S2tiItLc13HT+VPh9kvN1JUVFRDDJEREQSc6ZhIRzsS0RERJLFIENERESSxSBDREREksUgQ0RERJLFIENERESSxSBDREREksUgQ0RERJLFIENERESSxSBDREREksUgQ0RERJLFIENERESSxSBDREREksUgQ0RERJLFINNDNqeANQVVsDmFUJdCRETUb4U0yGzZsgW5ubkwmUyQyWRYu3at7z6Xy4U///nPGDNmDPR6PUwmE26++WZUV1eHruDj5BfVotHixIaiulCXQkRE1G+FNMhYrVZkZ2dj+fLlJ91ns9mwe/duPPTQQ9i9ezdWr16NkpISzJo1KwSVniwnKwlxBg0uy0oMdSlERET9lkwURTHURQCATCbDmjVrMGfOnFM+ZseOHZg4cSLKy8uRnp7erddtbW2F0WhES0sLoqKiAlQtERERBVN3r9/KXqzprLW0tEAmkyE6OvqUj3E4HHA4HL6vW1tbe6EyIiIiCgXJDPa12+3485//jOuvv/60yWzp0qUwGo2+W1paWi9WSURERL1JEkHG5XLhuuuugyiKePnll0/72MWLF6OlpcV3q6ys7KUqiYiIqLeFfdeSN8SUl5fjf//73xnHuWg0Gmg0ml6qjoiIiEIprIOMN8SUlpbiiy++QFxcXKhLIiIiojAS0iBjsVhw8OBB39dlZWXYs2cPYmNjkZycjGuvvRa7d+/GunXr4Ha7UVtbCwCIjY2FWq0OVdlEREQUJkI6/Xrz5s2YPn36SccXLFiARx99FIMHD+7yeV988QUuvvjibn0PTr8mIiKSHklMv7744otxuhwVJkvcEBERUZiSxKwlIiIioq4wyBAREZFkMcgQERGRZDHIEBERkWQxyBAREZFkMcgQERGRZDHIEBERkWQxyBAREZFkMcgQERGRZDHIEBERkWQxyBAREZFkMcgQERGRZDHIEBERkWQxyBAREZFkMcgQERGRZDHIEBERkWQxyBAREZFkMcgQERGRZDHIEBERkWQxyBAREZFkMcgQERGRZDHIEBERkWQxyBAREZFkMcgQERGRZDHIEBERkWQxyBAREZFkMcgQERGRZDHIEBERkWQxyISIzSlgTUEVbE4h1KUQERFJFoNMiOQX1aLR4sSGorpQl0JERCRZDDIhkpOVhDiDBpdlJYa6FCIiIslShrqA/ipCrcSccSmhLoOIiEjS2CJDREREksUgEwY48JeIiKhnGGTCAAf+EhER9QyDTBjgwF8iIqKe4WDfMMCBv0RERD3DFhkiIiKSLAYZIiIikiwGGSIiIpIsBhkiIiKSLAYZIiIikiwGGSIiIpKskAaZLVu2IDc3FyaTCTKZDGvXru10vyiKePjhh5GcnAydTocZM2agtLQ0NMUSERFR2AlpkLFarcjOzsby5cu7vP/pp5/GCy+8gBUrVuDbb7+FXq9HTk4O7HZ7L1dKRERE4SikC+JdccUVuOKKK7q8TxRFLFu2DA8++CBmz54NAHjnnXeQmJiItWvX4uc//3lvlkpERERhKGzHyJSVlaG2thYzZszwHTMajZg0aRK2b99+yuc5HA60trZ2uhEREVHfFLZBpra2FgCQmNh5/6HExETffV1ZunQpjEaj75aWlhbUOomIiCh0wjbI9NTixYvR0tLiu1VWVoa6JCIiIgqSsA0ySUlJAIC6urpOx+vq6nz3dUWj0SAqKqrTjYiIiPqmsA0ygwcPRlJSEjZt2uQ71traim+//RZTpkwJYWVEREQULkI6a8liseDgwYO+r8vKyrBnzx7ExsYiPT0dixYtwl//+ldkZGRg8ODBeOihh2AymTBnzpzQFU1ERERhI6RBZufOnZg+fbrv67vvvhsAsGDBArz99tv405/+BKvVittuuw3Nzc2YOnUq1q9fD61WG6qSiYiIKIzIRFEUQ11EMLW2tsJoNKKlpYXjZYiIiCSiu9fvsB0jQ0RERHQmDDJEREQkWQwyREREJFkMMkRERCRZDDJEREQkWQwyREREJFkMMkRERCRZDDJEREQkWQwyEmVzClhTUAWbUwh1KURERCHDICNR+UW1aLQ4saGo7swPJiIi6qMYZCQqJysJcQYNLstKDHUpREREIRPSTSOp5yLUSswZlxLqMoiIiEKKLTJEREQkWQwyREREJFkMMkRERCRZDDJEREQkWQwyREREJFkMMkRERCRZDDJEREQkWQwyREREJFkMMkRERCRZDDJEREQkWQwyREREJFkMMkRERCRZDDJEREQkWQwyREREJFkMMkRERCRZDDJEREQkWQwyREREJFkMMkRERCRZDDJERETUIzangDUFVbA5hZDVwCBDREREPZJfVItGixMbiupCVgODDBEREfVITlYS4gwaXJaVGLIalCH7zkRERCRpEWol5oxLCWkNbJEhIiIiyWKQ6cPCYRAWERFRMDHI9GHhMAiLiIgomBhk+rBwGIRFREQUTBzs24eFwyAsIiKiYGKLDBEREUkWgwwRERFJFoMMERERSRaDDBEREUkWgwwRERFJVlgHGbfbjYceegiDBw+GTqfD0KFD8Ze//AWiKIa6NCIiIgoDYT39+qmnnsLLL7+Mf/7zn8jKysLOnTvxy1/+EkajEXfccUeoyyMiIqIQC+sg8/XXX2P27Nm46qqrAACDBg3CypUr8d13353yOQ6HAw6Hw/d1a2tr0OskIiKi0AjrrqXzzz8fmzZtwoEDBwAAhYWF2LZtG6644opTPmfp0qUwGo2+W1paWm+VS0RERL1MJobxgBOPx4P7778fTz/9NBQKBdxuN5544gksXrz4lM/pqkUmLS0NLS0tiIqK6o2yiYiI6Cy1trbCaDSe8fod1l1LH374Id577z28//77yMrKwp49e7Bo0SKYTCYsWLCgy+doNBpoNJperpSIiIhCIayDzL333ov77rsPP//5zwEAY8aMQXl5OZYuXXrKIENERET9R1iPkbHZbJDLO5eoUCjg8XhCVBERERGFk7BukcnNzcUTTzyB9PR0ZGVloaCgAM8++yz+7//+L9SlERERURgI68G+bW1teOihh7BmzRrU19fDZDLh+uuvx8MPPwy1Wt2t1+juYCEiIiIKH929fod1kAkEBhkiIiLp6e71O6zHyBARERGdDoMMERERSRaDDBEREUmW30Gmvb0dNpvN93V5eTmWLVuGDRs2BLQw6n02p4A1BVWwOYVQl0JERNQtfgeZ2bNn45133gEANDc3Y9KkSfj73/+O2bNn4+WXXw54gdQ7bE4BT3xWjNoWOzYU1YW6HCIiom7xO8js3r0bF154IQDg448/RmJiIsrLy/HOO+/ghRdeCHiB1Dvyi2qRGq3D0aZ2XJaVGOpyiIiIusXvIGOz2RAZGQkA2LBhA+bOnQu5XI7JkyejvLw84AVS78jJSkJytA73X5UJAOxiIiIiSfA7yAwbNgxr165FZWUl8vPzcdlllwEA6uvruU6LhEWolZgzLgURaiXyi2rRaHGyi4mIiMKe30Hm4Ycfxj333INBgwZh4sSJmDJlCoCO1plx48YFvEDqfTlZSYgzaNjFRETUz0hx0kePVvatra1FTU0NsrOzfZs6fvfdd4iKisLIkSMDXuTZ4Mq+RERE3bOmoAqNFifiDBrMGZcS0lqCurJvUlISIiMjsXHjRrS3twMAJkyYEHYhhoiIiLpPii3yfgeZhoYGXHrppRg+fDiuvPJK1NTUAAB+9atf4Y9//GPACyQiIqLecfx4SanwO8jcddddUKlUqKioQEREhO/4/PnzsX79+oAWR0RERHQ6fkeuDRs2ID8/H6mpqZ2OZ2RkcPo1ERER9Sq/W2SsVmunlhivxsZGaDSagBRFRERE1B1+B5kLL7zQt0UBAMhkMng8Hjz99NOYPn16QIsjIiIiOh2/u5aefvppXHrppdi5cyecTif+9Kc/oaioCI2Njfjqq6+CUSOFEZtTQH5RLXKykiQ1GIyIiPomv1tkRo8ejQMHDmDq1KmYPXs2rFYr5s6di4KCAgwdOjQYNVIY4aq/REQUTnq0IJ6UcEG8wLI5BWwoqsNlWYlskSEioqDp7vXb7yvRli1bTnv/RRdd5O9LkoR41xggIiIKB34HmYsvvvikYzKZzPdvt9t9VgVReOMYGSIiCid+j5FpamrqdKuvr8f69esxYcIEbNiwIRg1UhjhGBkiIumS4qaQZ+L3R2qj0XjSsZkzZ0KtVuPuu+/Grl27AlIYhaecrCTfGBkiIpKW4z+M9pVhAgHrG0hMTERJSUmgXo7CFMfIEBFJk80pwCG4YdAq+9SHUb+DzPfff9/pa1EUUVNTgyeffBJjx44NVF1EREQUIDangCc+K0ZqtA7J0bo+NcbR7zMZO3YsZDIZTpy1PXnyZLz55psBK4yIiIgCI7+oFqkxOhxtaseCCwaFupyA8jvIlJWVdfpaLpdjwIAB0Gq1ASuKiIiIAsc7vnHB+YP6VGsM0IMgM3DgwGDUQRJjcwrIKzwKQIZZ2aY+94dBRNSX9OXxjd26+rzwwgvdfsE77rijx8WQdOQX1WL7wUbIZIBWqeizfyBERH1BX14DrFtbFAwePLh7LyaT4fDhw2ddVCBxi4LgsDkFfFpYDQDIZYsMEVFYW1NQhUaLE3EGjWQ+eHb3+s29loiIiPo4Ke6TF7S9loiIiEha+v0YmRNVVVUhLy8PFRUVcDqdne579tlnA1IYERER0Zn4HWQ2bdqEWbNmYciQIdi/fz9Gjx6NI0eOQBRFnHvuucGokYiIiKhLfm8auXjxYtxzzz3Yu3cvtFotPvnkE1RWVmLatGmYN29eMGokIiIi6pLfQaa4uBg333wzAECpVKK9vR0GgwGPP/44nnrqqYAXSERERHQqfgcZvV7vGxeTnJyMQ4cO+e4zm82Bq4yIiIjoDPweIzN58mRs27YNmZmZuPLKK/HHP/4Re/fuxerVqzF58uRg1EhERETUJb+DzLPPPguLxQIAeOyxx2CxWPDBBx8gIyODM5aIiIioV3FBPCIiIgo73b1++z1G5tZbb8XmzZvPpjYiIiKigPA7yBw7dgyXX3450tLScO+996KwsDAYdRERERGdkd9B5t///jdqamrw0EMPYceOHTj33HORlZWFJUuW4MiRI0EokYiIiKhrfgcZAIiJicFtt92GzZs3o7y8HLfccgveffddDBs2LND14ejRo7jxxhsRFxcHnU6HMWPGYOfOnQH/PhR4NqeANQVVsDmFUJdCRNQn8X22h0HGy+VyYefOnfj2229x5MgRJCYmBqouAEBTUxMuuOACqFQq/Oc//8EPP/yAv//974iJiQno96HgyC+qRaPFiQ1FdaEuhYioT+L7bA83jfziiy/w/vvv45NPPoHH48HcuXOxbt06XHLJJQEt7qmnnkJaWhreeust37HBgwef9jkOhwMOh8P3dWtra0Brou7LyUrybRtPRESBx/fZHrTIpKSk4Morr4TZbMarr76Kuro6vPnmm7j00kshk8kCWlxeXh7OO+88zJs3DwkJCRg3bhxee+210z5n6dKlMBqNvltaWlpAa6Lu824bH6HuUV4mIqJuENGnV1E5I7/XkXnttdcwb948REdHB6mkn2i1WgDA3XffjXnz5mHHjh248847sWLFCixYsKDL53TVIpOWlsZ1ZIiIqM9ZU1CFRosTcQYN5oxLCXU5AdXddWTCekE8tVqN8847D19//bXv2B133IEdO3Zg+/bt3XoNLohHRER9lc0p+LqW+lrrd9AWxOtNycnJGDVqVKdjmZmZqKioCFFFRERE4YNd+GEeZC644AKUlJR0OnbgwAEMHDgwRBURERFROAnrIHPXXXfhm2++wZIlS3Dw4EG8//77ePXVV7Fw4cJQl0ZERERhIKyDzIQJE7BmzRqsXLkSo0ePxl/+8hcsW7YMN9xwQ6hLIyIiojDQrcG+eXl53X7BWbNmnVVBgcbBvkRERNLT3et3t0YHzZkzp1vfVCaTwe12d+uxRERERGerW0HG4/EEuw4iIiIiv4X1GBkiIiLqwA0iu9ajiedWqxVffvklKioq4HQ6O913xx13BKQwIiIi+snxG0T2tVV8z4bfQaagoABXXnklbDYbrFYrYmNjYTabERERgYSEBAYZIiKiIOAGkV3zu2vprrvuQm5uLpqamqDT6fDNN9+gvLwc48ePxzPPPBOMGomIiPql47uTuIpv1/wOMnv27MEf//hHyOVyKBQKOBwOpKWl4emnn8b9998fjBqJiIj6peO7k6hrfgcZlUoFubzjaQkJCb59j4xGIyorKwNbHRERUT+Wk5WEOIOG3Umn4Xf71Lhx47Bjxw5kZGRg2rRpePjhh2E2m/Huu+9i9OjRwaiR+jCbU0B+US1yspLYXEpEdAJvdxKdmt8tMkuWLEFycjIA4IknnkBMTAx+97vf4dixY3j11VcDXiD1bd5m008LqzmtkIiI/NatLQqkjFsUhDebU8CGojrYBTesdgFxBg0/fRARUWC3KCAKFm+zqTfQsB+YiIj84XeQGTx4MGQy2SnvP3z48FkVRP0T+4GJiKgn/A4yixYt6vS1y+VCQUEB1q9fj3vvvTdQdRERERGdkd9B5s477+zy+PLly7Fz586zLoj6N85iIiIifwRs08grrrgCn3zySaBejvopzmIiIiJ/BCzIfPzxx4iNjQ3Uy1E/5V38SQS4miUREZ1RjxbEO36wryiKqK2txbFjx/CPf/wjoMVR/8NZTERE5A+/g8zs2bM7BRm5XI4BAwbg4osvxsiRIwNaHPVfnMVERETdwQXxiIiIKOx09/rt9xgZhUKB+vr6k443NDRAoVD4+3JEREREPeZ3kDlVA47D4YBarT7rgoi6YnMKnMVEREQn6fYYmRdeeAEAIJPJ8Prrr8NgMPjuc7vd2LJlC8fIUNB4p2VvKKrj2BkiIvLpdpB57rnnAHS0yKxYsaJTN5JarcagQYOwYsWKwFdIhI5p2ZzFREREJ+p2kCkrKwMATJ8+HatXr0ZMTEzQiiI6EWcxERFRV/weI/PFF18wxFBIcJwMERGdyO8gc8011+Cpp5466fjTTz+NefPmBaQooq4cP06GiEhq+GEsOPwOMlu2bMGVV1550vErrrgCW7ZsCUhRRF3xbl/AcTJEJEX8MBYcfgcZi8XS5TRrlUqF1tbWgBRF1BXvOBnuik1EUnRhRjzKzFZMzYgLdSl9it9BZsyYMfjggw9OOr5q1SqMGjUqIEURERH1NVtLzRgcr8e20oZQl9Kn+P3R9qGHHsLcuXNx6NAhXHLJJQCATZs2YeXKlfjoo48CXiAREZGU2ZwC8otqcWFGPLaVNrB7PMD8DjK5ublYu3YtlixZgo8//hg6nQ7nnHMO/vvf/2LatGnBqJGIiEiyvGNjtpU2cBmJIAjoppH79u3D6NGjA/VyAcFNI/smm1NAXuFRADLMyjZx3AwRhS2bU/At6Mn3qu4L2qaRJ2pra8Orr76KiRMnIjs7+2xfjqhb8otqsf1gI7451MAZAEQU1jhRIbh6/FPdsmULXn/9daxevRomkwlz587F8uXLA1kb0SnlZCXBKXjgFDywC27YnALfJIiI+iG/WmRqa2vx5JNPIiMjA/PmzYPRaITD4cDatWvx5JNPYsKECcGqk6iTCLUS8yekw6BVwmoX2CpDRNRPdTvI5ObmYsSIEfj++++xbNkyVFdX48UXXwxmbURnxEXyiIj6t263xf/nP//BHXfcgd/97nfIyMgIZk1EfhERsPHqREQB4Z1ynZOVxG7vIOt2i8y2bdvQ1taG8ePHY9KkSXjppZdgNpuDWRvRGR2/5Df3MSGicMHtCHpPt4PM5MmT8dprr6Gmpga/+c1vsGrVKphMJng8HmzcuBFtbW3BrJOoS8d3LZ34xsFgQ0ShYHMKcAgeGLRKdnv3grNaR6akpARvvPEG3n33XTQ3N2PmzJnIy8sLZH1njevI9B8nrtWwpqAKjRYn4gwaLkJFRL2G7z2B0SvryIwYMQJPP/00qqqqsHLlyrN5KaKzduJaDRwITES94cTWX7739K6AruwbjtgiQ0REwdRVCwwH+569XlvZtzc9+eSTkMlkWLRoUahLISIiAtB1CwwH+/YeyQSZHTt24JVXXsE555wT6lKIiIh8utqCgN1LvUcSQcZiseCGG27Aa6+9hpiYmFCXQ0REdFrcX6n3SCLILFy4EFdddRVmzJhxxsc6HA60trZ2uhEREQULl3oIrbAPMqtWrcLu3buxdOnSbj1+6dKlMBqNvltaWlqQKyQiov6M42FCK6yDTGVlJe68806899570Gq13XrO4sWL0dLS4rtVVlYGuUoKV/yURES9geNhQiusp1+vXbsWV199NRQKhe+Y2+2GTCaDXC6Hw+HodF9XOP26/+KUSCIi6eru9Tus38kvvfRS7N27t9OxX/7ylxg5ciT+/Oc/nzHEUP+Wk5WETwurYRcEmC12bC01wyF4YLV3rADMFTeJiKQvrINMZGQkRo8e3emYXq9HXFzcSceJThShVkKtlKPR4sSyjaUYHK+HQatkEzAR9RhbdcNPWI+RITpb3r7rRTMzEGfQIDfbxCmRROQ375i7vMJq1DbbseSzYo6/CxNhPUYmEDhGhoiIzpZ3zJ1Bq8TeqhakxOiQbNSxizqI+sQYGSIionCQk5WEDUV1mJoRBxEiZJCxizpMsGuJiIjoDLwr9W4tNcNqdwPoWD+G3UuhxyBDRETUTd5xdyLARfDCBIMMERHRGXgH+wLAnHEpmJVt4gzIMMEgQ/2a983JbLFzFWAiOqUTtyHgppDhg0GG+jXvm9OyjaVsJiaik3g/7FyYEc8WmDDFIEP92onrzPBNioiO5/2ws620gS0wYYpBhvo1b/NwvEHre5PiZpNE5OX9sDM1I47vC2GKQYboR8ev3MluJiLybkdwWVYitpaa+b4QphhkiH6UV3gUm/cfg0vwsJuJiDoN8PW2zPB9IfwwyBCh45NXQUUz3D/u2CGiT+/cQUSn0dUAX85SCl8MMkQA8gqrYbELMGgUcAgevLu9HJ/s6lgzgmNmiPoXDvCVFgYZIgCACJVCjnHpMSitb4NT8KCougXAyetHEFHfcKoPKRdmxKPMbMXUjLgQVUb+YJAhAjArOwUXj0hAbrYJ9+aMwNi0aNyTMwIA2DdO1Ed19SHF5hTw3MZSpMTosK20IYTVUXcxyBDhp2nYALC11Iz7r8pEvEHb6T42LxP1LV19SMkvqkVqtA5Hm9r54UUiGGSIjtPVJzSzxY4H1uyF2WIPYWVEFGhdfUjJyUpCcrQO91+VyQ8vEsEgQ3Scrj6hPbexFPWtdizbWBrCyogo0GxOAat2VGDVjnLfOBm2wEoP/6eIjnN8F5PNKSCv8ChSonU4WG/Bw7lDQlwdEQVSflEtth9qgAyAVvnT3z5JC1tkiE4hv6gW2w82YlNxHQYY1Hjly8Ocgk3Uh+RkJeH8oXGYMjQOl2UlcqkFiZKJotinV/5qbW2F0WhES0sLoqKiQl0OSYjNKeDTwmo4BQ/2VDbDLngweXAsDFolcrKS2PRM1MesKahCo8WJOIOGrTNhoLvXb74TE51ChFqJ+RPSAQAqZTm2lTbg34XVSIzSwil4fPcRUd+Qk5WEDUV1nK0kMexaIuqGWdkpMKgVcDjdONpoC3U5RNQDHYN7y7FqR0WX3Ucc6CtN/N8iOgPvDrijUqJgcbqhVcoh/nicb3hE0uEd9yaTAVqlgt1HfQRbZIjOwLu2jEapwIzMRJw3OAZWu8AtC4gk4PgBvDlZSTh/WBzGD4yBXXBzUG8fwSBDdAbetWVys00/9p3LYNAq2Y9OJAHHL3LpHfdm0Cr5YaQPYZAhOoPj15Z54rNiNFk7WmfYrUQU3mxOAQ7Bc9IHD+6f1rcwyBB1E/dgIZKW/KJaWO3CSR88OKi3b2GQIeom7sFCJC1seekfGGSIuqmrT3HcUJIofLHlpX9gkCE6C9xQkii8cduBvo9Bhugs3DUzA4lRWiyamQGAb5pE4cTmFPDEZ8WobbFzhlIfxiBDdBbiDVr89eoxiDdofW+aFQ02LPmsmGGGKMQ4QL9/YMch0VnyrvzbYHGgqLoFBxRyXDIiARuK6rhyKFEvsTkF5BVWAxAxK7tjXIx376QFFwziOJk+jP+zRGfJu+DWxh/qIbhFDBugQ3K0jp8AiXpRflEtth9qgAyAVvnTIF9+mOj72LVEdJa8UzyfmXcOxqZF474rM3FZViLyCquxakc5u5iIgsy78N15A2MwZWgcpmbEcaxaPyITRVEMdRHB1NraCqPRiJaWFkRFRYW6HOoHvGNlmtudUMnluHhEAj8VEgWJ9+8tNUaHWL0aaqUcDsENq92NOIOGf3sS1t3rN1tkiALMO8AwUqPk5nREQeQNMQmRGhxtaocIEY0WJ2SQcSG8foRBhijAvCsAP5ybBZVSju2HGvBpYTWnZhMFgPfvyGyxd4QYgwY1ze0YnWrEjMxE3wavXAiv/+D/MlGAdR5gKMLtEfFdWQN2VzRhUKyes5mIesBsseO5jaUYkRQJq0PA7e/tRrReDbfHg7EDY2C1C9hW2sC/rX6ILTJEQTQrOwXROhXsTg+sDjfKG6zsaiLqgb/ll6Cwshm7y5uwtdQMg0aFY60OjEuPwaxsE7uS+jEGGaIgilArcf9VmbhoxABcmBGPsQNj0GRxcsE8Ij9lmYzQKOUQRRFROiVa2l2YMy4Fudkm7qnUzzHIEPUCEYAIERcMjcPWg2YkRGm4ZDqRH64dn4qbpgzC+EExqG91IDFKg0itiuGFOEaGKNi8C3W5PSLy9tTg3PRofHWoAdERKticAt+Iic7AbLHjb+tLkJFogFopx9XjUqBWyn1dSd7VtXOykvj31A+xRYYoyHKyknD+0DgY1ApMGhSL3RXN0CjkeHp9Cf751ZFQl0cU1mxOAXes3IO9VS341zcV2F3ejEitCvMnpPtCi3d1bbZy9k9hHWSWLl2KCRMmIDIyEgkJCZgzZw5KSkpCXRaRXyLUSsyfkI6HZ2VhYLweL1w/FsU1LXAJHnywswJrCqpQ2WjFA2v2wmyxh7pcorCSX1SLSYNjEaVT4abJAzFlaNxJg3q9q2tzsG//FNYr+15++eX4+c9/jgkTJkAQBNx///3Yt28ffvjhB+j1+m69Blf2pXBjcwq496NCfHO4ARMGxUKjUqC2pR1apQLVLXasvG0SItRKNpUToePvZUNRHS7LSuTfQj/T3et3WAeZEx07dgwJCQn48ssvcdFFF3X5GIfDAYfD4fu6tbUVaWlpDDIUNtYUVKG22Y7yRitcggcOt4hxadFYtaMSiZEaiAByx5pgtQtcYp36HY53Ia8+uUVBS0sLACA2NvaUj1m6dCmMRqPvlpaW1lvlEXWLd+XfsenRSI/V41ibAyqlDG/ech5cogitRo4dZY0waJVsKqd+xzve5dPCaq6ETd0imRYZj8eDWbNmobm5Gdu2bTvl49giQ1JhcwpY8lkxzFYHjrU6cN2ENIgQ8dGOKiRH6zAjM5GtMdTveLuS7IIAq90Ng1YJtVLOFpp+qM+1yCxcuBD79u3DqlWrTvs4jUaDqKioTjeicORdLC9aq8KASA0KyptwwdB4DI6LgEYpx9SMuFCXSNTrvIvbzcpOQZyho6uVM5LodCQRZG6//XasW7cOX3zxBVJTU0NdDlHARKiVeHhWFuINGgyM1+OVLw/D5QHcHhHbShu40ST1KzangFU7yvH61kN47NMiTM2I4/YDdEZh3bUkiiL+8Ic/YM2aNdi8eTMyMjL8fg3OWiIpsDkFfFpYDYfggQyAWilHbrbJN16Ag36pr+lqUO+agips3n8MhVXNMGiUGJsWjb9ePSbElVKodPf6HdYdjgsXLsT777+Pf//734iMjERtbS0AwGg0QqfThbg6osCJUHeMA7AcN1PJ5hTgENwc9Et9Ul5hNbaWmvHN4QY8kpuFCLUSOVlJcAoenJNqRGl9GxbN9P/DK/U/Yd0iI5PJujz+1ltv4ZZbbunWa7BFhqTixPUy1hRUodHi9A12vDAjHltLzRz0SH3Cqh3l+HBHFUxGLWaMSuqyxZFTsfu3PrmOTE8wyJBU/TR7w40mixNbD5px4bB4JEfr2M1EkuftTgXg28H6xODiDfPsWu2f+tysJaL+5qfZGyYcabQiSqdEeaOV3Uwked7AkpttwqWZCXjis2KYLfaT9kzi1gPUHWyrIwpjNqeAvMJqWNqd2He0FePTYzvdx2Z3kqK8wmpsP9QAp+DB91UtqG+145n1JRg7MLrTmDBvmCc6HbbIEIWx/KJabD/UgMKjrWi0ulBmtvg+rXLHX5KKjmnVFVi1oxw2pwCn4EFlow1OwYO7ZmYgMUqLUSlRsNrd0CgVDObkF/62EIUx7ywOiMBhswVDBhhgFwTYnAIuzIjHso2lmD3OFOoyibrkbTV0CB5sP9QAt9uDgopmCIIHJqMOaqUc8QYt/nr1mE6D3Yn8wcG+RBLQ1bLt31e1IDVGh2QjB/9SePIO1lUr5dhX3QK704MyswUAMCo5Cg/PymLrC50SB/sS9SER6p/GDaiVcnxX1gCz1YHyhp8G/3IVYAonNqeANrsL31c1IW9PNUxROigVQHO7CyKArBQj8otq+ftKZ41BhkgCbE4BT3xWjCaLCyW1bbA7PTC3OpBl+ulicOKYGQYbCqX8olrsOtKMgooWVDbZ8K9vy2F3ChhlikKWKQoqpYxjvCgg2KZHJAH5RbVIjdHhaFM7Fs3MwKbiegBAm92FNduPwmIXcM34VGwoqsPUjDis2lGOgopmDIrVY0NRHbueqNflZCXBYhdwpMGCg/VWGDQKHG12YHSKEfdflQkAHBNDAcEWGSIJyMlKQrJRh/uvykS8QYv5E9KRm23Cxh/q0NruxIv/O4gGiwNzxqVga6kZ2w82wupwo7zRCrvgZqsM9boItRIGrRJJUVpER6ihVSuQm23C/VdlIkKt9E2t5hgZOlsMMkQS0NWbfn5RLc4fEo/6NicMGiXu/fh7rCmownkDY6BTyTFpcCzGpkfDahfYfE+9xruD9aodFThvYAya212I1aswJiUacQYNgwsFHGctEUmUdybTqORIPPbpD7hgWBxK6izQquQYnhDpWxH1+P2biILNu4O1TAYYNEqkxOhQ3mDFuPQY31YERN3BWUtEfZy3lWZ4UhTe+/VkxBrUkAEYbTLCoFXCLnR0J52q+Z6DgSlQzBY7HlizF2aLHRdmxEOrlmP8wBj8ZtoQHGmwYpQpiiGGgoa/VUQS5t3CABAxIzMRMsggAnAIHnx3uAF7ypt9a3V4ZzZ5d9F2CB5ftxMHA1NPmS12XP/qt0iI0uDOlXuQMzoRdpcHaqUcO8ubYHd5sLu8GVFaNX/PKCgYZIgkzLuFgQyADDJ8X9WCmAgV/rOvFjqVHEqFHJ8WVmP+hHTf9OxlG0sxOF4Pg1bJDfnIL13t7/W39SVQADhQ24pz0mOQt6caMRFqrN59FM/MO6djZWqAv2cUNOxaIpKwnKwknD80DuMHxmB3RRMSDBrkF9UhNVoHu8uN1Ghdp8fGGTRYNDMDcQYNcrNNnDVCfskvqkVtsx1LPiv2dUlmpURBr1Xi/GED0GxxYkCkBkcabNBrFHjly8OYPyEd8yek8/eMgoZBhkjCItRKzJ+QDoNWiUFxelQ22TAqORKRGjmyTEZMGhKHSzMTfGNh7IIb/y3m4F/yj3c81XkDY7D1oBkxepUvzFw7Pg03TRmEey4bjowEA84fGo83bzkPJqMOi2ZmhLp06gc4a4moD/DOYNp+yIxGqxNHGmwYFBeBOIMaogi02FxoanfBoFPiUJ0FN00eiFiDulMXAVFXbE4Bj+UVweoUoNd0BOZtpWZMHRaPWIMaaqUcOVlJvq7LOIOGY2EoIDhriagf8c5guvfyEUiM0mL5L8YhzqBGu9ONH2pbsa+mBe1OAYfr2xChVmDjDz9tZ3Di7CXOZiLAux5MBR5asxf7qltRWNkMu9ONWL0az18/FsnROogQfb9H3q5LjoWh3sYWGaI+ak1BFf5bXIcKsw0Wh4ARyZGYMDAWZWYrfjNtCL4+1ACH4EFhZRPsgoiLMuIxf0K6b8difrLu31btqMCHOyohuD1otDnR7nRj3MAYRGuUUCjluDdnBCLUSnxaWA0RwCxOr6YA6+71m791RH1UTlYSnIIHuzVNSInWobq5HSqlDKNTjdCpFdhd0YT9NW0QBA/anAKUMkD8cRr3ttIGfrLu90SYjDpolTKIMhla2p0oO2aBRqFATasdbreIZ64bC7VSjtqWjgHA3u0HiHoTW2SI+jjv+Bm7IGD7wUa4RRGNVieidErUtjigkgMOtwfNNhdGmaIQr++Y2bS11MwxNP2YzSng08JqtNkFFNe0QHCLcLg9KK5pg1PomBH3z19NAgAs+awYKdE6JEfr2IpHAdPd6zeDDFE/4b0w7a5ogilKh5rWdoxLj8GlmQl4Zn0JmttdaLY5MW1EAsrNVlicAjRKOSYMjsWsbE7T7uuOX1xxVnZHGHnis2IUVXeEGINGiSSjFmNSjNi0vx6Th8QiPVaPOeNSfGGZs+EokNi1RESdeKdq52absKGoDjdMScfWUjMi1Ercc/kI/C2/BOcNikWkVgm9RoF/F1TD7RHh9gBapdJ3wTpxQTTqG45fXFGr7NjiosHigBwyDEvQY3iiAeWNNsweZ8L1k9J9wQX4abA5UShw1hJRP+O96GwtNftmnPy3uB4ltW1Y9301AGD6iATI5TKkRGsxfmCM74LlnWLb1WwnkoZTzVK7MCPet7hiq92J7Qcb8ENNK2INatS3OVB0tBUOlwfbShu63I2dKFQYZIj6qc7TZUV4PCLMFge+K2vES18chNUhoKimDQCQV1iNVTvKcWFGvO85x4cako4T/9/yCquxueQYNhXX+xZX3FXejMNmKwCgxebE5CGxiNQpMWVoHAeBU9hhnCbqp47vDpiVnYJvSs042tyOJpsTsToVtCo5FDIZdh5pRGVT+48zWH56Tk5WUqfuBS92P4W3nKwkfFpYjVa7C+9sL0NhZTOcghs7DjdABDAjMwEWu4CjTTaMMkXhgqHxiNSqcOuFQ/j/SWGJv5VEhAi1EgqlAhqlHGVmKyaMT4Mok6G0rg17KpsxdIABZosDUzPifINCnYIHaqXspNc6/hM/x02Ep++rWtBsdeJoczvcHhE2lxsZAwz45scxMvuOtiAmQo1mqwtXjElCvEEb6pKJTolBhogAANlp0ThituLKMcmIM2ig1yqxq7wJOpUClU3tmDsuBdtKG2AX3PhwRyU8IjAoLqJTKw1w6pYaCp2O8HkUgAyAiNQYHax2FxqtTrTZXRiXHgODRgGFUg6H4IbVIaC03oLRJiO2lTYwkFJYY5AhIgDAteNTEaVV+abQmi12FFY2QwYRd84Yjl3lzZiaEYe/rS9BQqQGBo0C2WkxsAtu2JyCr9uBM1hCy2yx47mNpbhrZgYi1ErkF9XCIXiwuaQeP1S3YVxaNM4bFIujze1oaXdCJpPB5nTj4pEDsKu8GTLIcOHwAZg0JA5qpZyBlMIegwwRAegcQGxOAc9tLEWyUYudRxrxRckxXDs+FXmFR2F1CjBoFDhvcBwsdgEf7ajExzursOKmc9kFEQb+ll+CoqOteCa/BJOGxKHR4oRBq0Sz1QWX4MGu8mbUttoRpVNBr1XBJXjw+Ows7K5sggyAWinH/AnpoT4Nom7jrCUiOkl+US1SY3TIL6pDq13AvwuO/jjLRQalXA4RMmw/2IDP99bg0DErKhqtuP7Vb1HZaO32lGxO3w4ss8WOB9bsxcDYCGiUcmSZjMjJSoJBq4RD8GDGqESkxegQo1dBr1biSIMVSZEajEqOwmOf/oAZmYm4eEQCcrNNoT4VIr9wZV8iOol3pdbxA6Ox/IuDyDIZcc34VADAp4XV2FJSj73VrchIjERJbRvsLgEjEqNQ19qOYYkGGFRKnJMeDbVSfspVgbk5ZWB4Z4l9c6gBjVYnjDoVtCoFbp4yEG98VQa7U8DBYza02V24fmI6Dta3oaC8GaZoHRyCBwq5DFOGxmFgnJ7/DxRWuEXBjxhkiAJrTUEV3txWhqNN7RgUr0d6bATGpBjxv/310Cjl+P5oC+INalQ2tiM6QoXfThuKa8annjQlm8vaB4Y3EKqVchRVt8AleDAsIRJbD5rR2u5Cc7sLTVYHFDIZtCo5xqXHQCaX4VC9FbnZyUiM0kKjVPD/gcJOd6/f7FoiIr/kZCVhaFwEDBoVZv7YHXH9pHQ8f/1YWF1ujE2LRoPFCcHjQV2LHWsLjmLxJ4V4++sj+GRXle91jh+Twy4m/3i7kSobrXAIbhi0SlwxJgkKuRzpcXocbW7Ho7mjoFbKIJOJSIuOgAdAtF6NJpsLRp0Kb95yHtJj9cjNNnGVXpI0Bhki8ptGo0RWSiQOHmtDq92JvMKj+G9xPS4ePgDxBjVumJQOvVoJlVKG6uZ27CxvQbnZirzCo6hstOLPnxTine1HfN0iXCH4J90ZO/TcxlJUNdkw9x9f4/1vKuAUPNhaakZMhArrvq/p6FbaVob6NgcEQYReq8ClIwfgnJRo/Cw7GcMTIrGrvJkBhvoEBhki8kt+US0GxerR2i7A4fLg3wXV+OZQI1yCB0eb2jHKFIUysw0GrRJqhRwOwYN4vRpROhUMWhXm/uNrrN9biw++q8SSz4o7bXvQm8JlsPHxdZgtdtz6z52oMNt8wc57v9lix6od5Xhn+xGkROuwr6oFDRYnSmpa8ea2MmQlRyG/qA4moxaPfvoDrE4BHlGE092xbsw5qTGYmjEA145PC8nPmyhYGMWJyC/eBe+ev34sNhXX/7jCrxwiRAyO10OjVOCcVCMO1LSi1uXBoFgNZo01Qa2U4+2vywCZiHaXG20OF5rbnfjP3loYtMpe39qgt1cg9p7fhRnx2Fpq9p3n8XXsONIIuQx477sK3HbRYN8qyltLj+H9b8thc7pR02xHhEYBDwC5DHB6gAarA7/51y787Jxk7C5vxlPXjMHmkmOobGxHg9WBkUlGX3jhOj/U1zDIEJFfjr8QHr/eyPGDd5/4rBjlPy5/X97UDgDYU9kEnVIBh0LEAIMcHo+IigYbNEoFRiVHYdnGUgyO1/dasOjJCsRnE7a8geWZ9SVod3lgsQswaJW4MCMe20obcFlWIsYPjMa1K7YjWqfEf/bWIlKrwtaSOnxZ2oAUoxZmqxMutweNFjcGx+th1SphF9xobe9YkLDJ6sJrC85DhFoJg7bJu5Avyhut+N30YX7+hIikgbOWiCjgzBY7Hl27D9vLGpFg0GBUihGHjllgbnMiIVKDiiYb5DIRaoUSAyLVHTOfUo2I1KpwaWYC/ltcD6fgBtCxQNuMzER8vrcWRdUtuDdnhG/hPW+LBSD6pnkHs2XnbKaM25wCPi2sxndlDXAIIvQaBYYnRPpaSvIKq7GjrBGNNicO1Vtg1CqhVMhR1+pAg8UOg1aFyYNjUFRjgd0lIMkYgesnpqHR4sSHOytx3XlpWHDBoE6zwj7ZVYWi6hbcc9zPjEgqOP36RwwyRKGzakc5vjnUiPEDY+AUPPjs+xoIHg/anQIabS5EqJXQaxSwOdyINagxdIABgIgDdRY02zqWzz8nJRoGjQLbyxqhVylw7sAY/PXqMQCAd7aX4fWtZTBolIjUqrD8hnHYWmoO2vo0/k4ZPzFUrSmoQm2zHeWNVgxLMOBgnQULLxmGl/53EPvr2uB2i2i1u9BmF2B3uRGlVUJwe9Du8iA6QoVzUo3QKJWobrbh8tHJmD3OhDtW7oFWKYcpWuf7uRD1BZx+TUQhNys7BRePSMA141Oh1yrQYHPiWJsDNa0OCG4PPB4RUVolRpkiUdFgwzeHzSits6Cp3YW2dhda212obLKh3eXB8AQDoiJUWDQzA0BHq88/vy6Hw+XBwfo2tNicWLaxFDlZSb5WDn8G9B7/2OMH2B7/fG+32qlCTOeBuRV4LK8IFWYblnxWDJtTQE5WEmINaogAdhxpRGm9Bcv/dxAt7U40WZ0YFBeBAXo1tEo5VAoZrE43Mk1GaJVyADLsr2lDmdmCRKMOcQYNtpaaMXlILByCx/dzIepvOEaGiIKm88BSGTKTDKhtdUJpccAjiojVa5CbnYK8PUfh9ogQAQxPjIQIEdsPN6DJ6kJFowUeUUSWKQpLrxmD/xbXwSl4kLenGgoAHlHElCFxqGq24zfThnT6nqt2lGP7wUbfeJScrCQA6HLQbV5hNbYfavANXm60dASjlGgdlnxWjPuvyuzUdXXi880WOxa+V4AYvQrbSs04WG+BCBGHjlmRZNRg5bcVOFDfBrvTgyabE7vKm6CUAzF6FRqtTkTrVNCoFIgxqLG/zgJR9CA1NgLVTe2I0CohA6DXKtHc7sJwhcw3tmdDUR1uvXAIp1FTv8WuJSLqFd5uma0H6lFc0waDRoHcsSnYV92CRosTxTWtUCvlWHHjeHxT1oBPdlWhvNEGnVIBlVKOBVMG4bDZioY2B36obYXHI8ItilAr5BibFo2RSVFQK+X47PtquEUgJysRJbUWlJkt0KjkmDp0AKpb2wERGBSnx5FGqy/gPJKbhbzCo9h2wAyNSo7stGholApcmpnQEWZidEg26jBnXIpvnEyZ2YrB8XpfF9YDa/aioLwJNpcb2alGFFQ0I0qnhEIuR6PVAblMBrVShuomO5xuD1yejrG4MRFKJBq0KG9uR3q0FsesLijkgEouR1pcBGZkJmBPRTO0KgVEiHAIIi7MiOfGjtTn9akxMsuXL8ff/vY31NbWIjs7Gy+++CImTpzYrecyyBCFD5tTwENr9+LQMSuuOTcVBq3SN2akzGyDXq2A1emGXXCjweJAbIQaGpUciZE6GDQKDIzXY2NRHY40WpFgUKO6xYE4vRoyGZCZHIUvDxyDzeEGZIBSDsRHaqGQyRD14+yewQMMqG22w+oUMDwhEvuOtsADEanREXjwZ5m456PvMTA+Aga1EhePSMCccSkwW+xY+nkx7E43lAo5RiRForzRhtunD8Ou8mZMzYjDf4vr0Ghx4X/763FpZgIO1rWhxSHgcJ0F7S43WtpdSI+NQHVLO7RKOZrbBbjcIrQqOaYNH4Bd5U1obXdC8AARagW0KgV+O20orp/UEVY+LayGCBEzMhOxqbge4nGDm4n6qj4TZD744APcfPPNWLFiBSZNmoRly5bho48+QklJCRISEs74fAYZovCxpqAKFWYbvjvSiOevH4sItdI3eNbmFLBsYykGxUcgb081zBYnUmJ0ePa6bHx9qAEOwQONUo4tB45h64F6eCDD3HNNyC+qR2ZyJHYeaYIoirC7PJDLAZ1SgagIFZKMOrQ73Whtd2JApBb1bXZY7AJkMhlEjxttDg/0agUSojRIiY7A0WYbFpw/GFeMScLWUjPa7AJe33IYZosDaqUc0RFqjE4xwqBWYFhiJDb+UIuYCBWOtTkRq1djT2Uz4iLUUCllONpsh0IuwupwQ6tUYExKNCqbbFArZKi3OHDR8ARMHhqLo03tWPVdJWRyEYIAjE2Lhl3w4IXrxyLeoO00W0qEyM02qV/oM0Fm0qRJmDBhAl566SUAgMfjQVpaGv7whz/gvvvuO+nxDocDDofD93VrayvS0tIYZIjCgM0pYMlnxZ26arp6zGOfFqGkpg1J0VrE6zUYnRoFq71jT6EPvqvA3uoWKGQynJMajb9fl417PiqEVqlAeaMNIxINcLlF7KlqxsRBMXjgqlG456Pv0dLuRIRaAQCob3EgWq+GxSGg0eKAUtHRdbVubw0uyojDvuo2pMZokR6jx9aDx1Dd3I52p4C0WD3SYiJQ12ZHvF6DH2pa4XJ7oFF1tNR8VWqG0+2B2wNo1Qro1Qq0O92QywGnIEKnkmPK0DjUtzkRHaFClFaF7DSjrxvLu8BgXuFRWO1unJNmxJPXZHeaLQWAm21Sv9AnZi05nU7s2rULM2bM8B2Ty+WYMWMGtm/f3uVzli5dCqPR6LulpaX1VrlEdAYRaiXuvyoTyUZdlwvReQfS3pszAtdPSodRp0JKjA4yyHytEQMiNVDJZdCqFEiP0eGLknq0O92wOd24cXI6jlmcKKhshs3hxndlTXjpi4O4e2YGqpvb0Wpzod3lgUolx9XjUvDurybislGJmD4yATEGFeaOS8H/9h9DcXULNv1Qi7WFRxGpVcEheJASEwG1Uo6q5nbERKjRYHUgSqeEQ/DA5fagqLoVcrkMgrvjjTUpUgNAhpunDMTAWD1k6Ag3VU12JEZq0NYu4GC9Bd8dboRGqUCEWgm1Uo5rxqciN9sEnVqBLJPR93PzzpY608wpov4mrP8SzGYz3G43EhM7v+ElJiZi//79XT5n8eLFuPvuu31fe1tkiCg8nG6JfO/qt9tKGzB/Qjpys02dWh9sTgEuoWNsiUwmxznp0dh1pAmVTe1Ii9WhtN6ChEg16lvtsDtdaHcBDVYn7l+zD4AM1S3tkMnlSIvRobSuDbEGNUaaorDux/VtzhsYC71GifIGGxRyIMLlRrnZiuzUaJQ1WCEIIuosDihkgClGh6qmdhi0SkTrVAAAm8ONUcl6yOWARqXASJMGR5vtUCsVMOpUUCtkmJVtgkGrhEPoCFoGjQKXZSV22qrg2vFpiNKquR8SUTeEdZDpCY1GA41GE+oyiKgHTtw24MTQ07H0vhLnpMagzGyFSxChksswMFaPYQMicG/OCCzbWIrRqTLsOOyGWimHxS7gmnNTsWpHBWIi9EiI1KC5vaNl5r/FdahvteNYqwONVid0yo6ZQQqFDEo5IJPLodd0BKgRSZHYUdYInUoOU3QEalvtSInWIfecjmDy9cFjaLS6MCwhAlOGDYBDcEMGGdRKOdrsLvzrmwqMTIr0jW2xOQVfWIlQKzudO/dDIuq+sA4y8fHxUCgUqKur63S8rq4OSUlJIaqKiIKlOxfwCzPisWxjKX4zbQh2lTfjvqsysa20AVMz4rC11IxFMzOwqbgeEwbGorS+DQunD8NXh8xYOH2YbxzK7vImNNudMLc6cVHGAPzr23LEalRoF9wYNiASoluEVfBgZKIedW1OzByVBLVSDo1CBpWiI+zYnW7Utth9wcQhuFHVZMf4QXG+ABVnUPtCS6S2o9XmdCGN4YXIf2EdZNRqNcaPH49NmzZhzpw5ADoG+27atAm33357aIsjopDYWmrG4Hg9dpU3+y78x6/vsq20AbnZJuQX1eLhSVnIL6qF1e5GnEGDeIMWaqUcg+L0KG8Epg4dgHXfVyNWr0aD1Qm3W8SFw+NxzOJAklKOZrsbg+L0MGg7xq+ckxqDOIMGUzPi8Ex+CbJMRl8wObE76MSWJa77QhQcYT9r6YMPPsCCBQvwyiuvYOLEiVi2bBk+/PBD7N+//6SxM13h9GuivuVU+x15j0/NiMNzG0uRGq1DcnTHoOITx9kc/7iYCBV2ljdhZmYiDFolcrNNvqng3lYfzhYi6n19Zvo1ALz00ku+BfHGjh2LF154AZMmTerWcxlkiPqXNQVVqG2x42hTu29bAa/jN3HML6pFbbMdR5tPfhwRhV6fCjJng0GGqH853Q7Vxy8sd2JLDRGFFwaZHzHIEJHX6UIOEYWX7l6/+ZdMRP0GZwYR9T1hvbIvERER0ekwyBAREZFkMcgQERGRZDHIEBERkWQxyBAREZFkMcgQERGRZDHIEBERkWQxyBAREZFkMcgQERGRZDHIEBERkWQxyBAREZFkMcgQERGRZDHIEBERkWT1+d2vRVEE0LEdOBEREUmD97rtvY6fSp8PMm1tbQCAtLS0EFdCRERE/mpra4PRaDzl/TLxTFFH4jweD6qrqxEZGQmZTBbQ125tbUVaWhoqKysRFRUV0NcOdzx3njvPvf/guffPcwdCe/6iKKKtrQ0mkwly+alHwvT5Fhm5XI7U1NSgfo+oqKh++QsO8Nx57v0Pz53n3h+F6vxP1xLjxcG+REREJFkMMkRERCRZDDJnQaPR4JFHHoFGowl1Kb2O585z72947jz3/kgK59/nB/sSERFR38UWGSIiIpIsBhkiIiKSLAYZIiIikiwGGSIiIpIsBpkzWL58OQYNGgStVotJkybhu+++O+3jm5ubsXDhQiQnJ0Oj0WD48OH4/PPPe6nawPLn3C+++GLIZLKTbldddVUvVhw4/v6/L1u2DCNGjIBOp0NaWhruuusu2O32Xqo2sPw5d5fLhccffxxDhw6FVqtFdnY21q9f34vVBs6WLVuQm5sLk8kEmUyGtWvXnvE5mzdvxrnnnguNRoNhw4bh7bffDnqdweDvudfU1OAXv/gFhg8fDrlcjkWLFvVKncHg77mvXr0aM2fOxIABAxAVFYUpU6YgPz+/d4oNMH/Pfdu2bbjgggsQFxcHnU6HkSNH4rnnnuudYk+DQeY0PvjgA9x999145JFHsHv3bmRnZyMnJwf19fVdPt7pdGLmzJk4cuQIPv74Y5SUlOC1115DSkpKL1d+9vw999WrV6OmpsZ327dvHxQKBebNm9fLlZ89f8/9/fffx3333YdHHnkExcXFeOONN/DBBx/g/vvv7+XKz56/5/7ggw/ilVdewYsvvogffvgBv/3tb3H11VejoKCglys/e1arFdnZ2Vi+fHm3Hl9WVoarrroK06dPx549e7Bo0SLceuutkryo+XvuDocDAwYMwIMPPojs7OwgVxdc/p77li1bMHPmTHz++efYtWsXpk+fjtzc3H7xO6/X63H77bdjy5YtKC4uxoMPPogHH3wQr776apArPQORTmnixIniwoULfV+73W7RZDKJS5cu7fLxL7/8sjhkyBDR6XT2VolB4++5n+i5554TIyMjRYvFEqwSg8bfc1+4cKF4ySWXdDp29913ixdccEFQ6wwGf889OTlZfOmllzodmzt3rnjDDTcEtc5gAyCuWbPmtI/505/+JGZlZXU6Nn/+fDEnJyeIlQVfd879eNOmTRPvvPPOoNXTm/w9d69Ro0aJjz32WOAL6kU9Pferr75avPHGGwNfkB/YInMKTqcTu3btwowZM3zH5HI5ZsyYge3bt3f5nLy8PEyZMgULFy5EYmIiRo8ejSVLlsDtdvdW2QHRk3M/0RtvvIGf//zn0Ov1wSozKHpy7ueffz527drl64I5fPgwPv/8c1x55ZW9UnOg9OTcHQ4HtFptp2M6nQ7btm0Laq3hYPv27Z1+VgCQk5PT7b8R6hs8Hg/a2toQGxsb6lJ6XUFBAb7++mtMmzYtpHX0+U0je8psNsPtdiMxMbHT8cTEROzfv7/L5xw+fBj/+9//cMMNN+Dzzz/HwYMH8fvf/x4ulwuPPPJIb5QdED059+N999132LdvH954441glRg0PTn3X/ziFzCbzZg6dSpEUYQgCPjtb38rua6lnpx7Tk4Onn32WVx00UUYOnQoNm3ahNWrV0suvPdEbW1tlz+r1tZWtLe3Q6fThagy6k3PPPMMLBYLrrvuulCX0mtSU1Nx7NgxCIKARx99FLfeemtI62GLTAB5PB4kJCTg1Vdfxfjx4zF//nw88MADWLFiRahL61VvvPEGxowZg4kTJ4a6lF6xefNmLFmyBP/4xz+we/durF69Gp999hn+8pe/hLq0oHv++eeRkZGBkSNHQq1W4/bbb8cvf/lLyOV8a6G+7/3338djjz2GDz/8EAkJCaEup9ds3boVO3fuxIoVK7Bs2TKsXLkypPWwReYU4uPjoVAoUFdX1+l4XV0dkpKSunxOcnIyVCoVFAqF71hmZiZqa2vhdDqhVquDWnOg9OTcvaxWK1atWoXHH388mCUGTU/O/aGHHsJNN93k+1QyZswYWK1W3HbbbXjggQckc1HvybkPGDAAa9euhd1uR0NDA0wmE+677z4MGTKkN0oOqaSkpC5/VlFRUWyN6QdWrVqFW2+9FR999NFJXYx93eDBgwF0vNfV1dXh0UcfxfXXXx+yeqTxDhsCarUa48ePx6ZNm3zHPB4PNm3ahClTpnT5nAsuuAAHDx6Ex+PxHTtw4ACSk5MlE2KAnp2710cffQSHw4Ebb7wx2GUGRU/O3WaznRRWvGFWlNBWZmfz/67VapGSkgJBEPDJJ59g9uzZwS435KZMmdLpZwUAGzduPOPPiqRv5cqV+OUvf4mVK1dKdomJQPF4PHA4HKEtIqRDjcPcqlWrRI1GI7799tviDz/8IN52221idHS0WFtbK4qiKN50003ifffd53t8RUWFGBkZKd5+++1iSUmJuG7dOjEhIUH861//GqpT6DF/z91r6tSp4vz583u73IDy99wfeeQRMTIyUly5cqV4+PBhccOGDeLQoUPF6667LlSn0GP+nvs333wjfvLJJ+KhQ4fELVu2iJdccok4ePBgsampKURn0HNtbW1iQUGBWFBQIAIQn332WbGgoEAsLy8XRVEU77vvPvGmm27yPf7w4cNiRESEeO+994rFxcXi8uXLRYVCIa5fvz5Up9Bj/p67KIq+x48fP178xS9+IRYUFIhFRUWhKP+s+Hvu7733nqhUKsXly5eLNTU1vltzc3OoTqHH/D33l156SczLyxMPHDggHjhwQHz99dfFyMhI8YEHHgjVKYiiKIoMMmfw4osviunp6aJarRYnTpwofvPNN777pk2bJi5YsKDT47/++mtx0qRJokajEYcMGSI+8cQToiAIvVx1YPh77vv37xcBiBs2bOjlSgPPn3N3uVzio48+Kg4dOlTUarViWlqa+Pvf/16SF3NR9O/cN2/eLGZmZooajUaMi4sTb7rpJvHo0aMhqPrsffHFFyKAk27e812wYIE4bdq0k54zduxYUa1Wi0OGDBHfeuutXq87EHpy7l09fuDAgb1e+9ny99ynTZt22sdLib/n/sILL4hZWVliRESEGBUVJY4bN078xz/+Ibrd7tCcwI9koiihtm8iIiKi43CMDBEREUkWgwwRERFJFoMMERERSRaDDBEREUkWgwwRERFJFoMMERERSRaDDBEREUkWgwwRERFJFoMMEYWdW265BXPmzPF9ffHFF2PRokVn9ZqBeA0iCj8MMkTUbbfccgtkMhlkMhnUajWGDRuGxx9/HIIgBPX7rl69Gn/5y1+69djNmzdDJpOhubm5x69BRNKhDHUBRCQtl19+Od566y04HA58/vnnWLhwIVQqFRYvXtzpcU6nM2C7vsfGxobFaxBR+GGLDBH5RaPRICkpCQMHDsTvfvc7zJgxA3l5eb7uoCeeeAImkwkjRowAAFRWVuK6665DdHQ0YmNjMXv2bBw5csT3em63G3fffTeio6MRFxeHP/3pTzhxC7gTu4UcDgf+/Oc/Iy0tDRqNBsOGDcMbb7yBI0eOYPr06QCAmJgYyGQy3HLLLV2+RlNTE26++WbExMQgIiICV1xxBUpLS333v/3224iOjkZ+fj4yMzNhMBhw+eWXo6amxveYzZs3Y+LEidDr9YiOjsYFF1yA8vLyAP2kiag7GGSI6KzodDo4nU4AwKZNm1BSUoKNGzdi3bp1cLlcyMnJQWRkJLZu3YqvvvrKFwi8z/n73/+Ot99+G2+++Sa2bduGxsZGrFmz5rTf8+abb8bKlSvxwgsvoLi4GK+88goMBgPS0tLwySefAABKSkpQU1OD559/vsvXuOWWW7Bz507k5eVh+/btEEURV155JVwul+8xNpsNzzzzDN59911s2bIFFRUVuOeeewAAgiBgzpw5mDZtGr7//nts374dt912G2Qy2Vn/TImo+9i1REQ9IooiNm3ahPz8fPzhD3/AsWPHoNfr8frrr/u6lP71r3/B4/Hg9ddf913g33rrLURHR2Pz5s247LLLsGzZMixevBhz584FAKxYsQL5+fmn/L4HDhzAhx9+iI0bN2LGjBkAgCFDhvju93YhJSQkIDo6usvXKC0tRV5eHr766iucf/75AID33nsPaWlpWLt2LebNmwcAcLlcWLFiBYYOHQoAuP322/H4448DAFpbW9HS0oKf/exnvvszMzP9/0ES0VlhiwwR+WXdunUwGAzQarW44oorMH/+fDz66KMAgDFjxnQaF1NYWIiDBw8iMjISBoMBBoMBsbGxsNvtOHToEFpaWlBTU4NJkyb5nqNUKnHeeeed8vvv2bMHCoUC06ZN6/E5FBcXQ6lUdvq+cXFxGDFiBIqLi33HIiIifCEFAJKTk1FfXw+gIzDdcsstyMnJQW5uLp5//vlO3U5E1DvYIkNEfpk+fTpefvllqNVqmEwmKJU/vY3o9fpOj7VYLBg/fjzee++9k15nwIABPfr+Op2uR8/rCZVK1elrmUzWafzOW2+9hTvuuAPr16/HBx98gAcffBAbN27E5MmTe61Gov6OLTJE5Be9Xo9hw4YhPT29U4jpyrnnnovS0lIkJCRg2LBhnW5GoxFGoxHJycn49ttvfc8RBAG7du065WuOGTMGHo8HX375ZZf3e1uE3G73KV8jMzMTgiB0+r4NDQ0oKSnBqFGjTntOJxo3bhwWL16Mr7/+GqNHj8b777/v1/OJ6OwwyBBR0Nxwww2Ij4/H7NmzsXXrVpSVlWHz5s244447UFVVBQC488478eSTT2Lt2rXYv38/fv/735+0BszxBg0ahAULFuD//u//sHbtWt9rfvjhhwCAgQMHQiaTYd26dTh27BgsFstJr5GRkYHZs2fj17/+NbZt24bCwkLceOONSElJwezZs7t1bmVlZVi8eDG2b9+O8vJybNiwAaWlpRwnQ9TLGGSIKGgiIiKwZcsWpKenY+7cucjMzMSvfvUr2O12REVFAQD++Mc/4qabbsKCBQswZcoUREZG4uqrrz7t67788su49tpr8fvf/x4jR47Er3/9a1itVgBASkoKHnvsMdx3331ITEzE7bff3uVrvPXWWxg/fjx+9rOfYcqUKRBFEZ9//vlJ3UmnO7f9+/fjmmuuwfDhw3Hbbbdh4cKF+M1vfuPHT4iIzpZMPHHBBiIiIiKJYIsMERERSRaDDBEREUkWgwwRERFJFoMMERERSRaDDBEREUkWgwwRERFJFoMMERERSRaDDBEREUkWgwwRERFJFoMMERERSRaDDBEREUnW/wPfIXEK/u2r4gAAAABJRU5ErkJggg==\n"
},
"metadata": {}
}
],
"source": [
"# plotting actual values vs. predictions\n",
"ypred = model2(torch.from_numpy(np.float32(x_test)).to(device)).cpu().detach().numpy()\n",
"plt.scatter(ypred,y_non_test,s=.1)\n",
"plt.xlabel(\"Predictions\");\n",
"plt.ylabel(\"Actual values\");"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "L3l0fBJ2pZSt"
},
"source": [
"### Shallow/wide nonlinear ANN\n",
"As we've seen, linear regression can't capture a nonlinear relationship. The traditional approach to dealiung with this would be to do some manual feature engineering, like creating a quadratic term by multiplying our \"x\" by itself. This approach would actually work fine with the simulated data we're playing with here. However, it does not scale up well to datasets with more variables and more complex nonlinearities. A major part of the appeal of ANNs is that - under the right circumstances - they can learn optimal nonlinear mappings between inputs and outputs for us.\n",
"\n",
"Most of the ANNs you'll see making news these days are \"deep\" neural networks. The deep in this phrase refers to stacking a large number of layers on top of each other. However, before we do that, it's worth taking a look at \"shallow\" neural networks. Shallow networks can actually do everything that deep nets can do - in principle. In practice, they tend to do this in a different way, that is [often less than ideal](https://ojs.aaai.org/index.php/AAAI/article/view/10913/10772). Specifically, shallow networks are memorization machines. They learn many simple local approximations to the relationship between inputs and outputs. With enough capacity, they can memorize an arbitrarily complex relationship, making them universal function approximators. Let's see how this works! Below we'll train a shallow ANN with many more units, and a nonlinear activation function ([ReLU](https://en.wikipedia.org/wiki/Rectifier_(neural_networks)) - probably the most popular nonlinear activation function for ANNs)."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/"
},
"id": "zkEuH0aXpY01",
"outputId": "6669be48-a4f8-46a3-f402-aea80369291c"
},
"outputs": [
{
"output_type": "stream",
"name": "stdout",
"text": [
"ShallowWideANN(\n",
" (layer_stack): Sequential(\n",
" (0): Linear(in_features=1, out_features=1000, bias=True)\n",
" (1): ReLU()\n",
" (2): Linear(in_features=1000, out_features=1, bias=True)\n",
" )\n",
")\n"
]
}
],
"source": [
"# define model architecture\n",
"class ShallowWideANN(nn.Module):\n",
" def __init__(self):\n",
" super().__init__()\n",
" self.layer_stack = nn.Sequential(\n",
" nn.Linear(1,1000), ## layer with 1000 units!\n",
" nn.ReLU(), ## apply a nonlinear activation function\n",
" nn.Linear(1000,1), ## single linear output\n",
" )\n",
"\n",
" def forward(self, x):\n",
" pred = self.layer_stack(x)\n",
" return pred\n",
"\n",
"model3 = ShallowWideANN().to(device)\n",
"print(model3)"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/"
},
"id": "vITd_1Ltnqt3",
"outputId": "9e6da902-8fef-439b-dce1-afecf1a84733"
},
"outputs": [
{
"output_type": "stream",
"name": "stdout",
"text": [
"Epoch 1\n",
"-------------------------------\n",
"loss: 2.869074 [ 64/100000]\n",
"loss: 0.120316 [ 6464/100000]\n",
"loss: 0.041936 [12864/100000]\n",
"loss: 0.037916 [19264/100000]\n",
"loss: 0.028931 [25664/100000]\n",
"loss: 0.040988 [32064/100000]\n",
"loss: 0.178121 [38464/100000]\n",
"loss: 0.031798 [44864/100000]\n",
"loss: 0.067478 [51264/100000]\n",
"loss: 0.031456 [57664/100000]\n",
"loss: 0.021962 [64064/100000]\n",
"loss: 0.018875 [70464/100000]\n",
"loss: 0.023422 [76864/100000]\n",
"loss: 0.053598 [83264/100000]\n",
"loss: 0.023415 [89664/100000]\n",
"loss: 0.033236 [96064/100000]\n",
"Epoch 2\n",
"-------------------------------\n",
"loss: 0.016946 [ 64/100000]\n",
"loss: 0.024055 [ 6464/100000]\n",
"loss: 0.046232 [12864/100000]\n",
"loss: 0.016391 [19264/100000]\n",
"loss: 0.019041 [25664/100000]\n",
"loss: 0.017909 [32064/100000]\n",
"loss: 0.016895 [38464/100000]\n",
"loss: 0.022496 [44864/100000]\n",
"loss: 0.050606 [51264/100000]\n",
"loss: 0.016962 [57664/100000]\n",
"loss: 0.020621 [64064/100000]\n",
"loss: 0.017214 [70464/100000]\n",
"loss: 0.021813 [76864/100000]\n",
"loss: 0.025850 [83264/100000]\n",
"loss: 0.013514 [89664/100000]\n",
"loss: 0.026187 [96064/100000]\n",
"Epoch 3\n",
"-------------------------------\n",
"loss: 0.021621 [ 64/100000]\n",
"loss: 0.015872 [ 6464/100000]\n",
"loss: 0.014883 [12864/100000]\n",
"loss: 0.022556 [19264/100000]\n",
"loss: 0.016457 [25664/100000]\n",
"loss: 0.013740 [32064/100000]\n",
"loss: 0.019121 [38464/100000]\n",
"loss: 0.018989 [44864/100000]\n",
"loss: 0.034832 [51264/100000]\n",
"loss: 0.024770 [57664/100000]\n",
"loss: 0.013531 [64064/100000]\n",
"loss: 0.013870 [70464/100000]\n",
"loss: 0.018242 [76864/100000]\n",
"loss: 0.182599 [83264/100000]\n",
"loss: 0.021299 [89664/100000]\n",
"loss: 0.020903 [96064/100000]\n",
"Done!\n"
]
}
],
"source": [
"# new we'll train the model\n",
"# note that we've tuned down the learning rate\n",
"loss = nn.MSELoss()\n",
"optimizer = torch.optim.SGD(model3.parameters(),lr=.001)\n",
"epochs = 3\n",
"for t in range(epochs):\n",
" print(f\"Epoch {t+1}\\n-------------------------------\")\n",
" train(train_dataloader, model3, loss, optimizer)\n",
"print(\"Done!\")"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/"
},
"id": "2N7wquOTAKvE",
"outputId": "a35bbe97-c508-4cd0-904f-6e5aec297b4e"
},
"outputs": [
{
"output_type": "stream",
"name": "stdout",
"text": [
"Test performance: \n",
" R^2: 0.990787, RMSE: 0.171964 \n",
"\n"
]
}
],
"source": [
"# performance metrics\n",
"# the shallow network is doing quite well!\n",
"perf = test(test_dataloader, model3, loss)"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/",
"height": 449
},
"id": "r3RWinyAri4-",
"outputId": "90e47067-1194-4153-f36c-5c72b4295cee"
},
"outputs": [
{
"output_type": "display_data",
"data": {
"text/plain": [
""
],
"image/png": "iVBORw0KGgoAAAANSUhEUgAAAjIAAAGwCAYAAACzXI8XAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjcuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/bCgiHAAAACXBIWXMAAA9hAAAPYQGoP6dpAAA/UUlEQVR4nO3deXzU1b3/8fckk5nsO1kJECGyBERkVaBgpVCwLNWi7c9W1N7ah8UiolbxFq21glK11NYLaqt2EZdbhSK2BC4VkQrIEhAjQtjCGkISspOZzPL7A2eaQICETDLznbyej8c8yqz5MNV8357zOeeY3G63WwAAAAYU4u8CAAAALhdBBgAAGBZBBgAAGBZBBgAAGBZBBgAAGBZBBgAAGBZBBgAAGJbZ3wW0N5fLpePHjysmJkYmk8nf5QAAgBZwu92qrq5WRkaGQkIuPO4S9EHm+PHjysrK8ncZAADgMhw5ckRdu3a94PNBH2RiYmIknf0iYmNj/VwNAABoiaqqKmVlZXmv4xcS9EHGM50UGxtLkAEAwGAu1RZCsy8AADAsggwAADAsggwAADAsggwAADAsggwAADAsggwAADAsggwAADAsggwAADAsggwAADAsggwAADAsggwAADAsggwAADAsggwAADAsggwAALgsdXaHluUfVZ3d4bca/Bpk1q9fr8mTJysjI0Mmk0nLly/3PtfQ0KCHH35YAwYMUFRUlDIyMnT77bfr+PHj/isYAAB45RUUq7zGrtUFJ/1Wg1+DTG1trQYOHKgXX3zxvOfq6uq0fft2zZs3T9u3b9d7772nPXv2aMqUKX6oFAAAnGtCbpqSoq0an5vqtxpMbrfb7bef3ojJZNKyZcs0bdq0C75my5YtGjZsmIqKitStW7cWfW5VVZXi4uJUWVmp2NhYH1ULAADaU0uv3+YOrKnNKisrZTKZFB8ff8HX2Gw22Ww27/2qqqoOqAwAAPiDYZp96+vr9fDDD+t73/veRZPZggULFBcX571lZWV1YJUAAKAjGSLINDQ06JZbbpHb7dbixYsv+tq5c+eqsrLSezty5EgHVQkAADpawE8teUJMUVGR/vWvf12yz8VqtcpqtXZQdQAAwJ8COsh4QkxhYaE+/PBDJSUl+bskAAAQQPwaZGpqarRv3z7v/YMHD2rHjh1KTExUenq6vvOd72j79u1auXKlnE6niouLJUmJiYmyWCz+KhsAAAQIvy6/Xrduna6//vrzHp8xY4Z+8YtfKDs7u9n3ffjhhxo7dmyLfgbLrwEAMB5DLL8eO3asLpajAmSLGwAAEKAMsWoJAACgOQQZAABgWAQZAABgWAQZAABgWAQZAABgWAQZAABgWAQZAABgWAQZAABgWAQZAABgWAQZAABgWAQZAABgWAQZAABgWAQZAABgWAQZAABgWAQZAABgWAQZAABgWAQZAABgWAQZAABgWAQZAABgWAQZAABgWAQZAABgWAQZAABgWAQZAABgWAQZAABgWAQZAABgWAQZAABgWAQZAABgWAQZAABgWAQZAAACWJ3doWX5R1Vnd/i7lIBEkAEAIIDlFRSrvMau1QUn/V1KQCLIAAAQwCbkpikp2qrxuan+LiUgmf1dAAAAuLBIi1nTBmX6u4yAxYgMAAAwLIIMAAB+RDNv2xBkAADwI5p524YgAwCAH9HM2zY0+wIA4Ec087YNIzIAAMCwCDIAAMCwCDIAAMCwCDIAAMCwCDIAAMCwCDIAAMCw/Bpk1q9fr8mTJysjI0Mmk0nLly9v8rzb7dZjjz2m9PR0RUREaNy4cSosLPRPsQAAIOD4NcjU1tZq4MCBevHFF5t9fuHChXrhhRe0ZMkSbd68WVFRUZowYYLq6+s7uFIAABCI/Loh3sSJEzVx4sRmn3O73Vq0aJF+/vOfa+rUqZKkP//5z0pNTdXy5cv13e9+tyNLBQAAAShge2QOHjyo4uJijRs3zvtYXFychg8fro0bN17wfTabTVVVVU1uAAAgOAVskCkuLpYkpaY2PXsiNTXV+1xzFixYoLi4OO8tKyurXesEAAD+E7BB5nLNnTtXlZWV3tuRI0f8XRIAAGgnARtk0tLSJEknTzY91vzkyZPe55pjtVoVGxvb5AYAAIJTwAaZ7OxspaWlae3atd7HqqqqtHnzZl177bV+rAwAAAQKv65aqqmp0b59+7z3Dx48qB07digxMVHdunXT7Nmz9atf/Uo5OTnKzs7WvHnzlJGRoWnTpvmvaAAAEDD8GmS2bt2q66+/3nt/zpw5kqQZM2bo9ddf189+9jPV1tbq7rvvVkVFhUaNGqVVq1YpPDzcXyUDAIAAYnK73W5/F9GeqqqqFBcXp8rKSvplAAAwiJZevwO2RwYAAOBSCDIAAMCwCDIAAMCwCDIAAMCwCDIAAMCwCDIAAMCwCDIAAMCwCDIAAMCwCDIAAHylzu7QsvyjqrM7/F0KWoggAwDAV/IKilVeY9fqgpP+LgUtRJABAOArE3LTlBRt1fjcVH+Xghby66GRAAAEkkiLWdMGZfq7DLQCIzIAAMCwCDIAAMCwCDIAAMCwCDIAAMCwCDIAAMCwCDIAAMCwCDIAAMCwCDIAAMCwCDIAAMCwCDIAAMCwCDIAAMCwCDIAAMCwCDIAAMCwCDIAAMCwCDIAAMCwCDIAAMCwCDIAAMCwCDIAAMCwCDIAAMOoszu0LP+o6uwOf5eCAEGQAQAYRl5Bscpr7FpdcNLfpSBAEGQAAIYxITdNSdFWjc9N9XcpCBBmfxcAAEBLRVrMmjYo099lIIAwIgMAAAyLIAMAMAyafXEuggwAwDBo9sW5CDIAAMOg2RfnotkXAGAYNPviXIzIAAAAwyLIAAAAwyLIAAAAwyLIAAAAwyLIAAAAwwroION0OjVv3jxlZ2crIiJCPXv21JNPPim32+3v0gAAQAAI6OXXzzzzjBYvXqw//elPys3N1datW3XnnXcqLi5Os2bN8nd5AADAzwI6yHzyySeaOnWqbrzxRklSjx499Oabb+rTTz+94HtsNptsNpv3flVVVbvXCQAA/COgp5auu+46rV27Vnv37pUk7dy5Uxs2bNDEiRMv+J4FCxYoLi7Oe8vKyuqocgEAQAczuQO44cTlcunRRx/VwoULFRoaKqfTqaeeekpz58694HuaG5HJyspSZWWlYmNjO6JsAADQRlVVVYqLi7vk9Tugp5beeecdvfHGG1q6dKlyc3O1Y8cOzZ49WxkZGZoxY0az77FarbJarR1cKQAA8IeADjIPPfSQHnnkEX33u9+VJA0YMEBFRUVasGDBBYMMAADoPAK6R6aurk4hIU1LDA0Nlcvl8lNFAAAgkAT0iMzkyZP11FNPqVu3bsrNzVV+fr6ef/553XXXXf4uDQAABICAbvatrq7WvHnztGzZMpWUlCgjI0Pf+9739Nhjj8lisbToM1raLAQAAAJHS6/fAR1kfIEgAwCA8bT0+h3QPTIAAAAXQ5ABAACGRZABAACG1eogc+bMGdXV1XnvFxUVadGiRVq9erVPCwMABLY6u0PL8o+qzu7wdynoxFodZKZOnao///nPkqSKigoNHz5czz33nKZOnarFixf7vEAAQGDKKyhWeY1d7+88TqCB37Q6yGzfvl2jR4+WJP3tb39TamqqioqK9Oc//1kvvPCCzwsEAASmCblpSoq2yi23ymvsWl1w0t8loRNqdZCpq6tTTEyMJGn16tW66aabFBISohEjRqioqMjnBQIA/K+5aaRIi1nTBmVqysBMJUVbNT431Y8VorNqdZDp1auXli9friNHjigvL0/jx4+XJJWUlLBPCwAEKc80UnOjLp5AE2kJ6M3iEaRaHWQee+wxPfjgg+rRo4eGDRuma6+9VtLZ0ZlBgwb5vEAAgP95ppHG56bS5IuAclk7+xYXF+vEiRMaOHCg91DHTz/9VLGxserTp4/Pi2wLdvYFAN9aln9U5TV2JUVbNW1Qpr/LQZBq151909LSFBMTozVr1ujMmTOSpKFDhwZciAEA+F7j0RnA31odZMrKynTDDTfoyiuv1KRJk3TixAlJ0g9/+EM98MADPi8QABBY6IlBIGl1kLn//vsVFhamw4cPKzIy0vv4rbfeqlWrVvm0OAAAgItpdZxevXq18vLy1LVr1yaP5+TksPwaAAB0qFaPyNTW1jYZifEoLy+X1Wr1SVEAAAAt0eogM3r0aO8RBZJkMpnkcrm0cOFCXX/99T4tDgAA4GJaPbW0cOFC3XDDDdq6davsdrt+9rOfqaCgQOXl5fr3v//dHjUCANqgzu5QXkGxJuSm0aCLoNPqEZn+/ftr7969GjVqlKZOnara2lrddNNNys/PV8+ePdujRgBAG1xsV17A6C5rQzwjYUM8AJ1dnd2h1QUnNT43lREZGEZLr9+t/id6/fr1F33+a1/7Wms/EgDQjjz7vgDBqNVBZuzYsec9ZjKZvH92Op1tKggA4Bv0xqAzaHWPzOnTp5vcSkpKtGrVKg0dOlSrV69ujxoBAJehcW8MBz0iWLU6osfFxZ332De+8Q1ZLBbNmTNH27Zt80lhAIC2mZCb5u2NaRxqmGZCMLmsQyObk5qaqj179vjq4wAAbRRpMXtDzOicZA56RFBq9YjMZ5991uS+2+3WiRMn9PTTT+vqq6/2VV0AgMvUuDfGMxKzobCMkRgEpVYHmauvvlomk0nnrtoeMWKEXn31VZ8VBgC4PI2nkRpPLwHBqNVB5uDBg03uh4SEqEuXLgoPD/dZUQCAy9c4vLD0GsGu1UGme/fu7VEHAOAyNbfM2q2g3usU8GpRkHnhhRda/IGzZs267GIAAK3XeCppfG6qnvpgt7rGR7BCCZ1Ci44oyM7ObtmHmUw6cOBAm4vyJY4oABDsGh9BkFdQrOLKeh07fUaP3tiXjfBgWC29fnPWEgAEEc5VQrBot7OWAACBi+ZedDaXFWSOHj2qFStW6PDhw7Lb7U2ee/75531SGAAAwKW0OsisXbtWU6ZM0RVXXKEvv/xS/fv316FDh+R2u3XNNde0R40AAADNavURBXPnztWDDz6oXbt2KTw8XO+++66OHDmiMWPGaPr06e1RIwAAQLNaHWR2796t22+/XZJkNpt15swZRUdH65e//KWeeeYZnxcIAABwIa0OMlFRUd6+mPT0dO3fv9/7XGlpqe8qAwAAuIRW98iMGDFCGzZsUN++fTVp0iQ98MAD2rVrl9577z2NGDGiPWoEAABoVquDzPPPP6+amhpJ0hNPPKGamhq9/fbbysnJYcUSAADoUGyIBwAAAk5Lr9+t7pH5r//6L61bt64ttQEAAPhEq4PMqVOn9M1vflNZWVl66KGHtHPnzvaoCwAA4JJaHWT+/ve/68SJE5o3b562bNmia665Rrm5uZo/f74OHTrUDiUCAAA0r9VBRpISEhJ09913a926dSoqKtIdd9yhv/zlL+rVq5ev69OxY8f0/e9/X0lJSYqIiNCAAQO0detWn/8cAGhPdXaHluUfVZ3d0ex9AJfnsoKMR0NDg7Zu3arNmzfr0KFDSk1N9VVdkqTTp09r5MiRCgsL0z//+U998cUXeu6555SQkODTnwMA7S2voFjlNXatLjjZ7H0Al+eyDo388MMPtXTpUr377rtyuVy66aabtHLlSn3961/3aXHPPPOMsrKy9Nprr3kfy87Ovuh7bDabbDab935VVZVPawKAyzEhN02rC05qfG5qs/cBXJ5Wj8hkZmZq0qRJKi0t1csvv6yTJ0/q1Vdf1Q033CCTyeTT4lasWKEhQ4Zo+vTpSklJ0aBBg/TKK69c9D0LFixQXFyc95aVleXTmgDgckRazJo2KFOStCz/qCRp2qBMRVou678nAXyl1fvIvPLKK5o+fbri4+PbqaT/CA8PlyTNmTNH06dP15YtW3TfffdpyZIlmjFjRrPvaW5EJisri31kAASEZflHVV5jV1K01RtsAJyvpfvIBPSGeBaLRUOGDNEnn3zifWzWrFnasmWLNm7c2KLPYEM8AIGkzu7wTikxGgNcWLttiNeR0tPT1a9fvyaP9e3bV4cPH/ZTRQDQNp4pJkIM4BsBHWRGjhypPXv2NHls79696t69u58qAgAAgSSgg8z999+vTZs2af78+dq3b5+WLl2ql19+WTNnzvR3aQAAIAAEdJAZOnSoli1bpjfffFP9+/fXk08+qUWLFum2227zd2kAACAAtKjZd8WKFS3+wClTprSpIF+j2RcAAONp6fW7Rd1m06ZNa9EPNZlMcjqdLXotAABAW7UoyLhcrvauAwAAoNUCukcGAAIZBz8C/ndZGxnU1tbqo48+0uHDh2W325s8N2vWLJ8UBgCBrvHBj+zSC/hHq4NMfn6+Jk2apLq6OtXW1ioxMVGlpaWKjIxUSkoKQQZAp8HBj4D/tXpq6f7779fkyZN1+vRpRUREaNOmTSoqKtLgwYP17LPPtkeNAOA3F5s+YpdewP9aHWR27NihBx54QCEhIQoNDZXNZlNWVpYWLlyoRx99tD1qBAC/aTx9BCDwtDrIhIWFKSTk7NtSUlK85x7FxcXpyJEjvq0OAPxsQm6akqKtTB8BAarV46GDBg3Sli1blJOTozFjxuixxx5TaWmp/vKXv6h///7tUSMA+EWd3aG8gmJNyE1j+ggIUK0ekZk/f77S09MlSU899ZQSEhJ0zz336NSpU3r55Zd9XiAAdJRz+2GYVgICX4uOKDAyjigA0FLL8o+qvMaupGirpg3KVJ3d4V2VxIgM0LFaev1mQzwA+Mq5/TCsSgICX6v/7czOzpbJZLrg8wcOHGhTQQDgL57gAsA4Wh1kZs+e3eR+Q0OD8vPztWrVKj300EO+qgsAAOCSWh1k7rvvvmYff/HFF7V169Y2FwQAHYEVSUBw8FmPzMSJE/Xuu+/66uMAwCcutDMvK5KA4OCzIPO3v/1NiYmJvvo4APCJCwUWNroDgsNlbYjXuNnX7XaruLhYp06d0v/8z//4tDgAaKsLHexIYy8QHFodZKZOndokyISEhKhLly4aO3as+vTp49PiAKCtCCxAcGNDPAAAEHDabUO80NBQlZSUnPd4WVmZQkNDW/txAAAAl63VQeZCAzg2m00Wi6XNBQFAS1xoNRKAzqXFPTIvvPCCJMlkMukPf/iDoqOjvc85nU6tX7+eHhkAHabxaiR6YIDOq8VB5je/+Y2ksyMyS5YsaTKNZLFY1KNHDy1ZssT3FQJAMy60GglA59LiIHPw4EFJ0vXXX6/33ntPCQkJ7VYUAFwKq5EASJfRI/Phhx8SYgD4Ff0xADxaHWRuvvlmPfPMM+c9vnDhQk2fPt0nRQGAR3OhheMFAHi0OsisX79ekyZNOu/xiRMnav369T4pCgA8mgstHC8AwKPVQaampqbZZdZhYWGqqqrySVEA4DE6J1kHS2s1KifJ+5inP4ZTqwG0OsgMGDBAb7/99nmPv/XWW+rXr59PigIAj48LS5WdHKUNhWX+LgVAAGr1f87MmzdPN910k/bv36+vf/3rkqS1a9fqzTff1P/+7//6vEAAnUed3aG8gmJNyE3zjrawzBrAxVzWWUsffPCB5s+frx07digiIkJXXXWVHn/8cY0ZM6Y9amwTzloCjGNZ/lGV19iVFG1laTXQybX0+u3TQyM///xz9e/f31cf5xMEGcAY6uwOrdh5XCZJN/RN0ceFpU1GZgB0Lu12aOS5qqur9fLLL2vYsGEaOHBgWz8OQCeVV1Cs2nqHrOZQfVxYyvJqAC1y2UFm/fr1uv3225Wenq5nn31WX//617Vp0yZf1gagk6izO2RzuBQdbtb43FSWVwNosVaN2RYXF+v111/XH//4R1VVVemWW26RzWbT8uXLWbEE4LJ5RmOSoq3eqSR6ZAC0RItHZCZPnqzevXvrs88+06JFi3T8+HH97ne/a8/aAHQSjMAAuFwtHpH55z//qVmzZumee+5RTk5Oe9YEIAg1t7T6Yo8DQEu0eERmw4YNqq6u1uDBgzV8+HD9/ve/V2lpaXvWBiAIeM5KWrHzeLMNvJybBKAtWhxkRowYoVdeeUUnTpzQj3/8Y7311lvKyMiQy+XSmjVrVF1d3Z51AghQlzqJesXOY1r35Sk1OFzNTh8xrQSgLVq9aikqKkp33XWXNmzYoF27dumBBx7Q008/rZSUFE2ZMqU9agQQwC49omKSySRZzCHNno/EuUkA2qJN+8j07t1bCxcu1NGjR/Xmm2/6qiYABnF22bTTu2y6OVMGZmhs7xRNHpjRwdUB6Ax8urNvIGJnX6D9tPZIARp7AbRUh+3s25GefvppmUwmzZ4929+lAFDr+1to7AXga4YJMlu2bNFLL72kq666yt+lAPhKa/tbaOwF4GuGCDI1NTW67bbb9MorryghIcHf5QC4TDT2AvA1QwSZmTNn6sYbb9S4ceMu+VqbzaaqqqomNwC+d6ll1wDQEQI+yLz11lvavn27FixY0KLXL1iwQHFxcd5bVlZWO1cIdE70uwAIBAEdZI4cOaL77rtPb7zxhsLDw1v0nrlz56qystJ7O3LkSDtXCXROo3OSdbC0VqNykvxdCoBOLKCXXy9fvlzf/va3FRoa6n3M6XTKZDIpJCRENputyXPNYfk10HKtWR7d2qXXANAaQbH8+oYbbtCuXbu0Y8cO723IkCG67bbbtGPHjkuGGACtc+50UXN9MJ7HRuckswIJgN8F9NKBmJgY9e/fv8ljUVFRSkpKOu9xAJfPMxIzOidZGwrLvOGkcbDxjLp4HttQWMZIDAC/C+gRGQDt49yRlsbhZHxuqvIKilVndzS77wt7wQAIJAHdI+ML9MgA5zu3v6XO7tDqgpPeEEPvCwB/C4oeGQDt49xRFc9GdZIueQgkAAQSggzQCV1oh928gmLV1jtlNZ9tpGfDOwCBjiADwKvxSA0b3gEwAoIM0Ald6HiBxiM1NPUCMAKCDBCEPEGltKa+2cDSktEWDngEYAQEGSAIeYLKs6v2aN2Xp/T+zuNNnme0BUCwIMgAQcgTVPplxspkOv95RlsABAt+iwEG05LzkDxBpc7uUGy4pdmRl9acqwQAgYoRGcBgLtTf0riB1/NnSRcceWFVEoBgQJABDObc/hZPaFmx85g3mLQkpNAnAyAYEGQAA/FMB43PTfWOsuQVFKu4sl75hysUHW7WqJwk2RyuS+7OS58MgGBAkAECyIX2d/HIKyhWcUW9nni/QG9tKfIe7Hjs9Bn1SIySJP1mTaFO19plNYcSUgAEPYIMECDq7A499cFuFVfUX3BKaEJumo5VnFGNzaFN+8u9r+vfNU6J0Ra55VbX+AgdO32GKSMAnQJBBggQeQXF6poQoWMVFw4hkRazHr2xr0ZkJyncHKJROUlfnY/kkNUcqikDM5UeH6FHb+zLaAyAToEgAwSICblpSo+7dAiJtJgVHW5W96QoLVpTqNE5yd6mXfpeAHQ2BBkgQERazjbqPvXBbpXW1F/0tZ4ppsyECG0oLCO8AOi0CDJAAFnwwW59tKdET3+w+6Kv80wxpcdF0AsDoFMjyAABxOFyq8bm0IGyOu/KpZacVA0AnRVBBvCzxkFlcI8ExUdalBJj8a5IOndzu0st0QaAzoQgA3Swc4OIZ0O7+R/s1qQB6bpnbE9d3yfVO2V07g68HC0AAP9hcrvdbn8X0Z6qqqoUFxenyspKxcbG+rscdHKevWK6JkQoMcoiizlEo3OStWhNoTLjI5QeH6FpgzIv+RmrC0422d0XAIJNS6/fBBmgA9TZHVqx85jyD1coIzZCJyrPyG2SeiRGKT3+bMMu4QQA/qOl12+mloAOkFdQrA17S7WnuFonqs7o6u4J6pEU5d38ztO4K4n+FwBoBYIM0EYtab6dkJumaKtZXRMiNahbgqYMzFBilEX9u8Y1eR39LwDQOgQZoI1aEj4iLWY9NiVXo3OS5ZnLtZhDVFvvaPK+cxt7AQAXR5AB2qDO7pDN4VJ0uPmS4SPSYm4SXkbnJOtgaa1G5SQ1eQ17wwBAyxFkgDZofGBjS8JH4xGXjwtLlZ0cpQ2FZR1QKQAEJ4IM0AatnQpqPOLCNBIAtB3LrwEf8CyvlkyaMjCDqSEAaKOWXr/5bQu0kWeTu4pau8LMIQo3h15yUzsAgG8QZIAWqrM7lFdQrAm5aYq0mL33bQ6XuiZEyOlyaVC3BKaKAKADEWSAi2gcXlbsPKaN+8pld7h069Bu3mXX0eFmpcdFaMZ1PZhSAoAOxm9d4CuNQ4sk72jLf/Z6McnpduvTg+VySxrXN0UbCss0KidJHxeW+rV2AOisCDLAVxpvbOeW2zva0nhl0edHK3W61q5N+8u8vTDL8o9630dvDAB0LIIMOj3PSMzonGSt3V2iqvoGSW5Fh5s1udEKpDq7Q/27xqnB4ZLFHOINNxNy07wHPgIAOhb7yKBT86w4Kq6s14bCMlnMIdq8v0x/zz/ufY3nHCXP5ncx4WG6dWg3b8BhN14A8B+CDDqVcw94zCsoVkq0VR8XlmpUTtLZwx0jzEqPj/A+75k2YgM7AAg8BBkEpeZOpK6zO/TEigKt/eKk3t95dsRldE6yNh8q1zXd4rVoTaEk6fHJuRrXN1WTB2Y0CS+MvABA4GFnXxje2V11j0tya8rAs0HD04AbHX72oMbROcn6zZpCldXYVFxVr55dojU0O0GSSadr7Nqwr1SjeiUrPT6Chl0ACAAtvX4zIgPDyyso1sb9Zdqwt0zzP9itOrvj7BRRuFnbi07rcGmdZr25Q6mxVsVHhKl3WozqG5zatL9cJknp8RH67feuVnp8BNNGAGAwjJHD8CbkpsnucCm/6LQyEyK8K4i2Hz6tsjq7thad1qQBaSqpsumxKbmS5J1aarwqiZEYADAeggwML9Ji1q1Du2nywAxviMkrKFaNzaG9xdWKjwjTqUYhJq+gWJMHZnj/7DlyAABgPEwtwbBKa+r138t26Uh5rZblH5UkjcpJ0hMrClRWY9OI7CQN7ZGgzIQIuU1n39N4FVJeQbGKK+u901EAAOOh2ReG49nTZdOBMpXX2FVrd2rEFYnadKBcCZFh+mR/mbISInTXqCs0PjdV8z/Yrcz4CG8PTOPN6xo/x9QSAASOoGj2XbBggYYOHaqYmBilpKRo2rRp2rNnj7/Lgh/U2R16a8thvbWlSCt2Hld5jV1XpsSo1u7ULyb306YD5QoLMenTg2WymkMUHhbiXTL96I19vSGm8RLqc58DABhPQAeZjz76SDNnztSmTZu0Zs0aNTQ0aPz48aqtrfV3afCx5vZ98Tz+1pbDeuL9An1cWOpdaZQUbVWY2aQuMVblH6nQC9+7Wg0ut26+pqt6JEXpW1/1wEgX33mXvWEAwNgC+rf3qlWrmtx//fXXlZKSom3btulrX/tas++x2Wyy2Wze+1VVVe1aI3yjce/K+NxU774wkkkb95fJ6XQpJtysQd0SvCuN3tpSpK9aXxRpMWvK1ekyyaQeXaJUW+/kEEcA6AQCOsicq7KyUpKUmJh4wdcsWLBATzzxREeVBB/xHLw4KidJT32wWxVn7DLJpPBQk8whJg3pnqSbB3dtcoCj3eFSuDlEN/RN+eocJKd3F14OcQSAzsEwzb4ul0tTpkxRRUWFNmzYcMHXNTcik5WVRbOvQSzLP6riinoVlddKbqnG5pA5NERje6do2qDMr3bxPab8wxWqqXco7KvnGocXpokAwPha2uxrmN/4M2fO1Oeff37RECNJVqtVVqu1g6qCr03ITdP7O48rKjxUkkkmSRbz2cZdz0nVlXV2yWTyTjU1buIFAHQuhggy9957r1auXKn169era9eu/i4HPuJZRu3ZkM5z3+Zwann+caXEnD1S4OruCZLO9tF0TYiQ0+nSoO4JTXblBQB0TgF9FXC73frpT3+qZcuWad26dcrOzvZ3SWilc8NKY40bfKcNyvTeLzxZrYz4CJVW2yS39L9bjsgkeXfunXFdDwIMAEBSgC+/njlzpv76179q6dKliomJUXFxsYqLi3XmzBl/l4YWWrHzuP5vd4meeL/gvKXVnoMd6x0O70GPSdFWPfjN3hrXN1W/v22Q4iPDlB4fIYml0gCA8wV0kFm8eLEqKys1duxYpaene29vv/22v0tDC9kdTn1+rFKVdWdHXqT/7Bkjne1/8SyV9gSV5Ohw7/8+NiVX4/qmavLAjAvuNQMA6LwCOsi43e5mb3fccYe/S0MLWcyh6pseo4q6Bo3KSZLUdErJMwrjaeY9N6g0HoVp/D4AAKQADzIwpsaBZMrADCVHWTWmd4rW7i7Rsvyjyk2P1dovSzS4e3yrgkrj0AMAgESQgQ+cO5Jy7qnSV6ZFq6i0VtX1Dfq/gpO6543tCgsx6eF3d6nO7vC+f3RO8kWDCj0yAIBzGWZDvMvF6dft5+zmdMeVX3RaPZKivCdI19kdeuL9AtXUO8428za45HS6daqmXrYGl7rEWHWorE6TB6arW2KU3HKrvMaupGgre8EAACQFyenXCBzN9a/kFRRr4/4yVZ5p0Mf7SjW4e7ze2nJYK3YeU05KtE5U1uvKlBhd1zNJ0dZQjeyVrD7pMRrXL1Vv3j1c3RKjND43lSkjAMBlY0QGLbIs/+h5oyalNfX6dd4eOZwupcdFKK/gpHokRaqs1q7s5Eg5XdK1PZN069BuqrM7OEIAANBiQXdEAfyntKZemw6U6cqUGFXVN+jPGw/JYg6R5NaVKTGymEP0+icHFWIyqbCkRvERYeqXHqfocLPccqu0pl4fF5Y2uykeAABtwVUFki6+A+9v1hSqvMauf5WXKDnaqqKyWrncbplDTMruEq2+6TGKtJhVecaha7rGSiEhig43y2IOUXmNXYvWFCozPkLzP9itR2/sS5gBAPgMVxRI+s/eLu/vPK6zc41uTRl4doXQ/d/I0bN5ezQ8JVommXSsok7FlTZV1zeotMYuk9ut7olRiraG6sFv9taGwjJvv8vqgpOaOijjbJhJiPAeRwAAgC8QZKA6u0M2h+urqSBp4/4ymSSFm/+z0+7wK5JUXmPXFycqdcbuVK3dKZPciraa5TaZNPrKJO8hjo2DiufPj97Y19sjAwCAr7BqCcorKFZtvUNWc6imDMzQVZlxOlFZr37pMVqWf1SlNfWyOVyymEN0uOyMjlWckcvtVmxkuGIiw9SrS7Ss5tCLThmxBwwAoD1wVYEm5KY1WVF0oLRWseFm/eL9LzQ6J1mz3tyh0TnJOnb6jEblJEsmKTM+QoO7J2jigLQmU0kAAHQkgkyQuljz7rmvW7HzmCST937vtBjZGhwafkWi9p2s0fAeiSoqrVW/zFiZZNLNgzO9/TOS6HkBAPgNQSZINW7etZhDvIGmccCpszs0680dig03yxwaovzDp+VwulRrc+p0XYO6JUXJbZLS4sOVGh+u2nqH9pZUqd7uUoPDrehwM0uqAQB+RY9MkPLslmtzOLXuy1N6f+dxSU1Pnv71qj2qqLWrpMqmUzU2ldXa9OnBcm0+WKaj5XX6uLBUXaKsem/7UdXUNyg63KzcjDiZTNLnxys5iRoA4HcEmSDlaa61mENlMkl2h+u8gxlzM2MVYQlVdpcojc3popozDsVHWmSSSW5J3RMitP1IharqHVq584R2Ha3UpAFpGts7RQ9N6M2xAgAAv2NOIEh5ppBG9kzS50crZXM4tWl3mTYUlmpodoIkadKAdO0trtGPx1yhbUUV+uXU/vrvZZ9raHaCIkJD1CcjVn0ypC9PVMstKTMhQhsKy7w9MfTGAAD8jSATZDzNu/mHK5QRG6H/3XpUI65IVN7nxTpZbVNceJicLrcaHG6t/OyEEiLDtG7PKUWHm/XHfx/UmQanSqvd6pseq11Hq2QySWN7p2h8bir7wAAAAg5BxsAaN+5K0t+2HdX7O47pTINTkrT1ULkmDUjX5gPlSoqxqMHlVq8u0RqYFa+Vnx1XZW2DTlSe0ekzDRp7ZRddmRKjQ2V1Gt83VVMHZWjt7hJJ8i7LZgQGABBoOP3awBqfSF3vcOp/PtyniroGOZwuhYeFKj0uXH3TY9UnPVa7jlbKHCrNndRXHxeW6kh5nTbsPSWHW0qKtCgp2qKruyeott7R5IRrAAD8oaXXb5p9DcyzMmlUTpK2HChTg9Mlh9OpuMgwWcJMOl5ZrwOnarW1qFzbD5/WlkOn9ff847I5nEqNDddNQ7oqJdaqwlM1ys2M05SBGTTwAgAMhSBjQHV2h5blH5V0dtrnN2sKVWlzyNbgUlxEmKzmUCVHWeV2u2VzuVRaY1eNzSGbw6mlnx7W6ZqGr44jyFR8eJj6Z8TJYg7hGAEAgOFwxTIITz/M6Jxk/XrVHpXV2fX2p4c16aoMdU2I0N6TVYq0mFVrdyghKlQRllD1SI5Wry6RuiorXp/sK1VhSa0m9k/TsYozmjGyhyItZj02JZcmXgCAYRFkAljjZl7PRnaL1hSq1ubQjsMVSomx6ovjlcpJiVFptV1XpESput6hnslRmj6kq36+vEB3f62nuiZGKjbcol/lJHnPRfKMutDECwAwMqaWAoxn2sgTYoor6zX/g90anZOs6HCzeiRHyu12KTM+Qlemxuiukdl6af1+VdY3aMvBcg3rkaDwsFA9v6ZQJrn1xPtfeMNKcnQ4U0cAgKBCkAkwjY8QmJCbpkNltTpda9c/dxXrs6OV2nygXB8VlsnmcCo8LES/+CqonK5tUGqsRf/68pQyEyI0vl+qusSE6+mbB/j7rwQAQLshyPhR49EXD89KJM/0T25GrI5VnNHfth3VZ0crtLu4Sia3S6eqbDpjc6hverSKK88oOdqsEFOIXr1jiNLjIvS94d30xo9GKCsxyo9/QwAA2hdzDH7UePSlcZ9KvcOhv207Kos5RHaHS+W1dlXVN8gSalJ5bYPMoSbFWkJlc7q1YscJWUJD1OCSBnVLYA8YAECnwoiMHzUeffHIKyjWut2n9Nv/26ulm4q0+0S14iPDFB4WqtLaBskkOV1upcWGK9oSqtRYq0JCTEqItKhXl2hOowYAdCqMyPhRpMWs8bmpWrHzuCS3RvZM1qYDZTpacUZ1NodOVdtkDglRaGiIzCaTIsNMsjullBirusRaNbBbggZ2S1DB8UrNvL6XthVVsIwaANCpEGT8LK+gWB8XlurY6Tq9+GGhqs+4FGkJVVhYiMrr7HLJrdSYcMVFmFVSXa/kmHDlpEbryxPVGtkzWT+4tof3s+iHAQB0NkwtdYDmmnolqbSmXhsKS3W0vFYl1fUqrW5QTX2D6uwO2ewuxVjNanC4VW1zqMruVEiISWcaHDpSXqc+6TGymPm/DwDQuTEi0wGaa+otranX9MUbVVZbL5PpbFOvy+WSW1KDw6EGp+RwuRUbYVZYiEkxVrPMISb1S4/VlakxOlVt0+SBGf79iwEA4GcEmQ4wITfNewxAnd2hFTuP650th3W8ok52p2SSS5ZQk1ySwkKk2oaz76s641B6bLhO1do1tEe0JuSma3xuqlYXnNSPvnYFG9sBADo9roTtrPExA5EWsxat2aP/WbdPJkl2p+TW2fCSHGNVXESYDpXVKtHs1mmbW0mRZkVazRqTGaursxK8e8uwvBoAgLMIMu3EE2BsDpdO19g1b/kuSdI/Py+W3Xn2NWaTJJMUGmpSXESYjlWckclkUq3DpRHZiSqpsWnigDSZZFJMeBgjMAAAnIMrYxucO9rS+LGyGpv+satY3+ibqnV7T+nAqWqdqmlo8n6H++z/AW63dKisVk6XSzaHlBJjUWmtXd+5pqtiwsNkNYeyrBoAgGYQZNqgcRPv2f1gjin/cIV6JEVp9RcndaCkRgdP1cgSFnJeiPGIsIQqMdqiHolR2nWsQtYwKT0uXL/7f9d494VhJAYAgOaxfrcNRuck62BprUblJCmvoFgf7i7Rxn1l2n+yWhlx4TrT4FRdg1Mnq+wX/Ay706V377lOWYmR6poYqRirWX3SY5WVGMVJ1QAAXAJXyTb4uLBUmfERWrSmUNOHdNUn+8tkNZv07/2lOl1nl8MpuZp5X6jONvm6JP3uu1crOTpcj97YV+9uO6qC45V6cELvjv2LAABgUASZNpiQm6b5H+xWtDVUtyz5RA6XVGM7G1IuJETS8J6J2nOiWl1irFpfWKbx/TMUaTE32aUXAABcGkHmMpXW1OvXeXuUHGXRHz4+KJuzZe8LDZEaHC4NuyJRp+saNPsbOe1bKAAAQYwgc5l+nbdH/9pdolM1F+5/aU6IpK/3SVV6fASNvAAAtBHNvpcpJyWmxSEmzCRZQ02yhkpdYsOVGG2hkRcAAB8wRJB58cUX1aNHD4WHh2v48OH69NNP/V2SLt4J8x8mSalx4eqTHqOeydGyhIboup5J7VsaAACdRMAHmbfffltz5szR448/ru3bt2vgwIGaMGGCSkpK/FrXrqOVLXpdZFiIhnSL1+heXVR2pkFZiZF66aMD7VwdAACdQ8AHmeeff14/+tGPdOedd6pfv35asmSJIiMj9eqrrzb7epvNpqqqqia39lBZ3/wGdx6WUMlqNikzIUJmc6gqzjToliFd5XC5afAFAMBHAjrI2O12bdu2TePGjfM+FhISonHjxmnjxo3NvmfBggWKi4vz3rKystqltu0Hyy74XKgkc2iIclKiFWkJ08CseD16Y1/17BKjV2YMUXJ0eLvUBABAZxPQQaa0tFROp1OpqU3PGUpNTVVxcXGz75k7d64qKyu9tyNHjrRLbTZn8z0yoZLS463qlRKtrMRI9c+M1c2Du3pPrabBFwAA3wm6q6rVapXVam3/nxMWKpuz6eYxJklxkWb9eEwvWcxnM+LkgRmEFwAA2klAX2GTk5MVGhqqkydPNnn85MmTSktL81NVZ13dNV7r9/1neinGYlJ2lxhlJ0dq4oA0po8AAOgAAT21ZLFYNHjwYK1du9b7mMvl0tq1a3Xttdf6sTLpTEPT0ZgRPbvoqq7xuqprgjYUXrh/BgAA+E5ABxlJmjNnjl555RX96U9/0u7du3XPPfeotrZWd955p1/rev7Wq5UZf3bUpUdihKrrHfrxmCuUFG3V+NzUS7wbAAD4QkBPLUnSrbfeqlOnTumxxx5TcXGxrr76aq1ateq8BuCOlpUYpR+P6an3th+TOdSkMb27aFtRhaYNyvRrXQAAdCYmt9vdsi1qDaqqqkpxcXGqrKxUbGysTz/7rS1F2rS/XIO7JygmPIyzkwAA8JGWXr+56rbBlIGZCjebCTAAAPgJV9828OwNAwAA/CPgm30BAAAuhCADAAAMiyADAAAMiyADAAAMiyADAAAMiyADAAAMiyADAAAMiyADAAAMiyADAAAMiyADAAAMiyADAAAMiyADAAAMiyADAAAMK+hPv3a73ZKkqqoqP1cCAABaynPd9lzHLyTog0x1dbUkKSsry8+VAACA1qqurlZcXNwFnze5LxV1DM7lcun48eOKiYmRyWTyyWdWVVUpKytLR44cUWxsrE8+E83ju+44fNcdh++64/Bddxxff9dut1vV1dXKyMhQSMiFO2GCfkQmJCREXbt2bZfPjo2N5V+MDsJ33XH4rjsO33XH4bvuOL78ri82EuNBsy8AADAsggwAADAsgsxlsFqtevzxx2W1Wv1dStDju+44fNcdh++64/Bddxx/fddB3+wLAACCFyMyAADAsAgyAADAsAgyAADAsAgyAADAsAgyrfTiiy+qR48eCg8P1/Dhw/Xpp5/6u6Sgs2DBAg0dOlQxMTFKSUnRtGnTtGfPHn+X1Sk8/fTTMplMmj17tr9LCVrHjh3T97//fSUlJSkiIkIDBgzQ1q1b/V1W0HE6nZo3b56ys7MVERGhnj176sknn7zkuT24tPXr12vy5MnKyMiQyWTS8uXLmzzvdrv12GOPKT09XRERERo3bpwKCwvbrR6CTCu8/fbbmjNnjh5//HFt375dAwcO1IQJE1RSUuLv0oLKRx99pJkzZ2rTpk1as2aNGhoaNH78eNXW1vq7tKC2ZcsWvfTSS7rqqqv8XUrQOn36tEaOHKmwsDD985//1BdffKHnnntOCQkJ/i4t6DzzzDNavHixfv/732v37t165plntHDhQv3ud7/zd2mGV1tbq4EDB+rFF19s9vmFCxfqhRde0JIlS7R582ZFRUVpwoQJqq+vb5+C3GixYcOGuWfOnOm973Q63RkZGe4FCxb4sargV1JS4pbk/uijj/xdStCqrq525+TkuNesWeMeM2aM+7777vN3SUHp4Ycfdo8aNcrfZXQKN954o/uuu+5q8thNN93kvu222/xUUXCS5F62bJn3vsvlcqelpbl//etfex+rqKhwW61W95tvvtkuNTAi00J2u13btm3TuHHjvI+FhIRo3Lhx2rhxox8rC36VlZWSpMTERD9XErxmzpypG2+8sck/3/C9FStWaMiQIZo+fbpSUlI0aNAgvfLKK/4uKyhdd911Wrt2rfbu3StJ2rlzpzZs2KCJEyf6ubLgdvDgQRUXFzf5XRIXF6fhw4e327Uy6A+N9JXS0lI5nU6lpqY2eTw1NVVffvmln6oKfi6XS7Nnz9bIkSPVv39/f5cTlN566y1t375dW7Zs8XcpQe/AgQNavHix5syZo0cffVRbtmzRrFmzZLFYNGPGDH+XF1QeeeQRVVVVqU+fPgoNDZXT6dRTTz2l2267zd+lBbXi4mJJavZa6XnO1wgyCGgzZ87U559/rg0bNvi7lKB05MgR3XfffVqzZo3Cw8P9XU7Qc7lcGjJkiObPny9JGjRokD7//HMtWbKEIONj77zzjt544w0tXbpUubm52rFjh2bPnq2MjAy+6yDD1FILJScnKzQ0VCdPnmzy+MmTJ5WWluanqoLbvffeq5UrV+rDDz9U165d/V1OUNq2bZtKSkp0zTXXyGw2y2w266OPPtILL7wgs9ksp9Pp7xKDSnp6uvr169fksb59++rw4cN+qih4PfTQQ3rkkUf03e9+VwMGDNAPfvAD3X///VqwYIG/SwtqnuthR14rCTItZLFYNHjwYK1du9b7mMvl0tq1a3Xttdf6sbLg43a7de+992rZsmX617/+pezsbH+XFLRuuOEG7dq1Szt27PDehgwZottuu007duxQaGiov0sMKiNHjjxvK4G9e/eqe/fufqooeNXV1SkkpOklLjQ0VC6Xy08VdQ7Z2dlKS0trcq2sqqrS5s2b2+1aydRSK8yZM0czZszQkCFDNGzYMC1atEi1tbW68847/V1aUJk5c6aWLl2qv//974qJifHOq8bFxSkiIsLP1QWXmJiY83qPoqKilJSURE9SO7j//vt13XXXaf78+brlllv06aef6uWXX9bLL7/s79KCzuTJk/XUU0+pW7duys3NVX5+vp5//nnddddd/i7N8GpqarRv3z7v/YMHD2rHjh1KTExUt27dNHv2bP3qV79STk6OsrOzNW/ePGVkZGjatGntU1C7rIUKYr/73e/c3bp1c1ssFvewYcPcmzZt8ndJQUdSs7fXXnvN36V1Ciy/bl/vv/++u3///m6r1eru06eP++WXX/Z3SUGpqqrKfd9997m7devmDg8Pd19xxRXu//7v/3bbbDZ/l2Z4H374YbO/o2fMmOF2u88uwZ43b547NTXVbbVa3TfccIN7z5497VaPye1mm0MAAGBM9MgAAADDIsgAAADDIsgAAADDIsgAAADDIsgAAADDIsgAAADDIsgAAADDIsgAAADDIsgACDh33HFHk+3Mx44dq9mzZ7fpM33xGQACD0EGQIvdcccdMplMMplMslgs6tWrl375y1/K4XC0689977339OSTT7botevWrZPJZFJFRcVlfwYA4+DQSACt8s1vflOvvfaabDab/vGPf2jmzJkKCwvT3Llzm7zObrfLYrH45GcmJiYGxGcACDyMyABoFavVqrS0NHXv3l333HOPxo0bpxUrVning5566illZGSod+/ekqQjR47olltuUXx8vBITEzV16lQdOnTI+3lOp1Nz5sxRfHy8kpKS9LOf/UznHgF37rSQzWbTww8/rKysLFmtVvXq1Ut//OMfdejQIV1//fWSpISEBJlMJt1xxx3Nfsbp06d1++23KyEhQZGRkZo4caIKCwu9z7/++uuKj49XXl6e+vbtq+joaH3zm9/UiRMnvK9Zt26dhg0bpqioKMXHx2vkyJEqKiry0TcNoCUIMgDaJCIiQna7XZK0du1a7dmzR2vWrNHKlSvV0NCgCRMmKCYmRh9//LH+/e9/ewOB5z3PPfecXn/9db366qvasGGDysvLtWzZsov+zNtvv11vvvmmXnjhBe3evVsvvfSSoqOjlZWVpXfffVeStGfPHp04cUK//e1vm/2MO+64Q1u3btWKFSu0ceNGud1uTZo0SQ0NDd7X1NXV6dlnn9Vf/vIXrV+/XocPH9aDDz4oSXI4HJo2bZrGjBmjzz77TBs3btTdd98tk8nU5u8UQMsxtQTgsrjdbq1du1Z5eXn66U9/qlOnTikqKkp/+MMfvFNKf/3rX+VyufSHP/zBe4F/7bXXFB8fr3Xr1mn8+PFatGiR5s6dq5tuukmStGTJEuXl5V3w5+7du1fvvPOO1qxZo3HjxkmSrrjiCu/znimklJQUxcfHN/sZhYWFWrFihf7973/ruuuukyS98cYbysrK0vLlyzV9+nRJUkNDg5YsWaKePXtKku6991798pe/lCRVVVWpsrJS3/rWt7zP9+3bt/VfJIA2YUQGQKusXLlS0dHRCg8P18SJE3XrrbfqF7/4hSRpwIABTfpidu7cqX379ikmJkbR0dGKjo5WYmKi6uvrtX//flVWVurEiRMaPny49z1ms1lDhgy54M/fsWOHQkNDNWbMmMv+O+zevVtms7nJz01KSlLv3r21e/du72ORkZHekCJJ6enpKikpkXQ2MN1xxx2aMGGCJk+erN/+9rdNpp0AdAxGZAC0yvXXX6/FixfLYrEoIyNDZvN/fo1ERUU1eW1NTY0GDx6sN95447zP6dKly2X9/IiIiMt63+UICwtrct9kMjXp33nttdc0a9YsrVq1Sm+//bZ+/vOfa82aNRoxYkSH1Qh0dozIAGiVqKgo9erVS926dWsSYppzzTXXqLCwUCkpKerVq1eTW1xcnOLi4pSenq7Nmzd73+NwOLRt27YLfuaAAQPkcrn00UcfNfu8Z0TI6XRe8DP69u0rh8PR5OeWlZVpz5496tev30X/TucaNGiQ5s6dq08++UT9+/fX0qVLW/V+AG1DkAHQbm677TYlJydr6tSp+vjjj3Xw4EGtW7dOs2bN0tGjRyVJ9913n55++mktX75cX375pX7yk5+ctwdMYz169NCMGTN01113afny5d7PfOeddyRJ3bt3l8lk0sqVK3Xq1CnV1NSc9xk5OTmaOnWqfvSjH2nDhg3auXOnvv/97yszM1NTp05t0d/t4MGDmjt3rjZu3KiioiKtXr1ahYWF9MkAHYwgA6DdREZGav369erWrZtuuukm9e3bVz/84Q9VX1+v2NhYSdIDDzygH/zgB5oxY4auvfZaxcTE6Nvf/vZFP3fx4sX6zne+o5/85Cfq06ePfvSjH6m2tlaSlJmZqSeeeEKPPPKIUlNTde+99zb7Ga+99poGDx6sb33rW7r22mvldrv1j3/847zppIv93b788kvdfPPNuvLKK3X33Xdr5syZ+vGPf9yKbwhAW5nc527YAAAAYBCMyAAAAMMiyAAAAMMiyAAAAMMiyAAAAMMiyAAAAMMiyAAAAMMiyAAAAMMiyAAAAMMiyAAAAMMiyAAAAMMiyAAAAMP6/3HzpFPGLdF5AAAAAElFTkSuQmCC\n"
},
"metadata": {}
}
],
"source": [
"# plotting actual values vs. predictions\n",
"ypred = model3(torch.from_numpy(np.float32(x_test)).to(device)).cpu().detach().numpy()\n",
"plt.scatter(ypred,y_non_test,s=.1)\n",
"plt.xlabel(\"Predictions\");\n",
"plt.ylabel(\"Actual values\");"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "lNWhCAESwhOe"
},
"source": [
"As you can see, the shallow network has learned a pretty good approximation of the nonlinear relationship in our data, just by memorizing small bits of it in each unit. The results here aren't perfect, but with infinite data and infinite units, a shallow network can get arbitrarily close to perfection. In our more mundane world of limited data and models, deep neural networks typically offer a better solution."
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "kxzLLCNRw_W5"
},
"source": [
"### Deep neural network\n",
"In the example below, we'll use far fewer units to achieve similar performance, via a deep neural network. This network features 5 ReLU layers, with units decreasing in power of two. The final layer is a single linear unit, as in previous cases. An important addition here is batch normalization after each ReLU activation. Batch normalization is basically like z-scoring the data. This is really helpful to prevent what's known as the \"exploding gradient\" problem. Basically, nonlinear transformations have the potential to make some numbers really huge (or tiny) and this can cause problems for numerical computing with float point representations. Batch normalization helps to mitigate this problem, serving as a sort of regularization that improves training."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/"
},
"id": "cT278bxPxBMh",
"outputId": "eefc912b-64ab-483c-a51d-17d0188a7899"
},
"outputs": [
{
"output_type": "stream",
"name": "stdout",
"text": [
"DeepANN(\n",
" (layer_stack): Sequential(\n",
" (0): Linear(in_features=1, out_features=32, bias=True)\n",
" (1): ReLU()\n",
" (2): BatchNorm1d(32, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
" (3): Linear(in_features=32, out_features=16, bias=True)\n",
" (4): ReLU()\n",
" (5): BatchNorm1d(16, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
" (6): Linear(in_features=16, out_features=8, bias=True)\n",
" (7): ReLU()\n",
" (8): BatchNorm1d(8, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
" (9): Linear(in_features=8, out_features=4, bias=True)\n",
" (10): ReLU()\n",
" (11): BatchNorm1d(4, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
" (12): Linear(in_features=4, out_features=2, bias=True)\n",
" (13): ReLU()\n",
" (14): BatchNorm1d(2, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
" (15): Linear(in_features=2, out_features=1, bias=True)\n",
" )\n",
")\n"
]
}
],
"source": [
"# define model architecture\n",
"class DeepANN(nn.Module):\n",
" def __init__(self):\n",
" super().__init__()\n",
" self.layer_stack = nn.Sequential(\n",
" nn.Linear(1,32),\n",
" nn.ReLU(),\n",
" nn.BatchNorm1d(32),\n",
" nn.Linear(32,16),\n",
" nn.ReLU(),\n",
" nn.BatchNorm1d(16),\n",
" nn.Linear(16,8),\n",
" nn.ReLU(),\n",
" nn.BatchNorm1d(8),\n",
" nn.Linear(8,4),\n",
" nn.ReLU(),\n",
" nn.BatchNorm1d(4),\n",
" nn.Linear(4,2),\n",
" nn.ReLU(),\n",
" nn.BatchNorm1d(2),\n",
" nn.Linear(2,1)\n",
" )\n",
"\n",
" def forward(self, x):\n",
" pred = self.layer_stack(x)\n",
" return pred\n",
"\n",
"model4 = DeepANN().to(device)\n",
"print(model4)"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/"
},
"id": "wGma6Ndbxc1n",
"outputId": "80848c9e-9571-4f0c-bcea-6b9c84f31e10"
},
"outputs": [
{
"output_type": "stream",
"name": "stdout",
"text": [
"Epoch 1\n",
"-------------------------------\n",
"loss: 4.667707 [ 64/100000]\n",
"loss: 0.989088 [ 6464/100000]\n",
"loss: 0.216513 [12864/100000]\n",
"loss: 0.264145 [19264/100000]\n",
"loss: 0.306660 [25664/100000]\n",
"loss: 0.456342 [32064/100000]\n",
"loss: 0.057279 [38464/100000]\n",
"loss: 0.060745 [44864/100000]\n",
"loss: 0.394866 [51264/100000]\n",
"loss: 0.418425 [57664/100000]\n",
"loss: 0.314052 [64064/100000]\n",
"loss: 0.173316 [70464/100000]\n",
"loss: 0.039098 [76864/100000]\n",
"loss: 0.173622 [83264/100000]\n",
"loss: 0.153746 [89664/100000]\n",
"loss: 0.228229 [96064/100000]\n",
"Epoch 2\n",
"-------------------------------\n",
"loss: 0.071922 [ 64/100000]\n",
"loss: 0.055156 [ 6464/100000]\n",
"loss: 0.248226 [12864/100000]\n",
"loss: 0.049417 [19264/100000]\n",
"loss: 0.040127 [25664/100000]\n",
"loss: 0.057625 [32064/100000]\n",
"loss: 0.714862 [38464/100000]\n",
"loss: 0.200428 [44864/100000]\n",
"loss: 0.236930 [51264/100000]\n",
"loss: 0.031537 [57664/100000]\n",
"loss: 0.179619 [64064/100000]\n",
"loss: 0.115763 [70464/100000]\n",
"loss: 0.201975 [76864/100000]\n",
"loss: 0.044374 [83264/100000]\n",
"loss: 0.070829 [89664/100000]\n",
"loss: 0.019935 [96064/100000]\n",
"Epoch 3\n",
"-------------------------------\n",
"loss: 0.119832 [ 64/100000]\n",
"loss: 0.057674 [ 6464/100000]\n",
"loss: 0.081795 [12864/100000]\n",
"loss: 0.079885 [19264/100000]\n",
"loss: 0.036620 [25664/100000]\n",
"loss: 0.088968 [32064/100000]\n",
"loss: 0.167877 [38464/100000]\n",
"loss: 0.203338 [44864/100000]\n",
"loss: 0.018583 [51264/100000]\n",
"loss: 0.209267 [57664/100000]\n",
"loss: 0.175974 [64064/100000]\n",
"loss: 0.060453 [70464/100000]\n",
"loss: 0.039461 [76864/100000]\n",
"loss: 0.105169 [83264/100000]\n",
"loss: 0.085562 [89664/100000]\n",
"loss: 0.035103 [96064/100000]\n",
"Done!\n"
]
}
],
"source": [
"# new we'll train the model\n",
"loss = nn.MSELoss()\n",
"optimizer = torch.optim.SGD(model4.parameters(),lr=.01)\n",
"epochs = 3\n",
"for t in range(epochs):\n",
" print(f\"Epoch {t+1}\\n-------------------------------\")\n",
" train(train_dataloader, model4, loss, optimizer)\n",
"print(\"Done!\")"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/"
},
"id": "AJxrmvi1xkSc",
"outputId": "41ed8777-965b-4c89-a3f5-fc5a0327ab69"
},
"outputs": [
{
"output_type": "stream",
"name": "stdout",
"text": [
"Test performance: \n",
" R^2: 0.980514, RMSE: 0.292389 \n",
"\n"
]
}
],
"source": [
"# performance metrics\n",
"perf = test(test_dataloader, model4, loss)"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/",
"height": 449
},
"id": "_D7aGcR7x0y6",
"outputId": "f701b94e-19cf-478e-c1be-b11aca797bb1"
},
"outputs": [
{
"output_type": "display_data",
"data": {
"text/plain": [
""
],
"image/png": "iVBORw0KGgoAAAANSUhEUgAAAjIAAAGwCAYAAACzXI8XAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjcuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/bCgiHAAAACXBIWXMAAA9hAAAPYQGoP6dpAABBHElEQVR4nO3deXTU1f3/8ddkJjPZd7MSIErEEBCQTVyqVCSiZanWpT9bse1XeywWEbUVW7TWAspXLbVaUWu1ta5fFYraCpTKpqjsQkQIyJIASUhC9mQms/z+wEwTSGCGLDOf5Pk4Z85htk/eMG3m5b3ve6/J4/F4BAAAYEAhgS4AAADgTBFkAACAYRFkAACAYRFkAACAYRFkAACAYRFkAACAYRFkAACAYVkCXUBXc7vdOnz4sKKjo2UymQJdDgAA8IHH41FNTY3S09MVEtL+uEuPDzKHDx9WZmZmoMsAAABnoLCwUH369Gn3+R4fZKKjoyUd/4eIiYkJcDUAAMAX1dXVyszM9H6Pt6fHB5nm6aSYmBiCDAAABnO6thCafQEAgGERZAAAgGERZAAAgGERZAAAgGERZAAAgGERZAAAgGERZAAAgGERZAAAgGERZAAAgGERZAAAgGERZAAAgGERZAAAgGERZAAAgGERZAAAaKHe4dTiLUWqdzgDXQp8ENAgs2bNGk2aNEnp6ekymUxasmSJ97mmpib98pe/1JAhQxQZGan09HTdcsstOnz4cOAKBgD0eMvyi1VR69Dy/JJAlwIfBDTI1NXVaejQoXrmmWdOeq6+vl6bN2/WnDlztHnzZr377rvatWuXJk+eHIBKAQC9RV5uqhKjbJqQmxLoUuADk8fj8QS6CEkymUxavHixpk6d2u5rNmzYoNGjR+vAgQPq27evT9etrq5WbGysqqqqFBMT00nVAgCAruTr97elG2vqsKqqKplMJsXFxbX7GrvdLrvd7r1fXV3dDZUBAIBAMEyzb2Njo375y1/q+9///imT2fz58xUbG+u9ZWZmdmOVAACgOxkiyDQ1NemGG26Qx+PRs88+e8rXzp49W1VVVd5bYWFhN1UJAAC6W9BPLTWHmAMHDug///nPaftcbDabbDZbN1UHAAACKaiDTHOIKSgo0EcffaTExMRAlwQAAIJIQINMbW2t9uzZ472/b98+bd26VQkJCUpLS9P3vvc9bd68We+//75cLpeKi4slSQkJCbJarYEqGwAABImALr9etWqVxo0bd9Lj06ZN029+8xtlZWW1+b6PPvpIl19+uU8/g+XXAAAYjyGWX19++eU6VY4Kki1uAABAkDLEqiUAAIC2EGQAAIBhEWQAAIBhEWQAAIBhEWQAAIBhEWQAAIBhEWQAAIBhEWQAAIBhEWQAAIBhEWQAAIBhEWQAAIBhEWQAAIBhEWQAAIBhEWQAAIBhEWQAAIBhEWQAAIBhEWQAAIBhEWQAAIBhEWQAAIBhEWQAAIBhEWQAAIBhEWQAAIBhEWQAAIBhEWQAAIBhEWQAAIBhEWQAAIBhEWQAAIBhEWQAAIBhEWQAAOgi9Q6nFm8pUr3DGehSeiyCDAAAXWRZfrEqah1anl8S6FJ6LIIMAABdJC83VYlRNk3ITQl0KT2WJdAFAADQU0VYLZo6PCPQZfRojMgAAADDIsgAANCNaADuXAQZAAC6EQ3AnYsgAwBAN6IBuHPR7AsAQDeiAbhzMSIDAAAMiyADAAAMiyADAAAMiyADAAAMiyADAAAMiyADAAAMK6BBZs2aNZo0aZLS09NlMpm0ZMmSVs97PB49+OCDSktLU3h4uMaPH6+CgoLAFAsAAIJOQINMXV2dhg4dqmeeeabN5xcsWKCnnnpKixYt0meffabIyEjl5eWpsbGxmysFAADBKKAb4k2cOFETJ05s8zmPx6OFCxfq17/+taZMmSJJ+tvf/qaUlBQtWbJEN910U3eWCgAAglDQ9sjs27dPxcXFGj9+vPex2NhYjRkzRuvXr2/3fXa7XdXV1a1uAACgZwraIFNcXCxJSklpfRZFSkqK97m2zJ8/X7Gxsd5bZmZml9YJAAACJ2iDzJmaPXu2qqqqvLfCwsJAlwQAALpI0AaZ1NRUSVJJSetjzktKSrzPtcVmsykmJqbVDQAA9ExBG2SysrKUmpqqlStXeh+rrq7WZ599prFjxwawMgAAECwCumqptrZWe/bs8d7ft2+ftm7dqoSEBPXt21czZ87U7373O2VnZysrK0tz5sxRenq6pk6dGriiAQBA0AhokNm4caPGjRvnvT9r1ixJ0rRp0/Tyyy/rF7/4herq6nT77bersrJSl1xyiT788EOFhYUFqmQAABBETB6PxxPoIrpSdXW1YmNjVVVVRb8MAAAG4ev3d9D2yAAAAJwOQQYAABgWQQYAABgWQQYAABgWQQYAABgWQQYAABgWQQYAABgWQQYAABgWQQYAAAOpdzi1eEuR6h3OQJcSFAgyAAAYyLL8YlXUOrQ8vyTQpQQFggwAAAaSl5uqxCibJuSmBLqUoBDQQyMBAIB/IqwWTR2eEegyggYjMgAAwLAIMgAAwLAIMgAAwLAIMgAAwLAIMgAAwLAIMgAAwLAIMgAAwLAIMgAAwLAIMgAAwLAIMgAAwLAIMgAAwLAIMgAAwLAIMgAAwLAIMgAAwLAIMgAAwLAIMgAAwLAIMgAAwLAIMgAAwLAIMgAA4IzUO5xavKVI9Q5nwGogyAAAgDOyLL9YFbUOLc8vCVgNBBkAAHBG8nJTlRhl04TclIDVYAnYTwYAAIYWYbVo6vCMgNbAiAwAADAsggwAAD1QMDTidgeCDAAAPVAwNOJ2B4IMAAA9UDA04nYHmn0BAOiBgqERtzswIgMAAAyLIAMAAAyLIAMAAAyLIAMAAAyLIAMAAAwrqIOMy+XSnDlzlJWVpfDwcJ1zzjl65JFH5PF4Al0aAAAIAkG9/Pqxxx7Ts88+q7/+9a/Kzc3Vxo0b9aMf/UixsbGaMWNGoMsDAAABFtRB5pNPPtGUKVN0zTXXSJL69++v119/XZ9//nm777Hb7bLb7d771dXVXV4nAAAIjKCeWrrooou0cuVK7d69W5K0bds2rVu3ThMnTmz3PfPnz1dsbKz3lpmZ2V3lAgCAbmbyBHHDidvt1gMPPKAFCxbIbDbL5XJp7ty5mj17drvvaWtEJjMzU1VVVYqJiemOsgEAQAdVV1crNjb2tN/fQT219NZbb+nVV1/Va6+9ptzcXG3dulUzZ85Uenq6pk2b1uZ7bDabbDZbN1cKAAACIaiDzH333af7779fN910kyRpyJAhOnDggObPn99ukAEAAL1HUPfI1NfXKySkdYlms1lutztAFQEAgGAS1CMykyZN0ty5c9W3b1/l5uZqy5YtevLJJ/XjH/840KUBAIAgENTNvjU1NZozZ44WL16s0tJSpaen6/vf/74efPBBWa1Wn67ha7MQAAAIHr5+fwd1kOkMBBkAAIzH1+/voO6RAQAAOBWCDAAAMCyCDAAAMCy/g0xDQ4Pq6+u99w8cOKCFCxdq+fLlnVoYAABGUO9wavGWItU7nIEupVfyO8hMmTJFf/vb3yRJlZWVGjNmjJ544glNmTJFzz77bKcXCABAMFuWX6yKWoeW55dIIth0N7+DzObNm3XppZdKkt5++22lpKTowIED+tvf/qannnqq0wsEACCY5eWmKjHKpgm5KZJODjboWn4Hmfr6ekVHR0uSli9frmuvvVYhISG68MILdeDAgU4vEACAYNDeSEuE1aKpwzMUYT2+x+yJwQZdy+8gM2DAAC1ZskSFhYVatmyZJkyYIEkqLS1lnxYAQI/l60jLicEGXcvvIPPggw/q3nvvVf/+/TV69GiNHTtW0vHRmeHDh3d6gQAABIPuHGmhz8Z3Z7Szb3FxsY4cOaKhQ4d6D3X8/PPPFRMTo/POO6/Ti+wIdvYFABjN4i1Fqqh1KDHKpqnDMwJdTkB06c6+qampio6O1ooVK9TQ0CBJGjVqVNCFGAAAjIg+G9/5HWTKy8t1xRVX6Nxzz9XVV1+tI0eOSJJ+8pOf6J577un0AgEA6G3os/Gd30Hm7rvvVmhoqA4ePKiIiAjv4zfeeKM+/PDDTi0OAADgVPyOesuXL9eyZcvUp0+fVo9nZ2ez/BoAAHQrv0dk6urqWo3ENKuoqJDNZuuUogAAAHzhd5C59NJLvUcUSJLJZJLb7daCBQs0bty4Ti0OAADgVPyeWlqwYIGuuOIKbdy4UQ6HQ7/4xS+Un5+viooKffzxx11RIwAAp1XvcGpZfrHyclNpku1F/B6RGTx4sHbv3q1LLrlEU6ZMUV1dna699lpt2bJF55xzTlfUCADAaXHGUe90RhviGQkb4gFA71DvcGp5fokm5KYwItMD+Pr97fcnvWbNmlM+/61vfcvfSwIA0GHNe6+gd/E7yFx++eUnPWYymbx/drlcHSoIAAB/0BvTu/ndI3Ps2LFWt9LSUn344YcaNWqUli9f3hU1AgDQru7qjeEgx+Dkd3SNjY096bErr7xSVqtVs2bN0qZNmzqlMAAAfJGXm+rtjelKLQMTU1jB44wOjWxLSkqKdu3a1VmXAwDAJ115LlHLURgOcgxOfn/qX3zxRav7Ho9HR44c0aOPPqphw4Z1Vl0AAHS7E/ttThyFYSQm+PgdZIYNGyaTyaQTV21feOGF+stf/tJphQEA0N1ODC7dNW2FM+d3kNm3b1+r+yEhITrrrLMUFhbWaUUBABAIJwYXlnQHP7+DTL9+/bqiDgAA2tTW8uquWnJNcDEenz79p556yucLzpgx44yLAQDgRG2tFuqqFUTsSWM8Ph1RkJWV5dvFTCZ9/fXXHS6qM3FEAQAYW1tHD3TVcQSLtxSpotahxCgbIzMB5uv3N2ctAQDwDc5rCh5ddtYSAAA9FT0yxnNGQaaoqEhLly7VwYMH5XA4Wj335JNPdkphAAAAp+N3kFm5cqUmT56ss88+W1999ZUGDx6s/fv3y+Px6IILLuiKGgEAANrk9xEFs2fP1r333qvt27crLCxM77zzjgoLC3XZZZfp+uuv74oaAQAA2uR3kNm5c6duueUWSZLFYlFDQ4OioqL029/+Vo899linFwgAANAev4NMZGSkty8mLS1Ne/fu9T5XVlbWeZUBAACcht89MhdeeKHWrVunnJwcXX311brnnnu0fft2vfvuu7rwwgu7okYAAIA2+R1knnzySdXW1kqSHn74YdXW1urNN99UdnY2K5YAAEC3YkM8AAAQdHz9/va7R+Z//ud/tGrVqo7UBgAA0Cn8DjJHjx7VVVddpczMTN13333atm1bV9QFAABwWn4HmX/84x86cuSI5syZow0bNuiCCy5Qbm6u5s2bp/3793dBiQAAAG3zO8hIUnx8vG6//XatWrVKBw4c0K233qpXXnlFAwYM6Oz6dOjQIf3gBz9QYmKiwsPDNWTIEG3cuLHTfw4A4MzUO5xavKVI9Q5nt7wPaOmMgkyzpqYmbdy4UZ999pn279+vlJSUzqpLknTs2DFdfPHFCg0N1b/+9S99+eWXeuKJJxQfH9+pPwcAcOaW5Rerotah5fkl3fI+oKUzOjTyo48+0muvvaZ33nlHbrdb1157rd5//319+9vf7tTiHnvsMWVmZuqll17yPpaVlXXK99jtdtntdu/96urqTq0JANBaXm6qlueXaEKuf/8xe6bvA1rye/l1RkaGKioqdNVVV+nmm2/WpEmTZLPZuqS4QYMGKS8vT0VFRVq9erUyMjL0s5/9TLfddlu77/nNb36jhx9++KTHWX4NAMGr3uHUsvxi5eWmKsJ6Rv+NjR7G1+XXfgeZF154Qddff73i4uI6WuNphYWFSZJmzZql66+/Xhs2bNBdd92lRYsWadq0aW2+p60RmczMTIIMAASxxVuKVFHrUGKUTVOHZwS6HASBLgsy3clqtWrkyJH65JNPvI/NmDFDGzZs0Pr16326BhviAUDwq3c4vdNMjMhA6sIN8bpTWlqaBg0a1OqxnJwcHTx4MEAVAQC6QoTVoqnDMwgx8FtQB5mLL75Yu3btavXY7t271a9fvwBVBAAAgklQB5m7775bn376qebNm6c9e/botdde0/PPP6/p06cHujQAABAEgjrIjBo1SosXL9brr7+uwYMH65FHHtHChQt18803B7o0AAAQBHxq9l26dKnPF5w8eXKHCupsNPsCAGA8vn5/+9RVNXXqVJ9+qMlkksvl8um1AAAAHeVTkHG73V1dBwAAgN+CukcGABDcOPgRgXZGC/br6uq0evVqHTx4UA6Ho9VzM2bM6JTCAADBr+XBj+zIi0DwO8hs2bJFV199terr61VXV6eEhASVlZUpIiJCycnJBBkA6EU4+BGB5vfU0t13361Jkybp2LFjCg8P16effqoDBw5oxIgRevzxx7uiRgBAN/N1yogdeRFofgeZrVu36p577lFISIjMZrPsdrsyMzO1YMECPfDAA11RIwCgm7WcMgKCmd9BJjQ0VCEhx9+WnJzsPfcoNjZWhYWFnVsdACAg8nJTlRhlY8oIQc/vscDhw4drw4YNys7O1mWXXaYHH3xQZWVleuWVVzR48OCuqBEAcBr1DqeW5RcrLze1U6Z5mqeMgGDn94jMvHnzlJaWJkmaO3eu4uPjdccdd+jo0aN6/vnnO71AAEBrbfWvMBWE3sqnIwqMjCMKAPQ0i7cUqaLWocQom3fUpN7h9K4eovEWPYGv399siAcABtNW/wqrh9Bb+f2/+KysLJlMpnaf//rrrztUEADg1OhfAf7L7yAzc+bMVvebmpq0ZcsWffjhh7rvvvs6qy4AAIDT8jvI3HXXXW0+/swzz2jjxo0dLggAcFxnr0QCeqJO65GZOHGi3nnnnc66HAD0CqfaQZeVSMDpdVqQefvtt5WQkNBZlwOAXuFUYYVN6YDTO6MN8Vo2+3o8HhUXF+vo0aP605/+1KnFAUBPd6pDF2nqBU7P7yAzZcqUVkEmJCREZ511li6//HKdd955nVocAPR0hBWgY9gQDwAABJ0u2xDPbDartLT0pMfLy8tlNpv9vRwAAMAZ8zvItDeAY7fbZbVaO1wQABjZqVYhAeh8PvfIPPXUU5Ikk8mkP//5z4qKivI+53K5tGbNGnpkAPR6LVch0fsCdD2fg8zvf/97ScdHZBYtWtRqGslqtap///5atGhR51cIAAZyqlVIADqfz0Fm3759kqRx48bp3XffVXx8fJcVBQBGxSokoHv53SPz0UcfEWIA4AT0xgCB4XeQue666/TYY4+d9PiCBQt0/fXXd0pRAGA0pztOgKADdA2/g8yaNWt09dVXn/T4xIkTtWbNmk4pCgCM5nTHCXBuEtA1/A4ytbW1bS6zDg0NVXV1dacUBQBG09wb094p1ZdmJ2lfWZ0uyU7s5sqAns3vIDNkyBC9+eabJz3+xhtvaNCgQZ1SFAD0NGsLypSVFKl1BeWBLgXoUfw+a2nOnDm69tprtXfvXn3729+WJK1cuVKvv/66/u///q/TCwQAo6h3OLUsv1h5uaknjcywLBvoGmd01tIHH3ygefPmaevWrQoPD9f555+vhx56SJdddllX1NghnLUEoLss3lKkilqHEqNsLMEGOsjX7+9OPTRyx44dGjx4cGddrlMQZAD468SRlVONtJz4vuZRl1O9DsDpddmhkSeqqanR888/r9GjR2vo0KEdvRwABNyJK4x8XXF0uoZfAJ3vjIPMmjVrdMsttygtLU2PP/64vv3tb+vTTz/tzNoAoNvVO5yyO92KCrN4+1lOt7QaQOD49Z8NxcXFevnll/Xiiy+qurpaN9xwg+x2u5YsWcKKJQA9wrL8YtU1OpUYZfOOrHDsABC8fB6RmTRpkgYOHKgvvvhCCxcu1OHDh/XHP/6xK2sDgG7H6AtgLD6PyPzrX//SjBkzdMcddyg7O7srawKAgDjTJl8AgePziMy6detUU1OjESNGaMyYMXr66adVVlbWlbUBQKfx5ayjM23yBRA4PgeZCy+8UC+88IKOHDmin/70p3rjjTeUnp4ut9utFStWqKampivrBIA2+XoYoy+hpHla6ZLsRC3eUqRLs5OYZgKCXIf2kdm1a5defPFFvfLKK6qsrNSVV16ppUuXdmZ9HcY+MkDP5usmdP7s8cLGdkDgdcs+MgMHDtSCBQtUVFSk119/vSOXAoAzcqrm3JajNf7s8ULDL2AcnbqzbzBiRAbovTprZIWmX6D7ddvOvt3p0Ucflclk0syZMwNdCgAD6KyRFZp+geBlmCCzYcMGPffcczr//PMDXQoAg+isIwOYagKClyGCTG1trW6++Wa98MILio+PD3Q5AHoZzlACgpchgsz06dN1zTXXaPz48ad9rd1uV3V1dasbgN7H12XZAIwt6IPMG2+8oc2bN2v+/Pk+vX7+/PmKjY313jIzM7u4QgDBiL4WoHcI6iBTWFiou+66S6+++qrCwsJ8es/s2bNVVVXlvRUWFnZxlQAC4XQjLvS1AL1DUC+/XrJkib773e/KbDZ7H3O5XDKZTAoJCZHdbm/1XFtYfg0Yg79LnNm0DujZesTy6yuuuELbt2/X1q1bvbeRI0fq5ptv1tatW08bYgAYx4lTQe2NuDQ/zvEBACQ/Tr8OhOjoaA0ePLjVY5GRkUpMTDzpcQDGlpebquX5Jd5zjuxOt+oajx8r0HLEpTnwrCsoZyQGQHCPyADo2do6QmBtQZkqah0ySa0OcGwemaH3BUBLQd0j0xnokQGCV1t9Lice7kgvDNA79YgeGQA9W1ujK80jM5LohQFwWgQZAAFzqh1zT+yFibBa2OQOwEkIMgCCUlujNWxyB+BEBBkAAXOqEZa2Rmto9AVwIoIMgA45MYz4M/3j7wgLhzcCOBFBBkCHnBhGmu+/t+3waQMNIywAOoogA6BDTgwjzfc90mlHWxhhAdBRBBkAbfJ1iujEMNJ8f/LQdJ9GW1iJBKAjCDIA2nSq/pXTnYPUcqfe0422sBIJQEcQZAC0qa3+leagsnTboTbDx5mEEvpkAHQEQQbASeodTi3LL/YeE9BsWX6xiqsateVgpaLCLJ1yDhJ9MgA6giAD9CK+9qMsyy9WcWWj5n2wU2W1jd735OWm6tCxBvVPiJQk/X5FgYqrGr0jMIQSAN2NIAP0EvUOp+Z+sFPFlY2nnfrJy03VocoGZcSHa+GKAu90UYTVoplXZutQZYPsTpf6xIXr0LEGpoUABAxBBuglluUXq098uA5Vnj54RFgteuCaHCVEWnVuapSiwize96wtKFNWUqRsFrPS4sL1wDU5jMAACBiCDNBL5OWmKi3W9+ARYbXIaglRXaNL24uqWl0nMcqmSUPTmUYCEHAEGaCXiLAeb86d+03fiy9aTjHRBwMgGBFkgF5k/gc7tXpXqR79YKdPr2+eYkqLDacPBkBQIsgAvYjT7VGdw6VGl+ek1UvtrWhiBAZAMCPIAD3YieFkRP94nZ0UqTCz6aSN61puZsexAQCMgiAD9CAnBpDmDezmfbBT9Q6nvjciUz8c21/3X5Nz0sZ1LTez49gAAEZh8ng8nkAX0ZWqq6sVGxurqqoqxcTEBLocoMs07xPTJz5cCZFWWS0hujQ7SQtXFCgjLlxpceGaOjzD52stzy85aWdfAOguvn5/8xsKMLjm4wRqGp2qqnfI5XIr0mbWJ3vK5XC69cA1Od5Q4qvmvhgACHYEGcDglm47pPV7KmQxm2Qxh2h4v3h5JJlMx59vGUqaQ09ebiojLQB6BH6TAUHMt+BhkskkDcuMU3RYqHfkxSTJ8801mt/bsveFERcAPQHNvkAQ86XpdvLQdF0+MFkTh6TKo+Mtb//dldfZ6r1ncjo1AAQzggwQhJpXH12anXTa4NE8dbS2oKzV8mm709XqjKSWr2VaCUBPQZABglDzSMy6gnKfg8eJy6frGl2yWcyEFgA9GkEGCEJnMgXUcrSFKSQAvQVBBghSHnnOeIddppAA9Bb8lgOCUPPU0uMf7lJDk1sOp1s3juob6LIAIOgwIgMEwKlGWo436roVFWbRoIwY734wAICTEWSAbtIyvCzddkirvjqq97YdPul1xxt1nbJZzPreiExdPjBZk4amB6BiAAh+BBmgi5w46rJ022Gt2tUcXkxyeTz6fF+F3thwsNXIzKXZSdpXVqdLshPpdQGA0yDIAF3k5M3sPGqeJZo8NF1x4aFqdLj06d7yVpvWrS0oU1ZSpNYVlHd7zQBgNAQZoJOV1TbqV4u3KzctxjuyIkmTh2Z4p4kirBY9cE2OLjwnUbbQEO9rJHbfBQB/EGSATtI8lfS/y3aptLpRD7/3pXdkpay2UXM/2OmdLpKOL5GOCrPo3OToVqMvTCcBgO8IMkAHNIeX5qBSXNmowemxSokJ06PXDfGOrPx+RYFKqxu1cEVBq/cz+gIAHUOQAU6jZVhp2bxb73Bq7gc7dbCsXjNe36qUGJsOVTbouhF99MA1Odp44Jgm5KYowmrR3VdmKyUmTDOvzG51bUZfAKBjTB6PxxPoIrpSdXW1YmNjVVVVpZiYmECXgyB3fGn0YUkeTR56PGAs3lKkilqHdpfWqLHJrZH94hVqCdGGfRWqczh1rL5JF52dqKM1dj1wTU6r9yRG2TR1eEag/1oAYDi+fn8zIgO0sCy/WOv3lmvd7nLN+2Cn6h1OXZqddDzE2J1yuTzacbhK6/eWa09prY5W2/WdIWnqlxjpDTESU0YA0F0YzwZayMtNlcPp1oavy1XZ4NB72w7LaglRrd2lwop6RYRa9MuJ5+uTveUa2S9eVkuIdxVSS81TRgCArkWQAVqIsFp046i+8kj6dG+5HE63PJLCLCaZZFJSlFWf7C33Bhjp+ChOXm4qfS4AEAD85gX0394Yh9Mth9OlnUeqNSQjVjsOVSk9LlwWc4i+N6KPrJYQeSTvRnceebx/njo8Q/UOJ8EGALoRv2nR69U7nJqzZIc27K9QlM2igxX16hMfruIquy7JTtK6gjJdMiBJ0WGh3rCyPL/E2//S8s8td/NlagkAul5QN/vOnz9fo0aNUnR0tJKTkzV16lTt2rUr0GXBoOodTr2x4aDe2HDgpCXUu4pr1ORyq7CiXqnRNjldHj163RClxYbrD98fprS4cG9Yablk+sTl0zT5AkD3Curl11dddZVuuukmjRo1Sk6nUw888IB27NihL7/8UpGRkT5dg+XXvUdb0zotl1NLJq3fWy6TpMsHJmtCbormfrBTKTE2HSyvV9Gxeg3LjNPWwipdfX6qvjcik+khAAgQX7+/gzrInOjo0aNKTk7W6tWr9a1vfavN19jtdtntdu/96upqZWZmEmR6gZZ7t0zITdGy/GLZnW6t3X1UxVWNmjo8Q1bL8UHISUPTtSy/WMWVjTpU2aCZV2br3ztLZJJJHnlU1+hiDxgACKAeuY9MVVWVJCkhIaHd18yfP1+xsbHeW2ZmZneVhwBrnta5JDvx+I675fXaeuCYwkLNSosLl9USohtH9dWkoelauu2QymvtOlBep5lXZmttQZnqGl2yWcyaPDSD6SEAMAjDjMi43W5NnjxZlZWVWrduXbuvY0QGi7cUqbiq0dukmxBllc1i9gaTuR/sVFWDQ0cqG5WZEKGx5yR+M/GkNveEAQB0P19HZAzzG3v69OnasWPHKUOMJNlsNtlstm6qCsEoLzdV7207rHHnnaU9JbW6flQfJUWFSZLe2HBAVfUOhVlCvFNNHkl1jU4lRtkIMQBgMIaYWrrzzjv1/vvv66OPPlKfPn0CXQ6CRPNhjs0rkJpFWC2yWkK0/VC1au0uLVxR4D3wsaK2STsOVysnPVY/HNtfN47qq8lD05lKAgCDCuog4/F4dOedd2rx4sX6z3/+o6ysrECXhCDRvGy6uLJRy/NLTno+LzdVF52TqCibWckxNs14fauKqxq1evdRRdksKiip8b6WE6gBwLiCOshMnz5df//73/Xaa68pOjpaxcXFKi4uVkNDQ6BLQ4At3XZY5XV2rd59VCP6xZ00MtN81MCDk3NVWm3XmP4JOnSsQY9ff76GZcbp3qsGBrB6AEBnCepmX5PJ1ObjL730km699VafrsE+MsbW3pb/f1u/X39Zt0/npUYrKcqmrKTIdpdLt9yJt+U1OE4AAIJXj2j2DeKMhW7SvOV/8ynUzaHDajFpcEasoqxmzbwyW+sKytvscTlVWOE4AQAwvqCeWgKa94bxSDpYXq//+etGldU2avLQDI3PSdGDk3OVFBWmCbkpWrrtcKvjB6TWYaW9a9PkCwDGRZBBwLW3+kg63usyITdFDqdLS7Ye0tGaRt3x9016e1ORGp0u7+uW5Rdr/d5yfbq3whta6h1O2Z0uRYVZ2gwrNPkCgPHxGxwBd+IUT8vzkSYPzdCy/GJtOlCpCKtZB8rrVedw6t3NReqfGKkwi1lTh2coLzdVDqdbklqdRN181ABhBQB6Jn67I2Ca+1dG9ovXc6u/1pTh6ZL+O7ricnu0o6haP73sbK0rKJPNHKL+CREKCTFpwFmRGpWV2OpE6htH9W11/bzcVG+TLwCgZwrqVUudgVVLwav5kMd9ZXXKiA/XoWMNeuCaHNU7nPrfZbvkdLnVNyFSn+2rUEy4RUcqG5WdHKVRWQkcJQAAPVyPPDQSxtNe/0vL/pVbxvbTWxsLtbnwmN7ZVKR/7yxVY5NbwzLjdbCiXtUNTSqttiszPlyjshK8Iy/t9dUAAHoPggw65FSNulL7q4aWbjus9XsrJEl/XX9A9ia3Dh1r0Evr9mn9nqNyuY8PFBYdq5fb49HZSRG69Nyz5NF/p6TaW40EAOg9CDLokJaBot7h1BsbDrZaAt3eEmeH06Wvj9bq7Y2Fmja2n0b3j5fVHCKLWTpQ3qC48FCFWky66OwkxUdYdf/VObJaQlTXeHxzO5ZOAwAkemTQAc2ri0ySJg1N17L8Yq3adVQmSZcPTG5zk7my2kb974e71Njk0pbCSkXZLDq/T6zMISGKspm1Ir9EQ/rEaM6kXEVYLa125G1vh14AQM/TI3b2RXA7vrzZ6V3efGl2kj79ulyD02O9IyXN00CXZifp3ztLtHjzIdU0OGUNDdHo/gmSPGpyeZQWa9Nn+yqUkxErk8mkdQXlmjo8o1UYat73BQCAZgQZnLETlzevLSjTucnRig4L9Y6YHO+FKdenX5er0eFWfKRVJpl0zflpum5EHy3LL1ZxVaMOHWvQU98fppU7SyWJKSMAgE/okemlTtek64sTd8a9NDtJu0trVFbbqDc2HPzm2h6ZJA1Oj9XI/vGKDQ/V0zcP1w/H9veO4hw61qCfXna21haUadLQdN04qi9TRwAAn9Aj00s17+HSfGJ0e4cr+nJCdL3Dqbc3Fem9rYfU6HTLbDKpT0KEIm1m5abHyGYx64qcZP1+RYFSYmwqrbZr5pXZWltQJrvTrbpGp/aV1Z3yBGsAQO/i6/c3QaaXOrFx9sRg02zxliIVVzbqUOXxzeraCjNvbDigP320V5V1Drk80rhzk2SzWrSntFbpcWGKDQuVxySlx4Tr4z1HZXd5NCA5UoPSYhUVZpHNYtYl2YneE6wZjQEAsCEeTunEaaH2ljPn5abqUGWDMuLD9d62w+1MR5kUGx4ql0eKCbcoKixUo7ISlBkfror6JtXanapqaNJn+yuUFhcml9sjk0xKjLJp0tB0TR2eoaSoMA5wBAD4jSADSScHm7LaRv1q8XaV19o1uE+somwWbT54TAfL6/Xwe/ktemCk8TnJGpQeo9svPVt9EyL140uy5HC6FRlm0SOTcxVlsyjym36Ykf0TNSwzTvdffR7BBQDQYUwt9WJt9b/UO5x67bODen7NXp2bHC2PpG+fl6x9ZXVKjrHpvW1H1C8xXGW1Dp2XEq0HJ+d6VyaFhYbo3ORo7SurU02jUyaTFGWzKCsp0juFxNQRAMAXTC3Bq+UKpZZ/bl76PO+DnSqrbdTiLUV6e1ORnl21R/UOl3aV1GjceWfJagnRualROnysQd85P01V9U2qqHOosqFJy/NL5HC6VFhRr3OTo5UYZdPMK7N10YBEjT0nUTOvzG41hUSIAQB0Jr5VeoGWxwg0Ol1aW1CmdQVlGpoZpz2ltWpscul/l+3SucnR2l1ao5Qomw5WNmh0/ziZZNKOw1WqtTsVZjErIdKq9Lgw2Z1uRVmPj7As3XZY6bFh2l1ao4fGHF863XywoyRWIQEAugwjMr1AcyPviH5xentjobYVVuqrw9X6x5ZDcrpcOlLV6B1NuS9voHIyYjXgrCgNzojX7tIaNTqcKqyo197SWm0trNT+igaFhJg06uxERVgtmjw0XXERVvVPjDxFQzAAAJ2PINPDNU8hXZKdqHv/b5uKjtXLJCnSZlZaXLgsZrP6JkTIaglRdWOT5v9zp5wul/okRGjP0VoVlNTK7vQoItSiPvERMnk8SouxyRZ6fG8Y6Xij8APX5CgtNlweeTiVGgDQbZha6qGOH+h4SBv2VcjudOuzveWKDg+Vw+lReqxNT9w4TJsOVGpEvzg9t/preSQt2XJIpdWNSoqyaXBGrM5NjfJOIf3uu4O1rqBcl2QnauGKAg3NDPeehyT9d9VTy/1pAADoaozI9CAnNvKu31OhvaV1OlLZqNyMWEWGWmQySYlRVn2yt1weefTRrtLjK4wkXTMkTemxYZo0NF0zr8yWZFKkzax7rxro3eclKSrMO/rSVlg5cRk3AABdiW8bA2t5svTagjLVNDq1af8xOZxuTRqartpGp5wul0prHbp84FnauL9CLrdHFXV21TQ26Z/bi5UZHy6XR9pxuErD+8Zp3Hkp2ldWp3/vLNGm/cdkMqnVyIvEKdQAgOBBkDGw5tVIC1cUKCspUrtLa2QySbWNTj28NF+NTS5tP1StqDCLnvlojwor6iVJmfERKiitlcPplslkUkWtXdlnJcgkk3cXX5NMumhAoiROogYABC+CjMG03MQuLzdV7207rP5JkfrySLVy0qIVHRaqz/eV67OvK+R2u9XodCsuwqKdh6t1tNahqDCLRvRP0MQhqXp82S41udwak5WgozV23XbZ2Zo0NL3VGUwAAAQzvqkMpuWeMBNyU/RFUZXKaxu143C1Pvu6XCP6JcjpcsktjxqcHrk9HlU1uuRocqmqoUmDM2I0cUiq/r2zRB5J6XHh+nxfhf7w/WHe4MK0EQDAKGj2NZhLs5NUUFKj6kaHlm47pJQYm3aX1Ki20aWyWrv+vbNEByoaZDWbFW4NkdkUopyUSIVZzTr7rEhNGpqhtQVlWr+nQnV2lzYfqNQlA5K0rqA80H81AAD8xohMEDu+hPqwJI/G56RobUGZ7E63Khsd+vPafRqeGSdJcjg9klwySTKHSFZLiH4yKku1jU6t3n1U52cmyGwxK9pm0XUj+nzzHrck6YqcZK0rKKcPBgBgSASZILYsv1hrC8p0pLJBWw5W6tzkaEWFWVRe41B1Q5N2ldSoqKJOTS6PTCaTEiKsMptNujInRaGWEO0prVFcRKj+81WpUmPDNLxvvHf6iCMEAAA9AUEmSLRs4pWkpdsOyeF0K8xiUnpsmAanx8pqCZHd6VZydJiOVDWqsLxOTW7J4ZLCzFKD061+sRHaX1GnHYertL2oSjJJg9JiFBceqklD0wP8twQAoHMRZIJEcxPve9sO64uiKlXWORRqCdGIfvHKP1Slmkan/v1lsY5UNaqyoUnyuBVqMauu6fiUkuSRPB71iY/QfXkD9b/LdmlQRoyirBaNykrQpKHprEICAPQ4fLMFibzcVO/p1H3iw2VvcslsNskjj2rtTv1l3T41OptUa/d43+PyuBQRapbkUVhoiFweqaS6UUlRYXpoUi7LqAEAPR6rlgKo+UiBwoo6zf1gp0b0i5PkUZTNInOISf0TIr2b1JlMahViJCnKFqpxAxN19ZA0fX90X4WaQ/Ttgf89yJGjAgAAPR3fct2oZR9MhNXinU669/+2qbzWoXc2FWp0VqLsTpcuHpCkXSU12lZYqfomp47WOrzXCZF0VoxVI/vFSx6TPJKKjjWoT3y4osL4SAEAvQffep3gxIDS1vNvbyrUB18Ua1S/eM3bv1Mzr8xWTWOTvjxSJas5RHuO1kmSPtlbpohQs/IPV6t/YoTqHE5V1jtkkuSRZJIUaTNLMml433j9a3uxXB6P5JHMISZZLQyyAQB6D5PH4/Gc/mXGVV1drdjYWFVVVSkmJqZLfsbiLUWqqHUoMcrmXcp8fA+YQ9I3rbh/X39AByvq5HZ7NOrsRFXVO1RSbVe93ak6h1MO1/FrhYZITe7j78qID5PNYtb4nGS9taFQlQ1OeSQlR4fqxlF91Sc+QtJ/94SxWkJo6gUA9Ai+fn/zjdcBLU+fPnFTuWX5xVq7q0yHKht0TnKkLCEmNTa5FWoJ0epdR5UcZVVVo1NOl1smfTPSYjXJ6fLILclkkob1iVPWWZF6a2ORrKEh8jQcv3Zjk0fnnBVNIy8AoNdjHqIDmntc1hWUn9RYO7JfvLYWHdP2Q1UqKK7Rl0eq5HF7VGd3yeORqhublBhpVXhoyDejLFaFWy2KDg+V2WzS4PQY/XLiefpwR4kirRa5XB4lR1pkMUl//P4wGnkBABBBpkPyclOVGGXzjsQ0r0Kqdzi1aPXXKq60y+WRvjhcI4dLavpmEs8jySOTIq1m2Z0exYVb5PF4dO0FfdTQ5JY1RDp4rEHL80t046hMJcfY9NptF2rC4HTdM2GgjtU7A/eXBgAgiBBkOqC81q63NhaqvNaueodTcxZv10vr9mn2O18oKymiuT2mTY4mt4qONSguPFQeU4hSY8P1wfYjMn/TI2OzhGjFlyVqcro14Kwo9UmI0APX5CgtLpxzkQAA+AbNvh1w7Z8+1peHq5QcE6actGj9Z2ep3B7JajYpLNSs6ganXO281yQpKSpUDU1uXXxOokLNITpc2ahGp0s1jU6N6Beveyacq+dWf62M+HClxYZzJhIAoNfw9fubEZkOcDS5ZHd6VFrVoI++KpXTLbk8UoPTo2OnCDE2i5QeF6Ywq0UxYRaFWy0aPyhVz08boeiwUEXbLAqzhCgzIfL4KEwsozAAALTFEEHmmWeeUf/+/RUWFqYxY8bo888/D3RJqnc4Vdd0fDl0o0tqch3vfTkdW4g0OD1Ol2Qn6brhfWS1mDUkI1ZTh2coKSpM15yfqnCrWdkp0Vq8pUiSaOwFAKAdQR9k3nzzTc2aNUsPPfSQNm/erKFDhyovL0+lpaUBrWtZfrH2lTV47/s8P2eSrshJ0bnJ0TpS3aBhmXGtduO9ekiactJiFGoxqaLWoeX5JZ1bOAAAPUjQB5knn3xSt912m370ox9p0KBBWrRokSIiIvSXv/ylzdfb7XZVV1e3unWFS7OT/PrHs4Qc753plxipr4/WKjHKpvvyBurygcmaNDTd+7q1BWXKSoqUzWJutSIKAACcLKiDjMPh0KZNmzR+/HjvYyEhIRo/frzWr1/f5nvmz5+v2NhY7y0zM7NLavv3zlK5/Xh9lNWilFibwkLNGpoZ551KOnHaqHlJ96Sh6UwpAQBwGkEdZMrKyuRyuZSS0npUIiUlRcXFxW2+Z/bs2aqqqvLeCgsLu6S22kbf93IxS4qPsursxChFh4XquhF92n0tp1YDAOC7HvdtabPZZLPZuvznFJTWtPuc2XR89ZJ0fJl1XKRFf/vxaD23+mvNvDKbkAIAQCcJ6hGZpKQkmc1mlZS0bngtKSlRampqgKo67r68gUqI+G8gMen4P+bAlCiNz0nW2UkRumlkhqLDLPrTzSOUmRCp3313iJKiwgJWMwAAPU1QBxmr1aoRI0Zo5cqV3sfcbrdWrlypsWPHBrAyKSkqTOaQ/27dG24x6YpByYqLtGpY33j99LJzdPBYo0b2i9d7244EsFIAAHquoA4ykjRr1iy98MIL+utf/6qdO3fqjjvuUF1dnX70ox8FujT9edooRdnMSo6yaFxOisbnpOiZ/zdcabHhsjtdigk3q97h0swrswNdKgAAPVLQN2vceOONOnr0qB588EEVFxdr2LBh+vDDD09qAA6EoZnx+vxX4zXvg506K9qm7UVV3tVGb2w4qDCLRd+9IIXpJAAAughnLXWCeodT8z7YqYy4cKXFHT8Tqd7h1PL8Ek3ITaG5FwAAP3HWUjeKsFpOOpmaZdQAAHQ9vmU7SXNwAQAA3YcRGQAAYFgEGQAAYFgEGQAAYFgEGQAAYFgEGQAAYFgEGQAAYFgEGQAAYFgEGQAAYFgEGQAAYFgEGQAAYFgEGQAAYFgEGQAAYFgEGQAAYFg9/vRrj8cjSaqurg5wJQAAwFfN39vN3+Pt6fFBpqamRpKUmZkZ4EoAAIC/ampqFBsb2+7zJs/poo7Bud1uHT58WNHR0TKZTB2+XnV1tTIzM1VYWKiYmJhOqBDdgc/NePjMjInPzXiC9TPzeDyqqalRenq6QkLa74Tp8SMyISEh6tOnT6dfNyYmJqg+cPiGz814+MyMic/NeILxMzvVSEwzmn0BAIBhEWQAAIBhEWT8ZLPZ9NBDD8lmswW6FPiBz814+MyMic/NeIz+mfX4Zl8AANBzMSIDAAAMiyADAAAMiyADAAAMiyADAAAMiyDjp2eeeUb9+/dXWFiYxowZo88//zzQJaEd8+fP16hRoxQdHa3k5GRNnTpVu3btCnRZ8NOjjz4qk8mkmTNnBroUnMKhQ4f0gx/8QImJiQoPD9eQIUO0cePGQJeFU3C5XJozZ46ysrIUHh6uc845R4888shpzzYKNgQZP7z55puaNWuWHnroIW3evFlDhw5VXl6eSktLA10a2rB69WpNnz5dn376qVasWKGmpiZNmDBBdXV1gS4NPtqwYYOee+45nX/++YEuBadw7NgxXXzxxQoNDdW//vUvffnll3riiScUHx8f6NJwCo899pieffZZPf3009q5c6cee+wxLViwQH/84x8DXZpfWH7thzFjxmjUqFF6+umnJR0/xykzM1M///nPdf/99we4OpzO0aNHlZycrNWrV+tb3/pWoMvBadTW1uqCCy7Qn/70J/3ud7/TsGHDtHDhwkCXhTbcf//9+vjjj7V27dpAlwI/fOc731FKSopefPFF72PXXXedwsPD9fe//z2AlfmHERkfORwObdq0SePHj/c+FhISovHjx2v9+vUBrAy+qqqqkiQlJCQEuBL4Yvr06brmmmta/X8OwWnp0qUaOXKkrr/+eiUnJ2v48OF64YUXAl0WTuOiiy7SypUrtXv3bknStm3btG7dOk2cODHAlfmnxx8a2VnKysrkcrmUkpLS6vGUlBR99dVXAaoKvnK73Zo5c6YuvvhiDR48ONDl4DTeeOMNbd68WRs2bAh0KfDB119/rWeffVazZs3SAw88oA0bNmjGjBmyWq2aNm1aoMtDO+6//35VV1frvPPOk9lslsvl0ty5c3XzzTcHujS/EGTQK0yfPl07duzQunXrAl0KTqOwsFB33XWXVqxYobCwsECXAx+43W6NHDlS8+bNkyQNHz5cO3bs0KJFiwgyQeytt97Sq6++qtdee025ubnaunWrZs6cqfT0dEN9bgQZHyUlJclsNqukpKTV4yUlJUpNTQ1QVfDFnXfeqffff19r1qxRnz59Al0OTmPTpk0qLS3VBRdc4H3M5XJpzZo1evrpp2W322U2mwNYIU6UlpamQYMGtXosJydH77zzToAqgi/uu+8+3X///brpppskSUOGDNGBAwc0f/58QwUZemR8ZLVaNWLECK1cudL7mNvt1sqVKzV27NgAVob2eDwe3XnnnVq8eLH+85//KCsrK9AlwQdXXHGFtm/frq1bt3pvI0eO1M0336ytW7cSYoLQxRdffNLWBrt371a/fv0CVBF8UV9fr5CQ1jHAbDbL7XYHqKIzw4iMH2bNmqVp06Zp5MiRGj16tBYuXKi6ujr96Ec/CnRpaMP06dP12muv6R//+Ieio6NVXFwsSYqNjVV4eHiAq0N7oqOjT+pjioyMVGJiIv1NQeruu+/WRRddpHnz5umGG27Q559/rueff17PP/98oEvDKUyaNElz585V3759lZubqy1btujJJ5/Uj3/840CX5h8P/PLHP/7R07dvX4/VavWMHj3a8+mnnwa6JLRDUpu3l156KdClwU+XXXaZ56677gp0GTiF9957zzN48GCPzWbznHfeeZ7nn38+0CXhNKqrqz133XWXp2/fvp6wsDDP2Wef7fnVr37lsdvtgS7NL+wjAwAADIseGQAAYFgEGQAAYFgEGQAAYFgEGQAAYFgEGQAAYFgEGQAAYFgEGQAAYFgEGQAAYFgEGQBB59Zbb9XUqVO99y+//HLNnDmzQ9fsjGsACD4EGQA+u/XWW2UymWQymWS1WjVgwAD99re/ldPp7NKf++677+qRRx7x6bWrVq2SyWRSZWXlGV8DgHFwaCQAv1x11VV66aWXZLfb9c9//lPTp09XaGioZs+e3ep1DodDVqu1U35mQkJCUFwDQPBhRAaAX2w2m1JTU9WvXz/dcccdGj9+vJYuXeqdDpo7d67S09M1cOBASVJhYaFuuOEGxcXFKSEhQVOmTNH+/fu913O5XJo1a5bi4uKUmJioX/ziFzrxCLgTp4Xsdrt++ctfKjMzUzabTQMGDNCLL76o/fv3a9y4cZKk+Ph4mUwm3XrrrW1e49ixY7rlllsUHx+viIgITZw4UQUFBd7nX375ZcXFxWnZsmXKyclRVFSUrrrqKh05csT7mlWrVmn06NGKjIxUXFycLr74Yh04cKCT/qUB+IIgA6BDwsPD5XA4JEkrV67Url27tGLFCr3//vtqampSXl6eoqOjtXbtWn388cfeQND8nieeeEIvv/yy/vKXv2jdunWqqKjQ4sWLT/kzb7nlFr3++ut66qmntHPnTj333HOKiopSZmam3nnnHUnSrl27dOTIEf3hD39o8xq33nqrNm7cqKVLl2r9+vXyeDy6+uqr1dTU5H1NfX29Hn/8cb3yyitas2aNDh48qHvvvVeS5HQ6NXXqVF122WX64osvtH79et1+++0ymUwd/jcF4DumlgCcEY/Ho5UrV2rZsmX6+c9/rqNHjyoyMlJ//vOfvVNKf//73+V2u/XnP//Z+wX/0ksvKS4uTqtWrdKECRO0cOFCzZ49W9dee60kadGiRVq2bFm7P3f37t166623tGLFCo0fP16SdPbZZ3ufb55CSk5OVlxcXJvXKCgo0NKlS/Xxxx/roosukiS9+uqryszM1JIlS3T99ddLkpqamrRo0SKdc845kqQ777xTv/3tbyVJ1dXVqqqq0ne+8x3v8zk5Of7/QwLoEEZkAPjl/fffV1RUlMLCwjRx4kTdeOON+s1vfiNJGjJkSKu+mG3btmnPnj2Kjo5WVFSUoqKilJCQoMbGRu3du1dVVVU6cuSIxowZ432PxWLRyJEj2/35W7duldls1mWXXXbGf4edO3fKYrG0+rmJiYkaOHCgdu7c6X0sIiLCG1IkKS0tTaWlpZKOB6Zbb71VeXl5mjRpkv7whz+0mnYC0D0YkQHgl3HjxunZZ5+V1WpVenq6LJb//hqJjIxs9dra2lqNGDFCr7766knXOeuss87o54eHh5/R+85EaGhoq/smk6lV/85LL72kGTNm6MMPP9Sbb76pX//611qxYoUuvPDCbqsR6O0YkQHgl8jISA0YMEB9+/ZtFWLacsEFF6igoEDJyckaMGBAq1tsbKxiY2OVlpamzz77zPsep9OpTZs2tXvNIUOGyO12a/Xq1W0+3zwi5HK52r1GTk6OnE5nq59bXl6uXbt2adCgQaf8O51o+PDhmj17tj755BMNHjxYr732ml/vB9AxBBkAXebmm29WUlKSpkyZorVr12rfvn1atWqVZsyYoaKiIknSXXfdpUcffVRLlizRV199pZ/97Gcn7QHTUv/+/TVt2jT9+Mc/1pIlS7zXfOuttyRJ/fr1k8lk0vvvv6+jR4+qtrb2pGtkZ2drypQpuu2227Ru3Tpt27ZNP/jBD5SRkaEpU6b49Hfbt2+fZs+erfXr1+vAgQNavny5CgoK6JMBuhlBBkCXiYiI0Jo1a9S3b19de+21ysnJ0U9+8hM1NjYqJiZGknTPPffohz/8oaZNm6axY8cqOjpa3/3ud0953WeffVbf+9739LOf/UznnXeebrvtNtXV1UmSMjIy9PDDD+v+++9XSkqK7rzzzjav8dJLL2nEiBH6zne+o7Fjx8rj8eif//znSdNJp/q7ffXVV7ruuut07rnn6vbbb9f06dP105/+1I9/IQAdZfKcuGEDAACAQTAiAwAADIsgAwAADIsgAwAADIsgAwAADIsgAwAADIsgAwAADIsgAwAADIsgAwAADIsgAwAADIsgAwAADIsgAwAADOv/Ax67S9hWGwT/AAAAAElFTkSuQmCC\n"
},
"metadata": {}
}
],
"source": [
"# plotting actual values vs. predictions\n",
"ypred = model4(torch.from_numpy(np.float32(x_test)).to(device)).cpu().detach().numpy()\n",
"plt.scatter(ypred,y_non_test,s=.1)\n",
"plt.xlabel(\"Predictions\");\n",
"plt.ylabel(\"Actual values\");"
]
},
{
"cell_type": "code",
"source": [
"# compute the number of trainable parameters in shallow and deep net\n",
"m3p = sum(p.numel() for p in model3.parameters() if p.requires_grad)\n",
"m4p = sum(p.numel() for p in model4.parameters() if p.requires_grad)\n",
"print(\"Shallow net parameter number: \" + str(m3p))\n",
"print(\"Deep net parameter number: \" + str(m4p))"
],
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/"
},
"id": "A1IzvAZsSsYk",
"outputId": "face491c-4d56-40b7-c104-a7745b34344a"
},
"execution_count": null,
"outputs": [
{
"output_type": "stream",
"name": "stdout",
"text": [
"Shallow net parameter number: 3001\n",
"Deep net parameter number: 901\n"
]
}
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "KQalPfpB0kNz"
},
"source": [
"As you can see, the deep net has achieved similar performance to the shallow network, with only a fraction of the parameters!"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "TNIfRm2h3BMq"
},
"source": [
"### Bells and whistles\n",
"The deep net we used in the previous example was quite \"bare bones\" by contemporary standards. In practice, most deep nets rely on a variety of other features to improve their performance. Showcasing all of these features is beyond the scope of this tutorial, but the example below demonstrates a few popular examples, including:\n",
"\n",
"\n",
"* Skip-layer connections : In the most prototypical case, each layer in an ANN is connected only to the adjacent layers (e.g., layer 2 is connected to layer 1 and layer 3). This is known as a sequential structure, and the sequential class in pytorch makes this very convenient. However, some of the most important milestones in deep learning have been achieved by abandoning this simple structure. For example, resnets (residual networks) are an architecture that allowed computer vision to equal humans in tasks like object identification. These networks rely on connections that skip layers. The network below demonstrates this: in addition to receiving input from the previous layer, the final outputs here also receive input from the first layer. The potential of the nn.Module class extends well beyond this too: it can be used to stitch together different ANNs performing different tasks in different ways.\n",
"* Alternate activation functions : Although ReLU is probably the most popular default choice of nonlinear activation function, there are many options to choose from, and some may be better suited to particular problems. Here we demonstrate the use of the hyperbolic tangent (tanh) function as an alternative to ReLU. For niche applications (e.g., cognitive modeling) you could even craft your own bespoke activation functions.\n",
"* Dropout : ANNs can use the same regularization, like ridge/lasso, that typical linear models can use. However, dropout is a method unique to ANNs. This method effectively \"lesions\" a random subset of units during the forward pass of the model. This forces the model to acquire a more robust structure - effectively, many different assemblies that can accomplish the same end goal - that tends to reduce overfitting.\n",
"* Early stopping : Another strategy to prevent overfitting is early stopping. This strategy only makes sense for optimization strategies that proceed in a sequential manner (like gradient descent) rather than a single step of matrix math. Early stopping compares the performance of the model in training set to its performance in a validation set. As long as validation performance continues to improve, the model keep training. But when training performance improves while validation performance stalls, this is a sign of overfitting. Early stopping tracks this potential divergence between training and validation, and stops training when validation performance stops improving.\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/"
},
"id": "9RDXf_L_9HUY",
"outputId": "2ad4145c-68db-4de4-b0f6-d7a4fd442ac9"
},
"outputs": [
{
"output_type": "stream",
"name": "stdout",
"text": [
"FancyANN(\n",
" (layerlist): ModuleList(\n",
" (0): Linear(in_features=1, out_features=32, bias=True)\n",
" (1): Tanh()\n",
" (2): BatchNorm1d(32, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
" (3): Dropout(p=0.01, inplace=False)\n",
" (4): Linear(in_features=32, out_features=16, bias=True)\n",
" (5): Tanh()\n",
" (6): BatchNorm1d(16, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
" (7): Dropout(p=0.01, inplace=False)\n",
" (8): Linear(in_features=16, out_features=8, bias=True)\n",
" (9): Tanh()\n",
" (10): BatchNorm1d(8, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
" (11): Dropout(p=0.01, inplace=False)\n",
" (12): Linear(in_features=8, out_features=4, bias=True)\n",
" (13): Tanh()\n",
" (14): BatchNorm1d(4, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
" (15): Dropout(p=0.01, inplace=False)\n",
" (16): Linear(in_features=4, out_features=2, bias=True)\n",
" (17): Tanh()\n",
" (18): BatchNorm1d(2, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
" (19): Dropout(p=0.01, inplace=False)\n",
" (20): Linear(in_features=34, out_features=1, bias=True)\n",
" )\n",
")\n"
]
}
],
"source": [
"# define model architecture\n",
"class FancyANN(nn.Module):\n",
" def __init__(self, ninput=1, noutput=1, nhlayer=5, fextint=True):\n",
" super().__init__()\n",
" self.fextint = 0\n",
" if fextint:\n",
" self.fextint = 2**nhlayer\n",
" self.nhlayer = nhlayer\n",
" self.layerlist = nn.ModuleList()\n",
" for i in range(nhlayer):\n",
" if i == 0:\n",
" self.layerlist.append(nn.Linear(ninput,2**(nhlayer-i)))\n",
" else:\n",
" self.layerlist.append(nn.Linear(2**(nhlayer-i+1),2**(nhlayer-i)))\n",
" self.layerlist.append(nn.Tanh()) # here's our alternative activation function (tanh)\n",
" self.layerlist.append(nn.BatchNorm1d(2**(nhlayer-i)))\n",
" self.layerlist.append(nn.Dropout(p=.01)) ## here's where we add dropout\n",
" self.layerlist.append(nn.Linear(self.fextint+2,noutput))\n",
"\n",
" def forward(self, x):\n",
" for i in range(len(self.layerlist)-1):\n",
" x = self.layerlist[i](x)\n",
" if i == 2:\n",
" x0 = torch.clone(x)\n",
" pred = self.layerlist[-1](torch.cat((x0,x),1)) ## here's where the first layer and penultimate layer get joined together as inputs to the final layer\n",
" return pred\n",
"\n",
"model5 = FancyANN().to(device)\n",
"print(model5)"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "JnPm153TQ48K"
},
"outputs": [],
"source": [
"# this class defines the logic of early stopping\n",
"# higher patience means that improvement on the validation set has to stall for longer before training is terminated\n",
"class EarlyStopper:\n",
" def __init__(self, patience=1, min_delta=0):\n",
" self.patience = patience\n",
" self.min_delta = min_delta\n",
" self.counter = 0\n",
" self.min_validation_loss = np.inf\n",
"\n",
" def early_stop(self, validation_loss):\n",
" if validation_loss < self.min_validation_loss:\n",
" self.min_validation_loss = validation_loss\n",
" self.counter = 0\n",
" elif validation_loss > (self.min_validation_loss + self.min_delta):\n",
" self.counter += 1\n",
" if self.counter >= self.patience:\n",
" return True\n",
" return False"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/"
},
"id": "bnfctx2N8Zoz",
"outputId": "251c68f6-a1a9-4f37-ad3b-6e766292ac56"
},
"outputs": [
{
"output_type": "stream",
"name": "stdout",
"text": [
"Epoch 1\n",
"-------------------------------\n",
"loss: 3.888131 [ 64/100000]\n",
"loss: 3.917098 [ 6464/100000]\n",
"loss: 2.937879 [12864/100000]\n",
"loss: 5.071634 [19264/100000]\n",
"loss: 3.716036 [25664/100000]\n",
"loss: 2.398747 [32064/100000]\n",
"loss: 1.351015 [38464/100000]\n",
"loss: 1.365817 [44864/100000]\n",
"loss: 1.875856 [51264/100000]\n",
"loss: 1.061013 [57664/100000]\n",
"loss: 1.337905 [64064/100000]\n",
"loss: 1.853628 [70464/100000]\n",
"loss: 0.764149 [76864/100000]\n",
"loss: 0.915835 [83264/100000]\n",
"loss: 0.340542 [89664/100000]\n",
"loss: 0.346805 [96064/100000]\n",
"Test performance: \n",
" R^2: 0.780519, RMSE: 0.845239 \n",
"\n",
"Epoch 2\n",
"-------------------------------\n",
"loss: 0.396910 [ 64/100000]\n",
"loss: 0.205753 [ 6464/100000]\n",
"loss: 0.607033 [12864/100000]\n",
"loss: 1.113464 [19264/100000]\n",
"loss: 0.097006 [25664/100000]\n",
"loss: 0.100322 [32064/100000]\n",
"loss: 0.931997 [38464/100000]\n",
"loss: 0.178657 [44864/100000]\n",
"loss: 0.537316 [51264/100000]\n",
"loss: 0.205982 [57664/100000]\n",
"loss: 0.245557 [64064/100000]\n",
"loss: 0.080524 [70464/100000]\n",
"loss: 0.222449 [76864/100000]\n",
"loss: 0.213365 [83264/100000]\n",
"loss: 0.096103 [89664/100000]\n",
"loss: 0.591012 [96064/100000]\n",
"Test performance: \n",
" R^2: 0.979044, RMSE: 0.285466 \n",
"\n",
"Epoch 3\n",
"-------------------------------\n",
"loss: 0.064183 [ 64/100000]\n",
"loss: 0.243895 [ 6464/100000]\n",
"loss: 0.307392 [12864/100000]\n",
"loss: 0.179972 [19264/100000]\n",
"loss: 0.183356 [25664/100000]\n",
"loss: 0.188181 [32064/100000]\n",
"loss: 0.091715 [38464/100000]\n",
"loss: 0.677460 [44864/100000]\n",
"loss: 0.177787 [51264/100000]\n",
"loss: 0.066920 [57664/100000]\n",
"loss: 0.166385 [64064/100000]\n",
"loss: 0.066744 [70464/100000]\n",
"loss: 0.129679 [76864/100000]\n",
"loss: 0.137251 [83264/100000]\n",
"loss: 0.135885 [89664/100000]\n",
"loss: 0.153805 [96064/100000]\n",
"Test performance: \n",
" R^2: 0.986803, RMSE: 0.227382 \n",
"\n",
"Epoch 4\n",
"-------------------------------\n",
"loss: 0.329610 [ 64/100000]\n",
"loss: 0.309667 [ 6464/100000]\n",
"loss: 0.238928 [12864/100000]\n",
"loss: 0.049671 [19264/100000]\n",
"loss: 0.172278 [25664/100000]\n",
"loss: 0.167100 [32064/100000]\n",
"loss: 0.202609 [38464/100000]\n",
"loss: 0.105646 [44864/100000]\n",
"loss: 0.214544 [51264/100000]\n",
"loss: 0.181315 [57664/100000]\n",
"loss: 0.105952 [64064/100000]\n",
"loss: 0.054405 [70464/100000]\n",
"loss: 0.230735 [76864/100000]\n",
"loss: 0.109101 [83264/100000]\n",
"loss: 0.124816 [89664/100000]\n",
"loss: 0.099370 [96064/100000]\n",
"Test performance: \n",
" R^2: 0.988537, RMSE: 0.204467 \n",
"\n",
"Epoch 5\n",
"-------------------------------\n",
"loss: 0.063645 [ 64/100000]\n",
"loss: 0.123861 [ 6464/100000]\n",
"loss: 0.083427 [12864/100000]\n",
"loss: 0.250894 [19264/100000]\n",
"loss: 0.298372 [25664/100000]\n",
"loss: 0.049779 [32064/100000]\n",
"loss: 0.054952 [38464/100000]\n",
"loss: 0.233167 [44864/100000]\n",
"loss: 0.297191 [51264/100000]\n",
"loss: 0.102348 [57664/100000]\n",
"loss: 0.036531 [64064/100000]\n",
"loss: 0.079810 [70464/100000]\n",
"loss: 0.119193 [76864/100000]\n",
"loss: 0.105115 [83264/100000]\n",
"loss: 0.118437 [89664/100000]\n",
"loss: 0.212578 [96064/100000]\n",
"Test performance: \n",
" R^2: 0.989374, RMSE: 0.194593 \n",
"\n",
"Epoch 6\n",
"-------------------------------\n",
"loss: 0.104637 [ 64/100000]\n",
"loss: 0.124436 [ 6464/100000]\n",
"loss: 0.227147 [12864/100000]\n",
"loss: 0.285004 [19264/100000]\n",
"loss: 0.306534 [25664/100000]\n",
"loss: 0.126568 [32064/100000]\n",
"loss: 0.062833 [38464/100000]\n",
"loss: 0.070517 [44864/100000]\n",
"loss: 0.049845 [51264/100000]\n",
"loss: 0.080152 [57664/100000]\n",
"loss: 0.142207 [64064/100000]\n",
"loss: 0.082779 [70464/100000]\n",
"loss: 0.067380 [76864/100000]\n",
"loss: 0.080468 [83264/100000]\n",
"loss: 0.099579 [89664/100000]\n",
"loss: 0.149001 [96064/100000]\n",
"Test performance: \n",
" R^2: 0.991916, RMSE: 0.170523 \n",
"\n",
"Epoch 7\n",
"-------------------------------\n",
"loss: 0.090363 [ 64/100000]\n",
"loss: 0.105410 [ 6464/100000]\n",
"loss: 0.040804 [12864/100000]\n",
"loss: 0.054256 [19264/100000]\n",
"loss: 0.027425 [25664/100000]\n",
"loss: 0.207500 [32064/100000]\n",
"loss: 0.090952 [38464/100000]\n",
"loss: 0.086845 [44864/100000]\n",
"loss: 0.079099 [51264/100000]\n",
"loss: 0.275742 [57664/100000]\n",
"loss: 0.028412 [64064/100000]\n",
"loss: 0.116104 [70464/100000]\n",
"loss: 0.120119 [76864/100000]\n",
"loss: 0.070722 [83264/100000]\n",
"loss: 0.085920 [89664/100000]\n",
"loss: 0.083501 [96064/100000]\n",
"Test performance: \n",
" R^2: 0.991818, RMSE: 0.170135 \n",
"\n",
"Epoch 8\n",
"-------------------------------\n",
"loss: 0.223899 [ 64/100000]\n",
"loss: 0.205298 [ 6464/100000]\n",
"loss: 0.073132 [12864/100000]\n",
"loss: 1.529429 [19264/100000]\n",
"loss: 0.101772 [25664/100000]\n",
"loss: 0.157559 [32064/100000]\n",
"loss: 0.073842 [38464/100000]\n",
"loss: 0.123590 [44864/100000]\n",
"loss: 0.053510 [51264/100000]\n",
"loss: 0.150473 [57664/100000]\n",
"loss: 0.092963 [64064/100000]\n",
"loss: 0.050164 [70464/100000]\n",
"loss: 0.299292 [76864/100000]\n",
"loss: 0.059084 [83264/100000]\n",
"loss: 0.048018 [89664/100000]\n",
"loss: 0.114542 [96064/100000]\n",
"Test performance: \n",
" R^2: 0.992695, RMSE: 0.156656 \n",
"\n",
"Epoch 9\n",
"-------------------------------\n",
"loss: 0.048064 [ 64/100000]\n",
"loss: 1.965292 [ 6464/100000]\n",
"loss: 0.050489 [12864/100000]\n",
"loss: 0.111465 [19264/100000]\n",
"loss: 0.129185 [25664/100000]\n",
"loss: 0.453362 [32064/100000]\n",
"loss: 0.169424 [38464/100000]\n",
"loss: 0.096436 [44864/100000]\n",
"loss: 0.081334 [51264/100000]\n",
"loss: 0.086524 [57664/100000]\n",
"loss: 0.108220 [64064/100000]\n",
"loss: 0.026716 [70464/100000]\n",
"loss: 0.330060 [76864/100000]\n",
"loss: 0.081770 [83264/100000]\n",
"loss: 0.102128 [89664/100000]\n",
"loss: 0.380724 [96064/100000]\n",
"Test performance: \n",
" R^2: 0.992518, RMSE: 0.160280 \n",
"\n",
"Epoch 10\n",
"-------------------------------\n",
"loss: 0.067976 [ 64/100000]\n",
"loss: 0.061944 [ 6464/100000]\n",
"loss: 0.093701 [12864/100000]\n",
"loss: 0.062002 [19264/100000]\n",
"loss: 0.124478 [25664/100000]\n",
"loss: 0.267889 [32064/100000]\n",
"loss: 0.094276 [38464/100000]\n",
"loss: 0.100655 [44864/100000]\n",
"loss: 0.194106 [51264/100000]\n",
"loss: 1.437881 [57664/100000]\n",
"loss: 0.037840 [64064/100000]\n",
"loss: 0.100686 [70464/100000]\n",
"loss: 0.042177 [76864/100000]\n",
"loss: 0.150511 [83264/100000]\n",
"loss: 0.025171 [89664/100000]\n",
"loss: 0.045438 [96064/100000]\n",
"Test performance: \n",
" R^2: 0.991603, RMSE: 0.166102 \n",
"\n",
"Epoch 11\n",
"-------------------------------\n",
"loss: 0.113018 [ 64/100000]\n",
"loss: 0.364865 [ 6464/100000]\n",
"loss: 0.086984 [12864/100000]\n",
"loss: 0.089000 [19264/100000]\n",
"loss: 0.048422 [25664/100000]\n",
"loss: 0.111554 [32064/100000]\n",
"loss: 0.170104 [38464/100000]\n",
"loss: 0.456190 [44864/100000]\n",
"loss: 0.430310 [51264/100000]\n",
"loss: 0.137799 [57664/100000]\n",
"loss: 0.087476 [64064/100000]\n",
"loss: 0.034155 [70464/100000]\n",
"loss: 0.051317 [76864/100000]\n",
"loss: 0.139362 [83264/100000]\n",
"loss: 0.162727 [89664/100000]\n",
"loss: 0.079858 [96064/100000]\n",
"Test performance: \n",
" R^2: 0.992894, RMSE: 0.148741 \n",
"\n",
"Epoch 12\n",
"-------------------------------\n",
"loss: 0.156293 [ 64/100000]\n",
"loss: 0.093481 [ 6464/100000]\n",
"loss: 0.044689 [12864/100000]\n",
"loss: 0.069560 [19264/100000]\n",
"loss: 0.049289 [25664/100000]\n",
"loss: 0.330965 [32064/100000]\n",
"loss: 0.060987 [38464/100000]\n",
"loss: 0.041339 [44864/100000]\n",
"loss: 0.157445 [51264/100000]\n",
"loss: 0.097228 [57664/100000]\n",
"loss: 0.154629 [64064/100000]\n",
"loss: 0.153998 [70464/100000]\n",
"loss: 0.262758 [76864/100000]\n",
"loss: 0.049504 [83264/100000]\n",
"loss: 0.030409 [89664/100000]\n",
"loss: 0.068077 [96064/100000]\n",
"Test performance: \n",
" R^2: 0.992230, RMSE: 0.156347 \n",
"\n",
"Epoch 13\n",
"-------------------------------\n",
"loss: 0.205354 [ 64/100000]\n",
"loss: 0.252282 [ 6464/100000]\n",
"loss: 0.046880 [12864/100000]\n",
"loss: 0.122652 [19264/100000]\n",
"loss: 0.040483 [25664/100000]\n",
"loss: 0.332348 [32064/100000]\n",
"loss: 0.209426 [38464/100000]\n",
"loss: 0.071124 [44864/100000]\n",
"loss: 0.103392 [51264/100000]\n",
"loss: 0.065575 [57664/100000]\n",
"loss: 0.078010 [64064/100000]\n",
"loss: 0.036993 [70464/100000]\n",
"loss: 0.177545 [76864/100000]\n",
"loss: 0.105658 [83264/100000]\n",
"loss: 0.056419 [89664/100000]\n",
"loss: 0.047285 [96064/100000]\n",
"Test performance: \n",
" R^2: 0.991006, RMSE: 0.164956 \n",
"\n",
"Epoch 14\n",
"-------------------------------\n",
"loss: 0.142373 [ 64/100000]\n",
"loss: 0.032840 [ 6464/100000]\n",
"loss: 0.054341 [12864/100000]\n",
"loss: 0.045234 [19264/100000]\n",
"loss: 0.072157 [25664/100000]\n",
"loss: 0.146573 [32064/100000]\n",
"loss: 0.042619 [38464/100000]\n",
"loss: 0.093733 [44864/100000]\n",
"loss: 0.126631 [51264/100000]\n",
"loss: 0.174637 [57664/100000]\n",
"loss: 0.021763 [64064/100000]\n",
"loss: 0.069974 [70464/100000]\n",
"loss: 0.037953 [76864/100000]\n",
"loss: 0.215359 [83264/100000]\n",
"loss: 0.056612 [89664/100000]\n",
"loss: 0.086736 [96064/100000]\n",
"Test performance: \n",
" R^2: 0.991521, RMSE: 0.160487 \n",
"\n",
"Done!\n"
]
}
],
"source": [
"# new we'll train the model\n",
"early_stopper = EarlyStopper(patience=3, min_delta=0)\n",
"loss = nn.MSELoss()\n",
"optimizer = torch.optim.Adam(model5.parameters(),lr=.0001) # we're trying a different optimizer here too (ADAM)\n",
"epochs = 100 # this is just an upperbound - the actual epoch # is determined by early stopping\n",
"for t in range(epochs):\n",
" print(f\"Epoch {t+1}\\n-------------------------------\")\n",
" train(train_dataloader, model5, loss, optimizer)\n",
" val_loss = test(val_dataloader, model5, loss)[1]\n",
" if early_stopper.early_stop(val_loss):\n",
" break\n",
"print(\"Done!\")"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/"
},
"id": "HmLpYXOM7hE3",
"outputId": "63b59af5-7102-4ae7-cd9d-72f9c49f90b9"
},
"outputs": [
{
"output_type": "stream",
"name": "stdout",
"text": [
"Test performance: \n",
" R^2: 0.992058, RMSE: 0.155241 \n",
"\n"
]
}
],
"source": [
"# performance metrics\n",
"perf = test(test_dataloader, model5, loss)"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/",
"height": 449
},
"id": "U5YH-_8pM5v9",
"outputId": "8987259c-c719-4da5-e6ed-147be00b394a"
},
"outputs": [
{
"output_type": "display_data",
"data": {
"text/plain": [
""
],
"image/png": "iVBORw0KGgoAAAANSUhEUgAAAjIAAAGwCAYAAACzXI8XAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjcuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/bCgiHAAAACXBIWXMAAA9hAAAPYQGoP6dpAAA/+UlEQVR4nO3de3iU1bn+8XsOmZkchyRADhAgYkQIiAiCeKgHkBQsQrVqu21F7a79WdyAWKvYorVWEKrW2rql2qqtFdBdhSK2BjYVkBYQhCCmCEEhHEKAEHJOZjIz7+8PmtkEEkxgkpl38v1c11zbOebJbMp7s9az1rIYhmEIAADAhKzhLgAAAOBsEWQAAIBpEWQAAIBpEWQAAIBpEWQAAIBpEWQAAIBpEWQAAIBp2cNdQEcLBAIqKSlRYmKiLBZLuMsBAABtYBiGqqurlZmZKau19XGXqA8yJSUlysrKCncZAADgLOzfv1+9e/du9fmoDzKJiYmSTnwRSUlJYa4GAAC0RVVVlbKysoLX8dZEfZBpmk5KSkoiyAAAYDJf1hZCsy8AADAtggwAADAtggwAADAtggwAADAtggwAADAtggwAADAtggwAADAtggwAADAtggwAADAtggwAADAtggwAADAtggwAADAtggwAADAtggwAADgrdV6flmw9oDqvL2w1hDXIrF27VhMnTlRmZqYsFouWLl0afK6xsVEPPfSQhgwZovj4eGVmZuqOO+5QSUlJ+AoGAABB+YWlKq/xakXh4bDVENYgU1tbq6FDh+qFF1447bm6ujpt2bJFs2fP1pYtW/TOO+9o586duvHGG8NQKQAAOFVebrpSE5wal5sWthoshmEYYfvpJ7FYLFqyZIkmT57c6ms2bdqkkSNHqri4WH369GnT51ZVVcntdquyslJJSUkhqhYAAHSktl6/7Z1Y0zmrrKyUxWJRt27dWn2Nx+ORx+MJ3q+qquqEygAAQDiYptm3oaFBDz30kL71rW+dMZnNnTtXbrc7eMvKyurEKgEAQGcyRZBpbGzUrbfeKsMw9OKLL57xtbNmzVJlZWXwtn///k6qEgAAdLaIn1pqCjHFxcX6+9///qV9Lk6nU06ns5OqAwAA4RTRQaYpxBQVFemDDz5QampquEsCAAARJKxBpqamRrt37w7e37NnjwoKCpSSkqKMjAx94xvf0JYtW7R8+XL5/X6VlpZKklJSUuRwOMJVNgAAiBBhXX69evVqXXvttac9PmXKFP30pz9VdnZ2i+/74IMPdM0117TpZ7D8GgAA8zHF8utrrrlGZ8pREbLFDQAAiFCmWLUEAADQEoIMAAAwLYIMAAAwLYIMAAAwLYIMAAAwLYIMAAAwLYIMAAAwLYIMAAAwLYIMAAAwLYIMAAAwLYIMAAAwLYIMAAAwLYIMAAAwLYIMAAAwLYIMAAAwLYIMAAAwLYIMAAAwLYIMAAAwLYIMAAAwLYIMAAAwLYIMAAAwLYIMAAAwLYIMAAAwLYIMAAAwLYIMAAAwLYIMAAAwLYIMAAAwLYIMAAAwLYIMAAAmU+f1acnWA6rz+sJdStgRZAAAMJn8wlKV13i1ovBwuEsJO4IMAAAmk5ebrtQEp8blpoW7lLCzh7sAAADQPnEOuyYP6xXuMiICIzIAAMC0CDIAAEQYmnnbjiADAECEoZm37QgyAABEGJp5245mXwAAIgzNvG3HiAwAADAtggwAADAtggwAADAtggwAADAtggwAADAtggwAADCtsAaZtWvXauLEicrMzJTFYtHSpUubPW8Yhh599FFlZGQoNjZWY8eOVVFRUXiKBQAAESesQaa2tlZDhw7VCy+80OLz8+fP1/PPP68FCxZo48aNio+PV15enhoaGjq5UgAAEInCuiHe+PHjNX78+BafMwxDzz33nH7yk59o0qRJkqQ//vGPSktL09KlS/XNb36zM0sFAAARKGJ7ZPbs2aPS0lKNHTs2+Jjb7daoUaO0fv36Vt/n8XhUVVXV7AYAAKJTxAaZ0tJSSVJaWvNzJtLS0oLPtWTu3Llyu93BW1ZWVofWCQAAwidig8zZmjVrliorK4O3/fv3h7skAADQQSI2yKSnp0uSDh9ufoT54cOHg8+1xOl0KikpqdkNAABEp4gNMtnZ2UpPT9eqVauCj1VVVWnjxo0aPXp0GCsDAACRIqyrlmpqarR79+7g/T179qigoEApKSnq06ePZsyYoZ///OfKyclRdna2Zs+erczMTE2ePDl8RQMAgIgR1iCzefNmXXvttcH7M2fOlCRNmTJFr732mn70ox+ptrZW99xzjyoqKnTllVfq/fffl8vlClfJAAAgglgMwzDCXURHqqqqktvtVmVlJf0yAACYRFuv3xHbIwMAAPBlCDIAAMC0CDIAAMC0CDIAAMC0CDIAAMC0CDIAAMC0CDIAAMC0CDIAAMC0CDIAALRBndenJVsPqM7rC3cpOAlBBgCANsgvLFV5jVcrCg+HuxSchCADAEAb5OWmKzXBqXG5aeEuBScJ66GRAACYRZzDrsnDeoW7DJyCERkAAGBaBBkAAGBaBBkAAGBaBBkAAGBaBBkAAGBaBBkAAGBaBBkAAGBaBBkAAGBaBBkAAGBaBBkAAGBaBBkAAGBaBBkAAGBaBBkAAGBaBBkAAGBaBBkAAGBaBBkAAGBaBBkAAGBaBBkAAGBaBBkAQJdQ5/VpydYDqvP6wl0KQoggAwDoEvILS1Ve49WKwsPhLgUhRJABAHQJebnpSk1walxuWrhLQQjZw10AAACdIc5h1+RhvcJdBkKMERkAAGBaBBkAQNShsbfrIMgAAKIOjb1dB0EGABB1aOztOmj2BQBEHRp7uw5GZAAAgGkRZAAAgGkRZAAAgGkRZAAAgGkRZAAAgGlFdJDx+/2aPXu2srOzFRsbq/79++uJJ56QYRjhLg0AAESAiF5+PW/ePL344ov6wx/+oNzcXG3evFl33XWX3G63pk2bFu7yAABAmEV0kPnnP/+pSZMm6YYbbpAk9evXT4sWLdJHH33U6ns8Ho88Hk/wflVVVYfXCQAAwiOip5Yuv/xyrVq1Srt27ZIkbdu2TevWrdP48eNbfc/cuXPldruDt6ysrM4qFwAAdDKLEcENJ4FAQI888ojmz58vm80mv9+vJ598UrNmzWr1PS2NyGRlZamyslJJSUmdUTYAADhHVVVVcrvdX3r9juippbfeektvvPGGFi5cqNzcXBUUFGjGjBnKzMzUlClTWnyP0+mU0+ns5EoBAEA4RHSQefDBB/Xwww/rm9/8piRpyJAhKi4u1ty5c1sNMgAAoOuI6B6Zuro6Wa3NS7TZbAoEAmGqCAAARJKIHpGZOHGinnzySfXp00e5ubnaunWrnn32Wd19993hLg0AAESAiG72ra6u1uzZs7VkyRIdOXJEmZmZ+ta3vqVHH31UDoejTZ/R1mYhAAAQOdp6/Y7oIBMKBBkAAMynrdfviO6RAQAAOBOCDAAAMC2CDAAAMK12B5n6+nrV1dUF7xcXF+u5557TihUrQloYACD61Hl9WrL1gOq8vnCXgijR7iAzadIk/fGPf5QkVVRUaNSoUXrmmWc0adIkvfjiiyEvEAAQPfILS1Ve49W720oINAiJdgeZLVu26KqrrpIk/fnPf1ZaWpqKi4v1xz/+Uc8//3zICwQARI+83HSlJjhlyFB5jVcrCg+HuySYXLuDTF1dnRITEyVJK1as0E033SSr1arLLrtMxcXFIS8QAGBep04lxTnsmjysl24c2kupCU6Ny00Lc4Uwu3YHmfPPP19Lly7V/v37lZ+fr3HjxkmSjhw5wj4tAIBmmqaSTh15aQo0cY6I3mAeJtDuIPPoo4/qhz/8ofr166eRI0dq9OjRkk6MzgwbNizkBQIAzKtpKunKnFR6YtAhzmpn39LSUh06dEhDhw4NHur40UcfKSkpSRdeeGHIizwX7OwLAOG3ZOsBldd4lZrg1ORhvcJdDkygQ3f2TU9PV2JiolauXKn6+npJ0qWXXhpxIQYAEBmaRmboiUGotTvIHDt2TGPGjNEFF1ygCRMm6NChQ5Kk7373u3rggQdCXiAAwPzoiUFHaXeQuf/++xUTE6N9+/YpLi4u+Phtt92m999/P6TFAQAAnEm7o/GKFSuUn5+v3r17N3s8JyeH5dcAAKBTtXtEpra2ttlITJPy8nI5nc6QFAUAANAW7Q4yV111VfCIAkmyWCwKBAKaP3++rr322pAWBwAAcCbtnlqaP3++xowZo82bN8vr9epHP/qRCgsLVV5ern/84x8dUSMAIAzqvD7lF5YqLzedJl1ErHaPyAwePFi7du3SlVdeqUmTJqm2tlY33XSTtm7dqv79+3dEjQCAMGhtV14gkpzVhnhmwoZ4AHB26rw+rSg8rHG5aYzIoNO19frd7j+Za9euPePzX/nKV9r7kQCACNS09wsQydodZK655prTHrNYLMH/9vv951QQACD86I+BWbS7R+b48ePNbkeOHNH777+vSy+9VCtWrOiIGgEAnaTO69OSrQe0bFsJ/TEwhXbHbLfbfdpj119/vRwOh2bOnKmPP/44JIUBADpfU4NvgsvO2UgwhZCNF6alpWnnzp2h+jgAQCdqmkq6Kqe71hUdo8EXptHuP6WffPJJs/uGYejQoUN66qmndPHFF4eqLgBABzq1B6ZpJGZd0TEafGEq7Q4yF198sSwWi05dtX3ZZZfplVdeCVlhAICOc/IeMZOH9VJebnpwqTVgJu0OMnv27Gl232q1qkePHnK5XCErCgDQsU4NLiy1hlm1O8j07du3I+oAAHSipuDStEopLzddklhyDdNp05/U559/vs0fOG3atLMuBgDQOZp6ZDy+gGobTuzga8hoNt0EmEGbjijIzs5u24dZLPriiy/OuahQ4ogCADjdkq0HgsusnXZbcIqJIwkQKdp6/easJQDogjhHCZGuw85aAgCYH829iBZnFWQOHDigZcuWad++ffJ6vc2ee/bZZ0NSGAAAwJdpd5BZtWqVbrzxRp133nn67LPPNHjwYO3du1eGYeiSSy7piBoBAABa1O5DI2fNmqUf/vCH2r59u1wul95++23t379fV199tW655ZaOqBEAAKBF7Q4yO3bs0B133CFJstvtqq+vV0JCgn72s59p3rx5IS8QAACgNe0OMvHx8cG+mIyMDH3++efB58rKykJXGQAAwJdod4/MZZddpnXr1mngwIGaMGGCHnjgAW3fvl3vvPOOLrvsso6oEQAAoEXtDjLPPvusampqJEmPP/64ampq9OabbyonJ4cVSwAAoFOxIR4AAIg4bb1+t7tH5j//8z+1evXqc6kNAAAgJNodZI4ePaqvfvWrysrK0oMPPqht27Z1RF0AAABfqt1B5i9/+YsOHTqk2bNna9OmTbrkkkuUm5urOXPmaO/evR1QIgAAQMvaHWQkKTk5Wffcc49Wr16t4uJi3XnnnXr99dd1/vnnh7o+HTx4UN/+9reVmpqq2NhYDRkyRJs3bw75zwGAzlTn9WnJ1gOq8/o69b1AtDmrINOksbFRmzdv1saNG7V3716lpaWFqi5J0vHjx3XFFVcoJiZGf/vb3/Svf/1LzzzzjJKTk0P6cwCgs+UXlqq8xqsVhYc79b1AtDmrQyM/+OADLVy4UG+//bYCgYBuuukmLV++XNddd11Ii5s3b56ysrL06quvBh/Lzs4+43s8Ho88Hk/wflVVVUhrAoBQyMtN14rCwxqX2/5/AJ7Le4Fo0+4RmV69emnChAkqKyvTSy+9pMOHD+uVV17RmDFjZLFYQlrcsmXLNGLECN1yyy3q2bOnhg0bppdffvmM75k7d67cbnfwlpWVFdKaACAU4hx2TR7WS5LaPU3U9N44x1n9WxSIKu3eR+bll1/WLbfcom7dunVQSf/H5XJJkmbOnKlbbrlFmzZt0vTp07VgwQJNmTKlxfe0NCKTlZXFPjIAItKSrQdUXuNVaoIzGGwAtH0fmYjeEM/hcGjEiBH65z//GXxs2rRp2rRpk9avX9+mz2BDPACRrM7rC04TMcIC/J8O2xCvM2VkZGjQoEHNHhs4cKD27dsXpooAILSYJgLOTUQHmSuuuEI7d+5s9tiuXbvUt2/fMFUEAAAiSUQHmfvvv18bNmzQnDlztHv3bi1cuFAvvfSSpk6dGu7SAABABIjoIHPppZdqyZIlWrRokQYPHqwnnnhCzz33nG6//fZwlwYAACJAm5p9ly1b1uYPvPHGG8+poFCj2RcAAPNp6/W7Td1lkydPbtMPtVgs8vv9bXotAADAuWpTkAkEAh1dBwAAQLtFdI8MAJgBhzgC4XNWGxfU1tZqzZo12rdvn7xeb7Pnpk2bFpLCAMAsTj7Ekd15gc7V7iCzdetWTZgwQXV1daqtrVVKSorKysoUFxennj17EmQAdDkc4giET7unlu6//35NnDhRx48fV2xsrDZs2KDi4mINHz5cTz/9dEfUCABh05ZpI3bnBcKn3UGmoKBADzzwgKxWq2w2mzwej7KysjR//nw98sgjHVEjAITNydNGACJPu4NMTEyMrNYTb+vZs2fw3CO32639+/eHtjoACLO83HSlJjiZNgIiVLvHQYcNG6ZNmzYpJydHV199tR599FGVlZXp9ddf1+DBgzuiRgAIm6Zpo7ao8/qUX1iqvNx0ppmATtLuEZk5c+YoIyNDkvTkk08qOTlZ9957r44ePaqXXnop5AUCQCRqqXeGaSig87XpiAIz44gCAB1hydYDKq/xKjXBGRyxqfP6gquXGJEBzk1br99siAcAZ6Gl3hlWLwGdr93/a8vOzpbFYmn1+S+++OKcCgIAM2hP7wyAjtPuIDNjxoxm9xsbG7V161a9//77evDBB0NVFwAAwJdqd5CZPn16i4+/8MIL2rx58zkXBACRhJVIQGQLWY/M+PHj9fbbb4fq4wCg05xp915WIgGRLWRB5s9//rNSUlJC9XEA0GnOFFbYEA+IbGe1Id7Jzb6GYai0tFRHjx7Vf//3f4e0OADoDGc69JGmXiCytTvITJo0qVmQsVqt6tGjh6655hpdeOGFIS0OADoDYQUwLzbEAwAAEafDNsSz2Ww6cuTIaY8fO3ZMNputvR8HAABw1todZFobwPF4PHI4HOdcEACcqzOtQgIQXdrcI/P8889LkiwWi373u98pISEh+Jzf79fatWvpkQEQEU5ehUTvCxDd2hxkfvnLX0o6MSKzYMGCZtNIDodD/fr104IFC0JfIQC005lWIQGILm0OMnv27JEkXXvttXrnnXeUnJzcYUUBwLlgFRLQdbS7R+aDDz4gxACIWPTHAF1Lu4PMzTffrHnz5p32+Pz583XLLbeEpCgAXUsowwdHCgBdS7uDzNq1azVhwoTTHh8/frzWrl0bkqIAdC2hDB8cKQB0Le0OMjU1NS0us46JiVFVVVVIigLQteTlpivBZVeDz3/OozJN/TGcVA10De0OMkOGDNGbb7552uOLFy/WoEGDQlIUgK4lzmGXw25VbYOPKSEA7dLuf7LMnj1bN910kz7//HNdd911kqRVq1Zp0aJF+p//+Z+QFwggutR5fcovLFVebnqzUROWTAM4G2d11tJ7772nOXPmqKCgQLGxsbrooov02GOP6eqrr+6IGs8JZy0BkWXJ1gMqr/EqNcHJEmkArWrr9Tukh0Z++umnGjx4cKg+LiQIMkBkqfP6giMvZ+pjOXXkprWRHADRqcMOjTxVdXW1XnrpJY0cOVJDhw49148DEOXa2ox76komllUDaMlZB5m1a9fqjjvuUEZGhp5++mldd9112rBhQyhrA9AFNe0pc1VO92bLqFlWDaAl7RqfLS0t1Wuvvabf//73qqqq0q233iqPx6OlS5eyYglASDSNvKwrOtash4ZjBwC0pM0jMhMnTtSAAQP0ySef6LnnnlNJSYl+/etfd2RtALogRl4AtEebR2T+9re/adq0abr33nuVk5PTkTUBMKFQNOPS0Augvdo8IrNu3TpVV1dr+PDhGjVqlH7zm9+orKysI2sDYCJf1ozblvOUaOgF0F5tDjKXXXaZXn75ZR06dEjf//73tXjxYmVmZioQCGjlypWqrq7uyDoBRKA6r0+LN+3T4k3FpzXnnqotIYVpJQDtdU77yOzcuVO///3v9frrr6uiokLXX3+9li1bFsr6zhn7yAAdZ8nWA1q986gskq4Z0POMzbht3T8GAKRO2kdmwIABmj9/vg4cOKBFixady0cBiCBtmQaSToygXN4/VaP7p37pKAqHOQLoCCHd2TcSMSIDtN/ZHiNAsy6AUOm0nX0701NPPSWLxaIZM2aEuxQgqp1trwrNugA6m2mCzKZNm/Tb3/5WF110UbhLAaLe2U4D0awLoLOZIsjU1NTo9ttv18svv6zk5ORwlwOgFfTBAOhspggyU6dO1Q033KCxY8d+6Ws9Ho+qqqqa3QC0XVsbfQEgEkR8kFm8eLG2bNmiuXPntun1c+fOldvtDt6ysrI6uEIgutDnAsBMIjrI7N+/X9OnT9cbb7whl8vVpvfMmjVLlZWVwdv+/fs7uEogurS1z4WRGwCRIKKXXy9dulRf//rXZbPZgo/5/X5ZLBZZrVZ5PJ5mz7WE5dfAmZ3tkumzXaINAG0RFcuvx4wZo+3bt6ugoCB4GzFihG6//XYVFBR8aYgB8OVamkpqbbTl5MdZoQQgEkR0kElMTNTgwYOb3eLj45WamqrBgweHuzwgorVnd95TA0lrfTInP84KJQCRIKKDDIC2OzW4nBw6zhRqWgokrY22MAoDINJEdI9MKNAjg67i1J6Vkw9pbAo19LMAMIuo6JEB0HanjpY0jbRIksfnV4LLzkgKgKhDkAGiRGs9K/mFpapt8MtpP9Ecz5JpANGEIANEuZNHatjsDkC0IcgAUaK1ht6TR2po1gUQbQgygImdHF7yC0tVWtGgOe/taHXqiCXTAKINQQYwkTMtsc7LTdfBinr1So5l6ghAl0GQAUzk1B6Xk6eK4hx2PXLDQGW4Y5k6AtBlEGSACNRav0trS6zjHPbg9FJTqAGAroAgA0SgU0demoKNpODqo1NDDiuSAHRFBBkgAp068nJyI++ybQdbDCysSALQFRFkgAhU5/Vp097y4KjLyY28FlmUmuDUlTmpzaafWJEEoCsiyACdrC2nUv/i/Z3atr9CT/11R3BKqamRd+LQTI3LTdMvVxaptLKBqSQAXRpBBuggrQWWtvSy5KQlqMbjU6PfCL725BGX/MJS9e4Wq4PH65lKAtClEWSADtJaYGmtl+Xk4JPgitHFWd00ol9Ki6/Ny01XRrdYPXLDQKaSAHRpFsMwjHAX0ZHaegw4EGp1Xp9WFB5u83LoJVsPqLzGGwwuTe+VToSivNx0QguALqOt129GZIAO0lrzbVlNg368ZLvKahqaPX7q5nYnTyOxrBoAWkaQATpQndenxZv2afGm4mCvzC9XFulIVYOeW1nU7LWtBR+WVQNA6wgyQAfKLyzV+s+PacPn5cERlf939XmqavCpX/e4Zo3AbTm9GgDQHEEG6EB5ueka0TdZLrtVV+akSpI2Fx9XepJLnx6s0orCw8EAs2xbCVNIANBOBBmgA5x8pECCy66+3eP13Moi1Xl9ystN1+Xnp2p0/9TgcQPlNV5ZJKaQAKCdWLUEhFid16cn39uh3smxSol3yJChrfsq1C8lXhndYjV5WK/TXt+e1U0A0BW09frN35rAWarz+rRs20FJFt04NDMYQpo2qysuq9XeY7XqlxKvS/oky2m3tTja0tQDAwBoP4IMcJbyC0u1fne5LBbJ9e+Qkl9Yqqtyumtd0TElJzh0vNarg8frNeWKfsGgU+f1sS8MAIQIPTJAK77sTKSWel1KK04sqx6Xm6Ybh2Yqw3367rvsCwMAoUOQAVrxZYEjzmHXbZf20W2X9lGcwx48obpHolNz3tshSewLAwAdjCADtKDO65PHF1CCy97mwBHnsOuRGwbqaLVHvZJjmy2tPnlUh31hACB0CDJAC/ILS1Xb4JPTbmtX4GgKMxnu2GZLq5lGAoCOQZABWnAu0z8nj7gwjQQAHYt9ZIA2am25NQAg9Dj9GjhHp/a3NC233vD5MaaKACBC8E9K4CRNe7xcldNdv1xZpN7/btqdPKyX8nLT5fUFJImpIgCIEAQZdHknb1C3bNvBE6MuXxxTv5T4E5vZXd5P0v8ttwYARA6mltBlnDxVdPJ/N19ZZJHfMNTg9Wtvea1mXJ+jOIf9SzfHAwCEB0EGXcbJgeXk/z55ZdGNQzPVLTZGPr8hT2NA64qOnfZeAEDkIMgg6jWNplyV012pCU5dmZMqjy8gh92qBp9fkoJ7vkjSIzcM1FcG9AgePSCxGy8ARCqWXyOq1Xl9evK9HeqdHKsMd6wmD+ulxZuKtX53uWJjrMpJS1SCy65PDlQ2ew0AILxYfo0u69T+l97dYlVcVqsGn//fPS4WWSxSbi+3UhOcMmSod7dYHTxez4gLAJgMq5ZgaievOGraoO7PH+/X0q0lqmnw6ebhvfXuthLtPlqjD4vKZJF049BMuew2jctNCzbyrig8rClX9GOTOwAwGUZkYCplNQ368ZLtKqtpkHSiCbe0skFz3tuhspoGLdl6QNv2V8rrC2jb/got21aiLfuOq9bbqNKKekmnH9rIIY4AYF70yMBUfrxkuw5V1MvjC+jlKSMkSY+/W6iaep9cDpt8AUMX9XJrT1mtLkhP1MfFx+X3B5TgtGtY32RN5GgBADCFtl6/+RsdpnL/9TmavqhAI7NTtKLwsMblpqnB69P2kkoNyUxSjN2mBJddP//6ENV5fXLaTww6EmAAIDoxtQRT6Z7g0stTRqhvanxwyfTeY/UKBIwTIcZp15iBPYO9MxOHZmri0EzlF5aymR0ARCH+iQpTKKtp0C9XFun/XX2eNhcf11U53bVsW4lqGhrVLzVOOT0TdGFGorYfrNKqHUfksFuDG9g1+Pxa//kxeX0BjhgAgCjDiAwiXp3Xp2mLCnSool4Pvb1d+8vr9K2XNir/01ItWPOFvP6ArszpoQSXXZZ/v6f5BnZG8HEAQHSJ6BGZuXPn6p133tFnn32m2NhYXX755Zo3b54GDBgQ7tLQweq8Pi3bViLJkGTRqOwUrf/8mMYO6qmV/zqitESnPi4uV3qiS8frGoP7v7js9uCy6qaN7W4c2iv4OAAgukT0iMyaNWs0depUbdiwQStXrlRjY6PGjRun2tracJeGEDn1MMY6r0+LN+3T4+8W6sOiMm34vFwWSelul/p2j1Oj39DXLspQQNJ/jOyjlASnJgxJl9T6MmqWVwNA9DLV8uujR4+qZ8+eWrNmjb7yla+0+BqPxyOPxxO8X1VVpaysLJZfR6glWw+ovMarBJddhqStxcdV4/XJYkgJLruG9UkONuuWVjToYEW9Zlyfo//dcVgWWWTIUG2DX6kJTo4WAIAoEpVHFFRWVkqSUlJSWn3N3Llz5Xa7g7esrKzOKg9noamXxePz661N+1Ve69WRqgY5Y6z6Yd6AYHNudUOjio/Vasb1OfqwqEy1DX457TbdOLQXhzkCQBdmmhGZQCCgG2+8URUVFVq3bl2rr2NExpwWbyrWuqJjKqv2qHuCQ3abVdcM6Kkrc1I1bVGBEmPtirXbdM2AnhqXmxbcQ4bpIgCITlG3Id7UqVP16aefnjHESJLT6ZTT6eykqhAqYwemaeu+Cl2clSan3SqH3apxuWl6/N1CHa/1yucP6LrhvU9r5AUAdG2mmFq67777tHz5cn3wwQfq3bt3uMvBOWqpwfcX+Tu1s7RaG784pr99WqoxA3sqzmFXbqZbcQ6bJg7N1G2X9mEEBgDQTEQHGcMwdN9992nJkiX6+9//ruzs7HCXhHY6NbRIJw56bNqsrul+Tb1PkkVflNUqxmrR9EUFqvP69I3hvfWd0f1083ACLADgdBEdZKZOnao//elPWrhwoRITE1VaWqrS0lLV19eHuzS00bJtJfrfHUf0+LuFwTCTl5uuBJddDT6f6rw+5eWm6ysDeuhbI7P02l2XyhcwgmcpsXQaAHAmEd3sa7G0vB/rq6++qjvvvLNNn8Hp1+FT5/Xp8XcL9dmhamUlx2rsoPTg+Ugen7/VZdN1Xp9WFB7WlTmp+rCoTHm56QQZAOhiomL5tWEYLd7aGmIQXvmFpcp0xyo2xqpR56UGQ0x5jVcWWYLLpk+dfmoahfmwqKzZFBQAAKeK6CAD86rz+uTxBXSool5XD+gph92q/MJS5WYkadVnR3R5/9TglNGpPTNNmp+XBADA6QgyCIlTR1XyC0t1vNarel9Aq3ce0bEaj/638LC+/6ePVVHr1Qt/3x18z1U53VsMLPTHAAC+DEEG56ypF+Z/Cw/r3W0lkk6Mpuw9Vqst+46rsr5Rb285oIMV9Yp32FXX6FduL3dwJGZd0TECCwDgrHDlQLvVeX3KLywNNuHmF5aqsr5Rnx2q1iV9k7V40z55fX75/AENSEvU3rIaTRiSoaPVHuVmuuWwWzVxaKYkBXfoBQDgbET0qqVQYNVS6DUd9Ni04qispkFTF25VksuuitpGeXx+VdQ3alBmkrrHOzXj+hytKzrGkQIAgDaLuiMKEBnKahq04Ytj6pcSrw+LjqiqwSuH3arLs1O1fPsh9U6J1ef7ahTrsMllt2lw7xN/+AxFdV4GAIQJQQYtOnX6qMkvVxapvMarLcUVirFa9NGe4+qR4JAsUt/ucaqsa1Rmt1jZrRa57FbVNvj19Ps7Vd8YkNcXCJ5mDQBAKBBk0KKmRtx3t5X8eyzF0I1De+n+63P0dP5ODevTTf/7ryM6VNmgynqv7BaLAgHp5uG95bCf6CEfM7Cn1hUdU4zdoi3FFWH8bQAA0YoggxZdldNdz60s0gXpCfq4uEIWSS77ieXQo85L1b6yOh2v98oVY1VFXaPi4pyy260aPyRd3RNcwc+ZPKyX6rw+JbkcNPUCAEKO5ddo0YdFZcruHi+n3aaLerl1qLJBgzIStXjTPlU3+LRu91E1eP3y+ALqnhgrh92q9ESX1hUdO+2z2A8GANBRuLKgRXm56cGl0U++t0NJLrse/cun8voNZXaLVZ/UODX6DfXrHq/hfZMlSQ67lVEXAECnIsh0Qa018jYpq2nQL1cW6f7rcyRJA9IT5Wn0Kc5hk8cXkMtm0UVZ3XRxVrJi7BbdOJTRFgBAeLCPTBfUtA9Mgssuh92qvNx0SScafEf0Tdbdr21WzySn/H5DvZJdqvX4VVbjUfdEp7rFxujiPsk6XuvVh0VlGtUvRUdrPHrkhoGEGQBAyLCPDFqVl5uud7eVaMu+4+qXEq8VhYdlyFBpRYPu/mCz0hKdKjpcrdQEpw58Uadar0+GpDqvX2UOm87vmaC9x2o1KjtFG/eU66qc7lpReFiTh/UK968GAOhiaPbtguIcJ0ZiMt2x+nB3ma7MSVVebroOVtQrLzdNhqTvfaW/HDarZJEssijWblNZjVe1Xr+WbzskSUp3u/T8ty5WhjuW3hgAQFgwItOFnNwbc1VOd03bXKAhvZI0bVGB5t08RIN7J6nRZ2jixRkaOzBNFknvbS9Rj0ynYiySIYti7DbZrBb1Sz2xoql7gouRGABA2BBkuog6r0+PLytUrdcnry8gQ1JyfIzyC0vl9QU0881tuuqC7vpLQYnO6x6vguIKGRbJ6zNUVu2VM8ameIdNyfF2PXLDQA57BABEBIJMlGoafRnRN1kL1nyh7O7xKiypVElFg2wWi4b3S1GM1aqUeKeOVDUoLcmpd7cdks0ibdl3XFkp8coblKa9x+o0bmCaxuWm6bdrvtCM63OC+8IAABBuBJkolV9YqtLKBt392mZlpcRq99Ealdd6VVnfqJ2l1bIYhg5WefSVnO4qrWzQoF5JuiA9Uet2HVWsw65AIKDdR6s1bmCaUhOcykqJ18+/PiTcvxYAAM3Q7Bul8nLTdfB4va4b0EP7jtVpdHaKvD6/YmMsOnC8ThuKj+vTg5X688cH1Bg4sQI/Lcmlm0b0VkY3l6oafBqU4VZqgpMpJABAxCLIRKGmaaUZ1+eosr5Rrhib/vDPPbLbbHLF2JWVGq+GRr8kQ+W1Hm3cc0wffVEup92mG4f2UjdXjAZnupXg4mgBAEBk4woVReq8Pi3cuE9vbChW/7QE1TT4ZFikynqvPD5DhvzKSUtQWlKsBvRM0IHKBtV7/bJZpASXXeNy0xTnsOvRG3Np5gUAmAJBxsROPWogv7BUr28o1vEaj6o9PrldMWrw+jUgPUkF+45rRL8U3XtNf8356w5dPyhdP5mYqVU7jkiSJg7NDI680MwLADALgozJ1Hl9WratRJIhyaLjNV7NeW+HHrlhoPJy01Va0aBFH+1Tosuu3Ydr1NPtVMH+4/I0+rXrcLUWbtynWo9ff9t+SN0TnLrt0j7h/pUAADhrBBmTyS8s1frPj8kiaXT/VO0tr1VlfaNmL/1Ul2anKN5ll8Vq0cGKBjljrNpTXque8THaU3uiV2ZwL7csFik3083UEQDA9AgyEaylU6rzctPl9QXk9QXk8QXU4PXpXyVVSnDYtWlvuZLjYpQQY5M7xa6Sinolx8WotNqrC3okaFBGkm4e3lvfGd0vvL8YAAAhQpCJYPmFpSqv8Z52IKPH59eyghJ5GgMKGIZ8fkP7jtcpNsamsmqPUuKd6pnkUHpSrL4oq1aMzarEOLt+NnkwK5AAAFGFq1oEy8tND64eKqtp0C9XFmlAeoKWbi1RSUW9qhsalRznVGVDo9ISXTpSXa8Ym1V13kY1+mNkt1vUI8kln99Qn5Q4QgwAIOpwZYtATaHl/utzNC43TfmFpVpXdFSfHqzSut1lGpyRqLIaj6wWiw5X1Ss5PkZWi/S9q/rrzx8fUGq8UxOHZijBZdfl/VODRwsAABBtCDIR6Bf5O1V4sEpP5+/UqPNSVV7jlc8f0P7yWjX4DHkb/Sd2361vlDPGpnqvX7/+1mD9In+XDIvUPdGhPWW1euSGgYpz2DlaAAAQtdjZN8zqvD4t2XpAdV6f6rw+Ld5UrAavX3arRbmZbo3om6wV/ypVjcenhkZDAUMqq/GooTEgm82iGKtFGe44TVtcoJqGRlkl1TT41Cs5VisKD4f71wMAoEMxIhNmTQ29724r0ScHKlVR65XXH1C1x6fDVQ36+gvrTqxO8gVkkWSzSMnxMSoqrZbLYVP/Hglyxdh07YU9tG1/pW6/KEPjh6RrXdExllcDAKKexTAMI9xFdKSqqiq53W5VVlYqKSkp3OWcps7r04rCw2rw+XWkqkEfFh1VaaVHx+s8qvMEZLdJXv+J19otksUixTmsqm4IyGaTJg3tpZ9NHhxsCqahFwAQDdp6/eaqFyZNe8SM6Jus9V+UKadngnaWVmv34WrVNwbU4DNks0iGcWL+L85hkWRRZnKsahp8avQ3qn+PeD084UKOFAAAdFkEmU7Q0sZ2TVNKU9/YosKSKsW77IqxWFTd4Jfv32NkFotkt1rksFslizRldD99tKdcsTE2JbnsinPFMAIDAOjSaPY9Byc36rb23P7yWv3nHzZrX1md3t1WoiVbD6ispkHVDY3aUnxMn5ZUyW9IVfU+VdU3KmBIJ8ZeTozG2KxSfaNfvd0ubfjimAKGoQx3rPyGRZefl0pDLwCgS6NH5hws2XpA5TVepSY4g1M7J29c1+gztOqzI7JapF2Ha3Rp325qCAT0xZFaOawW7T5aK/8p335sjFXxMVYdr/cp1i4FZFW806Y+KfHBn+GwWzVmYM9gQy+jMgCAaEOPTCe4Kqe7nltZpEnDMoOP/eL9ndp+oFK7Sqv0jRFZmnfzEN356keyWaWtByp1rNojWaSAYZwWYk4wVO050d1b2yi5YiRDFk0e1uu0M5LoiwEAdHUEmXPwYVGZsrvHB0dG8gtLldktVss/KVHflHh5fQG9s+WgDh6vk88vyZCME/9H0ol5vYAkh1XyBk4srbZbLcpIjlVJZYOy3LHqn56oUdkpunl477D9ngAARCqCzDk4+SykZdtKtP7zY/r8SI28voCKjlZr8UfF+uxQjfwtvLcpxMTFWORpNGSV5LJblJrg0kW93Pr6sN7K6BbLqAsAAGdAkDlLZTUN+kX+TuX0TNDCjfv0109OnEZ9sKJejYETr9lZWiO73SK/7/Q5pKaRGF9AMv69zNrjN/Sdy/rqW6P6BAMSAABoHUHmLP0if6dWf3ZUf958QBbpxGojy4npoSY+Q/L7jODoy6m8ASk9MUaGxaJj1R5dlp2qBJedfWEAAGgjll+fpZyeiTpa7ZHfOBFYApL8xolwcjJDzUOM0yoluWyySkp02tQrOU59kmN1fo8E+QKGJg7NFAAAaBtTBJkXXnhB/fr1k8vl0qhRo/TRRx+FuyQ57Ba1d9260yZZbRZZJHWLt8sd59CEIRmaeHGmXE6beiXHdkSpAABErYgPMm+++aZmzpypxx57TFu2bNHQoUOVl5enI0eOhLmy9gWZeIdVNqtNDptVyXEOXdInRWmJTiW47JowJENxDrv6JMexwR0AAO0Q8UHm2Wef1fe+9z3dddddGjRokBYsWKC4uDi98sorLb7e4/Goqqqq2a0j1DQ0tul1LrtFmd1cMgwpYASUFOvQd686T9cPStOtl2Zp4tBMfVhUpqtyuutojYcGXwAA2iGig4zX69XHH3+ssWPHBh+zWq0aO3as1q9f3+J75s6dK7fbHbxlZWV1SG1/3X6oTa8zDEOxMTb1S41X9wSXhme5FWO3auLQTN12aR/FOezKy01XhjtWj9wwkF16AQBoh4gOMmVlZfL7/UpLaz5KkZaWptLS0hbfM2vWLFVWVgZv+/fv75Daymu9Z3w+3mGV025RTlqSJgxJ10VZbv3XmPN1WU4P1Tb4mk0hNa1SIsQAANA+UXfldDqdcjqdHf5zUuOdKqn0tPhcjFVKjnNq1HnJOr9Hog5W1OvRibmKc9hV5/WxRwwAACES0UGme/fustlsOny4eQPs4cOHlZ6eHqaqTvjKBd21veT/+m+SnDYlxdqVEu9UVnKsLumbohi7RU67TVOu6BccbWGPGAAAQieip5YcDoeGDx+uVatWBR8LBAJatWqVRo8eHcbKpM3Fx5vd75Mapytyeug/LuurS/qmaE9ZrRp9hpx2G1NGAAB0kIgOMpI0c+ZMvfzyy/rDH/6gHTt26N5771Vtba3uuuuusNb19C1D1atbrGIs0nUDemhAeqJq6n1q9AWUmuDUjOtzlJrgZAoJAIAOFPFDBbfddpuOHj2qRx99VKWlpbr44ov1/vvvn9YA3NmyUuL1X2P6a13RMSU4bBrUK0lbiivksFuDU0dMIQEA0LEshmG0d4NaU6mqqpLb7VZlZaWSkpJC+tl1Xp/mvLdDvbrFKiXBIafdpnG5aUwlAQBwjtp6/eaKew7iHHY9csPA4CokAgwAAJ2LK+85YhUSAADhE/HNvgAAAK0hyAAAANMiyAAAANMiyAAAANMiyAAAANMiyAAAANMiyAAAANMiyAAAANMiyAAAANMiyAAAANMiyAAAANMiyAAAANMiyAAAANOK+tOvDcOQJFVVVYW5EgAA0FZN1+2m63hroj7IVFdXS5KysrLCXAkAAGiv6upqud3uVp+3GF8WdUwuEAiopKREiYmJslgsIf3sqqoqZWVlaf/+/UpKSgrpZ3d1fLcdh++24/Dddhy+244Tqd+tYRiqrq5WZmamrNbWO2GifkTGarWqd+/eHfozkpKSIur/+dGE77bj8N12HL7bjsN323Ei8bs900hME5p9AQCAaRFkAACAaRFkzoHT6dRjjz0mp9MZ7lKiDt9tx+G77Th8tx2H77bjmP27jfpmXwAAEL0YkQEAAKZFkAEAAKZFkAEAAKZFkAEAAKZFkDkHL7zwgvr16yeXy6VRo0bpo48+CndJpjd37lxdeumlSkxMVM+ePTV58mTt3Lkz3GVFnaeeekoWi0UzZswIdylR4eDBg/r2t7+t1NRUxcbGasiQIdq8eXO4yzI9v9+v2bNnKzs7W7Gxserfv7+eeOKJLz17B6dbu3atJk6cqMzMTFksFi1durTZ84Zh6NFHH1VGRoZiY2M1duxYFRUVhafYdiLInKU333xTM2fO1GOPPaYtW7Zo6NChysvL05EjR8JdmqmtWbNGU6dO1YYNG7Ry5Uo1NjZq3Lhxqq2tDXdpUWPTpk367W9/q4suuijcpUSF48eP64orrlBMTIz+9re/6V//+peeeeYZJScnh7s005s3b55efPFF/eY3v9GOHTs0b948zZ8/X7/+9a/DXZrp1NbWaujQoXrhhRdafH7+/Pl6/vnntWDBAm3cuFHx8fHKy8tTQ0NDJ1d6FgyclZEjRxpTp04N3vf7/UZmZqYxd+7cMFYVfY4cOWJIMtasWRPuUqJCdXW1kZOTY6xcudK4+uqrjenTp4e7JNN76KGHjCuvvDLcZUSlG264wbj77rubPXbTTTcZt99+e5gqig6SjCVLlgTvBwIBIz093fjFL34RfKyiosJwOp3GokWLwlBh+zAicxa8Xq8+/vhjjR07NviY1WrV2LFjtX79+jBWFn0qKyslSSkpKWGuJDpMnTpVN9xwQ7M/uzg3y5Yt04gRI3TLLbeoZ8+eGjZsmF5++eVwlxUVLr/8cq1atUq7du2SJG3btk3r1q3T+PHjw1xZdNmzZ49KS0ub/b3gdrs1atQoU1zTov7QyI5QVlYmv9+vtLS0Zo+npaXps88+C1NV0ScQCGjGjBm64oorNHjw4HCXY3qLFy/Wli1btGnTpnCXElW++OILvfjii5o5c6YeeeQRbdq0SdOmTZPD4dCUKVPCXZ6pPfzww6qqqtKFF14om80mv9+vJ598Urfffnu4S4sqpaWlktTiNa3puUhGkEHEmjp1qj799FOtW7cu3KWY3v79+zV9+nStXLlSLpcr3OVElUAgoBEjRmjOnDmSpGHDhunTTz/VggULCDLn6K233tIbb7yhhQsXKjc3VwUFBZoxY4YyMzP5bhHE1NJZ6N69u2w2mw4fPtzs8cOHDys9PT1MVUWX++67T8uXL9cHH3yg3r17h7sc0/v444915MgRXXLJJbLb7bLb7VqzZo2ef/552e12+f3+cJdoWhkZGRo0aFCzxwYOHKh9+/aFqaLo8eCDD+rhhx/WN7/5TQ0ZMkTf+c53dP/992vu3LnhLi2qNF23zHpNI8icBYfDoeHDh2vVqlXBxwKBgFatWqXRo0eHsTLzMwxD9913n5YsWaK///3vys7ODndJUWHMmDHavn27CgoKgrcRI0bo9ttvV0FBgWw2W7hLNK0rrrjitC0Cdu3apb59+4apouhRV1cnq7X5ZcpmsykQCISpouiUnZ2t9PT0Zte0qqoqbdy40RTXNKaWztLMmTM1ZcoUjRgxQiNHjtRzzz2n2tpa3XXXXeEuzdSmTp2qhQsX6i9/+YsSExOD87Nut1uxsbFhrs68EhMTT+szio+PV2pqKv1H5+j+++/X5Zdfrjlz5ujWW2/VRx99pJdeekkvvfRSuEszvYkTJ+rJJ59Unz59lJubq61bt+rZZ5/V3XffHe7STKempka7d+8O3t+zZ48KCgqUkpKiPn36aMaMGfr5z3+unJwcZWdna/bs2crMzNTkyZPDV3RbhXvZlJn9+te/Nvr06WM4HA5j5MiRxoYNG8JdkulJavH26quvhru0qMPy69B59913jcGDBxtOp9O48MILjZdeeincJUWFqqoqY/r06UafPn0Ml8tlnHfeecaPf/xjw+PxhLs00/nggw9a/Lt1ypQphmGcWII9e/ZsIy0tzXA6ncaYMWOMnTt3hrfoNrIYBlskAgAAc6JHBgAAmBZBBgAAmBZBBgAAmBZBBgAAmBZBBgAAmBZBBgAAmBZBBgAAmBZBBgAAmBZBBkDEufPOO5ttjX7NNddoxowZ5/SZofgMAJGHIAOgze68805ZLBZZLBY5HA6df/75+tnPfiafz9ehP/edd97RE0880abXrl69WhaLRRUVFWf9GQDMg0MjAbTLV7/6Vb366qvyeDz661//qqlTpyomJkazZs1q9jqv1yuHwxGSn5mSkhIRnwEg8jAiA6BdnE6n0tPT1bdvX917770aO3asli1bFpwOevLJJ5WZmakBAwZIkvbv369bb71V3bp1U0pKiiZNmqS9e/cGP8/v92vmzJnq1q2bUlNT9aMf/UinHgF36rSQx+PRQw89pKysLDmdTp1//vn6/e9/r7179+raa6+VJCUnJ8tisejOO+9s8TOOHz+uO+64Q8nJyYqLi9P48eNVVFQUfP61115Tt27dlJ+fr4EDByohIUFf/epXdejQoeBrVq9erZEjRyo+Pl7dunXTFVdcoeLi4hB90wDagiAD4JzExsbK6/VKklatWqWdO3dq5cqVWr58uRobG5WXl6fExER9+OGH+sc//hEMBE3veeaZZ/Taa6/plVde0bp161ReXq4lS5ac8WfecccdWrRokZ5//nnt2LFDv/3tb5WQkKCsrCy9/fbbkqSdO3fq0KFD+tWvftXiZ9x5553avHmzli1bpvXr18swDE2YMEGNjY3B19TV1enpp5/W66+/rrVr12rfvn364Q9/KEny+XyaPHmyrr76an3yySdav3697rnnHlkslnP+TgG0HVNLAM6KYRhatWqV8vPz9V//9V86evSo4uPj9bvf/S44pfSnP/1JgUBAv/vd74IX+FdffVXdunXT6tWrNW7cOD333HOaNWuWbrrpJknSggULlJ+f3+rP3bVrl9566y2tXLlSY8eOlSSdd955weebppB69uypbt26tfgZRUVFWrZsmf7xj3/o8ssvlyS98cYbysrK0tKlS3XLLbdIkhobG7VgwQL1799fknTffffpZz/7mSSpqqpKlZWV+trXvhZ8fuDAge3/IgGcE0ZkALTL8uXLlZCQIJfLpfHjx+u2227TT3/6U0nSkCFDmvXFbNu2Tbt371ZiYqISEhKUkJCglJQUNTQ06PPPP1dlZaUOHTqkUaNGBd9jt9s1YsSIVn9+QUGBbDabrr766rP+HXbs2CG73d7s56ampmrAgAHasWNH8LG4uLhgSJGkjIwMHTlyRNKJwHTnnXcqLy9PEydO1K9+9atm004AOgcjMgDa5dprr9WLL74oh8OhzMxM2e3/99dIfHx8s9fW1NRo+PDheuONN077nB49epzVz4+NjT2r952NmJiYZvctFkuz/p1XX31V06ZN0/vvv68333xTP/nJT7Ry5UpddtllnVYj0NUxIgOgXeLj43X++eerT58+zUJMSy655BIVFRWpZ8+eOv/885vd3G633G63MjIytHHjxuB7fD6fPv7441Y/c8iQIQoEAlqzZk2LzzeNCPn9/lY/Y+DAgfL5fM1+7rFjx7Rz504NGjTojL/TqYYNG6ZZs2bpn//8pwYPHqyFCxe26/0Azg1BBkCHuf3229W9e3dNmjRJH374ofbs2aPVq1dr2rRpOnDggCRp+vTpeuqpp7R06VJ99tln+sEPfnDaHjAn69evn6ZMmaK7775bS5cuDX7mW2+9JUnq27evLBaLli9frqNHj6qmpua0z8jJydGkSZP0ve99T+vWrdO2bdv07W9/W7169dKkSZPa9Lvt2bNHs2bN0vr161VcXKwVK1aoqKiIPhmgkxFkAHSYuLg4rV27Vn369NFNN92kgQMH6rvf/a4aGhqUlJQkSXrggQf0ne98R1OmTNHo0aOVmJior3/962f83BdffFHf+MY39IMf/EAXXnihvve976m2tlaS1KtXLz3++ON6+OGHlZaWpvvuu6/Fz3j11Vc1fPhwfe1rX9Po0aNlGIb++te/njaddKbf7bPPPtPNN9+sCy64QPfcc4+mTp2q73//++34hgCcK4tx6oYNAAAAJsGIDAAAMC2CDAAAMC2CDAAAMC2CDAAAMC2CDAAAMC2CDAAAMC2CDAAAMC2CDAAAMC2CDAAAMC2CDAAAMC2CDAAAMK3/D+E3Nf9nn80iAAAAAElFTkSuQmCC\n"
},
"metadata": {}
}
],
"source": [
"# plotting actual values vs. predictions\n",
"ypred = model5(torch.from_numpy(np.float32(x_test)).to(device)).cpu().detach().numpy()\n",
"plt.scatter(ypred,y_non_test,s=.1)\n",
"plt.xlabel(\"Predictions\");\n",
"plt.ylabel(\"Actual values\");"
]
},
{
"cell_type": "markdown",
"source": [
"The bells and whistles buy us only a little bit of extra performance here, but this is a very simple case. In more complex, multivariate situations, these features can have a major effect on model performance."
],
"metadata": {
"id": "AJEp4isLZdH1"
}
},
{
"cell_type": "markdown",
"source": [
"## Next steps\n",
"This tutorial has provided a basic example of how one can create one's own artificial neural network using pytorch. However, there's a lot more that you can do with deep nets beyond what's been shown here. This final section will point you to some directions for further learning that you may be interested in."
],
"metadata": {
"id": "gHElnK1saJmr"
}
},
{
"cell_type": "markdown",
"source": [
"### Other layer connectivity patterns\n",
"The layers in the networks you've seen here are all \"densely\" connected. This means that every unit in one layer is connected to every unit in another layer. Most of the connections are also sequential, with the exception of the skip layer connection in the last model. However, there are a wide variety of other connectivity patterns that can perform better for particular applications. Several of these patterns are illustrated in the figure below.\n",
"\n",
"![](https://mysocialbrain.org/misc/data/ann_tutorial/Fig1_DS_hires_bottom.png)\n",
"\n",
"* The bottleneck (red) in the autoencoder illustration is a layer that is narrower (i.e., has fewer units) than the ones before or after it. This forces this layer to learn a compressed representation of the data - a bit like PCA, but nonlinear, not (necessarily) orthogonal, and with potentially different loss functions than maximizing variance explained.\n",
"\n",
"* Convolutional networks are ubiquitous due to their effectiveness in dealing with image-like data (e.g., photos, video, fMRI, or even spectrograms of audio or electrophysiology). The connectivity pattern - and receptive fields that emerge - loosely approximate the human visual system.\n",
"\n",
"* Recurrent connectivity carries a unit's activity forward from one time point to another. Long short-term memory (LSTM) networks are probably the best known example of this type. These models are most often used for time series and other sequential data.\n",
"\n",
"* Perhaps the most important type of unit/connectivity (not pictured due to its complexity) is the attention mechanism of transformer architectures. Transformers are beyond the scope of this introduction, but they support many of the most influential ANNs as of time of writing (e.g., all the major large language models like GPT)."
],
"metadata": {
"id": "1I8406rj4Y_u"
}
},
{
"cell_type": "markdown",
"source": [
"### Applications\n",
"One way to organize your learning is to think about how you want to apply deep nets in your own research. The figure below illustrates some of the main uses cases that you might be interested in. These include (A) training statistical biomarkers to predict phenotypes from brain data, (B) using computer vision/audition models to annotate behavior in stimuli/recordings of participants, and (C) training models on the same tasks as participants, so that they can be used as cognitive models. Once you know which of these applications you want to use in your research, it can help you figure out where to go next. ![](https://mysocialbrain.org/misc/data/ann_tutorial/Fig2_DS.png)\n"
],
"metadata": {
"id": "EgeJxcpq2trZ"
}
},
{
"cell_type": "markdown",
"source": [
"### Learning resources\n",
"\n",
"Tutorial material:\n",
"* [Pytorch's official tutorials](https://pytorch.org/tutorials/)\n",
"\n",
"* [Tensorflow's official tutorials](https://www.tensorflow.org/tutorials)\n",
"\n",
"Websites:\n",
"\n",
"* If you're looking to find the best (pre-trained) model for a specific application, [Paper's with Code](https://paperswithcode.com/) is in an extremely useful source. It tracks the state of the art with respect to a huge number of machine learning benchmarks, with links their papers and github repos.\n",
"\n",
"* [Hugging Face](https://huggingface.co/) is a model repository where an increasing number of popular pretrained models are uploaded. It provides a consistent API for installing, using, and documenting models, most of which are programmed in PyTorch and/or Tensorflow.\n",
"\n",
"* [DeepMind Blog](https://www.deepmind.com/blog): Google DeepMind maintains a blog where they write about their papers in (relatively) accessible ways.\n",
"\n",
"Videos:\n",
"\n",
"* Neuromatch's video playlists on [deep learning](https://www.youtube.com/watch?v=IZvcy0Myb3M&list=PLkBQOLLbi18PZ2uw0p7G4EkzjzqP8l0Eg), and [autoencoders](https://www.youtube.com/watch?v=VwSnDJZekQ4&list=PLkBQOLLbi18Ojl1CV8W00JZ0C0hivBxw1), \n",
"\n",
"* [NYU deep learning course](https://www.youtube.com/watch?v=0bMe_vCZo30&list=PLLHTzKZzVU9eaEyErdV26ikyolxOsz6mq).\n",
"\n",
"* Berkeley [Deep Reinforcement Learning Bootcamp](https://sites.google.com/view/deep-rl-bootcamp/lectures)\n",
"\n",
"* [Two minute papers](https://www.youtube.com/@TwoMinutePapers) Short, easily digested videos on recent modeling\n",
"\n",
"Many of my favorite scientific papers on deep learning are cited within these two papers:\n",
"\n",
"* Thornton, M.A., & Sievers, B. (2023). Deep social neuroscience: The promise and peril of using artificial neural networks to study the social brain. PsyArXiv. [Preprint](https://psyarxiv.com/fr4cb)\n",
"\n",
"* Lin, C., Bulls, L. S., Tepfer, L. J., Vyas, A., Thornton, M. A., (2023).\n",
"Advancing naturalistic affective science with deep learning. Affective Science. [Preprint](https://psyarxiv.com/j5q9h/)"
],
"metadata": {
"id": "6A7FDhq9lLHw"
}
}
],
"metadata": {
"accelerator": "GPU",
"colab": {
"toc_visible": true,
"provenance": []
},
"kernelspec": {
"display_name": "Python 3",
"name": "python3"
},
"language_info": {
"name": "python"
}
},
"nbformat": 4,
"nbformat_minor": 0
}