{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## API tutorial"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Expression Building\n",
    "\n",
    "(This tutorial is tested on DyNet 2.0.4+ and Python 2.7.)\n",
    "\n",
    "If you find any issues while running, please try to restart the kernel and run it again. :)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from __future__ import print_function\n",
    "from __future__ import division\n",
    "from __future__ import absolute_import\n",
    "\n",
    "# Note: please import dynet_config before import dynet\n",
    "import dynet_config\n",
    "# set random seed to have the same result each time\n",
    "dynet_config.set(random_seed=0)\n",
    "import dynet as dy\n",
    "\n",
    "import numpy as np\n",
    "\n",
    "## ==== Create a new computation graph\n",
    "# (There is a single global computation graph that is used at any point.\n",
    "# dy.renew_cg() clears the current one and starts a new one)\n",
    "dy.renew_cg();"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Create Expressions\n",
    "Expressions are used as an interface to the various functions that can be used to build DyNet computation graphs. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# create a scalar expression.\n",
    "value = 5.0\n",
    "x = dy.scalarInput(value)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# create a vector expression.\n",
    "dimension = 3\n",
    "v = dy.vecInput(dimension)\n",
    "v.set([1,2,3])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# create a matrix expression from a list\n",
    "mat1 = dy.inputTensor([[1,2], [3,4]]) # Row major\n",
    "\n",
    "# or, using a numpy array\n",
    "mat2 = dy.inputTensor(np.array([[1,2], [3,4]]))\n",
    "\n",
    "mat3 = dy.inputTensor(np.zeros((2,3)))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# create a vector/matrix expression of special values\n",
    "# Different from other toolkits such as TensorFlow or PyTorch.\n",
    "# DyNet has a special \"batch\" dimension, see here for more\n",
    "# details: http://dynet.readthedocs.io/en/latest/minibatch.html\n",
    "\n",
    "# zeros\n",
    "dim = 5\n",
    "batch_size = 3\n",
    "e = dy.zeros(dim, batch_size=batch_size)\n",
    "print('zeors of dim {} and batch_size {}:\\n{}'.format(dim, batch_size, e.npvalue()))\n",
    "\n",
    "# ones\n",
    "e = dy.ones(dim, batch_size=batch_size)\n",
    "print('ones of dim {} and batch_size {}:\\n{}'.format(dim, batch_size, e.npvalue()))\n",
    "\n",
    "# constant\n",
    "val = 2\n",
    "e = dy.constant(dim, val, batch_size=batch_size)\n",
    "print('constant {} of dim {} and batch_size {}:\\n{}'.format(val, dim, batch_size, e.npvalue()))\n",
    "\n",
    "# random_normal\n",
    "mean = 0\n",
    "stddev = 1.0\n",
    "e = dy.random_normal(dim, mean=mean, stddev=stddev, batch_size=batch_size)\n",
    "print('A {} dim random_normal of mean {} and stddev {} and batch_size {}:\\n{}'.format(dim, mean, stddev, batch_size, e.npvalue()))\n",
    "\n",
    "# random_bernoulli\n",
    "p = 0.3 # The p in Bernoulli distribution\n",
    "scale = 2.0 # Scaling factor to apply to the sampled tensor (default: (1.0))\n",
    "e = dy.random_bernoulli(dim, p=p, scale=scale, batch_size=batch_size)\n",
    "print('A {} dim random bernoulli distribution of p={} and scale={}, batch_size={}:\\n{}'.format(dim, p, scale, batch_size, e.npvalue()))\n",
    "\n",
    "# random_uniform\n",
    "left = -1\n",
    "right = 1\n",
    "e = dy.random_uniform(dim, left=left, right=right, batch_size=batch_size)\n",
    "print('A {} dim random uniform distribution of left={} and right={}, batch_size={}:\\n{}'.format(dim, left, right, batch_size, e.npvalue()))\n",
    "\n",
    "# random_gumbel\n",
    "# Create a vector distributed according to a Gumbel distribution with the specified parameters. \n",
    "# (Currently only the defaults of mu=0.0 and beta=1.0 supported.\n",
    "mu = 0.0\n",
    "beta = 1.0\n",
    "e = dy.random_gumbel(dim, mu=mu, beta=beta, batch_size=batch_size)\n",
    "print('A {} dim random gumbel distribution of mu={} and beta={}, batch_size={}:\\n{}'.format(dim, mu, beta, batch_size, e.npvalue()))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "## ==== Calculate the value of an expression.\n",
    "# This will run the forward step of the neural network.\n",
    "print(mat1.value())   \n",
    "print(mat1.npvalue())    # as numpy array\n",
    "print(v.vec_value())     # as vector, if vector\n",
    "print(x.scalar_value())  # as scalar, if scalar\n",
    "print(x.value())         # choose the correct one"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Create Parameters\n",
    "Parameters are things need to be trained. In contrast to a system like Torch where computational modules may have their own parameters, in DyNet parameters are just parameters."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Parameters are things we tune during training.\n",
    "# Usually a matrix or a vector.\n",
    "\n",
    "# First we create a parameter collection and add the parameters to it.\n",
    "m = dy.ParameterCollection() \n",
    "W = m.add_parameters((8,8)) # an 8x8 matrix, return an expr\n",
    "b = m.add_parameters(8) # an 8x1 vector, return as expr"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "It should be noticed that in DyNet 2.0.4+, the dy.parameters() is depecated so explicitly adding parameters to the computation graph is no longer necessary. Any used parameter will be added automatically."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# There are several ways to initial parameters\n",
    "# Specifiying parameter initialization\n",
    "scale, mean, stddev = 1, 0, 1\n",
    "\n",
    "# Creates 3x5 matrix filled with 0 (or any other float)\n",
    "p1 = m.add_parameters((3,5), init=0)\n",
    "# Creates 3x5 matrix initialized with U([-scale, scale])\n",
    "p2 = m.add_parameters((3,5), init='uniform', scale=scale)\n",
    "# Creates 3x5 matrix initialized with N(mean, stddev)\n",
    "p3 = m.add_parameters((3,5), init='normal', mean=mean, std=stddev)\n",
    "# Creates 5x5 identity matrix\n",
    "p4 = m.add_parameters((5,5), init='identity')\n",
    "# Creates 3x5 matrix with glorot init\n",
    "p5 = m.add_parameters((3,5), init='glorot')\n",
    "p6 = m.add_parameters((3,5)) # By default, the init = 'glorot'\n",
    "# Creates 3x5 matrix with he init\n",
    "p7 = m.add_parameters((3,5), init='he')\n",
    "# Creates 3x5 matrix from a numpy array (size is inferred)\n",
    "p8 = m.add_parameters((3,5), np.ones((3,5)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Create LookupParameters\n",
    "LookupParameters represents a table of parameters. They are used to embed a set of discrete objects (e.g. word embeddings). They can be sparsely updated."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "## ===== Lookup parameters\n",
    "# Similar to parameters, but are representing a \"lookup table\" that maps numbers to vectors.\n",
    "# These are often used for things like word embeddings.\n",
    "# For example, this will have VOCAB_SIZE rows, each of DIM dimensions.\n",
    "VOCAB_SIZE = 100\n",
    "DIM = 10\n",
    "lp = m.add_lookup_parameters((VOCAB_SIZE, DIM))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Create expressions from lookup parameters.\n",
    "e5  = dy.lookup(lp, 5)   # create an Expression from row 5.\n",
    "e5  = lp[5]              # same\n",
    "e5c = dy.lookup(lp, 5, update=False)  # as before, but don't update when optimizing.\n",
    "\n",
    "e45  = dy.lookup_batch(lp, [4, 5])   # create a batched Expression from rows 4 and 5.\n",
    "e45  = lp.batch([4, 5])\n",
    "print('e45 dim:', e45.dim())\n",
    "\n",
    "e0_9 = dy.lookup_batch(lp, range(10))  # create a batched Expression from rows 0 to 9\n",
    "e0_9 = lp.batch(range(10))\n",
    "print('e0_9 dim:', e0_9.dim())\n",
    "\n",
    "e5.set(10)  # now the e5 expression contains row 10\n",
    "print('e5 dim after applying set method', e5.dim())\n",
    "print(e5.value())\n",
    "\n",
    "# We can check if it is actually containing row 10\n",
    "e10 = lp[10]\n",
    "print(e5.value() == e10.value())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Similar to Parameters, we have several ways to\n",
    "# initialize LookupParameters.\n",
    "scale, mean, stddev = 1, 0, 1\n",
    "\n",
    "# Creates 3x5 matrix filled with 0 (or any other float)\n",
    "lp1 = m.add_lookup_parameters((3,5), init=0)\n",
    "# Creates 3x5 matrix initialized with U([-scale, scale])\n",
    "lp2 = m.add_lookup_parameters((3,5), init='uniform', scale=scale)\n",
    "# Creates 3x5 matrix initialized with N(mean, stddev)\n",
    "lp3 = m.add_lookup_parameters((3,5), init='normal', mean=mean, std=stddev)\n",
    "# Creates 5x5 identity matrix\n",
    "lp4 = m.add_lookup_parameters((5,5), init='identity')\n",
    "# Creates 3x5 matrix with glorot init\n",
    "lp5 = m.add_lookup_parameters((3,5), init='glorot')\n",
    "lp6 = m.add_parameters((3,5)) # By default, the init = 'glorot'\n",
    "# Creates 3x5 matrix with he init\n",
    "lp7 = m.add_lookup_parameters((3,5), init='he')\n",
    "# Creates 3x5 matrix from a numpy array (size is inferred)\n",
    "lp8 = m.add_lookup_parameters((3,5), np.ones((3,5)))\n",
    "# Creates 3x5 matrix from a numpy array (size is inferred)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### More Expression Manipulation\n",
    "DyNet provides hundreds of operations on Expressions. The user can manipulate Expressions, or build complex Expression easily."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Fist we create some vector Expressions.\n",
    "e1 = dy.vecInput(4)\n",
    "e1.set([1, 2, 3, 4])\n",
    "\n",
    "e2 = dy.vecInput(4)\n",
    "e2.set([5, 6, 7, 8])\n",
    "\n",
    "# Concatenate list of expressions to a single batched expression.\n",
    "# All input expressions must have the same shape.\n",
    "e_batch = dy.concatenate_to_batch([e1, e2])\n",
    "\n",
    "mat1 = dy.inputTensor(np.array([[1, 2], [3, 4], [5, 6], [7, 8]]))  # A 4x2 matrix\n",
    "mat2 = dy.inputTensor(np.array([[1, 0], [0, 1]]))  # A 2x2 matrix"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Basic Math Operations\n",
    "\n",
    "# Add\n",
    "e = e1 + e2  # Element-wise addition\n",
    "\n",
    "# Minus\n",
    "e = e2 - e1 # Element-wise minus\n",
    "# Negative\n",
    "e = -e1  # Should be [-1.0, -2.0, -3.0, -4.0]\n",
    "\n",
    "# Multiply\n",
    "e = e1 * dy.transpose(e1)  #It's Matrix multiplication (like e1.dot(e2) in numpy)\n",
    "\n",
    "mat = mat1 * mat2\n",
    "\n",
    "# Dot product\n",
    "e = dy.dot_product(e1, e2)  # dot product = sum(component-wise multiply)\n",
    "\n",
    "# Component-wise multiply\n",
    "e = dy.cmult(e1, e2)\n",
    "\n",
    "# Component-wise division\n",
    "e = dy.cdiv(e1, e2)\n",
    "\n",
    "# Column-wise addition\n",
    "# colwise_add(x, y)\n",
    "#  x:  An MxN matrix\n",
    "#  y:  A length M vector\n",
    "mat = dy.colwise_add(mat1, e1)  # column-wise addition\n",
    "\n",
    "# Useful math operations\n",
    "# abs()\n",
    "e = dy.abs(e1)\n",
    "\n",
    "# cube()\n",
    "# Elementwise cubic\n",
    "e = dy.cube(e1)\n",
    "\n",
    "# exp()\n",
    "e = dy.exp(e1)\n",
    "\n",
    "# pow()\n",
    "# For each element in e1, calculate e1^{y}\n",
    "e = dy.pow(e1, dy.inputTensor([2]))\n",
    "e_ = dy.square(e1)\n",
    "assert e.value() == e_.value()\n",
    "\n",
    "# bmin()\n",
    "# Calculate an output where the ith element is min(x_i, y_i)\n",
    "e = dy.bmin(e1, e2)\n",
    "assert e.value() == e1.value()\n",
    "\n",
    "# bmax()\n",
    "# Calculate an output where the ith element is max(x_i, y_i)\n",
    "e = dy.bmax(e1, e2)\n",
    "assert e.value() == e2.value()\n",
    "\n",
    "# sin()\n",
    "e = dy.sin(e1)\n",
    "\n",
    "# cos()\n",
    "e = dy.cos(e1)\n",
    "\n",
    "# tan()\n",
    "e = dy.tan(e1)\n",
    "\n",
    "# asin()\n",
    "e = dy.asin(e1)\n",
    "\n",
    "# acos()\n",
    "e = dy.acos(e1)\n",
    "\n",
    "# atan()\n",
    "e = dy.atan(e1)\n",
    "\n",
    "# sinh()\n",
    "e = dy.sinh(e1)\n",
    "\n",
    "# cosh()\n",
    "e = dy.cosh(e1)\n",
    "\n",
    "# tanh()\n",
    "e = dy.tanh(e1)\n",
    "\n",
    "# asinh()\n",
    "e = dy.asinh(e1)\n",
    "\n",
    "# acosh()\n",
    "e = dy.acosh(e1)\n",
    "\n",
    "# atanh\n",
    "e = dy.atanh(e1)\n",
    "\n",
    "# square()\n",
    "e = dy.square(e1)\n",
    "\n",
    "# sqrt()\n",
    "e = dy.sqrt(e1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Matrix manipulation\n",
    "\n",
    "# Reshape\n",
    "new_dimension = (2, 2)\n",
    "e = dy.reshape(e1, new_dimension)  # Col major\n",
    "print('reshape a vector:\\n', e.value())\n",
    "\n",
    "# Transpose\n",
    "e = dy.transpose(e1)\n",
    "print('e1 dimension:', e1.dim())\n",
    "print('e1 transpose dimension', e.dim())\n",
    "\n",
    "# inverse\n",
    "# Not implemented on GPU yet.\n",
    "e = dy.inverse(dy.inputTensor([[1, 3], [3, 1]]))\n",
    "print('Inverse a matrix:\\n', e.npvalue())\n",
    "\n",
    "# logdet\n",
    "# Not implemented on GPU yet\n",
    "e = dy.logdet(dy.inputTensor([[1, 0], [0, 2]]))\n",
    "print('logdet diag(1,2) is log(2):\\n', e.npvalue())\n",
    "\n",
    "# trace_of_product\n",
    "# Not implemented on GPU yet\n",
    "diag_12 = dy.inputTensor([[1, 0], [0, 2]])\n",
    "e = dy.trace_of_product(diag_12, diag_12)\n",
    "# or on matrix\n",
    "e = dy.trace_of_product(mat1, mat1)\n",
    "\n",
    "# circ_conv\n",
    "sig_1 = dy.inputTensor([1,2,1,0])\n",
    "sig_2 = dy.inputTensor([0,1,1,1])\n",
    "e = dy.circ_conv(sig_1, sig_2)\n",
    "\n",
    "# circ_corr\n",
    "e = dy.circ_corr(sig_1, sig_2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Other Per-element unary functions.\n",
    "\n",
    "# erf()\n",
    "# Elementwise calculation of the Gaussian error function\n",
    "e = dy.erf(e1)\n",
    "\n",
    "# log()\n",
    "e = dy.log(e1)\n",
    "\n",
    "# log_sigmoid()\n",
    "e = dy.log_sigmoid(e1)\n",
    "\n",
    "# lgamma()\n",
    "# Definition of gamma function ca be found here\n",
    "# https://en.wikipedia.org/wiki/Gamma_function\n",
    "e = dy.lgamma(e1)\n",
    "e_ = dy.log(dy.inputTensor([1, 1, 2, 6]))\n",
    "assert e.value() == e_.value()\n",
    "\n",
    "# sigmoid()\n",
    "e = dy.logistic(e1)   # Sigmoid(x)\n",
    "\n",
    "# rectify()\n",
    "# Rectifier (or ReLU, Rectified Linear Unit)\n",
    "e = dy.rectify(e1)    # Relu (= max(x,0))\n",
    "\n",
    "# elu()\n",
    "# Exponential Linear Unit (ELU)\n",
    "# Definition can be found here:\n",
    "# https://en.wikipedia.org/wiki/Rectifier_(neural_networks)#ELUs\n",
    "e = dy.elu(e1)\n",
    "\n",
    "# selu()\n",
    "# Scaled Exponential Linear Unit (SELU)\n",
    "# Definition can be found here:\n",
    "# https://arxiv.org/abs/1706.02515\n",
    "e = dy.selu(e1)\n",
    "\n",
    "# silu()\n",
    "# Sigmoid Linear Unit / sigmoid-weighted linear unit\n",
    "# SILU / SiL / Swish\n",
    "# Definition can be found here:\n",
    "# https://openreview.net/pdf?id=Bk0MRI5lg\n",
    "e = dy.silu(e1)\n",
    "\n",
    "# sparsemax()\n",
    "# **Note:** This function is not yet implemented on GPU.\n",
    "# Similar to softmax, but induces sparse solutions where \n",
    "# most of the vector elements are zero.\n",
    "e = dy.sparsemax(e1)\n",
    "\n",
    "# softsign()\n",
    "e = dy.softsign(e1)    # x/(1+|x|)\n",
    "\n",
    "# softmax()\n",
    "e = dy.softmax(e1)\n",
    "print('softmax result:', e.value())\n",
    "\n",
    "# log_softmax\n",
    "# logsoftmax = logits - log(reduce_sum(exp(logits), dim))\n",
    "# restrict is a set of indices. if not empty, only entries \n",
    "# in restrict are part of softmax computation, others get -inf.\n",
    "e_log_softmax = dy.log_softmax(e1)\n",
    "e_log_softmax = dy.log_softmax(e1, restrict=[0,1,2])\n",
    "print('log_softmax result', e_log_softmax.value())\n",
    "\n",
    "# constrained_softmax()\n",
    "# **Note:** This function is not yet implemented on GPU.\n",
    "# similar to softmax, but defines upper bounds for the resulting probabilities. \n",
    "e = dy.constrained_softmax(e1, dy.inputTensor([0.01, 0.05, 0.10, 0.55]))\n",
    "print('constrained_softmax result', e.value())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Picking values from vector expressions\n",
    "\n",
    "k, v = 1, 3\n",
    "# Pick one element from a vector or matrix\n",
    "# similar to python's e1[k] for list.\n",
    "# k can be negative, which has exactly the same behavior\n",
    "# as it is in python\n",
    "e = dy.pick(e1, k)\n",
    "print('The {} element of vector is {}'.format(k+1, e.value())) # index starts from 0\n",
    "# which is also equivalent to:\n",
    "e = e1[k]\n",
    "# k can be negative. -1 means the last element\n",
    "e = e1[-1]\n",
    "print(e.value())\n",
    "\n",
    "mat = dy.pick(mat1, k)\n",
    "print('The {} element of matrix mat1 is {}'.format(k+1, mat.value()))\n",
    "# which is equivalent to:\n",
    "mat = mat1[k]\n",
    "\n",
    "# Pick several elements from a vector or matrix\n",
    "# similar to python's e1[k:v] for lists. \n",
    "# e1 is an Expression, k, v are integers.\n",
    "# Important: v should not exceed the e1's dimension.\n",
    "e = dy.pickrange(e1, k, v)\n",
    "print('Pick range[k, v) from a vector', e.value())\n",
    "# which is also equivalent to:\n",
    "e = e1[k:v]\n",
    "e = e1[:v]  # similar to python, you can neglect k\n",
    "e = e1[:]   # or even both k and v\n",
    "# ERROR: Don't try this\n",
    "# e = e1[0:10], the v value should not exceed the dimension.\n",
    "\n",
    "mat = dy.pickrange(mat1, k, v)\n",
    "print('Pick range[k, v) from a matrix:\\n', mat.value())\n",
    "\n",
    "# pickneglogsoftmax\n",
    "# which is equivalent to: dy.pick(-dy.log(dy.softmax(e1)), k)\n",
    "e = dy.pickneglogsoftmax(e1, k)\n",
    "e_ = dy.pick(-dy.log(dy.softmax(e1)), k)\n",
    "print('{0:.6f} and {0:6f}'.format(e.value(), e_.value()))\n",
    "\n",
    "# pickneglogsoftmax_batch\n",
    "# similar to pickneglogsoftmax, this is negative softmax log likelihood on a batch\n",
    "# The difference is, a list of intergers is required for True classes.\n",
    "e = dy.pickneglogsoftmax_batch(e1, [k])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Selecting vectors from matrix Expressions\n",
    "\n",
    "# select_rows\n",
    "# works similar to pickrange\n",
    "e = dy.select_rows(mat1, [0,1])\n",
    "e_ = mat1[:2]\n",
    "assert np.all(e.value() == e_.value())\n",
    "\n",
    "# select_cols\n",
    "e = dy.select_cols(mat1, [0])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Expressions concatenation & other useful manipuulations\n",
    "\n",
    "# This performs an elementwise sum over all the expressions included.\n",
    "# All expressions should have the same dimension.\n",
    "e = dy.esum([e1, e2])\n",
    "# which is equivalent to:\n",
    "e_ = e1 + e2\n",
    "assert e.value() == e_.value()\n",
    "\n",
    "# This performs an elementwise average over all the expressions included.\n",
    "# All expressions should have the same dimension.\n",
    "e = dy.average([e1, e2])\n",
    "# which is equivalent to:\n",
    "e_ = (e1 + e2)/2\n",
    "assert e.value() == e_.value()\n",
    "\n",
    "# Concate vectors/matrix column-wise\n",
    "# All expressions should have the same dimension.\n",
    "# e1, e2,.. are column vectors. return a matrix. (sim to np.hstack([e1,e2,...])\n",
    "e = dy.concatenate_cols([e1, e2])\n",
    "print('vector and vector concatenate_cols:\\n', e.value())\n",
    "\n",
    "mat = dy.concatenate_cols([mat1, e2])\n",
    "print('mattix and vector concatenate_cols:\\n', mat.value())\n",
    "\n",
    "# Concate vectors/matrix\n",
    "# All expressions should have the same dimension.\n",
    "# e1, e2,.. are column vectors. return a matrix. (sim to np.hstack([e1,e2,...])\n",
    "e = dy.concatenate([e1, e2])\n",
    "print('vector concatenate:', e.value())\n",
    "\n",
    "mat = dy.concatenate([mat2, mat2])\n",
    "print('matrix concatenate:\\n',mat.value())\n",
    "\n",
    "# affine transform\n",
    "e0 = dy.vecInput(2)\n",
    "e0.set([-1, 0])\n",
    "e = dy.affine_transform([e1,mat1,e0])\n",
    "print('affine_transform:', e.value())\n",
    "\n",
    "# sum_elems\n",
    "# Sum all elements\n",
    "e = dy.sum_elems(mat1)\n",
    "print('sum_elems:', e.value())\n",
    "\n",
    "# sum_dim\n",
    "# sum_dim(Expression x, list d, bool b=False, unsigned n=0)\n",
    "# d (list): Dimensions along which to reduce\n",
    "# b (bool): Whether to include batch dimension\n",
    "# Sum along an arbitrary dimension\n",
    "# Here, mat1 has dimension ((4, 2), 1),\n",
    "e = dy.sum_dim(mat1, [0])\n",
    "print('sum over the 0th dimension', e.value())\n",
    "\n",
    "e = dy.sum_dim(mat1, [1])\n",
    "print('sum over the 1st dimension', e.value())\n",
    "\n",
    "# sum_batches\n",
    "# Sum an expression that consists of multiple minibatches into \n",
    "# one of equal dimension but with only a single minibatch. \n",
    "# This is useful for summing loss functions at the end of minibatch training.\n",
    "e = dy.sum_batches(e1)\n",
    "\n",
    "# cumsum\n",
    "# usage: cumsum(Expression x, unsigned d=0)\n",
    "# Computes the cumulative sum along an arbitrary dimension.\n",
    "# d (int): Dimension along which to compute the cumulative sums (default: 0)\n",
    "e = dy.cumsum(mat1, 1)\n",
    "print('cumsum:\\n', e.value())\n",
    "\n",
    "# mean_elems\n",
    "# Computes the mean of all the elements of each minibatch.\n",
    "e = dy.mean_elems(mat1)\n",
    "print('mean_elems', e.value()) # will return 4.5 in this case.\n",
    "\n",
    "# mean_dim\n",
    "# usage:  mean_dim(Expression x, list d, bool b, unsigned n=0)\n",
    "#         x (dynet.Expression): Input expression\n",
    "#         d (list): Dimensions along which to reduce\n",
    "#         b (bool): Whether to include batch dimension\n",
    "#         n (int): If > 0, overwrite the n in the equation by this value, useful for masking\n",
    "# Computes the mean along an arbitrary dimension.\n",
    "e = dy.mean_dim(mat1, [0], True)\n",
    "print('mean_dim:', e.value())\n",
    "e_ = dy.mean_dim(mat1, [1], True)\n",
    "print('mean_dim:', e_.value())\n",
    "\n",
    "# mean_batches\n",
    "# Mean along the batch dimension\n",
    "e = dy.mean_batches(mat1)\n",
    "\n",
    "# std_elems\n",
    "# Computes the standard deviation of all the elements of each minibatch\n",
    "e = dy.std_elems(mat1)\n",
    "print('std_elems:', e.value())\n",
    "\n",
    "# std_dim\n",
    "# usage:  std_dim(Expression x, list d, bool b, unsigned n=0)\n",
    "#         x (dynet.Expression): Input expression\n",
    "#         d (int): Dimensions along which to reduce\n",
    "#         b (bool): Whether to include batch dimension\n",
    "#         n (int): If > 0, overwrite the n in the equation by this value, useful for masking\n",
    "# Computes the standard deviation along arbitrary dimensions.\n",
    "e = dy.std_dim(mat1, [0], True)\n",
    "print('std_dim', e.value())\n",
    "\n",
    "# std_batches\n",
    "# Standard deviation along the batch dimension\n",
    "e = dy.std_batches(mat1)\n",
    "\n",
    "# moment_elems\n",
    "# Statistical moment of elements of the tensor\n",
    "# usage:  moment_elems(Expression x, unsigned r)\n",
    "#         x (dynet.Expression): Input expression\n",
    "#         r (int): Moment order\n",
    "e = dy.moment_elems(mat1, 1)\n",
    "print('moment_elems:', e.value())\n",
    "\n",
    "# moment_dim\n",
    "# Statistical moment along an arbitrary dimension\n",
    "# usage:  moment_dim(Expression x, list d, unsigned r, bool b, unsigned n=0)\n",
    "#             x (dynet.Expression): Input expression\n",
    "#             d (list): Dimensions along which to reduce\n",
    "#             r (int): Moment order\n",
    "#             b (bool): Whether to include batch dimension\n",
    "#             n (int): If > 0, overwrite the n in the equation by this value, useful for masking\n",
    "e = dy.moment_dim(mat1, [0], 2, False)\n",
    "print('moment_dim:', e.value())\n",
    "\n",
    "# moment_batches\n",
    "# Statistical moment along the batch dimension\n",
    "e = dy.moment_batches(mat1, 2)\n",
    "\n",
    "# fold_rows\n",
    "# usage: fold_rows(Expression x, unsigned nrows=2)\n",
    "e = dy.fold_rows(mat1)\n",
    "print('fold_rows:\\n', e.value())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### DyNet in Neural Networks\n",
    "This part contains Neural Networks related issues."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Noise and Dropout Expressions\n",
    "\n",
    "# Add a noise to each element from a gausian distribution\n",
    "# with standard-dev = stddev\n",
    "stddev = 0.1\n",
    "e = dy.noise(e1, stddev)\n",
    "print('noise for stddev=0.1:', e.value())\n",
    "\n",
    "# Apply dropout to the input expression\n",
    "# There are two kinds of dropout methods \n",
    "# (http://cs231n.github.io/neural-networks-2)\n",
    "# Dynet implement the Inverted dropout where dropout with prob p \n",
    "# and scaling others by 1/p at training time, and do not need \n",
    "# to do anything at test time. \n",
    "p = 0.5\n",
    "e = dy.dropout(e1, p)    # apply dropout with probability p \n",
    "print('dropout at probability 0.5:', e.value()) # It should be [2.0, 4.0, 6.0, 0.0], the last element is dropped out and the rest are scaled\n",
    "\n",
    "# If we set p=1, everything will be dropped out\n",
    "e = dy.dropout(e1, 1)\n",
    "print('dropout at probbability 1:', e.value()) # Should be [nan, nan, ...]\n",
    "\n",
    "# If we set p=0, everything will be kept\n",
    "e = dy.dropout(e1, 0)\n",
    "assert e.value() == e1.value()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Loss Functions\n",
    "\n",
    "# DyNet provides several ways to calculate \"distance\"\n",
    "# between two expressions of the same dimension\n",
    "# This is square_distance, defined as\n",
    "# sum(square of(e1-e2)) for all elements\n",
    "# in e1 and e2.\n",
    "# Here e1 is a vector of [1,2,3,4]\n",
    "# And e2 is a vector of [5,6,7,8]\n",
    "# The square distance is sum((5-1)^2 + (6-2)^2+...)\n",
    "e = dy.squared_distance(e1, e2)\n",
    "print('squared_distance:', e.value())\n",
    "\n",
    "# This is the l1_distance, defined as \n",
    "# sum (abs(e1-e2)) for all elements in\n",
    "# e1 and e2.\n",
    "e = dy.l1_distance(e1, e2)\n",
    "print('l1_distance:', e.value())\n",
    "\n",
    "# This is the huber_distance, definition \n",
    "# found here. (https://en.wikipedia.org/wiki/Huber_loss)\n",
    "# The default threhold (delta) is 1.345.\n",
    "# Here e1 is a vector of [1,2,3,4]\n",
    "# And e2 is a vector of [5,6,7,8]\n",
    "# because for each pair-wised element in\n",
    "# e1 and e2, the abs(e1-e2)=4>delta=1.345,\n",
    "# so the output is sum(delta*(abs(4)-1/2*delta))\n",
    "e = dy.huber_distance(e1, e2, c=1.345)\n",
    "print('huber distance:', e.value())\n",
    "\n",
    "# Binary logistic loss function\n",
    "# This is similar to cross entropy loss function\n",
    "# e1 must be a vector that takes values between 0 and 1\n",
    "# ty must be a vector that takes values between 0 and 1\n",
    "# e = -(ty * log(e1) + (1 - ty) * log(1 - e1))\n",
    "ty = dy.vecInput(4)\n",
    "ty.set([0, 0.5, 0.5, 1])\n",
    "e_scale = ty = dy.vecInput(4)\n",
    "e_scale.set([0.5, 0.5, 0.5, 0.5])\n",
    "e = dy.binary_log_loss(e_scale, ty)\n",
    "print('binary_log_loss:', e.value())\n",
    "# Te binary_log_loss is equivalent to the following:\n",
    "e_equl = -(dy.dot_product(ty, dy.log(e_scale)) + dy.dot_product((dy.inputTensor([1,1,1,1]) - ty), dy.log(dy.inputTensor([1,1,1,1]) - e_scale)))\n",
    "assert e_equl.value() == e.value()\n",
    "\n",
    "# pairwise_rank_loss\n",
    "# e1 is row vector or scalar\n",
    "# e2 is row vector or scalar\n",
    "# m is number\n",
    "# e = max(0, m - (e1 - e2))\n",
    "e = dy.pairwise_rank_loss(dy.transpose(e1), dy.transpose(e2), m=1.0) # Row vector needed, so we transpose the vector.\n",
    "print('pairwise_rank_loss wrt e1:', e.value())  # Expect [[5. 5. 5. 5.]]\n",
    "\n",
    "e = dy.pairwise_rank_loss(dy.transpose(e2), dy.transpose(e1), m=1.0) # Row vector needed, so we transpose the vector.\n",
    "print('pairwise_rank_loss wrt e2:', e.value())  # Expect [[0. 0. 0. 0.]]\n",
    "\n",
    "# poisson_loss\n",
    "# The negative log probability of y according to a Poisson distribution with parameter x. \n",
    "# Useful in Poisson regression where, we try to predict the parameters of a Possion \n",
    "# distribution to maximize the probability of data y.\n",
    "# usage: poisson_loss(Expression log_lambda, unsigned x)\n",
    "#        log_lambda (dynet.Expression): The log of the Poisson distribution's lambda\n",
    "#        x (int): The target value\n",
    "e_scalar = dy.scalarInput(2)\n",
    "e = dy.poisson_loss(e_scalar, 1)\n",
    "print('poisson_loss:', e.value())\n",
    "\n",
    "# hinge, aka SVM loss\n",
    "# usage: hinge(Expression x, unsigned v, float m=1.0)\n",
    "#         x (dynet.Expression): Input scores\n",
    "#         v (int): True class\n",
    "#         m (float): The margin\n",
    "# here we have e1.value() equals to [1,2,3,4] and the true class is set to be 3 (index 2)\n",
    "# and we manually set m to be zero.\n",
    "# so, for each element in e1, we will calculate element-true_class_value+m\n",
    "# in this case, it is [max(0, 1-3+0), max(0, 2-3+0), max(0, 3-3+0), max(0, 4-3+0)]\n",
    "# and finally sum it uo, which is 1.\n",
    "e = dy.hinge(e1, 2, m=0)\n",
    "print('hinge loss:', e.value())\n",
    "\n",
    "# binary_log_loss\n",
    "# The log loss of a binary decision according to the sigmoid sigmoid function.\n",
    "e_scale1 = dy.vecInput(3)\n",
    "e_scale1.set([0.1, 0.2, 0.7])\n",
    "e_scale2 = dy.vecInput(3)\n",
    "e_scale2.set([0.5, 0.2, 0.3])\n",
    "e = dy.binary_log_loss(e_scale1, e_scale2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Convolutions\n",
    "\n",
    "# DyNet can do convolutions similar to PyTorch.\n",
    "# First we mock an image and a filter\n",
    "# mat is a 3D tensor of dim{4,4,3}\n",
    "# kernel is a 4d Tensor of shape {2,2,3,1}\n",
    "mat = dy.inputTensor(np.array([[[1,2,1], [0,1,2], [0,0,1], [0,1,0]], [[1,0,2], [0,0,0], [1,1,1], [2,2,2]], [[0,1,2], [1,1,0], [0,0,1], [2,2,1]], [[2,2,0], [2,1,2], [2,2,1], [1,1,0]]]))\n",
    "kernel = dy.inputTensor(np.array([[[[1], [0], [2]], [[1], [2], [0]]], [[[0], [1], [0]], [[2], [1], [1]]]]))\n",
    "print(mat.dim(), kernel.dim())\n",
    "\n",
    "# filter1d_narrow()\n",
    "# usage: e = dy.filter1d_narrow(e1, e2)\n",
    "#        e1: Expression of n x s\n",
    "#        e2: Expression of n x d\n",
    "# This function will calculate the convolution along each dimension.\n",
    "# For example, 0.8 = 1*0 + 2*0.1 + 3*0.2 + 0*0.5\n",
    "#              1.6 = 2*0 + 3*0.1 + 4*0.2 + 1*0.5\n",
    "#              1.4 = 3*0 + 4*0.1 + 0*0.2 + 2*0.5\n",
    "e_input = dy.inputTensor([[1,2,3], [2,3,4], [3,4,0], [0,1,2]])\n",
    "e_filter = dy.inputTensor([[0], [0.1], [0.2], [0.5]])\n",
    "e = dy.filter1d_narrow(e_input, e_filter)\n",
    "print('filter1d_narrow', e.npvalue()) # expect [[0.8 1.6 1.4]]\n",
    "\n",
    "\n",
    "# conv2d\n",
    "# This is 2D convolution operator without bias parameters.\n",
    "# dy.conv2d(Expression x, Expression f, vector[unsigned] stride, bool is_valid = True)\n",
    "# x: The input feature maps: (H x W x Ci) x N (ColMaj), 3D tensor with an optional batch dimension\n",
    "# f: 2D convolution filters: H x W x Ci x Co (ColMaj), 4D tensor\n",
    "# stride: the row and column strides in a list\n",
    "# is_valid: padding method. True for 'Valid' and False for 'Same'.\n",
    "#     'Valid': output size shrinks by `filter_size - 1`, and the filters always sweep at valid \n",
    "#              positions inside the input maps. No padding needed.\n",
    "#     'Same': output size is the same with input size. To do so, one needs to pad the input so \n",
    "#             the filter can sweep outside of the input maps.\n",
    "e = dy.conv2d(mat, kernel, stride=[1, 1], is_valid=True)\n",
    "print('con2d without bias:\\n', e.npvalue())\n",
    "\n",
    "# conv2d_bias\n",
    "# This is 2D convolution operator with bias parameters.\n",
    "# dy.conv2d_bias(Expression x, Expression f, Expression b, vector[unsigned] stride, bool is_valid = True)\n",
    "# b: A vector representing bias. (Ci x 1)\n",
    "bias = dy.inputTensor([1])\n",
    "e = dy.conv2d_bias(mat, kernel, bias, stride=[1, 1], is_valid=True)\n",
    "print('conv2d with bias:\\n', e.npvalue())\n",
    "\n",
    "# maxpooling2d\n",
    "# Usage: maxpooling2d(Expression x, vector[unsigned] ksize, vector[unsigned] stride, bool is_valid = True)\n",
    "# x: The input feature maps: (H x W x Ci) x N (ColMaj), 3D tensor with an optional batch dimension\n",
    "# ksize (list): the max pooling 2d window size\n",
    "# stride (list): the row and column strides\n",
    "# Here we use a [2,2] maxpooling window and stride is 1 in rows and 2 in cols.\n",
    "# Given a ((4, 4, 3), 1) matrix , we expect a (3,2,3) * 1 output \n",
    "e = dy.maxpooling2d(mat, [2, 2], [1, 2], is_valid=True)\n",
    "\n",
    "# kmax_pooling\n",
    "# Usage: kmax_pooling(Expression x, unsigned k, unsigned d=1\n",
    "# Select out k maximum values along a given dimension, in the same order as they appear. \n",
    "# This will result in the size of the given dimension being changed to k.\n",
    "#       unsigned k (dynet.Expression): Number of maximum values to retrieve along the given dimension\n",
    "#       unsigned d (int): Dimension on which to perform kmax-pooling (default: (1))\n",
    "# Given a ((4, 4, 3), 1) matrix:\n",
    "e = dy.kmax_pooling(mat, 2, 0) # we expect a ((2, 4, 3), 1) output.\n",
    "e = dy.kmax_pooling(mat, 2, 1) # we expect a ((4, 2, 3), 1) output.\n",
    "e = dy.kmax_pooling(mat, 2, 2) # we expect a ((4, 4, 2), 1) output.\n",
    "\n",
    "# kmh_ngram\n",
    "# usage: kmh_ngram(Expression x, unsigned v)\n",
    "# x should be an Expression. \n",
    "# v should be an int and should be less than number of cols\n",
    "e = dy.kmh_ngram(mat1, 2)\n",
    "print('kmh_ngram\\n',e.npvalue())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Backpropagation and Gradient related\n",
    "\n",
    "# nobackprop\n",
    "#  This node has no effect on the forward pass, but prevents gradients from flowing backward during the backward pass. \n",
    "# This is useful when there's a subgraph for which you don't want loss passed back to the parameters.\n",
    "e = dy.nobackprop(e1)\n",
    "\n",
    "# flip_gradient\n",
    "# This node has no effect on the forward pass, but takes negative on backprop process. \n",
    "# This operation is widely used in adversarial networks.\n",
    "e = dy.flip_gradient(e1)\n",
    "\n",
    "# scale_gradient\n",
    "# This node scales the gradient by a constant on backprop, with no effect on the forward pass\n",
    "e = dy.scale_gradient(e1, lambd=2)\n",
    "\n",
    "# argmax\n",
    "# This node takes an input vector x and returns a one hot vector y.\n",
    "# There are two gradient modes for this operation:\n",
    "# \"zero_gradient\": is the standard argmax operation. Note that this almost everywhere differentiable and its gradient is 0. **It will stop your gradient**\n",
    "# \"straight_through_gradient\": Its forward pass is the same as the argmax operation, but its gradient is the same as the identity function.\n",
    "# Find more information here at: https://arxiv.org/abs/1308.3432\n",
    "e = dy.argmax(dy.inputTensor([1,2,3]), gradient_mode='zero_gradient')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Normalization\n",
    "\n",
    "# Squared norm\n",
    "e = dy.squared_norm(e1)\n",
    "\n",
    "# l2_norm\n",
    "e = dy.l2_norm(e1)\n",
    "\n",
    "# layer_norm\n",
    "# Usage: layer_norm(Expression x, Expression g, Expression b)\n",
    "# x (dynet.Expression): Input expression (possibly batched)\n",
    "# g (dynet.Expression): Gain (same dimension as x, no batch dimension)\n",
    "# b (dynet.Expression): Bias (same dimension as x, no batch dimension)\n",
    "# details can be found here: https://arxiv.org/abs/1607.06450\n",
    "# mu = average(e1) = 2.5\n",
    "# delta = sqrt(1/4*sum(e1-mu)) = 1.118033988749895\n",
    "# e_norm = g/delta * (x-mu) + b\n",
    "e_norm = dy.layer_norm(e1, e1, e1)\n",
    "print('layer norm:', e_norm.npvalue())\n",
    "\n",
    "# weight_norm\n",
    "# Usage: weight_norm(Expression w, Expression g)\n",
    "#       w (dynet.Expression): Input expression (weight parameter)\n",
    "#       g (dynet.Expression): Gain (scalar expression, usually also a parameter)\n",
    "e_norm = dy.weight_norm(e1, dy.scalarInput(2))\n",
    "print('weight norm:', e_norm.npvalue() )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "### Write your own Neural Networks\n",
    "Now that you have a basic idea about APIs, you can try to write simple Neural Networks of your own.\n",
    "\n",
    "Here we have some toy models for your reference."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In the first example, we creat a simple Neural Networks which could learn to predict the minimal value given a 1 x 3 input vector."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from __future__ import print_function\n",
    "from __future__ import division\n",
    "from __future__ import absolute_import\n",
    "\n",
    "import dynet_config\n",
    "# Declare GPU:0 as the default device type\n",
    "# dynet_config.set_gpu()\n",
    "import dynet as dy\n",
    "\n",
    "# reset the global cg\n",
    "dy.renew_cg()\n",
    "# create parameter collection\n",
    "m = dy.ParameterCollection()\n",
    "\n",
    "# add parameters to parameter collection\n",
    "pW = m.add_parameters((10,30))\n",
    "pB = m.add_parameters(10)\n",
    "lookup = m.add_lookup_parameters((500, 10))\n",
    "print(\"Parameters added.\")\n",
    "\n",
    "# create trainer \n",
    "trainer = dy.SimpleSGDTrainer(m)\n",
    "\n",
    "# Regularization is set via the --dynet-l2 commandline flag.\n",
    "# Learning rate parameters can be passed to the trainer:\n",
    "# alpha = 0.1  # learning rate\n",
    "# trainer = dy.SimpleSGDTrainer(m, e0=alpha)\n",
    "\n",
    "# function for graph creation\n",
    "def create_network_return_loss(inputs, expected_output):\n",
    "    \"\"\"\n",
    "    inputs is a list of numbers\n",
    "    \"\"\"\n",
    "    dy.renew_cg()\n",
    "    emb_vectors = [lookup[i] for i in inputs]\n",
    "    net_input = dy.concatenate(emb_vectors)\n",
    "    net_output = dy.softmax( (pW*net_input) + pB)\n",
    "    loss = -dy.log(dy.pick(net_output, expected_output))\n",
    "    return loss\n",
    "\n",
    "# function for prediction\n",
    "def create_network_return_best(inputs):\n",
    "    \"\"\"\n",
    "    inputs is a list of numbers\n",
    "    \"\"\"\n",
    "    dy.renew_cg()\n",
    "    emb_vectors = [lookup[i] for i in inputs]\n",
    "    net_input = dy.concatenate(emb_vectors)\n",
    "    net_output = dy.softmax( (pW*net_input) + pB)\n",
    "    return np.argmax(net_output.npvalue())\n",
    "\n",
    "\n",
    "# train network\n",
    "for epoch in range(5):\n",
    "    for inp,lbl in ( ([1,2,3],1), ([3,2,4],2) ):\n",
    "        loss = create_network_return_loss(inp, lbl)\n",
    "        print(loss.value()) # need to run loss.value() for the forward prop\n",
    "        loss.backward()\n",
    "        trainer.update()\n",
    "\n",
    "print('Predicted smallest element among {} is {}:'.format([1,2,3], create_network_return_best([1,2,3])))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "You can also rewrite this example in a more fancy way. You create an object which has `create_network_return_loss()` and `create_network_return_best()` public method."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from __future__ import print_function\n",
    "from __future__ import division\n",
    "from __future__ import absolute_import\n",
    "\n",
    "import dynet_config\n",
    "# Declare GPU as the default device type\n",
    "# dynet_config.set_gpu()\n",
    "import dynet as dy\n",
    "\n",
    "dy.renew_cg()\n",
    "# create parameter collection\n",
    "m = dy.ParameterCollection()\n",
    "\n",
    "# create a class encapsulating the network\n",
    "class OurNetwork(object):\n",
    "    # The init method adds parameters to the parameter collection.\n",
    "    def __init__(self, pc):\n",
    "        self.pW = pc.add_parameters((10,30))\n",
    "        self.pB = pc.add_parameters(10)\n",
    "        self.lookup = pc.add_lookup_parameters((500,10))\n",
    "    \n",
    "    # the __call__ method applies the network to an input\n",
    "    def __call__(self, inputs):\n",
    "        lookup = self.lookup\n",
    "        emb_vectors = [lookup[i] for i in inputs]\n",
    "        net_input = dy.concatenate(emb_vectors)\n",
    "        net_output = dy.softmax((self.pW * net_input) + self.pB)\n",
    "        return net_output\n",
    "    \n",
    "    def create_network_return_loss(self, inputs, expected_output):\n",
    "        dy.renew_cg()\n",
    "        out = self(inputs)\n",
    "        loss = -dy.log(dy.pick(out, expected_output))\n",
    "        return loss\n",
    "       \n",
    "    def create_network_return_best(self, inputs):\n",
    "        dy.renew_cg()\n",
    "        out = self(inputs)\n",
    "        return np.argmax(out.npvalue())\n",
    "        \n",
    "        \n",
    "# create network\n",
    "network = OurNetwork(m)\n",
    "\n",
    "# create trainer \n",
    "trainer = dy.SimpleSGDTrainer(m)\n",
    "   \n",
    "# train network\n",
    "for epoch in range(5):\n",
    "    for inp,lbl in ( ([1,2,3],1), ([3,2,4],2) ):\n",
    "        loss = network.create_network_return_loss(inp, lbl)\n",
    "        print(loss.value()) # need to run loss.value() for the forward prop\n",
    "        loss.backward()\n",
    "        trainer.update()\n",
    "\n",
    "print('Predicted smallest element among {} is {}:'.format([1,2,3], network.create_network_return_best([1,2,3])))\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Or alternatively, have the training outside of the network class."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# create network\n",
    "network = OurNetwork(m)\n",
    "\n",
    "# create trainer \n",
    "trainer = dy.SimpleSGDTrainer(m)\n",
    "   \n",
    "# train network\n",
    "for epoch in range(5):\n",
    "    for inp,lbl in ( ([1,2,3],1), ([3,2,4],2) ):\n",
    "        dy.renew_cg()\n",
    "        out = network(inp)\n",
    "        loss = -dy.log(dy.pick(out, lbl))\n",
    "        print(loss.value()) # need to run loss.value() for the forward prop\n",
    "        loss.backward()\n",
    "        trainer.update()\n",
    "\n",
    "print(np.argmax(network([1,2,3]).npvalue()))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}