PGI-15 Tutorial

Recurrent Neural Networks

02.11.2023 | Emre Neftci, Susanne Kunkel, Jamie Lohoff, and Willem Wybo

<h2> How to Deal with Variable-Sized Dimensions?</h2>                                                     
        <ul>
          <li/> Time is a variable dimension. Until now, we dealt with data having static dimensions. There are two ways to deal with this:
          <ol>
          <li class=fragment /> Fix the size, truncate (pad) the data that has a larger (smaller) dimension. For example for size 2:
            <ul>
              <li> Three data samples:
                <pre><code data-trim data-noescape> [[1, 5, 6], [3,4], [3]] -- [[1, 5], [3,4], [3, None]] </code></pre> </li>
              <li /> Once the size is fixed, a feed-forward neural network can be used
            </ul>
            
          <li class=fragment /> Use a recursive algorithm: Feed in the data time step by time step and keep a memory of the previously fed data samples
            <ul>
              <li> <pre><code data-trim data-noescape> [[1, 5, 6], [3,4], [3]] := [[1,3,3],[5,4, EOS],[3,EOS, None] </code></pre> </li>
              <li > EOS is a special symbol meaning end-of-sequence. </li>
              <li > Recurrent neural networks (and more recently, transformers) are designed to deal with such data </li>
            </ul>
          </ol>
        </ul>

Recurrent Neural Networks and Working Memory

Working Memory:

A type of short-term memory
Rapid & selective updating, robust maintenance
Task- and sensory modality-dependent
Necessary for cognitive control

O'Reilly and Frank, 2006

Human brain areas for working memory of face identity and location

Example Tasks

Examples: Depending on the output structure, different problems can be solved:

Many-to-one: Classification, Sentiment analysis
Many-to-many: Machine translation
One-to-many: Image captioning
One-to-one: Time series prediction

Complexity of Gradient-Based Algorithms in RNNs

$n$ number of neurons
$L$ number of time steps
$h$ representing the number of prior time steps saved (truncation)

Williams and Zipser, 1995

End

Lecture 4: Learning and Plasticity