Intro to EIANN

Intro to EIANN#

Note

🚧 Work in Progress: This documentation site is currently under construction. Content may change frequently.

EIANN is a PyTorch-based library for building neural networks.

At its core, the syntax for building these networks uses python dicts to specify the network architecture and parameters. We provide two easy ways to generate these dicts: YAML files or a programmatic interface with a NetworkBuilder class. The first method is convenient for quickly prototyping and testing different network architectures. The second method is particularly useful for hyperparameter optimization, systematic architecture search, and experiment reproducibility. Here we will show you examples of how to do both.

Since thie library is based on PyTorch, a basic familiarity with PyTorch objects and syntax is helpful, but not required, to use EIANN. You can geet a quick overview of PyTorch by going through the PyTorch Quickstart Tutorial.

Basic building blocks#

To build a neural network using the EIANN library, you need to specify 3 things:

  1. The number neural populations within in each “layer” and how many neurons are in each population

  2. The projections (how the populations connect to each other)

  3. Global parameters that we want to apply to the whole network (e.g. learning rate)

This population-centric approach is flexible and allows you to recurrently connect any two populations, regardless of layer hierarchy.

Each of these things will be defined through 3 separate python dicts. EIANN will then build a PyTorch network the specified architecture.

1. Programming interface#

import EIANN.EIANN as eiann

Hide code cell output

 
network_config = eiann.NetworkBuilder()

# Define layers and populations
network_config.layer('Input').population('E', size=784)
network_config.layer('H1').population('E', size=500, activation='relu')
network_config.layer('Output').population('E', size=10, activation='softmax')

# Create connections between populations
network_config.connect(source='Input.E', target='H1.E')
network_config.connect(source='H1.E', target='Output.E')

# Build the network
network_config.print_architecture()

==================================================
Network Architecture:
Input.E (784) -> H1.E (500): No learning rule
H1.E (500) -> Output.E (10): No learning rule
==================================================

Rather than defining only the weight matrix dimensions, this population-based syntax allows you to also specify attributes at the population level. For example, if you want the network to respect Dale’s Law, you can specify whether each population is excitatory or inhibitory. Doing this will automatically bound the sign of outgoing weights, both at initialization and during training.

network_config.layer('H1').population('E', 500, 'relu').type('Exc') 
network_config.layer('H1').population('SomaI', 50, 'relu').type('Inh') 
network_config.print_architecture()

==================================================
Network Architecture:
Input.E (784) -> H1.E (500): No learning rule
H1.E (500) -> Output.E (10) [Exc]: No learning rule

Unconnected populations:
H1.SomaI (50) [Inh]
==================================================

Should you prefer, these weight bounds can also be set or overwritten for individual projections:

network_config.connect(source='H1.E', target='H1.SomaI').type('Exc') 
network_config.connect(source='H1.SomaI', target='H1.E').type('Inh') 
network_config.print_architecture()

==================================================
Network Architecture:
Input.E (784) -> H1.E (500): No learning rule
H1.SomaI (50) -> H1.E (500) [Inh]: No learning rule
H1.E (500) -> H1.SomaI (50) [Exc]: No learning rule
H1.E (500) -> Output.E (10) [Exc]: No learning rule
==================================================

So far we have not specified the type of learning rule we want to use. If we want to use a single learning rule (e.g. Backprop) throughout the whole network, this can be done globally:

# Set Backprop as the learning rule for the whole network
network_config.set_learning_rule('Backprop')
network_config.print_architecture()

==================================================
Network Architecture:
Input.E (784) -> H1.E (500): Backprop
H1.SomaI (50) -> H1.E (500) [Inh]: Backprop
H1.E (500) -> H1.SomaI (50) [Exc]: Backprop
H1.E (500) -> Output.E (10) [Exc]: Backprop
==================================================

If we instead want a different learning rule for a subset of the connections, we can also manually specify it for a given layer, population, or projection

network_config.connect(source='H1.SomaI', target='H1.E').type('Inh').learning_rule('BCM')
network_config.print_architecture()

==================================================
Network Architecture:
Input.E (784) -> H1.E (500): Backprop
H1.SomaI (50) -> H1.E (500) [Inh]: BCM
H1.E (500) -> H1.SomaI (50) [Exc]: Backprop
H1.E (500) -> Output.E (10) [Exc]: Backprop
==================================================

network_config.set_learning_rule_for_population(target_layer='H1', target_population='E', rule='Hebbian', learning_rate=0.01)
network_config.print_architecture()

==================================================
Network Architecture:
Input.E (784) -> H1.E (500): Hebbian (lr=0.01)
H1.SomaI (50) -> H1.E (500) [Inh]: Hebbian (lr=0.01)
H1.E (500) -> H1.SomaI (50) [Exc]: Backprop
H1.E (500) -> Output.E (10) [Exc]: Backprop
==================================================

network_config.set_learning_rule_for_layer(target_layer='H1', rule='Ojas_rule', learning_rate=0.01)
network_config.print_architecture()

==================================================
Network Architecture:
Input.E (784) -> H1.E (500): Ojas_rule (lr=0.01)
H1.SomaI (50) -> H1.E (500) [Inh]: Ojas_rule (lr=0.01)
H1.E (500) -> H1.SomaI (50) [Exc]: Ojas_rule (lr=0.01)
H1.E (500) -> Output.E (10) [Exc]: Backprop
==================================================

Note

You need to make sure that the learning rule you want to use actually exists in the EIANN.rules module. In later tutorials, we will show you how to write your own learning rule.

Once we have specified all the populations and projections that define the network architecture, we can use the build_network function to create the neural network (pytorch) object.

network_random_seed = 42 # Specifying a seed for the random number generator is optional, but enables your experiments to be reproducible.
network = network_config.build(seed=network_random_seed)

Network(
  (criterion): MSELoss()
  (module_dict): ModuleDict(
    (H1E_InputE): Projection(in_features=784, out_features=500, bias=False)
    (H1E_H1SomaI): Projection(in_features=50, out_features=500, bias=False)
    (H1SomaI_H1E): Projection(in_features=500, out_features=50, bias=False)
    (OutputE_H1E): Projection(in_features=500, out_features=10, bias=False)
  )
  (parameter_dict): ParameterDict(
      (H1E_bias): Parameter containing: [torch.FloatTensor of size 500]
      (H1SomaI_bias): Parameter containing: [torch.FloatTensor of size 50]
      (OutputE_bias): Parameter containing: [torch.FloatTensor of size 10]
  )
)

2. YAML interface#

When running many experiments, optimizing architectures or parameters, and comparing results, it can be helpful to separate model specifications from your code. One way to do this is by using a YAML file to describe the model. Then, when you run the model, you only need to specify the path or filename of the YAML file.

If you’re not familiar with YAML, it’s just a structured text format that is both human-readable and easy for code to parse.

Here’s an example of a YAML file describing a simple EIANN model:

layer_config:
  Input:
    E:
      size: 784
  H1:
    E:
      size: 500
      activation: relu
  Output:
    E:
      size: 10
      activation: relu

projection_config:
  H1.E:
    Input.E:
      learning_rule: Backprop

  Output.E:
    H1.E:
      learning_rule: Backprop

training_kwargs:
  learning_rate: 0.025

So instead of specifying the parameters of the model in your code, you can simply load this YAML file:

import EIANN.utils as ut
config_file_path = f"../EIANN/network_config/MNIST_templates/example_EIANN_config_1.yaml"

network_random_seed = 42 # Specifying a seed for the random number generator is optional, but enables your experiments to be reproducible.
network = ut.build_EIANN_from_config(config_file_path, network_seed=network_random_seed)
print(network)

Network(
  (criterion): MSELoss()
  (module_dict): ModuleDict(
    (H1E_InputE): Projection(in_features=784, out_features=500, bias=False)
    (OutputE_H1E): Projection(in_features=500, out_features=10, bias=False)
  )
  (parameter_dict): ParameterDict(
      (H1E_bias): Parameter containing: [torch.FloatTensor of size 500]
      (OutputE_bias): Parameter containing: [torch.FloatTensor of size 10]
  )
)

For more complex models, the YAML file also allows you to specify a number of optional parameters for each population and projection:

(For a full list of parameters, see the glossary.)

layer_config:
  Input:
    E:
      size: 784
  H1:
    E:
      size: 500
      activation: softplus
      activation_kwargs:
          beta: 20.0      
  Output:
    E:
      size: 10
      activation: softmax

projection_config:
  H1.E:
    Input.E:
      weight_init: half_kaiming
      type: exc
      weight_init_args: [2.4]
      direction: F
      learning_rule: Backprop
      learning_rule_kwargs:
        learning_rate: 0.2
    H1.SomaI:
      weight_init: half_kaiming
      type: inh
      weight_init_args: [1.1]
      direction: R
      learning_rule: Backprop
      
  H1.SomaI:
    H1.E:
      weight_init: half_kaiming
      type: exc
      weight_init_args: [1.]
      direction: F
      learning_rule: None

  Output.E:
    H1.E:
      learning_rule: Backprop

training_kwargs:
  tau: 3
  forward_steps: 15
  backward_steps: 3
  learning_rate: 0.05

In the next tutorial, we will see how to train a simple feedforward neural network to classify MNIST handwritten digits

Contents