Neural Networks as Morphisms

Abstract

This set of notes attempts to study possibility of constructing categories with neural nets as morphism and functors.
The reason for such construction comes stems from possible enriching connections between category theory and
deep learning. Usually Category Theory is used to formalize abstractions of the given concrete objects. On the
other hand, Deep Learning is mostly used in constructing explicit neural network based mappings between various
spaces given by sampled datasets. The notes explores if it is possible to leverage on strengths of category theory
and deep learning in order to represent more abstract concepts and processes.

Introduction

Motivation:

The main motivation for development of framework if two folds. Firstly, it might shed a new light for representing
more abstract objects, such as movements and processes. Secondly, we leverage on rich experience of building
deep neural nets accumulated in Machine Learning community.

Abstract Objects

Many objects that are considered abstract can be realized as a mapping between spaces. For instance. A video can
be thought as a mapping between some time interval say [0,1] and a space of images. As another example we can
look at decision making process. It is quite well-known that a decision can be thought as a mapping between a
space of states into a space of actions. On the other hand a concept of dynamic actions can be thought as a mapping
between some time interval say [0,1] into a former space.

Why is it important to consider representation of abstract objects?

A success of deep learning comes mainly from its ability to extract some deep invariant representations from given
objects. For instance a property of being a number 1 does not change across its different writings. Deep Learning
has managed to extract these features which are closely related to those perceived by humans. However, further
advancements in computational reasoning and artificial intelligence demand ability to represent much more
complex objects such as complex decision making, unsupervised learning and totally autonomous systems. This
idea is also mentioned by Yoshua Bengio in one of his presentations.

Leveraging on current developments in Deep Learning

Since there has been much progress in Deep Learning research and many techniques and methods were developed,
it is important to leverage on this developments. Roughly speaking, the supervised Deep Learning concerns itself
with construction of mappings based on neural networks. In other words given several instances of a desired map f,
Deep Learning teaches us how to construct it a neural network based map f* which approximates f, without being
too costly, where cost refers to regularization cost. This is done by performing a some kind of gradient descent on
the function space given some loss function. It is important to observe that this process usually involves two
spaces, one is input, while another is target space. So, diagrammatically, it can viewed as two object diagram:

(1) -> (2)

Ideas and Results

Formalization

Let us name category whose morphism neural networks NN.

Objects of NN: Euclidean Spaces, Conceptual Spaces, Probability distributions

Morphism if NN: Neural Nets

Checking if NN is indeed a category

Checking elementary rules for being a category:

1. Identity morphism is just identity linear transformation (no nonlinear activation function)
2. Composition of two neural networks is again a neural network.

Remarks:

• Most of neural networks are not reversible, so the category is not a groupoid

Functors:

Given two NN categories A and B we can define a functor between these categories which is required to be a
neural network itself.

Enriching Connections

In this section we wish to propose two potential connection between category theory and deep learning for a start.
First connection has to do with a category theoretic notions of limit and colimits.

The second connection goes other way around and seek a category theoretic equivalent for generative model for a
given classification model.

Limits and Colimits

The notions of limit and colimit is meant to extract certain information from a given diagram or subcategory.
This information is then encoded into a single object. The application of this idea might be possibility of going up
in abstractions. Suppose we have certain NN category, by considering its limit or colimit we could go up and refer
to the (co)limit instead of entire category. This is depicted in the following diagram

Understanding Generative Models in Deep Learning through a lense of Category Theory

Generative models in Deep Learning are meant to produce a realistic looking objects having seen many examples
of certain objects. In some sense this generation can be thought as a reverse of abstraction. So the approach might
be to learn some kind of abstract representation of some object and try to go back from abstract representation to
original object. Pictorially this process might look like the following,

Discussion

The Deep Learning community tends to perceive the understanding of an object as an ability to construct a neural
network related to that object. This set of notes tries to extend this idea by looking not just into a pair of objects
or spaces, but rather into a category of such objects or spaces connected with neural networks. Furthermore the
notes consider a possibility of going back and forth between these categories with the help of functors which are
also neural networks. Our hope is that this approach might shed a new light in representation of more abstract
concepts and processes using neural networks. It is hoped that this set of notes could open up a new venue for
future research in possible interactions between Category Theory and Deep Learning.