neuralogic.nn.module.general package

Submodules

neuralogic.nn.module.general.attention module

class Attention(embed_dim: int, output_name: str, query_name: str, key_name: str, value_name: str, mask_name: str | None = None, arity: int = 1)

Bases: Module

A single-head attention module based on “Attention Is All You Need”.

Parameters:

• embed_dim (int) – The number of expected features.

• output_name (str) – Output (head) predicate name of the module.

• query_name (str) – The name of the queries predicate.

• key_name (str) – The name of the keys predicate.

• value_name (str) – The name of the values predicate.

• mask_name (str, optional) – The name of the input mask predicate. Default: None

• arity (int) – Arity of the input and output predicates. Default: 1

class MultiheadAttention(embed_dim: int, num_heads: int, output_name: str, query_name: str, key_name: str, value_name: str, vdim: int | None = None, kdim: int | None = None, mask_name: str | None = None, arity: int = 1)

Bases: Module

A multi-head attention module based on “Attention Is All You Need”.

Parameters:

• embed_dim (int) – The number of expected features.

• num_heads (int) – The number of heads.

• output_name (str) – Output (head) predicate name of the module.

• query_name (str) – The name of the queries predicate.

• key_name (str) – The name of the keys predicate.

• value_name (str) – The name of the values predicate.

• vdim (Optional[int]) – Total number of features for values.

• kdim (Optional[int]) – Total number of features for keys.

• mask_name (str, optional) – The name of the input mask predicate. Default: None

• arity (int) – Arity of the input and output predicates. Default: 1

neuralogic.nn.module.general.gru module

class GRU(input_size: int, hidden_size: int, output_name: str, input_name: str, hidden_0_name: str, arity: int = 1, next_name: str = '_next__positive')

Bases: Module

One-layer Gated Recurrent Unit (GRU) module which is computed as:

\[\begin{split}r_t = \sigma(\mathbf{W}_{xr} \mathbf{x}_t + \mathbf{W}_{hr} \mathbf{h}_{t-1}) \\\end{split}\]

\[\begin{split}z_t = \sigma(\mathbf{W}_{xz} \mathbf{x}_t + \mathbf{W}_{hz} \mathbf{h}_{t-1}) \\\end{split}\]

\[\begin{split}n_t = \tanh(\mathbf{W}_{xn} \mathbf{x}_t + r_t \odot (\mathbf{W}_{hn} \mathbf{h}_{t-1})) \\\end{split}\]

\[h_t = (1 - z_t) \odot n_t + z_t \odot h_{t-1}\]

where \(t \in (1, sequence\_length + 1)\) is a time step. In the template, the \(t\) is referred to as V.T, and \(t - 1\) is referred to as V.Z. This module expresses the first equation as:

(R.<output_name>__r(<...terms>, V.T) <= (
    R.<input_name>(<...terms>, V.T)[<hidden_size>, <input_size>],
    R.<hidden_input_name>(<...terms>, V.Z)[<hidden_size>, <hidden_size>],
    R.<next_name>(V.Z, V.T),
)) | [Transformation.SIGMOID]

R.<output_name>__r / <arity> + 1 | [Transformation.IDENTITY]

The second equation is expressed in the same way, except for a different head predicate name. The third equation is split into three rules. The first two computes the element-wise product - \(r_t * (\mathbf{W}_{hn} \mathbf{h}_{t-1})\).

(R.<output_name>__n_helper_weighted(<...terms>, V.T) <= (
    R.<hidden_input_name>(<...terms>, V.Z)[<hidden_size>, <hidden_size>], R.<next_name>(V.Z, V.T),
)) | [Transformation.IDENTITY],

R.<output_name>__n_helper_weighted / (<arity> + 1) | [Transformation.IDENTITY],

(R.<output_name>__n_helper(<...terms>, V.T) <= (
    R.<output_name>__r(<..terms>, V.T), R.<>__n_helper_weighted(<...terms>, V.T)
)) | [Transformation.IDENTITY, Combination.ELPRODUCT],

R.<output_name>__n_helper / (<arity> + 1) | [Transformation.IDENTITY],

The third one computes the sum and applies the \(tanh\) activation function.

(R.<output_name>__n(<...terms>, V.T) <= (
    R.<input_name>(<...terms>, V.T)[<hidden_size>, <input_size>],
    R.<output_name>__n_helper(<...terms>, V.T)
)) | [Transformation.TANH]
R.<output_name>__n / (<arity> + 1) | [Transformation.IDENTITY],

The last equation is computed via three rules. The first two rules computes element-wise products. That is:

(R.<output_name>__left(<...terms>, V.T) <= (
    R.<output_name>__z(<...terms>, V.T), R.<output_name>__n(<...terms>, V.T)
)) | [Transformation.IDENTITY, Combination.ELPRODUCT]

(R.<output_name>__right(<...terms>, V.T) <= (
    R.<output_name>__z(<...terms>, V.T), R.<hidden_input_name>(<...terms>, V.Z), R.<next_name>(V.Z, V.T),,
)) | [Transformation.IDENTITY, Combination.ELPRODUCT]

R.<output_name>__left / <arity> + 1 | [Transformation.IDENTITY]
R.<output_name>__right / <arity> + 1 | [Transformation.IDENTITY]

The last output rule sums up the element-wise products.

(R.<output_name>(<...terms>, V.T) <= (
    R.<output_name>__left(<...terms>, V.T), R.<output_name>__right(<...terms>, V.T)
)) | [Transformation.IDENTITY]
R.<output_name> / <arity> + 1 | [Transformation.IDENTITY],

Additionally, we define a rule for the “stop condition”, that is:

(R.<output_name>(<...terms>, 0) <= R.<hidden_0_name>(<...terms>)) | [Transformation.IDENTITY]

Parameters:

• input_size (int) – Input feature size.

• hidden_size (int) – Output and hidden feature size.

• output_name (str) – Output (head) predicate name of the module.

• input_name (str) – Input feature predicate name to get features from.

• hidden_0_name (str) – Predicate name to get initial hidden state from.

• arity (int) – Arity of the input and output predicate. Default: 1

class GRUCell(input_size: int, hidden_size: int, output_name: str, input_name: str, hidden_input_name: str, arity: int = 1)

Bases: Module

Parameters:

• input_size (int) – Input feature size.

• hidden_size (int) – Output and hidden feature size.

• output_name (str) – Output (head) predicate name of the module.

• input_name (str) – Input feature predicate name to get features from.

• hidden_input_name (str) – Predicate name to get hidden state from.

• arity (int) – Arity of the input and output predicate. Default: 1

neuralogic.nn.module.general.linear module

class Linear(in_channels: int, out_channels: int, output_name: str, input_name: str, activation: TransformationFunction = <neuralogic.core.constructs.function.function.TransformationFunction object>, arity: int = 1)

Bases: Module

Apply linear transformation on the input. Can be expressed as:

\[h_{i_0, .., i_{n}} = W \cdot x_{i_0, .., i_{n}}\]

Where \(x\) is the input, \(W \in R^{(out\_channels \times in\_channels)}\) is a learnable parameter, and \(n\) is the arity of the input and output.

It is also possible to attach non-linearity via the activation parameter and compute:

\[h_{i_0, .., i_{n}} = act(W \cdot x_{i_0, .., i_{n}})\]

Example

The whole computation of this module (parametrized as Linear(1, 2, "h1", "h0")) is as follows:

(R.h1(V.X0)[2, 1] <= R.h0(V.X0)) | [Transformation.IDENTITY]
R.h1 / 1 | [Transformation.IDENTITY]

Module parametrized as Linear(1, 2, "h1", "h0", Transformation.SIGMOID, 2) translates into:

(R.h1(V.X0, V.X1)[2, 1] <= R.h0(V.X0, V.X1)) | [Transformation.IDENTITY]
R.h1 / 2 | [Transformation.SIGMOID]

Parameters:

• in_channels (int) – Input feature size.

• out_channels (int) – Output feature size.

• output_name (str) – Output (head) predicate name of the module.

• input_name (str) – Input name.

• activation (TransformationFunction) – Activation function of the output. Default: Transformation.IDENTITY

• arity (int) – Arity of the input and output predicate. Default: 1

neuralogic.nn.module.general.lstm module

class LSTM(input_size: int, hidden_size: int, output_name: str, input_name: str, hidden_0_name: str, cell_state_0_name: str, arity: int = 1)

Bases: Module

One-layer Long Short-Term Memory (LSTM) RNN module which is computed as:

\[i_t = \sigma(\mathbf{W}_{xi} \mathbf{x}_t + \mathbf{W}_{hi} \mathbf{h}_{t-1})\]

\[f_t = \sigma(\mathbf{W}_{xf} \mathbf{x}_t + \mathbf{W}_{hf} \mathbf{h}_{t-1})\]

\[o_t = \sigma(\mathbf{W}_{xo} \mathbf{x}_t + \mathbf{W}_{ho} \mathbf{h}_{t-1})\]

\[\begin{split}g_t = \tanh(\mathbf{W}_{xg} \mathbf{x}_t + \mathbf{W}_{hg} \mathbf{h}_{t-1}) \\\end{split}\]

\[c_t = f_t \odot c_{t-1} + i_t \odot g_t\]

\[h_t = o_t \odot \tanh(c_t)\]

Parameters:

• input_size (int) – Input feature size.

• hidden_size (int) – Output and hidden feature size.

• output_name (str) – Output (head) predicate name of the module.

• input_name (str) – Input feature predicate name to get features from.

• hidden_0_name (str) – Predicate name to get initial hidden state from.

• cell_state_0_name (str) – Predicate name to get initial cell state from.

• arity (int) – Arity of the input and output predicate. Default: 1

class LSTMCell(input_size: int, hidden_size: int, output_name: str, input_name: str, hidden_input_name: str, cell_state_0_name: str, arity: int = 1)

Bases: Module

Parameters:

• input_size (int) – Input feature size.

• hidden_size (int) – Output and hidden feature size.

• output_name (str) – Output (head) predicate name of the module.

• input_name (str) – Input feature predicate name to get features from.

• hidden_input_name (str) – Predicate name to get hidden state from.

• cell_state_0_name (str) – Predicate name to get initial cell state from.

• arity (int) – Arity of the input and output predicate. Default: 1

neuralogic.nn.module.general.mlp module

class MLP(units: ~typing.List[int], output_name: str, input_name: str, activation: ~neuralogic.core.constructs.function.function.TransformationFunction | ~typing.List[~neuralogic.core.constructs.function.function.TransformationFunction] = <neuralogic.core.constructs.function.function.TransformationFunction object>, arity: int = 1)

Bases: Module

Parameters:

• units (List[int]) – List of layer sizes.

• output_name (str) – Output (head) predicate name of the module.

• input_name (str) – Input name.

• activation (Union[TransformationFunction, List[TransformationFunction]]) – Activation function of all layers or list of activations for each layer. Default: Transformation.RELU

• arity (int) – Default: -1

neuralogic.nn.module.general.pooling module

class AvgPooling(output_name: str, input_name: str, input_arity: int = 1)

Bases: Pooling

Apply average pooling over the input specified by the input_name and the input arity parameters. Can be expressed as:

\[h = \frac{1}{|N|}\sum_{i_{0}, .., i_{n} \in N} x_{(i_{0}, .., i_{n})}\]

Where \(N\) is a set of tuples of length \(n\) (specified by the input arity parameter) that are valid arguments for the input predicate.

This module extends the generic pooling Pooling.

Examples

The whole computation of this module (parametrized as AvgPooling("h1", "h0")) is as follows:

(R.h1 <= R.h0(V.X0)) | [Aggregation.AVG, Transformation.IDENTITY]
R.h1 / 0 | [Transformation.IDENTITY]

Parameters:

• output_name (str) – Output (head) predicate name of the module.

• input_name (str) – Input name.

• input_arity (int) – Arity of the input predicate input_name. Default: 1

class MaxPooling(output_name: str, input_name: str, input_arity: int = 1)

Bases: Pooling

Apply max pooling over the input specified by the input_name and the input arity parameters. Can be expressed as:

\[h = max_{i_{0}, .., i_{n} \in N}(x_{(i_{0}, .., i_{n})})\]

Where \(N\) is a set of tuples of length \(n\) (specified by the input arity parameter) that are valid arguments for the input predicate.

This module extends the generic pooling Pooling.

Examples

The whole computation of this module (parametrized as MaxPooling("h1", "h0")) is as follows:

(R.h1 <= R.h0(V.X0)) | [Aggregation.MAX, Transformation.IDENTITY]
R.h1 / 0 | [Transformation.IDENTITY]

Parameters:

• output_name (str) – Output (head) predicate name of the module.

• input_name (str) – Input name.

• input_arity (int) – Arity of the input predicate input_name. Default: 1

class Pooling(output_name: str, input_name: str, aggregation: AggregationFunction, input_arity: int = 1)

Bases: Module

Apply generic pooling over the input specified by the input_name and the input arity parameters. Can be expressed as:

\[h = agg_{i_{0}, .., i_{n} \in N}(x_{(i_{0}, .., i_{n})})\]

Where \(N\) is a set of tuples of length \(n\) (specified by the input arity parameter) that are valid arguments for the input predicate.

For example, a classic pooling over graph nodes represented by relations of arity 1 (node id) would be calculated as:

\[h = agg_{i \in N}(x_{(i)})\]

Here \(N\) refers to a set of all node ids. Lifting the restriction of the input arity via the input_arity parameter allows for pooling not only nodes but also edges (input_arity=2) and other objects (hyperedges etc.)

Examples

The whole computation of this module (parametrized as Pooling("h1", "h0", Aggregation.AVG)) is as follows:

(R.h1 <= R.h0(V.X0)) | [Aggregation.AVG, Transformation.IDENTITY]
R.h1 / 0 | [Transformation.IDENTITY]

Module parametrized as Pooling("h1", "h0", Aggregation.MAX, 2) translates into:

(R.h1 <= R.h0(V.X0, V.X1)) | [Aggregation.MAX, Transformation.IDENTITY]
R.h1 / 0 | [Transformation.IDENTITY]

Parameters:

• output_name (str) – Output (head) predicate name of the module.

• input_name (str) – Input name.

• aggregation (AggregationFunction) – Aggregation function.

• input_arity (int) – Arity of the input predicate input_name. Default: 1

class SumPooling(output_name: str, input_name: str, input_arity: int = 1)

Bases: Pooling

Apply sum pooling over the input specified by the input_name and the input arity parameters. Can be expressed as:

\[h = \sum_{i_{0}, .., i_{n} \in N} x_{(i_{0}, .., i_{n})}\]

Where \(N\) is a set of tuples of length \(n\) (specified by the input arity parameter) that are valid arguments for the input predicate.

This module extends the generic pooling Pooling.

Examples

The whole computation of this module (parametrized as SumPooling("h1", "h0")) is as follows:

(R.h1 <= R.h0(V.X0)) | [Aggregation.SUM, Transformation.IDENTITY]
R.h1 / 0 | [Transformation.IDENTITY]

Parameters:

• output_name (str) – Output (head) predicate name of the module.

• input_name (str) – Input name.

• input_arity (int) – Arity of the input predicate input_name. Default: 1

neuralogic.nn.module.general.positional_encoding module

class PositionalEncoding(embed_dim: int, max_len: int, output_name: str, input_name: str, arity: int = 1, learnable: bool = False)

Bases: Module

Implements positional encoding for logic sequences. It generates a set of rules that add fixed or learnable positional embeddings to the input features.

neuralogic.nn.module.general.rnn module

class RNN(input_size: int, hidden_size: int, output_name: str, input_name: str, hidden_0_name: str, activation: TransformationFunction = <neuralogic.core.constructs.function.function.TransformationFunction object>, arity: int = 1)

Bases: Module

One-layer Recurrent Neural Network (RNN) module which is computed as:

\[h_t = act(\mathbf{W}_{ih} \mathbf{x}_t + \mathbf{W}_{hh} \mathbf{h}_{t-1})\]

where \(t \in (1, sequence\_length + 1)\) is a time step. In the template, the \(t\) is referred to as V.T, and \(t - 1\) is referred to as V.Z. This module expresses the first equation as:

(R.<output_name>(<...terms>, V.T) <= (
    R.<input_name>(<...terms>, V.T)[<hidden_size>, <input_size>],
    R.<hidden_input_name>(<...terms>, V.Z)[<hidden_size>, <hidden_size>],
    R.special.next(V.Z, V.T),
)) | [<activation>]

R.<output_name> / <arity> + 1 | [Transformation.IDENTITY]

Additionally, we define a rule for the “stop condition”, that is:

(R.<output_name>(<...terms>, 0) <= R.<hidden_0_name>(<...terms>)) | [Transformation.IDENTITY]

Parameters:

• input_size (int) – Input feature size.

• hidden_size (int) – Output and hidden feature size.

• output_name (str) – Output (head) predicate name of the module.

• input_name (str) – Input feature predicate name to get features from.

• hidden_0_name (str) – Predicate name to get initial hidden state from.

• activation (TransformationFunction) – Activation function. Default: Transformation.TANH

• arity (int) – Arity of the input and output predicate. Default: 1

class RNNCell(input_size: int, hidden_size: int, output_name: str, input_name: str, hidden_input_name: str, activation: TransformationFunction = <neuralogic.core.constructs.function.function.TransformationFunction object>, arity: int = 1)

Bases: Module

Parameters:

• input_size (int) – Input feature size.

• hidden_size (int) – Output and hidden feature size.

• output_name (str) – Output (head) predicate name of the module.

• input_name (str) – Input feature predicate name to get features from.

• hidden_input_name (str) – Predicate name to get hidden state from.

• activation (TransformationFunction) – Activation function. Default: Transformation.TANH

• arity (int) – Arity of the input and output predicate. Default: 1

neuralogic.nn.module.general.rvnn module

class RvNN(input_size: int, output_name: str, input_name: str, parent_map_name: str, max_children: int = 2, activation: TransformationFunction = <neuralogic.core.constructs.function.function.TransformationFunction object>, aggregation: AggregationFunction = <neuralogic.core.constructs.function.function.AggregationFunction object>, arity: int = 1)

Bases: Module

Recursive Neural Network (RvNN) module which is computed as:

\[\mathbf{h}_i = act(agg_{j \in \mathcal{Ch(i)}}(\mathbf{W_{id(j)}} \mathbf{h}_j))\]

Where \(act\) is an activation function, \(agg\) aggregation function and \(\mathbf{W}\)’s are learnable parameters. \(\mathcal{Ch(i)}\) represents the ordered list of children of node \(i\). The \(id(j)\) function maps node \(j\) to its index (position) in its parent’s children list.

Parameters:

• input_size (int) – Input feature size.

• output_name (str) – Output (head) predicate name of the module.

• input_name (str) – Input feature predicate name to get leaf features from.

• parent_map_name (str) – Name of the predicate to get mapping from parent to children

• max_children (int) – Maximum number of children (specify which <max_children>-ary tree will be considered). Default: 2

• activation (TransformationFunction) – Activation function of all layers. Default: Transformation.TANH

• aggregation (AggregationFunction) – Aggregation function of a layer. Default: Aggregation.SUM

• arity (int) – Arity of the input and output predicate (doesn’t include the node id term). Default: 1

neuralogic.nn.module.general.transformer module

class EncoderBlock(input_dim: int, num_heads: int, dim_feedforward: int, output_name: str, query_name: str, key_name: str, value_name: str, mask_name: str | None = None, arity: int = 1, mlp: bool = True)

Bases: Module

class Transformer(input_dim: int, num_heads: int, dim_feedforward: int, output_name: str, src_name: str, tgt_name: str, src_mask_name: str | None = None, tgt_mask_name: str | None = None, memory_mask_name: str | None = None, arity: int = 1)

Bases: Module

A transformer module based on “Attention Is All You Need”.

Parameters:

• input_dim (int) – The number of expected features.

• num_heads (int) – The number of heads in the multi-head attention module.

• dim_feedforward (int) – The dimension of the feedforward network.

• output_name (str) – Output (head) predicate name of the module.

• src_name (str) – The name of the predicate of the input to the encoder.

• tgt_name (str) – The name of the predicate of the input to the decoder.

• src_mask_name (str, optional) – The name of the predicate of the encoder input mask. Default: None

• tgt_mask_name (str, optional) – The name of the predicate of the decoder input mask. Default: None

• memory_mask_name (str, optional) – The name of the predicate of the encoder output mask. Default: None

• arity (int) – Arity of the input and output predicate. Default: 1

class TransformerDecoder(input_dim: int, num_heads: int, dim_feedforward: int, output_name: str, input_name: str, encoder_name: str, mask_name: str | None = None, memory_mask_name: str | None = None, arity: int = 1)

Bases: Module

A transformer decoder module based on “Attention Is All You Need”.

Parameters:

• input_dim (int) – The number of expected features.

• num_heads (int) – The number of heads in the multi-head attention module.

• dim_feedforward (int) – The dimension of the feedforward network.

• output_name (str) – Output (head) predicate name of the module.

• input_name (str) – The name of the predicate of the input sequence.

• input_name – The name of the input encoder.

• mask_name (str, optional) – The name of the predicate of the decoder input sequence mask. Default: None

• memory_mask_name (str, optional) – The name of the predicate of the encoder output mask. Default: None

• arity (int) – Arity of the input and output predicate. Default: 1

class TransformerEncoder(input_dim: int, num_heads: int, dim_feedforward: int, output_name: str, input_name: str, mask_name: str | None = None, arity: int = 1)

Bases: EncoderBlock

A transformer encoder module based on “Attention Is All You Need”.

Parameters:

• input_dim (int) – The number of expected features.

• num_heads (int) – The number of heads in the multi-head attention module.

• dim_feedforward (int) – The dimension of the feedforward network.

• output_name (str) – Output (head) predicate name of the module.

• input_name (str) – The name of the predicate of the input sequence.

• mask_name (str, optional) – The name of the predicate of the input sequence mask. Default: None

• arity (int) – Arity of the input and output predicate. Default: 1

Module contents