Welcome to our Support Center
Updated

PackedAttention

Description

This is the packed version of Attention.

 

 

Sequences in one batch usually don’t have same length and they are padded to have same length, e.g., below is a batch with 3 sequences and tokens* are padded. Sequence_0: 0, 1*, 2*, 3* Sequence_1: 4, 5, 6*, 7* Sequence_2: 8, 9, 10, 11

PackedAttention is designed to takes in packed input, i.e., only the real tokens without padding. An input as above will be packed into 3 tensors like below:

  • input ([h0, h4, h5, h8, h9, h10, h11])
  • token_offset: 0, 4, 5, 8, 9, 10, 11, 1*, 2*, 3*, 6*, 7*
  • cumulated_token_count: 0, 1, 1+2, 1+2+4

Input tensors contains the hidden embedding of real tokens. Token_offset records the offset of token in the unpacked input. cumulated_token_count records cumulated length of each sequence length.

The operator only supports BERT like model with padding on right now.

Input parameters

 

specified_outputs_name : array, this parameter lets you manually assign custom names to the output tensors of a node.

 Graphs in : cluster, ONNX model architecture.

input (heterogeneous) – T : object, input tensor with shape (token_count, input_hidden_size).
weights (heterogeneous) – T : object, merged Q/K/V weights with shape (input_hidden_size, hidden_size + hidden_size + v_hidden_size).
bias (heterogeneous) – T : object, bias tensor with shape (hidden_size + hidden_size + v_hidden_size) for input projection.
token_offset (heterogeneous) – M : object, in packing mode, it specifies the offset of each token(batch_size, sequence_length).
cumulative_sequence_length (heterogeneous) – M : object, a tensor with shape (batch_size + 1). It specifies the cumulative sequence length.
relative_position_bias (optional, heterogeneous) – T : object, a tensor with shape (batch_size or 1, num_heads or 1, sequence_length, sequence_length).It specifies the additional bias to QxK’

 Parameters : cluster,

num heads : integer, number of attention heads.
Default value “0”.
qkv hidden sizes : array, hidden dimension of Q, K, V: hidden_size, hidden_size and v_hidden_size.
Default value “empty”.
scale : float, custom scale will be used if specified.
Default value “0”.
 training? : boolean, whether the layer is in training mode (can store data for backward).
Default value “True”.
 lda coeff : float, defines the coefficient by which the loss derivative will be multiplied before being sent to the previous layer (since during the backward run we go backwards).
Default value “1”.

 name (optional) : string, name of the node.

Output parameters

 

output (heterogeneous) – T : object, 2D output tensor with shape (token_count, v_hidden_size).

Type Constraints

T in (tensor(float), tensor(float16)) : Constrain input and output types to float tensors.

M in (tensor(int32)) : Constrain mask index to integer types.

Example

All these exemples are snippets PNG, you can drop these Snippet onto the block diagram and get the depicted code added to your VI (Do not forget to install Deep Learning library to run it).
Table of Contents