Welcome to our Support Center
Updated

QOrderedMatMul

Description

Quantize (Int8) MatMul with order. Implement Y = alpha * A * B + bias + beta * C. Matrix A, B, C, Y are all int8 matrix. Two type of order combination supported: *) When order_B is ORDER_COL, order_A must be ORDER_ROW. bias is vector of {#cols of Y} of float32, C should be batch 1/batch_A. B could be of batch 1 or batch_A. Note B is reorder to ORDER_COL, or Transposed. Not Transposed first and then Reordered here. *) When order_B is specify ORDER_COL4_4R2_8C or ORDER_COL32_2R_4R4, orderA must be ORDER_COL32. MatMul will be implemented using alpha(A * B) + beta * C => Y. bias is not supported here. B in fact is transposed first then reordered into ORDER_COL4_4R2_8C or ORDER_COL32_2R_4R4 here. order_Y and order_C will be same as order_A. Support per column quantized weight, ie, scale_B is 1-D vector of size [#cols of matrix B].

 

Input parameters

 

specified_outputs_name : array, this parameter lets you manually assign custom names to the output tensors of a node.

 Graphs in : cluster, ONNX model architecture.

A (heterogeneous) – Q : object, 3-dimensional matrix A.
scale_A (heterogeneous) – S : object, scale of the input A.
B (heterogeneous) – Q : object, 2-dimensional matrix B. Transposed if order_B is ORDER_COL.
scale_B (heterogeneous) – S : object, scale of the input B. Scalar or 1-D float32.
scale_Y (heterogeneous) – S : object, scale of the output Y.
bias (optional, heterogeneous) – S : object, 1d bias, not scaled with scale_Y.
C (optional, heterogeneous) – Q : object, 3d or 2d matrix C. if 2d expand to 3d first. Shape[0] should be 1 or same as A.shape[0].
scale_C (optional, heterogeneous) – S : object, scale of the input A.

 Parameters : cluster,

 order_A : enum, cublasLt order of matrix A. See the schema of QuantizeWithOrder for order definition.
Default value “ORDER_ROW”.
 order_B : enum, cublasLt order of matrix B.
Default value “ORDER_COL”.
 order_Y : enum, cublasLt order of matrix Y and optional matrix C.
Default value “ORDER_ROW”.
 training? : boolean, whether the layer is in training mode (can store data for backward).
Default value “True”.
 lda coeff : float, defines the coefficient by which the loss derivative will be multiplied before being sent to the previous layer (since during the backward run we go backwards).
Default value “1”.

 name (optional) : string, name of the node.

Output parameters

 

 Y (heterogeneous) – Q : object, matrix multiply results from A * B.

Type Constraints

Q in (tensor(int8)) : Constrain input and output types to int8 tensors.

S in (tensor(float)) : Constrain bias and scales to float32.

Example

All these exemples are snippets PNG, you can drop these Snippet onto the block diagram and get the depicted code added to your VI (Do not forget to install Deep Learning library to run it).
Table of Contents