Linear algebra

NumPy includes many functions for linear algebra in the numpy.linalg package.

import numpy as np

Multiplication

There are several ways to multiply two vectors. The most natural way is the element-wise product. This works precisely how it sounds: multiply two vectors of the same size element-by-element.

\[\begin{split} \mathbf{x} * \mathbf{y} = \begin{bmatrix} x_1 y_1 \\ \vdots \\ x_{N} y_{N} \\ \end{bmatrix} \end{split}\]

In numpy, this operation is performed by the operator *.

x = np.array([2.1, -5.7, 13])
y = np.array([4.3,  9.2, 13])

p = x * y

print(p)

[  9.03 -52.44 169.  ]

Scalar product

The scalar product is another way to multiply two vectors of the same dimension. Here is how it works: multiply the vectors element-by-element, then add up the result. For two generic column vectors \(\mathbf{x}\in\mathbb{R}^N\) and \(\mathbf{y}\in\mathbb{R}^N\), the scalar product is mathematically defined as

\[\begin{split} \mathbf{x}^\top \mathbf{y} = \begin{bmatrix}x_1 & \dots & x_{N}\end{bmatrix} \begin{bmatrix}y_1\\\vdots\\y_{N}\end{bmatrix} = x_1 y_1 + \dots + x_{N} y_{N} = \sum_{n=1}^{N} x_n y_n. \end{split}\]

NumPy provides the operator @ and the function np.matmul() for matrix products.

dot = x @ y

print("  x:", x)
print("  y:", y)
print("x@y:", dot)

  x: [ 2.1 -5.7 13. ]
  y: [ 4.3  9.2 13. ]
x@y: 125.59

Note

With two 1D arrays, x @ y returns their inner product, which is a scalar. Use np.outer(x, y) for the outer product, which is a matrix of shape (len(x), len(y)).

Matrix-vector product

The operator @ is also capable of performing matrix-vector or vector-matrix products. Assume that \(A\in\mathbb{R}^{N\times K}\), \({\bf b}\in\mathbb{R}^{K}\), and \({\bf c}\in\mathbb{R}^{N}\):

\[\begin{split} A = \begin{bmatrix} a_{1,1} & \dots & a_{1,K}\\ \vdots & & \vdots\\ a_{N,1} & \dots & a_{N,K}\\ \end{bmatrix} \qquad\qquad {\bf b} = \begin{bmatrix}b_1\\\vdots\\b_{K}\end{bmatrix} \qquad\qquad {\bf c} = \begin{bmatrix}c_1\\\vdots\\c_{N}\end{bmatrix} \end{split}\]

Mathematically, the exact operation performed by @ depends on the order of inputs.

• A @ b multiplies the vector \({\bf b}\) by the rows of \(A\).

\[\begin{split} A{\bf b} = \begin{bmatrix} \sum_{k=1}^K a_{1,k} b_k \\ \vdots\\ \sum_{k=1}^K a_{N,k} b_k \\ \end{bmatrix} \end{split}\]

• c @ A multiplies the vector \({\bf c}\) by the columns of \(A\).

\[ {\bf c}^\top A = \left[ \sum_{n=1}^N a_{n,1} c_n \quad\dots\quad \sum_{n=1}^N a_{n,K} c_n \right] \]

The result is a vector in both cases.

A = np.array([[1,3,1],[2,5,1]])
b = np.array([1, 1, 1])
c = np.array([1, 1])

Ab = A @ b
cA = c @ A

print("--- A ---")
print(A)
print("--- b ---")
print(b)
print()
print("A @ b:", Ab)
print("c @ A:", cA)

--- A ---
[[1 3 1]
 [2 5 1]]
--- b ---
[1 1 1]

A @ b: [5 8]
c @ A: [3 8 2]

Matrix product

When both inputs are matrices, the operator @ performs a matrix multiplication. Specifically, given two matrices \(A\in\mathbb{R}^{M\times N}\) and \(B\in\mathbb{R}^{N\times K}\), their product is equal to the scalar product between the rows of \(A\) and the columns of \(B\):

\[\begin{split} AB = \begin{bmatrix} \_\!\_\; {\bf a}_1^\top \,\_\!\_ \\ \vdots\\ \_\!\_\; {\bf a}_M^\top \,\_\!\_ \\ \end{bmatrix} \begin{bmatrix} | & & |\\[-1em] {\bf b}_1 & \dots & {\bf b}_K\\ | & & |\\ \end{bmatrix} = \begin{bmatrix} {\bf a}_1^\top{\bf b}_1 & \dots & {\bf a}_1^\top{\bf b}_K\\ \vdots & & \vdots\\ {\bf a}_M^\top{\bf b}_1 & \dots & {\bf a}_M^\top{\bf b}_K \end{bmatrix}. \end{split}\]

X = np.array([[1,3,1],[2,5,1]])
Y = np.array([[5,1],[9,2],[14,1]])

Z = X @ Y

print("--- X ---")
print(X)
print("--- Y ---")
print(Y)
print("--- X @ Y ---")
print(Z)

--- X ---
[[1 3 1]
 [2 5 1]]
--- Y ---
[[ 5  1]
 [ 9  2]
 [14  1]]
--- X @ Y ---
[[46  8]
 [69 13]]

Function np.dot()

Numpy also provides the function np.dot() for computing the dot product of two arrays. However, its behavior differs from @ for arrays with more than two dimensions. Prefer @ or np.matmul() for linear-algebra products: leading dimensions are treated as batch dimensions and broadcast together, while the final two dimensions are matrix dimensions. Instead, np.dot() contracts the last axis of the first input with the second-to-last axis of the second input. For explicit inner, outer, and elementwise products, use np.inner, np.outer, and *, respectively.