Pytorch 张量计算

什么是 Tensor?

张量（Tensor）是物理的语言，在力学和广义相对论等领域发挥着不可替代的作用。但是在深度学习中，张量是基础数据结构，可以看作是多维数据容器。

现代深度学习框架（如 TensorFlow、PyTorch）的核心数据结构就是张量，所有模型运算（卷积、全连接、注意力机制）本质上都是张量操作：

• 卷积神经网络（CNN）通过二维张量卷积提取图像局部特征。
• Transformer 模型利用张量的多头注意力机制处理序列数据的长距离依赖。
• 张量分解（如 CP 分解、Tucker 分解）可用于模型压缩、特征降维，解决高维数据的 “维度灾难” 问题。

学习张量计算能帮助理解深度学习的底层数学逻辑，而非仅停留在 API 调用层面。

Attention: 本文假设读者拥有使用Python的经验，不对基础语法进行解释, 在教学部分使用英文教学。

为何选择 PyTorch?

作为主流深度学习框架，PyTorch 以简洁易用、动态计算图特性著称，兼顾科研灵活性与工业落地能力：

• 对初学者友好，API 设计直观，能快速上手搭建神经网络。
• 生态丰富，覆盖计算机视觉、自然语言处理、强化学习等多领域，适配科研实验与实际项目开发。
• 深度集成 Python 生态，支持即时调试，大幅降低模型开发与优化成本。

PyTorch的安装与引入

安装PyTorch:

首先进入PyTorch官网中的GetStart页面，我们通过pip包管理器来安装PyTorch。
对于版本，选择Stable版本即可，通常来说系统的选择不影响pip的安装，所以我们不做解释，对于语言我们选择Python语言，而版本，选择最新即可。

复制Run this Command下的内容，比如我这里是：

pip3 install torch torchvision --index-url https://download.pytorch.org/whl/cu126

这时候打开你的venv环境或者在终端下粘贴即可安装。

引入PyTorch:

使用torch.__version__来查看PyTorch的版本。
使用torch.cuda.is_available()来查看当前PyTorch是否支持Cuda计算。
什么是Cuda，我们可以把它理解为一个在GPU上加速计算的一个工具：

By Nvidia Cuda Zone:
CUDA® is a parallel computing platform and programming model developed by NVIDIA for general computing on graphical processing units (GPUs)
With CUDA, developers are able to dramatically speed up computing applications by harnessing the power of GPUs.

import torch
print(torch.__version__)
print(torch.cuda.is_available())

这将会输出类似的内容：

2.8.0+cu126
True

开始学习PyTorch

Before we begin our studies

This function allows us to quickly get the properties of PyTorch tensors.
We will not explain the purpose of this function in subsequent content.

def get_tensor_info(p_type : str, p_tensor : torch.Tensor) :
    print(p_type, "Tensor belong", p_tensor.ndim, "Dimension with Data Type", p_tensor.dtype, "On", p_tensor.device)
    print("and the Shape of", p_type, "is", p_tensor.shape)

PyTorch Tensor Unit

Before we continue, we need to understand PyTorch’s Dimension and Shape.
It is not a comprehensive data structure reference, as tensors may also have other data structures.
Therefore, we only present the key properties of tensors relevant to this chapter.

Property	Description	Examples
Tensor Dimension	Return this tensor’s dimension	tensor.ndim
	Scalar: 0 Dim	torch.tensor(5).ndim -> 0
	Vector: 1 Dim	torch.tensor([1,2,3]).ndim -> 1
	Matrix: 2 Dim	torch.tensor([[1,2],[3,4]]).ndim -> 2
Tensor Shape	Return this tensor’s shape (size)	tensor.shape or tensor.size()
	Scalar: will return torch.size([])	torch.tensor(7).shape -> torch.size([])
	Vector: will return length in this vector	torch.tensor([2,3]).shape -> torch.size([2])
	Matrix: will return items in this matrix and length in item	torch.tensor([[1,2,3],[4,5,6]].shape -> torch.size([2,3])

Get a Tensor

How to get a Tensor, and what is the Tensor’s Data Type?
Use torch.tensor() to get a Tensor
Use tensor.dtype to get tensor’s type
Use tensor.shape to get tensor’s shape
Use tensor.device to get tensor’s device

As for what dtype, shape, and device are here, we shall explain these later.

E.g.:

int_tensor = torch.tensor(1) # Get a tensor
get_tensor_info("int_tensor", int_tensor)

# Or Specify the type explicitly
float32_tensor = torch.tensor(1.0, dtype=torch.float32)
get_tensor_info("float32_tensor", float32_tensor)

# Explicit type conversion
float16_tensor = float32_tensor.type(torch.float16)
get_tensor_info("float16_tensor", float16_tensor)

# More additional condition
condition_tensor = torch.tensor([3.0, 6.0, 8.0],
                                dtype=torch.float32, # Data type for this tensor
                                device=torch.device("cuda:0"), # Put this tensor to where
                                requires_grad=True) # Is Track the gradients with this tensor operation

• Tensor’s dtype:
  The official PyTorch documentation provides a detailed list of the possible data types that a tensor can possess: torch.dtype, which we refer to as dtype. The data type defines the precision of a tensor during computations. If you have experience with programming, you will be familiar with concepts such as int and float types; the data types here are analogous. Further details will be discussed later.
  Generally, in PyTorch, the default data type for tensors is float32.

• Tensor’s shape:
  We shall explain it here; this is its ref:What is the Tensor’s Shape

• Tensor’s device:
  The device attribute enables developers to specify where tensors and models are executed, such as on the CPU or GPU. This is crucial for optimising code performance and leveraging hardware resources.
  In this example, we store the condition_tensor on my GPU.

This will output content similar to the following:

int_tensor Tensor belong 0 Dimension with Data Type torch.int64 On cpu
and the Shape of int_tensor is torch.Size([])
float32_tensor Tensor belong 0 Dimension with Data Type torch.float32 On cpu
and the Shape of float32_tensor is torch.Size([])
float16_tensor Tensor belong 0 Dimension with Data Type torch.float16 On cpu
and the Shape of float16_tensor is torch.Size([])

Scalar Tensor

What is the Scalar Tensor:
A scalar is a zero-dimensional tensor representing a single numerical value (such as an integer or floating-point number) without spatial dimensions (ndim=0).
It constitutes one of the most fundamental units within PyTorch’s tensor system, enabling seamless operations with higher-dimensional tensors (vectors, matrices, etc.).

E.g.:

# Scalar
scalar = torch.tensor(7)
# scalar.item() # use this method to convert scalar tensors into Python's native data types
get_tensor_info("Scalar", scalar)

This will output content similar to the following:

Scalar Tensor belong 0 Dimension with Data Type torch.int64 On cpu
and the Shape of Scalar is torch.Size([])

Vector Tensor

What is the Vector Tensor:
A Vector Tensor is a one-dimensional tensor (1-dimensional tensor) corresponding to the mathematical concept of a “vector”.
It consists of an ordered sequence of numerical values and possesses only one dimension (ndim=1).
It serves as an extension of the scalar (zero-dimensional) and forms the foundation for constructing higher-dimensional tensors such as matrices and 3D tensors.

E.g.:

# Vector
vector = torch.tensor([1, 3])
get_tensor_info("Vector", vector)

This will output content similar to the following:

Vector Tensor belong 1 Dimension with Data Type torch.int64 On cpu
and the Shape of Vector is torch.Size([2])

Matrix Tensor

What is the Matrix Tensor:
A Matrix Tensor is a two-dimensional tensor corresponding to the mathematical concept of a “matrix”.
It is a two-dimensional numerical array composed of rows and columns, possessing a dimension of 2 (ndim=2).

E.g.:

# Matrix
matrix = torch.tensor([[1, 2, 3],
                       [4, 5, 6]])
get_tensor_info("Matrix", matrix)

This will output content similar to the following:

Matrix Tensor belong 2 Dimension with Data Type torch.int64 On cpu
and the Shape of Matrix is torch.Size([2, 3])

What is the Tensor’s Shape

We will give an E.g. Tensor:

# Tensor
tensor = torch.tensor([[[1,2,3],
                        [4,5,6],
                        [7,8,9]]])
get_tensor_info("Tensor", tensor)

This will output content similar to the following:

Tensor Tensor belong 3 Dimension with Data Type torch.int64 On cpu
and the Shape of Tensor is torch.Size([1, 3, 3])

Therefore, in PyTorch, the shape of a tensor signifies:
A Tensor’s Shape refers to the number of elements across each dimension of the tensor.
It is represented as a tuple (of type torch.Size, a subclass of tuple) to describe the structure of the tensor (such as “how many rows, how many columns, how many channels”, etc.).

But why the tensor’s shape is torch.Size([1,3,3]) here:
1: Meaning that for the 3Dim tensor, the outer [] layer only contain one [] item
2: the middle [] layer contain three [] item
3: the inner [] layer contain three element item.

t_tensor = torch.tensor(
[                     # ← 第1层括号 → 第1维度（大小=1）
  [                   # ← 第2层括号 → 第2维度（大小=3）
    [1, 2, 3],        # ← 第3层括号 → 第3维度（大小=3）
    [4, 5, 6],
    [7, 8, 9]
  ]
])

Radom Tensor

Why Random Tensor:
Fundamentally, random numbers are indispensable in scenarios such as model training, algorithm validation, and data augmentation.
Amidst the continuous updating of data, our ultimate objective is to arrive at a result that meets our requirements.
Random numbers provide us with a starting point, thereby ensuring that computations commence correctly.

just like:
Start with random numbers -> look at data -> update random numbers -> look at data -> … -> Expected data

E.g.:

random_tensor = torch.randn(3, 4)
print(random_tensor)
get_tensor_info("Random", random_tensor)

This will output content similar to the following:

tensor([[ 0.0503,  0.0958, -0.6131, -0.2491],
        [ 0.0630,  0.0489,  0.4874, -1.5149],
        [-0.0070, -1.4396, -0.1872, -0.0911]])
Random Tensor belong 2 Dimension with Data Type torch.float32 On cpu
and the Shape of Random is torch.Size([3, 4])

Here is one example of the application of random tensors:
In the field of image processing, we can utilise a randomised noise image to approximate the target image.
Through this code, we can generate a 224×224 RGB-channel image tensor.

# Create a Radom Tensor with similar shape to an image tensor
random_image_tensor = torch.randn(size=(224, 224, 3)) # 224 height, 224 width, 3 color (R,G,B)
get_tensor_info("Random Image", random_image_tensor)

This will output content similar to the following:

Random Image Tensor belong 3 Dimension with Data Type torch.float32 On cpu
and the Shape of Random Image is torch.Size([224, 224, 3])

An interesting little experiment
Cool tools, trans the Random Image Tensor to the img

import matplotlib.pyplot as plt

# 步骤1：确保数据类型正确（转换为float32）
if random_image_tensor.dtype != torch.float32:
    image_tensor = random_image_tensor.float()

# 步骤2：归一化像素值到 [0, 1] 范围（如果尚未归一化）
if random_image_tensor.max() > 1.0:
    image_tensor = random_image_tensor / 255.0

# 步骤3：转换为NumPy数组并调整通道顺序
image_np = random_image_tensor.numpy()

# 步骤4：使用Matplotlib显示图像
plt.figure(figsize=(6, 6))
plt.imshow(image_np)
plt.axis('off')  # 隐藏坐标轴
plt.show()

Zero Tensor and One Tensor

Fundamentally: any tensor with all elements set to zero is termed a Zero Tensor, while any tensor with all elements set to one is termed a One Tensor.
They serve numerous purposes, some of which may be quite advanced. However, at their simplest, Zero Tensors are typically employed for parameter initialisation, whereas One Tensors are generally used for element-wise operations and normalisation.

E.g.:

zero_tensor = torch.zeros(3,4)
print(zero_tensor)

one_tensor = torch.ones(3,4)
print(one_tensor)

This will output content similar to the following:

tensor([[0., 0., 0., 0.],
        [0., 0., 0., 0.],
        [0., 0., 0., 0.]])
tensor([[1., 1., 1., 1.],
        [1., 1., 1., 1.],
        [1., 1., 1., 1.]])

Range of Tensor

Occasionally, we may need to generate tensors within a specific range or with a specific step.
This can be achieved as follows:
E.g.:

range_tensor = torch.arange(0,10)
print(range_tensor)
# Or addition condition
range_addition_tensor = torch.arange(start=0, end=10, step=2)
print(range_addition_tensor)

This will output content similar to the following:

tensor([0, 1, 2, 3, 4, 5, 6, 7, 8, 9])
tensor([0, 2, 4, 6, 8])

Tensors-Like

Tensors-Like enables you to swiftly create a new tensor based on the shape, data type, and device of an existing reference tensor, without requiring manual specification of these attributes.
E.g.:

# Tensors-Like
# Create a tensor like what you choose tensor
zero_like_tensor = torch.zeros_like(range_tensor)
print(zero_like_tensor)

This will create a new vector with the same properties as the range_tensor attribute from the previous section Range of Tensor, without requiring us to redefine the attribute.

This will output content similar to the following:

tensor([0, 0, 0, 0, 0, 0, 0, 0, 0, 0])

Tensor’s Operations

Make sure that calculate tensor with this:

• The right Data Type
• The right Device
• The right Shape (like: float16 tensor * float16 tensor)

The rules governing tensor operations bear a striking resemblance to those of matrix operations.
Should you have studied linear algebra, you will be thoroughly familiar with them.

The following code examples illustrate the expressions for addition, multiplication, and subtraction operations on a tensor, along with their results, enabling us to grasp these concepts quickly.

Please note that we have not included element-wise division or inverse operations on two-dimensional tensors (i.e., matrices).

E.g.:

op_tensor = torch.tensor([1.5,2.4,6.9])
print("Tensor:", op_tensor)

op_plus_tensor = op_tensor + 10 # Or use: op_plus_tensor = torch.add(op_tensor, 10)
print("Plus 10 to tensor:", op_plus_tensor)

op_mul_tensor = op_tensor * 10 # Or use: op_mul_tensor = torch.mul(op_tensor, 10)
print("Mul 10 to tensor:", op_mul_tensor)

op_sub_tensor = op_tensor - 10 # Or use: op_sub_tensor = torch.sub(op_tensor, 10)
print("Sub 10 to tensor:", op_sub_tensor)

This will output content similar to the following:

Tensor: tensor([1.5000, 2.4000, 6.9000])
Plus 10 to tensor: tensor([11.5000, 12.4000, 16.9000])
Mul 10 to tensor: tensor([15., 24., 69.])
Sub 10 to tensor: tensor([-8.5000, -7.6000, -3.1000])

Conditions for Matrix Multiplication

If you have not studied linear algebra, here is a brief introduction to the rules governing matrix multiplication.
Dimensional representation

• Let the dimensions of matrix A be m × n.
• Let the dimensions of matrix B be p × q.
• Conditions for multiplication: n = p

Dimensional Properties

• Number of rows in C = Number of rows in A
• Number of columns in C = Number of columns in B
• Dimension formula: (m × n) · (n × p) = (m × p)

Example Explanation

Matrix A Dimensions	Matrix B Dimensions	Can Be Multiplied	Result Dimensions
2×3	3×4	✅ Yes	2×4
4×2	3×4	❌ No	-
5×5	5×1	✅ Yes	5×1
3×2	2×3	✅ Yes	3×3

Matrix multiplication comprises two types: Cross multiplication and Dot multiplication:
Dot multiplication: The dot product of two vectors is obtained by multiplying corresponding elements and summing the results, yielding a scalar quantity denoted as a·b.
Cross multiplication: The cross product of two three-dimensional vectors is a new vector, denoted as axb.

E.g.:

# Wise multiplication axb (aka. Cross Multiplication)
# Simply use the * operator
op_wisemul_tensor = op_tensor * op_tensor
print("CrossMul tensor x tensor:", op_wisemul_tensor)

# Matrix Multiplication a.b (aka. Dot Product)
op_matmul_tensor = torch.matmul(op_tensor, op_tensor)
print("MatMul tensor . tensor:", op_matmul_tensor)

This will output content similar to the following:

CrossMul tensor x tensor: tensor([ 2.2500,  5.7600, 47.6100])
MatMul tensor . tensor: tensor(55.6200)

Matrix Transpose

Matrix transposition involves swapping the rows and columns of a matrix.
Its primary function is to rearrange the dimensions of the matrix to accommodate computational requirements or align with data formats.

E.g.:

# Use tensor.T to get this matrix's Transpose matrix
before_tensor = torch.tensor([[1.5,2.4,6.9],
                             [2.4,8.3,9.2]])
after_tensor = before_tensor.T
print("Before Transpose:\n", before_tensor, "\nShape:", before_tensor.shape)
print("After Transpose:\n", after_tensor, "\nShape:", after_tensor.shape)

This will output content similar to the following:

Before Transpose:
 tensor([[1.5000, 2.4000, 6.9000],
        [2.4000, 8.3000, 9.2000]]) 
Shape: torch.Size([2, 3])
After Transpose:
 tensor([[1.5000, 2.4000],
        [2.4000, 8.3000],
        [6.9000, 9.2000]]) 
Shape: torch.Size([3, 2])

Tensor Aggregation

For data analysis, simplifying data, or summarising statistical characteristics, we can use tensor aggregation operations: minimum, maximum, mean aka. avg, and sum.

E.g.:

agg_tensor = torch.tensor([1.5,2.4,6.9,42,6.4,25,1])
print(agg_tensor)
agg_min_tensor = agg_tensor.min() # Or use: agg_min_tensor = torch.min(agg_tensor)
print("Tensor's Min elem:", agg_min_tensor)
agg_max_tensor = agg_tensor.max() # Or use: agg_max_tensor = torch.max(agg_tensor)
print("Tensor's Max elem:", agg_max_tensor)
agg_avg_tensor = agg_tensor.mean() # Or use: agg_avg_tensor = torch.mean(agg_tensor)
print("Tensor's Mean(Avg):", agg_avg_tensor)
agg_sum_tensor = agg_tensor.sum() # Or use: agg_sum_tensor = torch.sum(agg_tensor)
print("Tensor's Sum:", agg_sum_tensor)

This will output content similar to the following:

tensor([ 1.5000,  2.4000,  6.9000, 42.0000,  6.4000, 25.0000,  1.0000])
Tensor's Min elem: tensor(1.)
Tensor's Max elem: tensor(42.)
Tensor's Mean(Avg): tensor(12.1714)
Tensor's Sum: tensor(85.2000)

Tensor’s item position

Indexing corresponding values in a tensor, such as retrieving the indices (positions) of the minimum and maximum values in the following code:
E.g.:

# Find min item index
min_pos = agg_tensor.argmin()
print("Tensor's minimal item", agg_tensor[min_pos], "with index:", min_pos)
max_pos = agg_tensor.argmax()
print("Tensor's maximum item", agg_tensor[max_pos], "with index:", max_pos)

This will output content similar to the following:

Tensor's minimal item tensor(1.) with index: tensor(6)
Tensor's maximum item tensor(42.) with index: tensor(3)

Reshape

Reshape: By adjusting the dimensionality of a tensor, its shape is altered whilst preserving the total number of elements, thereby reorganising its dimensions.

E.g.:

# Reshape the input tensor to a defined shape
x_tensor = torch.arange(0, 10).float()
print(x_tensor, x_tensor.shape)
x_reshape_tensor = x_tensor.reshape(2,5) # tensor.reshape(n,m) -> n*m must be equal to torch size
print(x_reshape_tensor)

This will output content similar to the following:

tensor([0., 1., 2., 3., 4., 5., 6., 7., 8., 9.]) torch.Size([10])
tensor([[0., 1., 2., 3., 4.],
        [5., 6., 7., 8., 9.]])

View

Its core functionality is similar to reshape() — adjusting the dimensionality of a tensor while preserving the total number of elements — but it imposes a crucial restriction: it is only applicable to “memory-contiguous” tensors.
If, after creating a tensor, its memory is contiguous, then view() can be used immediately.
However, if other operations are performed, such as transposition or permuting, once the tensor becomes non-contiguous, view() cannot be used. Instead, reshape() must be employed.

Concurrently, another interpretation is that this bears some resemblance to references in C++.
For instance, in the following code, we convert x_tensor into a view_tensor. Thereafter, our subsequent operations will also affect x_tensor.

E.g.:

x_view_tensor = x_tensor.view(1,10) # now x_view_tensor linked with x_tensor
print(x_view_tensor)

x_view_tensor[:,0] = 123
print(x_view_tensor,"\n", x_tensor) # also changed the x_tensor

This will output content similar to the following:

tensor([[0., 1., 2., 3., 4., 5., 6., 7., 8., 9.]])
tensor([[123.,   1.,   2.,   3.,   4.,   5.,   6.,   7.,   8.,   9.]]) 
 tensor([123.,   1.,   2.,   3.,   4.,   5.,   6.,   7.,   8.,   9.])

Stack Tensor

torch.stack() is used to concatenate multiple tensors of the same shape along a newly created dimension.
It adds a new dimension and sequentially ‘stacks’ the input tensors along this new dimension.

The following code creates a tensor with a range from 0 to 4, which we stack horizontally and vertically using torch.stack():

E.g.:

# Stack tensors together
tensor = torch.arange(0, 5).float()
stack_tensor_H = torch.stack([tensor, tensor, tensor],dim=1)  # horizontally stack
stack_tensor_V = torch.stack([tensor, tensor, tensor],dim=0)  # vertically stack
print("HStack:\n", stack_tensor_H, "\nVStack:\n", stack_tensor_V)

This will output content similar to the following:

HStack:
 tensor([[0., 0., 0.],
        [1., 1., 1.],
        [2., 2., 2.],
        [3., 3., 3.],
        [4., 4., 4.]]) 
VStack:
 tensor([[0., 1., 2., 3., 4.],
        [0., 1., 2., 3., 4.],
        [0., 1., 2., 3., 4.]])

Squeeze & Unsqueeze

torch.squeeze() is an operation designed to remove all dimensions of size 1 from a tensor.
Its core function is to simplify the tensor structure by eliminating those “superfluous” dimensions containing only a single element (i.e., dimensions of size 1), thereby rendering the tensor shape more compact.

Conversely, torch.unsqueeze() is used to insert a new dimension of size 1 at a specified position (dimension) within a tensor.
This operation does not alter the number of elements in the tensor, merely increasing the number of dimensions.

E.g.:

# Delete the tensor's 1 dim
z_tensor = torch.tensor([[1,2,3,1,2,2]]).float()
print("Before Squeeze:", z_tensor, "\n", z_tensor.shape)
z_squeeze_tensor = torch.squeeze(z_tensor)
print("After Squeeze:", z_squeeze_tensor, "\n", z_squeeze_tensor.shape)
# Unsqueeze
# Add 1 dim to tensor
z_unsqueeze_tensor = torch.unsqueeze(z_squeeze_tensor, dim=0)
print("After Unsqueeze:", z_unsqueeze_tensor, "\n", z_unsqueeze_tensor.shape)

This will output content similar to the following:

Before Squeeze: tensor([[1., 2., 3., 1., 2., 2.]]) 
 torch.Size([1, 6])
After Squeeze: tensor([1., 2., 3., 1., 2., 2.]) 
 torch.Size([6])
After Unsqueeze: tensor([[1., 2., 3., 1., 2., 2.]]) 
 torch.Size([1, 6])

Permute

torch.permute() is an operation used to rearrange the dimension order of a tensor. We shall illustrate this with a simple example using the code below. Initially, we created a tensor with a resolution of 224×224 (x, y) and an RGB channel (z).
Should we then require the colour channel to be placed at the front, resulting in the order (z, x, y), we can employ permute to swiftly swap the multidimensional data.

E.g.:

img_tensor = torch.randn(size=(224, 224, 3)) # 224 height, 224 width, 3 color (R,G,B)
permute_img_tensor = img_tensor.permute(2, 0, 1) # Shift to 3 color, 224 height, 224 width
print("Permute(Shift)", img_tensor.shape, "to", permute_img_tensor.shape)

This will output content similar to the following:

Permute(Shift) torch.Size([224, 224, 3]) to torch.Size([3, 224, 224])