Tensor Objects and Operaitons

A Tensor represents a multi-dimensional matrix where all elements must share the same data type. PyTorch supports floating-point, signed integer, and unsigned integer types, which can reside on either CPU or GPU devices. The dtype attribute specifies the data type, while device determines the hardware location.

import torch
import numpy as np

# Default data types differ between torch.Tensor and torch.tensor
print(f'torch.Tensor default: {torch.Tensor(1).dtype}')
print(f'torch.tensor default: {torch.tensor(1).dtype}')

# Creating tensors from lists
matrix_a = torch.tensor([[1, 2], [3, 4]], dtype=torch.float64)

# Creating tensors from numpy arrays
numpy_arr = np.array([[1, 2], [3, 4]])
matrix_b = torch.tensor(numpy_arr, dtype=torch.uint8)
print(matrix_a)
print(matrix_b)

# Specifying device placement
cuda_device = torch.device('cuda:0')
ones_matrix = torch.ones((2, 2), device=cuda_device)
print(ones_matrix)

Arithmetic operations on tensors perform element-wise computations between matrices. Key operations include: torch.mm for matrix multiplication, torch.clamp for cilpping values to a specified range, torch.round for rounding to integers, and torch.tanh for hyperbolic tangent activation mapping values to (0, 1).

PyTorch provides factory functions for direct tensor creation. torch.arange mirrors Python's range with configurable step size, while torch.linspace generates evenly spaced values based on a count parameter. torch.ones and torch.zeros create uniform matrices.

Random tensor generation includes: torch.rand for uniform distribution in [0,1], torch.randn for standard normal distribution, and torch.randint for random integers within a specified range.

Tensor Indexing and Slicing

PyTorch tensors support standard indexing, slicing, and advanced indexing operations similar to NumPy.

import torch

# Basic indexing
tensor_a = torch.arange(9).view(3, 3)
print(tensor_a[2, 2])

# Slicing operations
print(tensor_a[1:, :-1])

# Strided slicing
print(tensor_a[::2])
print(tensor_a[::2, ::2])

# Integer array indexing
rows = [0, 1]
cols = [2, 2]
print(tensor_a[rows, cols])

# Boolean indexing
mask = tensor_a > 4
print(mask)
print(tensor_a[mask])

torch.nonzero returns indices of non-zero elements.

large_tensor = torch.arange(16).view(4, 4)
indices = torch.nonzero(large_tensor >= 8)
print(indices)

binary_tensor = torch.randint(0, 2, (3, 3))
nonzero_indices = torch.nonzero(binary_tensor)
print(nonzero_indices)

torch.where(condition, x, y) selects elements from x or y based on condition:

x = torch.randn(3, 2)
y = torch.ones(3, 2)
print(x)
result = torch.where(x > 0, x, y)
print(result)

Tensor Transformation, Concatenation, and Splitting

Metadata inspection methods:

a = torch.rand(1, 2, 3, 4, 5)
print(f'Element count: {a.nelement()}')
print(f'Axis count: {a.ndimension()}')
print(f'Shape: {a.size()}, {a.shape}')

Both Tensor.reshape and Tensor.view modify tensor dimensions. The critical distinction: view requires contiguous memory layout or raises an error, while reshape handles non-contiguous tensors. view always returns a view of the original data (changes affect source), whereas reshape may return either a view or copy. Both preserve element count and support -1 for automatic dimension inference.

a = torch.rand(1, 2, 3, 4, 5)
b = a.view(2 * 3, 4 * 5)
print(b.shape)
c = a.reshape(-1)
print(c.shape)
d = a.reshape(2 * 3, -1)
print(d.shape)

torch.squeeze removes dimensions of size 1, while torch.unsqueeze inserts a size-1 dimension at a specified position.

a = torch.rand(1, 2, 3, 4, 5)
b = torch.squeeze(a)
print(b.shape)
c = torch.unsqueeze(b, 1)
print(c.shape)

For 2D matrices, torch.t and torch.transpose perform transposition. torch.t serves as a simplified version of torch.transpose:

a = torch.tensor([[2]])
b = torch.tensor([[2, 3]])
print(torch.transpose(a, 1, 0))
print(torch.t(a))
print(torch.transpose(b, 1, 0))
print(torch.t(b))

Higher-dimensional tensors use permute for dimension reordering:

a = torch.rand((1, 224, 224, 3))
print(a.shape)
b = a.permute(0, 3, 1, 2)
print(b.shape)

PyTorch offers two concatenation methods. torch.cat(tensors, dim) merges tensors along an existing dimension where only the concatenation dimension may differ in size. torch.stack(tensors, dim) requires all tensors to share identical dimensions and creates a new dimension at the specified index:

a = torch.randn(2, 3)
b = torch.randn(3, 3)
c = torch.cat((a, b))
d = torch.cat((b, b, b), dim=1)
print(c.shape)
print(d.shape)

For splitting, torch.chunk(input, chunks, dim) divides tensors into equal parts; the final chunk may be smaller if the dimension isn't evenly divisible. torch.split() provides enhanced functionality—besides equal partitioning, it accepts a list specifying exact split sizes:

import torch

a = torch.randn(10, 3)

# Equal chunks
chunks = torch.chunk(a, 3, dim=0)
print(len(chunks))

# Custom split sizes
result = torch.split(a, [2, 3, 5], dim=0)
print([t.shape for t in result])

PyTorch Reduction Operations

Reduction operations aggregate tensor values, typically producing scalars or reduced-dimension tensors. Common reduction functions include torch.sum, torch.mean, torch.prod, torch.min, torch.max, and torch.std. These operations support dim parameters for specifying reduction axes and keepdim for preserving output dimensionality.

x = torch.randn(4, 5)
print(torch.sum(x))
print(torch.sum(x, dim=0))
print(torch.mean(x, dim=1, keepdim=True))
print(torch.max(x, dim=1))
print(torch.min(x, dim=0))

Logical reductions like torch.all and torch.any evaluate conditions across specified dimensions.

condition = torch.tensor([[True, False], [True, True]])
print(torch.all(condition))
print(torch.any(condition, dim=1))

PyTorch Automatic Differentiation

The autograd module enables automatic gradient computation for neural network training. Setting requires_grad=True on tensors tracks operations for gradient calculation.

x = torch.tensor([1.0, 2.0, 3.0], requires_grad=True)
y = x ** 2 + 2 * x + 1
loss = y.sum()
loss.backward()
print(x.grad)

The computational graph builds dynamically, with backward() propagating gradients through the graph. The grad attribute accumulates gradients on leaf tensors. For gradient computation without updating parameters, use torch.no_grad() context or detach tensors with .detach() to halt tracking.

x = torch.tensor([2.0], requires_grad=True)
y = x ** 3
z = y.detach()  # Creates new tensor without grad tracking
w = x ** 2

with torch.no_grad():
    frozen_result = x * 2

Higher-order derivatives require create_graph=True in backward calls. Gradient accumulation occurs across multiple backward passes—call .zero_grad() before each iteration to reset gradients.