Tensors — the core data structure in deep learning — generalize vectors and matrices to higher diemnsions. Frameworks like PyTorch, TensorFlow, and MXNet provide tensor types (Tensor in PyTorch/TensorFlow, ndarray in MXNet) that closely resemble NumPy's ndarray, but extend it with critical capabilities such as GPU acceleration and automatic differentiation.

Creating Tensors

A tensor is defined by three primary attributes:

• Shape: dimensions (e.g., a 3×4 matrix),
• Data type: e.g., torch.float32,
• Values: initialized as zeros, ones, random values, or explicit data.

import torch

# Element-count aware vector: shape (12,)
x = torch.arange(12)
print(x.shape)     # torch.Size([12])
print(x.numel())   # 12

# Reshape into 3×4 matrix without changing values
X = x.reshape(3, 4)
# Alternative auto-inference: X = x.reshape(-1, 4) or x.reshape(3, -1)

# Initialize with specific values
zeros = torch.zeros((2, 3, 4))
ones = torch.ones((2, 3, 4))
randn = torch.randn(3, 4)  # Standard normal distribution
explicit = torch.tensor([[2., 1., 4., 3.],
                         [1., 2., 3., 4.],
                         [4., 3., 2., 1.]])

Elementwise Operations & Arithmetic

Standard arithmetic operators operate elementwise on tensors of compatible shapes.

a = torch.tensor([1.0, 2.0, 4.0, 8.0])
b = torch.tensor([2.0, 2.0, 2.0, 2.0])

sum_ab = a + b
diff_ab = a - b
prod_ab = a * b
quot_ab = a / b
pow_ab = a ** b  # exponentiation
exp_a = torch.exp(a)  # e^a elementwise

For matrix multiplication, use @ or torch.matmul:

A = torch.randn(2, 3)
B = torch.randn(3, 4)
C = A @ B  # shape (2, 4)

Concatenation and Logical Comparisons

Multiple tensors can be joined along a specified axis:

M1 = torch.arange(12, dtype=torch.float32).reshape(3, 4)
M2 = torch.tensor([[2., 1., 4., 3.],
                   [1., 2., 3., 4.],
                   [4., 3., 2., 1.]])

# Concatenate along rows (axis 0) → shape (6, 4)
concat_0 = torch.cat((M1, M2), dim=0)

# Concatenate along columns (axis 1) → shape (3, 8)
concat_1 = torch.cat((M1, M2), dim=1)

Logical operations produce boolean-valued tensors:

equal_mask = M1 == M2  # 1 where equal, 0 otherwise
total_sum = M1.sum()   # scalar tensor with sum of all elements

Broadcasting

Broadcasting allows elementwise operations on tensors with compatible but differing shapes by implicitly replicating data along size-1 axes.

A = torch.arange(3).reshape(3, 1)  # shape (3, 1)
B = torch.arange(2).reshape(1, 2)  # shape (1, 2)

C = A + B  # Broadcasts to shape (3, 2)
# Internally:
# A becomes [[0,0], [1,1], [2,2]]
# B becomes [[0,1], [0,1], [0,1]]

Broadcasting rules require that for each dimension, sizes must either match or one of them is 1.

Indexing and Assignment

Standard indexing and slicing apply to tansors:

X = torch.arange(12).reshape(3, 4).float()

last_row = X[-1]              # last row
submatrix = X[1:3]           # rows 1 and 2 (0-indexed)

X[1, 2] = 9                   # assign single element
X[0:2, :] = 12               # assign entire rows

In-place Operations and Memory Efficiency

Modifying a tensor in-place avoids unnecessary allocations:

# Non-in-place: creates new tensor, Y loses original reference
 erst = id(Y)
 Y = Y + X
 assert id(Y) != erst  # different memory

# In-place via slice assignment
Z = torch.zeros_like(Y)
id_Z_start = id(Z)
Z[:] = X + Y
assert id(Z) == id_Z_start  # same reference

# Or directly update
orig_id_X = id(X)
X += Y  # modifies X in-place
assert id(X) == orig_id_X

⚠️ Use in-place operations carefully: they can interfere with autograd if not used correctly.

Interoperability with NumPy and Python Scalars

Tensors and NumPy arrays share memory (on CPU), enabling efficient convertions:

# Tensor → NumPy array (shares memory if on CPU)
np_array = X.numpy()

# NumPy array → Tensor
tensor_back = torch.tensor(np_array)

# Single-element tensor → Python scalar
value = torch.tensor([3.5])
py_int = int(value)
py_float = float(value)
py_scalar = value.item()

Tensors form the backbone of deep learning workflows by providing a unified, efficient interface for Numerical Linear Algebra, memory management, and hardware acceleration.