Subscribe to our newsletter
📬 Receive new lessons straight to your inbox (once a month) and join 40K+ developers in learning how to responsibly deliver value with ML.
We'll import PyTorch and set seeds for reproducibility. Note that PyTorch also required a seed since we will be generating random tensors.
1
2 | import numpy as np
import torch
|
1 | SEED = 1234
|
1
2
3 | # Set seed for reproducibility
np.random.seed(seed=SEED)
torch.manual_seed(SEED)
|
We'll first cover some basics with PyTorch such as creating tensors and converting from common data structures (lists, arrays, etc.) to tensors.
1
2
3
4
5 | # Creating a random tensor
x = torch.randn(2, 3) # normal distribution (rand(2,3) -> uniform distribution)
print(f"Type: {x.type()}")
print(f"Size: {x.shape}")
print(f"Values: \n{x}")
|
1
2
3
4
5 | # Zero and Ones tensor
x = torch.zeros(2, 3)
print (x)
x = torch.ones(2, 3)
print (x)
|
1
2
3
4 | # List → Tensor
x = torch.Tensor([[1, 2, 3],[4, 5, 6]])
print(f"Size: {x.shape}")
print(f"Values: \n{x}")
|
1
2
3
4 | # NumPy array → Tensor
x = torch.Tensor(np.random.rand(2, 3))
print(f"Size: {x.shape}")
print(f"Values: \n{x}")
|
1
2
3
4
5 | # Changing tensor type
x = torch.Tensor(3, 4)
print(f"Type: {x.type()}")
x = x.long()
print(f"Type: {x.type()}")
|
Now we'll explore some basic operations with tensors.
1
2
3
4
5
6 | # Addition
x = torch.randn(2, 3)
y = torch.randn(2, 3)
z = x + y
print(f"Size: {z.shape}")
print(f"Values: \n{z}")
|
1
2
3
4
5
6 | # Dot product
x = torch.randn(2, 3)
y = torch.randn(3, 2)
z = torch.mm(x, y)
print(f"Size: {z.shape}")
print(f"Values: \n{z}")
|
1
2
3
4
5
6
7 | # Transpose
x = torch.randn(2, 3)
print(f"Size: {x.shape}")
print(f"Values: \n{x}")
y = torch.t(x)
print(f"Size: {y.shape}")
print(f"Values: \n{y}")
|
1
2
3
4
5 | # Reshape
x = torch.randn(2, 3)
z = x.view(3, 2)
print(f"Size: {z.shape}")
print(f"Values: \n{z}")
|
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19 | # Dangers of reshaping (unintended consequences)
x = torch.tensor([
[[1,1,1,1], [2,2,2,2], [3,3,3,3]],
[[10,10,10,10], [20,20,20,20], [30,30,30,30]]
])
print(f"Size: {x.shape}")
print(f"x: \n{x}\n")
a = x.view(x.size(1), -1)
print(f"\nSize: {a.shape}")
print(f"a: \n{a}\n")
b = x.transpose(0,1).contiguous()
print(f"\nSize: {b.shape}")
print(f"b: \n{b}\n")
c = b.view(b.size(0), -1)
print(f"\nSize: {c.shape}")
print(f"c: \n{c}")
|
1
2
3
4
5
6
7 | # Dimensional operations
x = torch.randn(2, 3)
print(f"Values: \n{x}")
y = torch.sum(x, dim=0) # add each row's value for every column
print(f"Values: \n{y}")
z = torch.sum(x, dim=1) # add each columns's value for every row
print(f"Values: \n{z}")
|
Now we'll look at how to extract, separate and join values from our tensors.
1
2
3
4 | x = torch.randn(3, 4)
print (f"x: \n{x}")
print (f"x[:1]: \n{x[:1]}")
print (f"x[:1, 1:3]: \n{x[:1, 1:3]}")
|
1
2
3
4
5
6
7
8
9
10
11
12 | # Select with dimensional indices
x = torch.randn(2, 3)
print(f"Values: \n{x}")
col_indices = torch.LongTensor([0, 2])
chosen = torch.index_select(x, dim=1, index=col_indices) # values from column 0 & 2
print(f"Values: \n{chosen}")
row_indices = torch.LongTensor([0, 1])
col_indices = torch.LongTensor([0, 2])
chosen = x[row_indices, col_indices] # values from (0, 0) & (1, 2)
print(f"Values: \n{chosen}")
|
We can also combine our tensors via concatenation or stacking operations, which are consistent with NumPy's joining functions' behaviors as well.
1
2
3 | x = torch.randn(2, 3)
print (x)
print (x.shape)
|
1
2
3
4 | # Concatenation
y = torch.cat([x, x], dim=0) # concat on a specified dimension
print (y)
print (y.shape)
|
1
2
3
4 | # Stacking
z = torch.stack([x, x], dim=0) # stack on new dimension
print (z)
print (z.shape)
|
We can determine gradients (rate of change) of our tensors with respect to their constituents using gradient bookkeeping. The gradient is a vector that points in the direction of greatest increase of a function. We'll be using gradients in the next lesson to determine how to change our weights to affect a particular objective function (ex. loss).
1
2
3
4
5
6
7 | # Tensors with gradient bookkeeping
x = torch.rand(3, 4, requires_grad=True)
y = 3*x + 2
z = y.mean()
z.backward() # z has to be scalar
print(f"x: \n{x}")
print(f"x.grad: \n{x.grad}")
|
We also load our tensors onto the GPU for parallelized computation using CUDA (a parallel computing platform and API from Nvidia).
1
2 | # Is CUDA available?
print (torch.cuda.is_available())
|
If False (CUDA is not available), let's change that by following these steps: Go to Runtime > Change runtime type > Change Hardware accelerator to GPU > Click Save
1 | import torch
|
1
2 | # Is CUDA available now?
print (torch.cuda.is_available())
|
1
2
3 | # Set device
device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
print (device)
|
1
2
3
4 | x = torch.rand(2,3)
print (x.is_cuda)
x = torch.rand(2,3).to(device) # Tensor is stored on the GPU
print (x.is_cuda)
|
To cite this content, please use:
1
2
3
4
5
6 | @article{madewithml,
author = {Goku Mohandas},
title = { PyTorch - Made With ML },
howpublished = {\url{https://madewithml.com/}},
year = {2023}
}
|