Pytorch – Index-based Operation

PyTorch is a python library developed by Facebook to run and train deep learning and machine learning algorithms. Tensor is the fundamental data structure of the machine or deep learning algorithms and to deal with them, we perform several operations, for which PyTorch library offers many functionalities.

Tensor operations that handle indexing on some particular row or column for copying, adding, filling values/tensors are said to be index-based developed operation. There are two types of index-based operations in PyTorch, one is in-place operations and the other is out-of-place operations.  The basic difference between the two is in-place operation directly changes the values of the tensors without making any copy of that whereas out of place operations don’t. Following are the operations:-

1. index_add_
2. index_add
3. index_copy_
4. index_copy
5. index_fill_
6. index_fill
7. index_put_
8. index_put
9. index_select

Now we discuss each of the functions with proper examples.

1. index_add_:  Adds the given tensor elements to the self tensor along the order given in the matrix.

Syntax:

 index_add_(dim,index,ensor)---> Tensor

 Parameters:

• dim: dimension along which index to add. ‘0’ stands for column and ‘1’ stands for row.
• index: indices of the tensor to select from. It can be LongTensor or IntTensor.
• tensor: tensor containing the values to add.

Example 1: We take a zero vector ‘x’, te tensor of size(3,5) and index tensor. Accumulating the resultant vector along rows we get the output.

Output:

tensor([[1., 3., 5., 7., 9.],
        [0., 0., 0., 0., 0.],
        [1., 3., 5., 7., 9.],
        [0., 0., 0., 0., 0.],
        [1., 3., 5., 7., 9.]])

 Example 2:

Output:

tensor([[0.9460, 0.4762, 1.2219, 1.0000, 1.0000],
        [0.9460, 0.4762, 1.2219, 1.0000, 1.0000],
        [0.9460, 0.4762, 1.2219, 1.0000, 1.0000],
        [0.9460, 0.4762, 1.2219, 1.0000, 1.0000],
        [0.9460, 0.4762, 1.2219, 1.0000, 1.0000]])

2.index_add: It is the out-of place version of the above function. This adds a given tensor to the self tensor temporarily. Parameters and syntax are the out-of-placesame as above.

Example 3:

Output:

Indexed Matrix:
 tensor([[-0.2811, -1.0776,  2.2697,  1.0000,  1.0000],
        [-0.2811, -1.0776,  2.2697,  1.0000,  1.0000],
        [-0.2811, -1.0776,  2.2697,  1.0000,  1.0000],
        [-0.2811, -1.0776,  2.2697,  1.0000,  1.0000],
        [-0.2811, -1.0776,  2.2697,  1.0000,  1.0000]])
Printing Indexed Matrix again:
 tensor([[1., 1., 1., 1., 1.],
        [1., 1., 1., 1., 1.],
        [1., 1., 1., 1., 1.],
        [1., 1., 1., 1., 1.],
        [1., 1., 1., 1., 1.]])

3.index_copy_:  Copies the elements of a given tensor to the input tensor by selecting the indices in the order given in ‘index’.

Syntax:

index_copy_(dim,index,tensor)---> Tensor

Parameters:

• dim: dimension along which index to copy. It is of ‘int’ format.
• index: indices of the tensor to select from. It can be either IntTensor or LongTensor.
• tensor: tensor containing the values to be copied.

Example 4:  Here elements of ‘a’ is being replaced by ‘t1’ tensor in an order given in the ‘index3’.

Output:

tensor([[ 1.0000,  1.0000,  1.0000,  1.0000],
        [-0.1918, -1.2089,  0.3229, -0.1831],
        [ 1.0000,  1.0000,  1.0000,  1.0000],
        [ 0.7521,  0.8424, -0.8698, -0.3908]])

Example 5: In the below example we will receive an error. This is because the dim^th dimension of the tensor is not equal to the length of the index. So we have to keep in mind that dim^th dimension of the tensor must have the same size as the length of the index.

Output:

RuntimeError                              Traceback (most recent call last)
<ipython-input-8-25e4150d5bd7> in <module>
      1 y=torch.ones(5,5)
      2 index1=torch.tensor([0,1,2,3,4])
----> 3 y.index_copy_(1,index1,te)

RuntimeError: index_copy_(): Source/destination tensor must have same slice shapes.
Destination slice shape: 5 at dimension 1 and source slice shape: 3 at dimension 0.

Example 6: In this, we will copy the given tensor to the self tensor keeping in mind that dim^th dimension of the tensor must have the same size as the length of the index.

Output:

tensor([[-0.3964, -0.3859,  1.0000,  1.0000],
        [ 2.6910, -0.9394,  1.0000,  1.0000],
        [ 0.3591, -0.2262,  1.0000,  1.0000],
        [ 1.2102, -0.8340,  1.0000,  1.0000]])

 4.index_copy: This is the out-of-place index-based operation for replacing elements of input tensor with a given tensor. Syntax, parameters are the same as the above.

5.index_fill_:  ‘Val’ value is filled with the elements of ‘x’ along with the order of indices given in the vector ‘index’.

Syntax:

index_fill_(dim, index, val) → Tensor

Parameters:

• dim: dimension along which to fill. It is of ‘int’ format.
• index:  filling in order of the indices given in this index vector. It can be either IntTensor or LongTensor.
• val(float): value with which to fill in.

Example 7: In this example, we will declare a tensor with random elements and then fill it with ‘4’ along with the given indices.

Output:

 tensor([[ 4.0000,  4.0000,  4.0000,  4.0000],
        [ 0.4762,  0.0056,  0.3258,  1.1345],
        [ 4.0000,  4.0000,  4.0000,  4.0000],
        [-0.1490, -0.6543,  0.9755,  1.8087]])

 Example 8: Similarly we perform a filling operation along with the columns.

Output:

 tensor([[ 0.5978, -1.2461, -0.8794, -1.0175,  0.8938],
        [-0.6374,  1.0848,  0.1291,  0.6658,  0.3081],
        [-0.9686, -0.8212, -0.5223, -0.3208, -1.7718],
        [-0.1153, -1.2552, -0.4119, -1.1293,  0.2266],
        [ 1.2610,  0.2618, -1.5528,  0.7805,  1.3730]])

 6. index_fill: This is the out of place index-based operation for filling the elements with tensor with ‘val’. Syntax, parameters are the same as the above.

7.index_put_: This operation puts the value of ‘val’ into the self tensor using the indices of the given ‘index’.

Syntax:

index_put_(indices, values, accumulate=False) → Tensor

Parameters:

• indices: It is the tuple of LongTensor which is used to index to self.
• values: Tensor with values that need to be put into the target.
• accumulate: whether to accumulate or not.

Example 9: Here we take the target vector and replaces the values of  value tensor as mentioned in the 

Output:

tensor([[0., 1., 0., 0.],
       [1., 0., 1., 0.],
       [0., 0., 0., 0.],
       [0., 1., 0., 0.]])

 Note: We have to take the transpose of the indices tensor or else an error will occur.

Example 10: In this example, we keep the accumulate function as true which means the elements in value are added to the target.

Output:
tensor([[1., 1., 1., 1.],
       [1., 1., 1., 1.],
       [1., 1., 1., 1.],
       [1., 1., 1., 1.]])

8. index_fill: This is the out of place version of index_fill_. Syntax, parameters are the same as above.

9. index_select: A tensor is returned with indices as mentioned, by selecting from the target tensor.

Syntax:

torch.index_select(input, dim, index, out=None) 

Parameters:

• input(Tensor): Tensor from which the indices will be selected.
• dim(int): dimension along which to select.
• index: It contains indices to the index.

Example 11: We pick up 0,1 indices of the tensor ‘m’ along dim=0 i.e. rows and print the output.

Output:

Original matrix:
 tensor([[ 0.2008, -0.2637,  2.1216, -0.2892],
        [-0.4059, -1.6054, -2.5022, -0.2912],
        [-0.3246,  0.4751, -0.1018, -0.6707]])
Indexed Matrix:
 tensor([[ 0.2008, -0.2637,  2.1216, -0.2892],
        [-0.4059, -1.6054, -2.5022, -0.2912]])

  Reference: https://pytorch.org/docs/stable/tensors.html