Skip to content
Tutorials Center
My learning

Topics

Lesson 6 of 40 ยท Tensors and Automatic Differentiation

Tensors and Shapes

Learn TensorFlow tensors: rank, shape and dtype, creating tensors, indexing, reshaping, stacking and converting to NumPy, with runnable NumPy stand-ins.

A tensor is a multi-dimensional array of numbers that all share one data type. Scalars, vectors, matrices, batches of images and batches of sentences are all tensors; they differ only in how many axes they have. Almost every bug you will meet in deep learning is a shape bug, so being fluent with tensors and their shapes pays off immediately.

TensorFlow tensors behave much like NumPy arrays, with two important differences: they can live on a GPU, and TensorFlow can differentiate through operations on them. This lesson shows the real tf calls as read-only code, and uses NumPy (whose API for these operations is nearly identical) for the runnable examples.

Rank, shape and dtype

Every tensor has three properties:

  • Rank: the number of axes (ndim in NumPy).
  • Shape: the size along each axis, such as (32, 28, 28, 1).
  • dtype: the numeric type, such as float32 or int32.
Rank Name Example Shape
0 scalar a single loss value ()
1 vector one example's 5 features (5,)
2 matrix a table of 32 examples x 5 features (32, 5)
3 3D tensor 32 sentences x 20 words x 64 numbers (32, 20, 64)
4 4D tensor 32 color images, 28x28 (32, 28, 28, 3)

The conventions matter. The first axis is the batch (examples). Images in Keras default to channels_last: (batch, height, width, channels). Sequences are (batch, time steps, features).

Python
import tensorflow as tf

a = tf.constant(7)                         # rank 0
b = tf.constant([1.0, 2.0, 3.0])           # rank 1
c = tf.constant([[1, 2], [3, 4]])          # rank 2

print(a.shape, b.shape, c.shape)           # () (3,) (2, 2)
print(a.dtype, b.dtype, c.dtype)           # <dtype: 'int32'> <dtype: 'float32'> <dtype: 'int32'>
print(tf.rank(c).numpy())                  # 2
print(c)
# tf.Tensor(
# [[1 2]
#  [3 4]], shape=(2, 2), dtype=int32)

Notice the defaults: Python integers become int32 and Python floats become float32, unlike NumPy's int64 and float64. That mismatch causes subtle bugs, which Variables and Data Types covers.

Creating tensors

Python
tf.zeros((2, 3))                 # all zeros
tf.ones((2, 3))                  # all ones
tf.fill((2, 2), 9.0)             # all 9.0
tf.range(5)                      # [0 1 2 3 4]
tf.linspace(0.0, 1.0, 5)         # 5 evenly spaced numbers
tf.eye(3)                        # identity matrix
tf.random.normal((2, 3), seed=1) # normal random numbers
tf.convert_to_tensor(np_array)   # from NumPy

The NumPy twins are np.zeros, np.ones, np.full, np.arange, np.linspace, np.eye and rng.normal. Run them and look at shapes:

Python Stand-in: creating arrays and reading rank, shape, dtype (NumPy)

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Indexing and slicing

Tensors use NumPy-style indexing. A colon takes a whole axis, a:b a range, and a negative index counts from the end.

Python
t = tf.constant([[1, 2, 3],
                 [4, 5, 6],
                 [7, 8, 9]])

t[0]          # first row   -> [1 2 3]
t[:, 1]       # second column -> [2 5 8]
t[1:, :2]     # rows 1.., columns 0..1 -> [[4 5] [7 8]]
t[-1, -1]     # last element -> 9
Python Stand-in: indexing and slicing (NumPy)

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Unlike NumPy arrays, tf.Tensor objects are immutable: t[0, 0] = 5 is an error. To change values you create a new tensor (for example with tf.tensor_scatter_nd_update) or use a tf.Variable.

Reshaping

Reshaping changes how the same numbers are grouped without changing their order. The total number of elements must stay the same, and -1 means "figure this axis out".

Python
x = tf.range(12)                  # shape (12,)
tf.reshape(x, (3, 4))             # shape (3, 4)
tf.reshape(x, (2, -1))            # shape (2, 6)
tf.reshape(x, (-1,))              # flatten to (12,)

Related shape operations:

  • tf.expand_dims(x, axis=0) adds an axis of size 1 (turns (3,) into (1, 3)).
  • tf.squeeze(x) removes axes of size 1.
  • tf.transpose(x, perm=[1, 0]) reorders axes (a matrix transpose for rank 2).

A very common use is flattening images before a Dense layer, which Keras does with layers.Flatten(): (batch, 28, 28, 1) becomes (batch, 784).

Python Stand-in: reshape, expand_dims, squeeze, transpose (NumPy)

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Stacking and splitting

Python
a = tf.constant([1, 2])
b = tf.constant([3, 4])

tf.stack([a, b])                # shape (2, 2): new axis
tf.concat([a, b], axis=0)       # shape (4,): joins along an existing axis
tf.split(tf.range(6), 3)        # three tensors of 2

stack creates a new axis (combining 32 single images into a batch); concat joins along an existing one.

flowchart LR
    A["a: (2,)"] --> S["stack<br/>result (2, 2)"]
    B["b: (2,)"] --> S
    A --> C["concat<br/>result (4,)"]
    B --> C

Stack adds an axis; concat extends one.

From and to NumPy

Python
arr = np.array([[1.0, 2.0], [3.0, 4.0]])
t = tf.constant(arr)        # NumPy -> tensor
back = t.numpy()            # tensor -> NumPy

Most TensorFlow operations also accept NumPy arrays directly and return tensors. Keras fit and predict accept NumPy arrays.

Special kinds of tensors

Real data is not always rectangular. TensorFlow has tf.RaggedTensor for rows of different lengths (sentences of different word counts), tf.SparseTensor for mostly-zero data, and tf.string tensors for text. Chapter 6 uses ragged data and padding.

Reading shapes: the debugging habit

When a model errors with something like Input 0 of layer "dense" is incompatible with the layer: expected shape=(None, 784), found shape=(32, 28, 28), the message is telling you what shape it wanted and what it got. Print x.shape at each stage, and write the expected shape in a comment next to every line while you are learning. It saves hours.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. What is the rank of a tensor with shape (32, 28, 28, 3)?

    Rank is the number of axes, and this shape has four.
  2. 2. A batch of 32 grayscale 28x28 images has shape (32, 28, 28, 1). What does reshape(32, -1) give?

    The -1 is inferred as 28 x 28 x 1 = 784.
  3. 3. What is the difference between stack and concat?

    Stacking two (2,) tensors gives (2, 2); concatenating gives (4,).
  4. 4. Which statement about tf.Tensor is true?

    Tensors are immutable. Use a tf.Variable if you need mutable state.

Summary

  • A tensor has a rank (number of axes), a shape and one dtype.
  • The first axis is the batch; images are (batch, height, width, channels).
  • Indexing and slicing follow NumPy; tensors are immutable.
  • reshape (with -1), expand_dims, squeeze, transpose, stack and concat rearrange tensors.
  • Printing shapes is the fastest way to debug.

Next: Tensor Operations and Broadcasting