Skip to content
Tutorials Center
My learning

Topics

Lesson 8 of 40 ยท Tensors and Automatic Differentiation

Variables and Data Types

Learn why TensorFlow has tf.Variable for model weights, how assign works, how dtypes and tf.cast behave, and why float32 precision matters, with NumPy stand-ins.

A tf.constant never changes. But a model's weights must change thousands of times during training. TensorFlow solves this with variables: tensors with state that operations can update in place. Every weight and bias inside every Keras layer is a tf.Variable (or the Keras equivalent wrapper). This lesson explains variables, then turns to the other half of the title: data types, where a wrong choice means errors, wasted memory or numbers that quietly lose precision.

Constants versus variables

Python
import tensorflow as tf

c = tf.constant([1.0, 2.0, 3.0])       # immutable
v = tf.Variable([1.0, 2.0, 3.0])       # mutable state

v.assign([10.0, 20.0, 30.0])           # replace the values
v.assign_add([1.0, 1.0, 1.0])          # in-place +=
v.assign_sub([0.5, 0.5, 0.5])          # in-place -=
print(v.numpy())                       # [10.5 20.5 30.5]
print(v.shape, v.dtype, v.trainable)   # (3,) <dtype: 'float32'> True

Key facts about a variable:

  • Its shape and dtype are fixed when created. assign with a different shape raises an error.
  • It is trainable by default, meaning tf.GradientTape watches it automatically (next lessons).
  • It has a name, handy when saving and inspecting models.
  • Create tf.Variable(..., trainable=False) for counters and running averages that the optimizer should not touch.

An optimizer's update w = w - lr * grad is literally w.assign_sub(lr * grad).

Why variables exist

Think of a variable as a mailbox. The model's computation reads the number inside; the optimizer replaces it with a better one after each step. A constant is like a number printed on paper. Because the model only ever sees the current contents, you must reset or checkpoint variables if you want to go back to an earlier state.

flowchart LR
    W["Variable w"] -->|"read"| F["Forward pass"]
    F --> L["Loss"]
    L --> G["Gradient dL/dw"]
    G --> O["Optimizer"]
    O -->|"w.assign_sub(lr * grad)"| W

The optimizer closes the loop by writing a new value back into the variable.

A stand-in for tf.Variable

NumPy arrays are already mutable, so a small class can mimic the key behaviors: fixed shape and dtype, assign, assign_add, assign_sub.

Python Stand-in: a minimal Variable class (NumPy)

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Data types

TensorFlow dtypes mirror NumPy's: float16, bfloat16, float32, float64, int8, int16, int32, int64, uint8, bool, string. The defaults to remember:

Python value NumPy default TensorFlow default
1.5 float64 float32
3 int64 int32

Deep learning uses float32 almost everywhere: half the memory of float64, fast on GPUs, and precise enough. float16 and bfloat16 are used for mixed-precision training to go faster still (Chapter 8). uint8 is how images are stored (0-255) before scaling.

Casting

TensorFlow does not silently convert types in operations. Adding an int32 tensor to a float32 tensor is an error, which protects you from accidental precision loss. Convert explicitly:

Python
i = tf.constant([1, 2, 3])               # int32
f = tf.constant([0.5, 0.5, 0.5])         # float32

# i + f   -> error: dtypes do not match
tf.cast(i, tf.float32) + f               # [1.5 2.5 3.5]
tf.cast(tf.constant([1.9, -1.9]), tf.int32)   # [ 1 -1 ]  truncates toward zero
tf.cast(tf.constant([0, 3, 0]), tf.bool)      # [False True False]

Note that casting a float to an integer truncates toward zero; it does not round. Use tf.round first if you want rounding.

Python Stand-in: dtypes, casting and truncation (NumPy)

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Precision and range

Floating-point numbers store a limited number of significant digits. float32 keeps about 7 decimal digits, float64 about 15, and float16 only about 3, with a maximum value near 65,504. Two surprises follow:

  1. Adding a tiny number to a big one can do nothing (it vanishes in rounding).
  2. Very large or small intermediate values overflow to inf or underflow to 0, which is why softmax subtracts the maximum and why losses are computed in numerically stable forms.
Python Stand-in: float16, float32 and float64 precision (NumPy)

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The practical rule: never test floating-point results with ==. Use tf.experimental.numpy.isclose or np.isclose, or compare with a tolerance (atol). Tests of training code check that a loss goes down, not that it equals an exact number.

Variables in Keras layers

When you create layers.Dense(3), its weights are variables created when the layer is built (when it first knows the input size). You can inspect them and set them directly, which is useful for loading pretrained values or for tests.

Python
import numpy as np
import keras

layer = keras.layers.Dense(2)
layer.build((None, 3))

kernel, bias = layer.weights
print(kernel.shape, bias.shape)          # (3, 2) (2,)

layer.set_weights([np.ones((3, 2)), np.zeros(2)])
print(layer.get_weights()[0])            # three rows of [1. 1.]

get_weights() returns NumPy arrays; set_weights() takes a list of NumPy arrays in the same order and shapes.

Check your understanding

Quick check ยท 5 questions

of 5 answered You scored / 5 ยท perfect!

  1. 1. Why does TensorFlow have tf.Variable in addition to tf.constant?

    Training repeatedly changes the weights, so they live in mutable variables.
  2. 2. What does tf.cast(tf.constant([1.9, -1.9]), tf.int32) return?

    Casting to an integer truncates toward zero; it does not round.
  3. 3. What is the default dtype of tf.constant(1.5)?

    TensorFlow defaults Python floats to float32 (NumPy would use float64).
  4. 4. Why should you avoid == when comparing floating-point results?

    Floating-point numbers are approximations. Compare with a tolerance such as np.isclose.
  5. 5. What happens if you add an int32 tensor to a float32 tensor in TensorFlow?

    TensorFlow requires matching dtypes and does not convert silently.

Summary

  • tf.Variable holds trainable state; assign, assign_add and assign_sub update it in place, and an optimizer step is exactly that.
  • Variables keep a fixed shape and dtype.
  • Defaults are float32 and int32; TensorFlow does not auto-convert, so use tf.cast.
  • Float-to-int casting truncates; floating-point math needs tolerance-based comparisons.
  • float32 is the standard; float16/bfloat16 trade precision and range for speed.

Next: Automatic Differentiation