Skip to content
Tutorials Center
My learning

Topics

Lesson 3 of 46 ยท Intelligent Agents

Types of Environments and Agents

Classify task environments (observable, deterministic, episodic, static, discrete, multi-agent) and compare reflex, model-based, goal-based and utility-based agents in Python.

Not every problem is equally hard for an agent. Solving a crossword on paper is a very different challenge from driving through a city, even though both involve perceiving and acting. The difficulty comes from the properties of the task environment, and those properties tell you which kind of agent design is even worth trying. A handful of yes-or-no questions lets you size up almost any problem in a minute.

This lesson gives you those questions, then walks through four agent designs of increasing power: simple reflex, model-based reflex, goal-based and utility-based. You will classify real examples in code and run the four designs side by side on a tiny thermostat world.

Six properties of task environments

Fully versus partially observable. If the sensors reveal the complete state relevant to the decision at every moment, the environment is fully observable (a chess board). If sensors are noisy, limited or incomplete, it is partially observable (a vacuum that sees only its own square, a poker hand, a taxi that cannot see around a corner). Partial observability forces the agent to keep an internal estimate of the hidden state.

Deterministic versus stochastic. If the next state is completely determined by the current state and the action, the environment is deterministic. If there is randomness (dice, wheel slip, other drivers), it is stochastic. A related word, non-deterministic, means the outcomes are a set of possibilities with no probabilities attached.

Episodic versus sequential. In an episodic environment, each decision is independent of earlier ones, such as classifying a part on an assembly line as good or defective. In a sequential one, today's choice affects tomorrow's options (chess, driving), so the agent must think ahead.

Static versus dynamic. A static environment waits while the agent deliberates. A dynamic one keeps changing, so slow thinking has a cost (a taxi in traffic). Semi-dynamic means the world is static but the score depends on time, as in timed chess.

Discrete versus continuous. This applies to states, time and actions. Chess has a finite set of positions and moves. A robot arm's joint angles are continuous.

Single-agent versus multi-agent. Solving a crossword is single-agent. Chess has two competing agents, and a team of delivery drones has several cooperating ones. With other agents, behavior must account for what they will do.

The hardest case combines all the awkward answers: partially observable, stochastic, sequential, dynamic, continuous and multi-agent. Driving is the usual example. The easiest is the opposite corner, and most textbook algorithms start there.

Python Classifying environments in code

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Real tasks are often arguable. Image classification of a single photo is episodic, but the world the camera sees is not discrete. Decide on the level of detail you model and state your assumptions. The point of the exercise is to notice which assumptions your chosen algorithm needs.

Four agent designs

The agent function is the same abstract object in all cases. What changes is how much structure the agent program has.

graph TD
    A["Simple reflex<br/>condition-action rules"] --> B["Model-based reflex<br/>+ internal state"]
    B --> C["Goal-based<br/>+ goals, looks ahead"]
    C --> D["Utility-based<br/>+ how good each outcome is"]

Each design adds one capability to the previous one.

Simple reflex agents

A simple reflex agent maps the current percept to an action with condition-action rules: "if the car in front is braking, brake". It is tiny and fast, and it works well only when the correct action can be chosen from the current percept alone, that is, when the environment is fully observable. In a partially observable world it can loop forever, such as a vacuum that cannot tell whether the other square is clean.

Model-based reflex agents

To handle partial observability, the agent keeps internal state: a summary of the parts of the world it cannot see now. Updating that state needs a model with two pieces of knowledge: how the world changes by itself, and how the agent's own actions change it. The model-based vacuum remembers which squares it has already cleaned.

Goal-based agents

Knowing the state is not always enough to decide what to do. A taxi at a junction needs to know where it is going. A goal-based agent adds an explicit goal description and chooses actions by considering their consequences: "which sequence of actions reaches the goal?" That is search and planning, the core of this course. Goal-based agents are flexible: change the goal, and the behavior changes without rewriting rules.

Utility-based agents

Goals give only a yes-or-no distinction between happy and unhappy. Many routes reach the destination, but some are faster, safer or cheaper. A utility function scores states on a scale, which lets the agent trade off conflicting goals (speed versus safety) and weigh uncertain outcomes by expected utility. You will see this again in decision making and MDPs.

Note: A fifth kind, the learning agent, can be added to any of the four: it improves its own rules, model or utility function from experience. This course concentrates on how the four designs decide; machine learning is covered in its own tracks.

The four designs side by side

The world below is a thermostat. The room temperature drifts down by one degree each step, but the sensor only reports a coarse reading (cold, ok, hot), so the agent never sees the exact temperature. The heater adds three degrees when on. The performance measure penalizes any step outside 19 to 23 degrees, and costs one point per step the heater is on.

Python Reflex, model-based, goal-based and utility-based thermostats

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Run it and read the scores: the reflex agent gets -29, the model-based and goal-based agents both get -16, and the utility-based agent gets -17. The reflex agent reacts only when the sensor says cold, by which time the room has already left the comfort zone, so it spends many steps outside it. The three agents with internal state predict the drift and keep the temperature in range. The goal-based and utility-based versions look one step ahead, and here that ties with the model-based rule, because this world is simple. The utility agent is one point behind: it weighs the energy cost against comfort, a trade-off the others do not even represent. Try raising the discomfort penalty from 5 to 20 or lowering the heater cost, and watch that trade-off move. Notice what did not change between the agents: the world and the performance measure. Only the agent program improved, and each improvement needed a richer description of the problem.

Choosing a design

If the environment is... Then try...
Fully observable, rules obvious Simple reflex
Partially observable Model-based, with a state estimate
Many steps to a clear target Goal-based: search, planning
Conflicting objectives or uncertainty Utility-based, with expected utility

Most of the rest of this course builds goal-based agents. Next you will see how to describe a problem precisely so that a search algorithm can attack it, in Formulating Search Problems.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. A chess program sees the whole board, and nothing random happens. How would you classify chess?

    The full board is visible, moves have exact results, each move affects later ones (sequential) and there are two competing agents.
  2. 2. Why can a simple reflex agent fail in a partially observable environment?

    Without internal state, identical-looking percepts lead to identical actions, even when the hidden situation is different.
  3. 3. What does a utility-based agent add over a goal-based agent?

    Goals only separate satisfied from unsatisfied. A utility function ranks outcomes and supports expected-value decisions.
  4. 4. Which environment property means decisions made now do NOT affect later decisions?

    In an episodic environment each percept-action episode is independent, like inspecting separate items on a conveyor belt.

Summary

  • Six questions size up a task: observable, deterministic, episodic, static, discrete, single or multi-agent.
  • The hardest tasks are partially observable, stochastic, sequential, dynamic, continuous and multi-agent.
  • Simple reflex agents use only the current percept; model-based agents add internal state.
  • Goal-based agents choose actions by looking ahead toward a goal; utility-based agents rank outcomes by how good they are.
  • The same world and performance measure can reward very different agent programs.

Next: Formulating Search Problems.