Skip to content
Tutorials Center
My learning

Topics

Lesson 18 of 36 ยท Fields and Extraction

Splunk Lookups, Field Aliases, Calculated Fields, Tags and Event Types

Enrich events with CSV lookups and automatic lookups, rename fields with aliases, define calculated fields, and label events with tags and event types for reuse in searches.

Extracting fields gets you the values that are in the log. Real analysis also needs context that is not in the log: a country for an IP address, a department for a user, a product name for an id. Lookups add it. This lesson also covers three other knowledge objects that make fields consistent and searches reusable: field aliases, calculated fields, and tags and event types.

Lookups

A lookup matches a field in your events against a table and adds columns from the matching row. Types:

  • CSV lookup: a .csv file uploaded to Splunk (the common type).
  • KV Store lookup: a table in Splunk's built-in MongoDB-based store, which supports updates.
  • External and geospatial lookups: a script, or a KMZ/geo file (the iplocation command also maps IPs to places using a built-in database).

Create a CSV lookup

Example product_names.csv:

TEXT
sku,product_name,category
A1,Espresso machine,kitchen
B2,Desk lamp,office
C3,Water bottle,outdoor
  1. Settings > Lookups > Lookup table files > New: upload the file and choose the app.
  2. Settings > Lookups > Lookup definitions > New: name it (products) and select the file. (Definitions let you add options such as case-insensitive matching.)
  3. Use it in a search:
SPL
index=orders
| lookup products sku OUTPUT product_name, category
| stats sum(quantity) AS units by product_name

lookup <name> <input field> OUTPUT <new fields>: Splunk matches sku in the event with sku in the table and adds the output fields. If the column in the table has a different name: lookup products sku AS item_code OUTPUT product_name. Without OUTPUT, all other columns are added (and OUTPUTNEW only fills fields that do not exist yet).

Python Stand-in: a CSV lookup enriching events

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Managing lookups from searches

SPL
| inputlookup products
| inputlookup products where category="office"
| inputlookup products | eval name_len=len(product_name) | outputlookup products_with_len.csv
| outputlookup append=true new_values.csv

inputlookup reads a table into results (it is a generating command, so it starts the search with a |). outputlookup writes results to a table, which is how scheduled searches build "who has been seen" tables or watch lists.

Automatic lookups

An automatic lookup applies a lookup to every search of a source type without writing | lookup. Define it in Settings > Lookups > Automatic lookups > New, or in props.conf:

INI
[orders_json]
LOOKUP-products = products sku OUTPUT product_name category

Now product_name is simply a field for anyone searching that source type. Use it for stable, widely needed context.

Lookup limits: a CSV lookup file that gets very large (hundreds of megabytes) slows searches and replication to the indexers; use the KV Store or a database lookup for big or frequently changing tables. Keep lookup files small and tidy.

Field aliases

A field alias gives an existing field another name, so different sources can share one name. Define it under Settings > Fields > Field aliases, or in props.conf:

INI
[firewall_logs]
FIELDALIAS-src = src_address AS src
FIELDALIAS-dest = dst_address AS dest

Now a search on src=198.51.100.9 finds events from every source type that has such an alias. The original field stays; aliases are one-to-one renames at search time. This is the main tool to map data to the Common Information Model.

Calculated fields

A calculated field is a permanent eval that every event in a source type gets automatically:

INI
[access_combined]
EVAL-kb = round(bytes/1024, 1)
EVAL-is_error = if(status>=500, 1, 0)

You can also create them in Settings > Fields > Calculated fields. They run at search time, can use extracted fields and aliases (order: extractions, then aliases, then calculated fields, then lookups), but cannot reference other calculated fields reliably, so keep each expression self-contained.

Python Stand-in: aliases, calculated fields and a lookup applied in order

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Tags and event types

An event type is a named saved search that classifies events. Define it with a base search and give it a name:

SPL
# Event type "failed_login": 
sourcetype=linux_secure "Failed password"

Then eventtype=failed_login finds those events anywhere. Event types can have a priority and a color. A tag is a label attached to a field value (for example host=web01 tagged production, or action=failure tagged authentication). Search with tag=production. The CIM relies on tags and event types: its data models search tag=authentication tag=failure.

Create tags from the sidebar (click a field value, Edit Tags) or at Settings > Tags. Create event types from Save As > Event Type on the search page.

Python Stand-in: event types and tags classifying events

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

These objects (extractions, aliases, calculated fields, lookups, tags, event types) are together called knowledge objects. Package them in an app so you can share and version them.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. Which command adds a product name from a CSV table to events that have an sku field?

    lookup matches an event field against the table and adds the output fields.
  2. 2. What does a field alias do?

    Aliases map field names so that different sources share the same name, as the CIM expects.
  3. 3. What is an event type?

    You search it with eventtype=<name>.
  4. 4. Which command starts a search by reading a lookup table into results?

    inputlookup is a generating command that reads the table; outputlookup writes one.

Summary

  • Lookups enrich events with context: CSV, KV Store, external; add them with lookup, read with inputlookup, write with outputlookup.
  • Automatic lookups, field aliases and calculated fields apply to every event of a source type at search time.
  • Event types and tags label events so searches and CIM data models can find them.
  • Share knowledge objects through an app with proper permissions.

Next: the next chapters continue on the Splunk track page.