> ## Documentation Index
> Fetch the complete documentation index at: https://docs.elementum.io/llms.txt
> Use this file to discover all available pages before exploring further.

# Data Mining

> Monitor CloudLink-backed table data on a schedule, track per-record match state, and start automations when conditions are met or cleared.

## What is a Data Mine?

**Data Mining** is a way to start [automations](/workflows/automation-system) from data that lives in your data warehouse and is exposed in Elementum through [CloudLinks](/administration/setup-cloudlink). You define a **Data Mine** on a [Table](/data/tables) that reads that data: Elementum runs your criteria on a schedule, tracks whether each row **matches** or **does not match**, and fires automations when a row **transitions** between those states (for example, from unmatched to matched, or the reverse).

A Data Mine does not run arbitrary one-off queries for reporting—it **watches** the same logical rows over time and pairs that behavior with **state management** so you are not notified on every scan when nothing has changed for a given row.

When your criteria match a row, that row is **ON** (matched); when they do not, it is **OFF** (unmatched). [Automations](/workflows/automation-system) run on **transitions** between those states, not on every evaluation where the state is unchanged.

**Key components:**

* **Data source** — Rows from a Table backed by a [CloudLink](/administration/setup-cloudlink) to your data warehouse (for example Snowflake or BigQuery, depending on your connection).
* **Matching criteria** — Filters that define which rows count as matched.
* **State management** — Per-row tracking of matched vs. unmatched across runs.
* **Automations** — Workflows you attach via the **Data Mine** trigger.

<Warning>
  **Important:** A Data Mine alone does not complete work in your process. Connect it to automations that take the next steps (notifications, updates, handoff to an agent, and so on). Decide those steps before you finalize the Data Mine.
</Warning>

## When to Implement Data Mines

Use Data Mining when you need **scheduled, condition-based** reactions to warehouse data: thresholds, SLAs, inventory levels, open claims, and similar rules. It fits **batch-style** monitoring (for example hourly or daily), which is also how many CloudLink-backed tables are refreshed. For very fast reactions when a record changes inside an Elementum app, [record triggers and other automation triggers](/workflows/automation-triggers-reference) may be a better fit—see [Automations](/getting-started/fundamentals/core-concepts#automations) and [Change monitoring](/data/data-best-practices#change-monitoring).

**Outcomes you can expect:**

* Automations start when a condition becomes true (or false) for a row, not on every run.
* Teams spend less time scanning reports to find exceptions.
* You can chain work: notify, assign, update records, or call [Agents](/ai-agents/ai-overview) when your automation design includes them.

<Info>
  Data Mining connects warehouse-backed table data to automations. CloudLinks provide access to that data; Data Mines define **what** to watch and **when** to run; [automations](/workflows/automation-system) define **what** happens next.
</Info>

## How Data Mines Operate

1. **Table** — Use a Table whose source is CloudLink data you want to monitor (see [Tables](/data/tables)).
2. **Data Mine** — In that Table, create a Data Mine (see [Create a Data Mine](#create-a-data-mine-in-the-ui)), set identifying columns, matching criteria, and a schedule.
3. **Automation** — Create an automation with a **Data Mine** trigger and choose whether it runs when data **meets** or **no longer meets** your criteria (see [Data Mine trigger](/workflows/automation-triggers-reference#data-mine)).

## Understand state management: ON and OFF

Instead of firing an automation on every scheduled run for every matching row, Elementum tracks state for each row (using your **identifying columns**). Automations tied to the Data Mine run when **state changes** in a way you configured.

**How state transitions map to events:**

```
Record state: OFF → ON  = Automation can run for "Data Mine Triggered" (condition newly met)
Record state: ON → ON   = No transition; no new trigger for that row
Record state: ON → OFF  = Automation can run for "Data Mine Cleared" (condition no longer met)
Record state: OFF → OFF = No transition; no new trigger for that row
```

This supports workflows such as: alert when a case first crosses a threshold, stay quiet while it stays above the threshold, then run a different path when it drops back below.

<AccordionGroup>
  <Accordion title="Example: High-value claims" icon="shield-check">
    **Scenario:** Claims over \$10,000 while status is open need immediate attention.

    **Data Mine setup:**

    * **Source:** Claims table (CloudLink)
    * **Criteria:** `claim_amount > 10000 AND status = 'open'`
    * **Schedule:** Every 15 minutes

    **State behavior:**

    1. **New high-value open claim (OFF → ON):** Automation runs (for example assign adjuster, notify a queue).
    2. **Claim still matches (ON → ON):** No additional automation from this transition.
    3. **Claim closed or amount no longer in range (ON → OFF):** Automation runs for clearance steps (for example update a dashboard record, notify that the exception cleared).
  </Accordion>

  <Accordion title="Example: Inventory threshold" icon="package">
    **Scenario:** Reorder when stock falls below a threshold for active SKUs.

    **Data Mine setup:**

    * **Source:** Inventory table (CloudLink)
    * **Criteria:** `stock_level < reorder_threshold AND status = 'active'`
    * **Schedule:** Hourly

    **State behavior:**

    1. **OFF → ON:** Start reorder or approval automation.
    2. **ON → ON:** No duplicate reorder signal for the same row until state changes.
    3. **ON → OFF:** Run follow-up when stock is back above the threshold (for example notify sales).
  </Accordion>

  <Accordion title="Example: SLA breach" icon="timer">
    **Scenario:** Tickets open past the response window without a first response.

    **Data Mine setup:**

    * **Source:** Support tickets table (CloudLink)
    * **Criteria:** Example pattern: open tickets where the clock has passed your SLA and `first_response_date` is still null—adjust fields to match your schema.
    * **Schedule:** Every 30 minutes

    **State behavior:**

    1. **OFF → ON:** Escalation or reassignment automation.
    2. **ON → ON:** No repeated escalation for the same breach state.
    3. **ON → OFF:** Automation when the ticket is responded to or no longer breaches.
  </Accordion>
</AccordionGroup>

## Working with Agents

Data Mine triggers can start automations whose actions include [Agents](/ai-agents/ai-overview): for example summarizing the matched row, classifying it, or drafting a reply. Agent behavior depends on how you configure prompts, tools, and steps in that automation—not on automatic “learning” stored inside the Data Mine itself. For details, see [Agent tools, deployment, and integrations](/ai-agents/agents-tools-and-deployment) and the [AI overview](/ai-agents/ai-overview).

## Types of Data Mining

When you create a Data Mine, you choose a **type** that matches how you want to detect rows of interest. The product may show one or more of the following.

<Tabs>
  <Tab title="Logic-based rules">
    **Best for:** Clear business rules and known patterns.

    **Example:** Alert when any order exceeds \$5,000.

    ```
    Criteria: order_total > 5000
    Actions: Route to approval workflow, notify finance team
    ```

    You define explicit filters (comparisons, ranges, null checks, time windows) that rows must satisfy. Most guided documentation (including the UI steps below) uses this type.
  </Tab>

  <Tab title="ML anomaly detection">
    **Best for:** Discovering unexpected patterns or behaviors.

    **Example:** Detect unusual spending patterns in expense reports.

    ```
    AI Model: Learns normal spending patterns
    Detection: Flags expenses that deviate significantly from learned norms
    Actions: Flag for review, request additional documentation
    ```
  </Tab>

  <Tab title="Statistical anomaly">
    **Best for:** Finding numerical outliers using statistical methods.

    **Example:** Identify processing times that are unusually long.

    ```
    Statistical Method: Z-score analysis
    Threshold: 2 standard deviations above mean
    Actions: Alert operations team, investigate bottlenecks
    ```
  </Tab>
</Tabs>

<Info>
  Available Data Mine types depend on your workspace and product configuration. If an option is not shown in the UI, use the types your environment supports and follow the on-screen steps for that type.
</Info>

## Create a Data Mine in the UI

These steps align with guided flows elsewhere in the docs (for example file and document monitoring). Exact labels may vary slightly by workspace.

1. Open the **[Table](/data/tables)** that uses your CloudLink data.
2. Go to **Data Mining** → **Create Data Mine** → **Logic-Based Rules Mining**.
3. **Identifying columns** — Choose stable columns so the same business row is recognized across runs (IDs, composite keys). See [Define identifying columns](#define-identifying-columns).
4. **Matching criteria** — Set filters for rows that should count as matched. See [Build matching criteria](#build-matching-criteria).
5. **Name and schedule** — Name the Data Mine and set how often it runs.

Then build an automation:

1. Go to **Automations** → **Create Automation**.
2. Add a **Data Mine** trigger and select this Data Mine.
3. Set the trigger to **Trigger when data meets requirement** and/or **Trigger when data no longer meets requirement**, depending on whether you need ON or OFF transition handling. Details and option names are in [Automation triggers reference — Data Mine](/workflows/automation-triggers-reference).

## Best Practices for Data Mine Creation

<AccordionGroup>
  <Accordion title="Plan before you build" icon="clipboard-list">
    <span id="plan-before-you-build" />

    Before creating a Data Mine, decide:

    * What should happen the first time a row **matches** (OFF → ON)?
    * What should happen when it **stops** matching (ON → OFF), if anything?
    * Who or what receives the work (users, records, [Agents](/ai-agents/ai-overview))?
    * Which fields from the row the automation needs?
  </Accordion>

  <Accordion title="Select your data source" icon="database">
    <span id="select-your-data-source" />

    Choose a CloudLink-backed Table where:

    * Data is sufficiently current for your process.
    * You have permission to read the objects you reference.
    * Refresh timing matches how often you need to evaluate conditions (see [Change monitoring](/data/data-best-practices#change-monitoring)).
  </Accordion>

  <Accordion title="Define identifying columns" icon="fingerprint">
    <span id="define-identifying-columns" />

    **Purpose:** Identifying columns let the system recognize the **same** logical row across Data Mine runs so state (OFF/ON) is stable.

    **Practices:**

    * Prefer stable unique identifiers (IDs, UUIDs).
    * Add business keys when they help disambiguate (order number, case ID).
    * Avoid using columns that change every run unless they are part of your identity rule.

    **Example (support tickets):**

    ```
    Identifying columns:
    - ticket_id (primary identifier)
    - customer_id (context)
    - created_date (if required for your keying strategy)
    ```
  </Accordion>

  <Accordion title="Build matching criteria" icon="filter">
    <span id="build-matching-criteria" />

    **Simple conditions:**

    ```
    WHERE priority = 'High'
    WHERE amount > 1000
    WHERE status IN ('pending', 'review')
    ```

    **Grouped logic:**

    ```
    WHERE (
        (priority = 'High' AND amount > 500)
        OR
        (priority = 'Critical' AND amount > 100)
    )
    AND status = 'active'
    ```

    **Time-based conditions (adjust functions to your warehouse SQL dialect):**

    ```
    WHERE created_date > NOW() - INTERVAL '7 days'
    WHERE last_updated < NOW() - INTERVAL '24 hours'
    ```
  </Accordion>

  <Accordion title="Schedule and volume" icon="clock">
    <span id="schedule-and-volume" />

    **Scheduling:** Match frequency to the process—tighter schedules (for example 15–30 minutes) when latency matters; hourly or daily when batch refresh and cost matter more.

    **Volume:** Each Data Mine can process up to **100,000 rows** per run (see [Field updates](/data/data-best-practices#field-updates)). For performance and predictable run times, keep the **matching** set as small as practical with selective criteria; very large match sets may need longer runs or a different split of rules.
  </Accordion>

  <Accordion title="Practices for maintainable Data Mines" icon="settings">
    <span id="practices-for-maintainable-data-mines" />

    * Start with one clear business rule per Data Mine when possible.
    * Name Data Mines and automations so owners know what is monitored.
    * Document assumptions (SLA definition, thresholds, timezone).
    * Revisit criteria when business rules change; archive Data Mines you no longer use.
  </Accordion>

  <Accordion title="Test and validate" icon="circle-check">
    <span id="test-and-validate" />

    1. Use **VIEW MATCHING DATA** (or the equivalent in your Table) to confirm which rows match.
    2. Start with a narrow criterion or time window if possible.
    3. Confirm automations fire on **OFF → ON** and **ON → OFF** as intended.
    4. If the automation uses an Agent, run end-to-end tests with realistic rows.
  </Accordion>
</AccordionGroup>

## Schedule an automation with a Data Mine

A Data Mine evaluates its Table on a schedule, so it can also be used purely as a way to run an automation on a recurring cadence — every 15 minutes, hourly, daily at 8:00 UTC, and so on. This is the only way to schedule an automation in Elementum; the [Time-Based trigger](/workflows/automation-triggers-reference#time-based) fires relative to a date field on a specific record and is not a recurring scheduler.

<Tip>
  This is a sidecar use of Data Mining — the primary purpose is monitoring warehouse data for state transitions. Reach for this pattern only when you genuinely need a scheduled automation; otherwise prefer event-based [automation triggers](/workflows/automation-triggers-reference).
</Tip>

To schedule with a Data Mine, use the **trigger record pattern**:

1. **Create a small dedicated Table** for scheduled jobs and add a single record per job, named clearly (for example `Next Run` or `Daily 8:00 Job`).
2. **Define a Data Mine on that Table** with the cadence you want as the schedule, and matching criteria that the trigger record satisfies.
3. **Include a column whose value changes each run** (for example `Updated On`) in the Data Mine's identifying columns alongside the record's stable ID. Each scheduled run then treats the row as a new logical row and re-evaluates its state from OFF.
4. **Update the trigger record from the automation** — typically the field captured by the volatile identifying column. This guarantees the next scheduled run produces an OFF → ON transition and fires the automation again.

Each scheduled run still consumes a Data Mine evaluation and the automation runs once per transition, so pick the longest cadence that meets the requirement and design the automation to be idempotent in case a run overlaps a previous one or is retried.

## Advanced patterns

**Cascade:** One automation updates data that a second Data Mine or process depends on—design order and idempotency carefully.

**Threshold ladders:** Separate Data Mines or criteria bands for different severities (for example response time tiers).

**Trends over time:** Compare snapshots or use conditions that encode “worse than prior period” if your table holds the needed history.

## Troubleshooting

<AccordionGroup>
  <Accordion title="Data Mine not triggering" icon="alert-circle">
    * Confirm CloudLink and Table access.
    * Validate criteria against sample rows.
    * Check schedule and that a **state transition** occurred (not ON → ON).
    * Review the [Data Mine](/workflows/automation-triggers-reference) trigger configuration on the automation.
  </Accordion>

  <Accordion title="Too many notifications" icon="bell">
    * Tighten criteria.
    * Run less often if business rules allow.
    * Confirm you are not relying on repeated ON → ON when you need only the first match.
  </Accordion>

  <Accordion title="Performance" icon="gauge">
    * Reduce the matched row count with stricter filters.
    * Align schedule with warehouse refresh and off-peak windows where applicable.
    * See [Data best practices](/data/data-best-practices) for limits and change strategies.
  </Accordion>

  <Accordion title="Agent steps not behaving as expected" icon="bot">
    * Review agent configuration and prompts in the automation.
    * Confirm the trigger passes the fields the agent needs.
  </Accordion>
</AccordionGroup>

## Measuring results

Track what matters for your process: time to act on exceptions, number of manual touches avoided, and error rates. A simple cost view is:

```
ROI ≈ (time saved × loaded cost rate × frequency) − setup and maintenance effort
```

Use it only with numbers your organization accepts; the value is in consistent measurement, not the formula itself.

For automation design, see [Automation System](/workflows/automation-system). For trigger options and variables, see [Automation triggers reference](/workflows/automation-triggers-reference).