Data Point Filters

Data Point Filters: Noise Reduction and Data Quality

Data Point Filters provide essential data quality control by filtering out noise, outliers, and invalid readings from sensor data. With four filter types—Deadband, Debounce, Discard Above, and Discard Below—you can ensure clean, reliable data for your automation workflows.

Overview

Real-world sensors often produce noisy, fluctuating, or occasionally erroneous readings. Data Point Filters sit between your sensors and your automation logic, cleaning and validating data before it triggers workflows or gets stored. This reduces false triggers, conserves storage space, and improves overall system reliability.

Filter Types

Krill provides four complementary filter types:

Filter	Icon	Purpose	Use Case
Deadband		Ignore small changes	Reduce noise from jittery sensors
Debounce		Wait for stability	Filter rapid fluctuations
Discard Above		Remove high outliers	Ignore sensor errors/spikes
Discard Below		Remove low outliers	Ignore invalid low readings

Filter Processing Flow

graph TD
    A[Raw Sensor Reading] --> B{Discard Below<br/>Filter}
    B -->|Too Low| C[Reject Value]
    B -->|Valid| D{Discard Above<br/>Filter}
    D -->|Too High| C
    D -->|Valid| E{Deadband<br/>Filter}
    E -->|Change Too Small| C
    E -->|Significant Change| F{Debounce<br/>Filter}
    F -->|Unstable| G[Wait for Stability]
    F -->|Stable| H[Accept Value]
    G --> F
    H --> I[Store in Data Point]
    I --> J[Trigger Automation]

---

Deadband Filter

Purpose: Ignore small value changes to reduce noise and excessive updates.

How Deadband Works

Deadband filtering prevents recording values that haven’t changed significantly. Only when a value changes by more than the deadband threshold is it recorded.

Example:

Deadband Threshold: 2.0°C
Current Value: 23.0°C

New Reading: 23.5°C → Rejected (change = 0.5 < 2.0)
New Reading: 24.0°C → Rejected (change = 1.0 < 2.0)  
New Reading: 25.5°C → Accepted (change = 2.5 > 2.0)

Use Cases

• Temperature Sensors: Ignore ±0.1°C fluctuations around setpoint
• Pressure Transducers: Filter out pressure oscillations
• Flow Meters: Reduce noise from turbulent flow
• Analog Inputs: Stabilize ADC readings
• Position Sensors: Ignore micro-movements

Configuration

• threshold: Minimum change required to record value

---

Debounce Filter

Purpose: Wait for value stability before recording to filter rapid fluctuations.

How Debounce Works

Debounce requires values to remain stable for a specified duration before being recorded. If the value changes during the debounce period, the timer resets.

Example:

Debounce Period: 5 seconds
Time 0s: Value = 23.0°C → Start timer
Time 2s: Value = 23.5°C → Reset timer (value changed)
Time 4s: Value = 23.5°C → Continue waiting
Time 7s: Value = 23.5°C → Accept (stable for 5s)

Use Cases

• Digital Inputs: Debounce mechanical switches
• Rapid Fluctuations: Filter oscillating sensor readings
• Settling Time: Wait for sensor warm-up
• State Changes: Confirm persistent state changes
• Noise Filtering: Remove high-frequency noise

Configuration

• delayMs: Stability period in milliseconds (e.g., 5000 for 5 seconds)

---

Discard Above Filter

Purpose: Filter out and ignore values exceeding a specified maximum.

How Discard Above Works

Any reading above the configured maximum threshold is discarded and not recorded. This prevents sensor errors, spikes, or impossible values from affecting your automation.

Example:

Maximum Threshold: 100°C
Reading: 75°C → Accepted
Reading: 98°C → Accepted
Reading: 150°C → Rejected (sensor error)
Reading: 82°C → Accepted

Use Cases

• Sensor Range Limits: Enforce physical sensor limits
• Error Detection: Catch sensor failures reporting impossible values
• Spike Filtering: Remove transient high spikes
• Safety Limits: Prevent recording dangerous values
• Data Quality: Remove outliers from datasets

Configuration

• maximum: Maximum acceptable value (readings above are discarded)

---

Discard Below Filter

Purpose: Filter out and ignore values falling below a specified minimum.

How Discard Below Works

Any reading below the configured minimum threshold is discarded. This catches sensor failures, disconnections, or physically impossible low values.

Example:

Minimum Threshold: -10°C
Reading: 15°C → Accepted
Reading: 5°C → Accepted
Reading: -50°C → Rejected (sensor error)
Reading: 8°C → Accepted

Use Cases

• Sensor Range Limits: Enforce minimum valid readings
• Disconnect Detection: Catch sensor disconnections (often report 0 or very low)
• Physical Limits: Remove impossible low values
• Quality Control: Filter out calibration drift
• Safety Monitoring: Ignore erroneous low readings

Configuration

• minimum: Minimum acceptable value (readings below are discarded)

---

Combining Filters

Filters can be chained together for comprehensive data cleaning:

Example: Temperature Sensor with Full Filtering

Temperature Sensor (raw)
  └─> Discard Below (-50°C)
      └─> Discard Above (150°C)
          └─> Deadband (±1.0°C)
              └─> Debounce (3 seconds)
                  └─> Clean Data Point

This configuration:

1. Rejects impossible temperatures outside sensor range
2. Ignores changes smaller than 1°C
3. Waits for 3 seconds of stability
4. Results in clean, reliable temperature data

Example: Pressure Sensor

Pressure Sensor (0-200 PSI)
  └─> Discard Below (0 PSI)
      └─> Discard Above (210 PSI)
          └─> Deadband (±2 PSI)
              └─> Validated Pressure

Example: Flow Meter

Flow Meter
  └─> Discard Below (0 GPM)
      └─> Discard Above (500 GPM)
          └─> Deadband (±0.5 GPM)
              └─> Debounce (2 seconds)
                  └─> Stable Flow Rate

Configuration Guidelines

Deadband Thresholds

Sensor Type	Typical Deadband	Reasoning
Temperature	±0.5-2.0°C	Reduce oscillation around setpoint
Pressure	±1-5 PSI	Filter transducer noise
Humidity	±2-5% RH	Ignore normal fluctuations
Flow Rate	±0.1-1.0 GPM	Smooth turbulent flow
Voltage	±0.1-0.5V	Stabilize ADC readings

Debounce Periods

Application	Typical Period	Reasoning
Mechanical Switches	10-50 ms	Eliminate contact bounce
Sensor Settling	1-5 seconds	Allow sensor stabilization
State Changes	3-10 seconds	Confirm persistent changes
High Noise	5-30 seconds	Filter rapid fluctuations

Range Limits

Set Discard Above/Below based on:

• Physical Limits: Sensor’s rated range
• Application Limits: Expected operating range
• Safety Margins: Add 10-20% beyond normal range
• Error Detection: Values outside range indicate problems

Integration Points

• Data Points: Attach filters directly to data points
• Triggers: Filtered data provides reliable trigger conditions
• Executors: Use clean data for calculations and control
• Storage: Reduce storage by not saving filtered values
• Logging: Only log validated, filtered data

Best Practices

• Start Conservative: Begin with wide filters, tighten as needed
• Test Thoroughly: Validate filters don’t remove valid data
• Layer Filters: Use multiple filters for comprehensive cleaning
• Monitor Performance: Track filter rejection rates
• Document Settings: Record why specific thresholds were chosen
• Sensor-Specific: Tune filters for each sensor’s characteristics
• Validation Period: Run parallel unfiltered logging initially

Performance Impact

Filters provide significant benefits:

• Reduced Storage: Fewer values written to time-series database
• Fewer Triggers: Less false alarms and unnecessary automation
• Better Performance: Less processing of noisy data
• Cleaner Analytics: More reliable trend analysis
• Lower Bandwidth: Less data transmitted across network

Data Point Filters are essential for transforming noisy, unreliable sensor data into clean, trustworthy measurements that drive reliable automation.