Operations & Monitoring

This guide focuses on Neutron-specific operational requirements for the Connect oracle. For comprehensive troubleshooting and metrics details, see Skip’s Connect documentation.

Performance Requirements

Oracle performance directly impacts validator rewards. Maintain ≥95% success rate for full compensation.

Reward Tiers

Success Rate	Reward Level
≥98%	Full rewards ($3,000/month)
95-98%	Proportional rewards
<95%	No rewards

Validators below 95% for 2 consecutive months or 3 months in a 6-month period may be rotated out. See Validator Incentives.

Monitoring Oracle Success Rate

Quick Health Check

Create a monitoring script:

#!/bin/bash
# /usr/local/bin/check-oracle.sh

echo "=== Neutron Oracle Health Check ==="

# Check Connect service
if systemctl is-active --quiet connect; then
    echo "✓ Connect service running"
else
    echo "✗ Connect service stopped"
    exit 1
fi

# Check Connect API
if curl -sf http://localhost:8080/connect/oracle/v2/prices > /dev/null; then
    TIMESTAMP=$(curl -s http://localhost:8080/connect/oracle/v2/prices | jq -r '.timestamp')
    echo "✓ Connect API responding (last update: $TIMESTAMP)"
else
    echo "✗ Connect API not responding"
fi

# Calculate success rate
TOTAL=$(curl -s http://localhost:26660/metrics 2>/dev/null | grep 'app_oracle_responses{.*status="Success"}' | awk '{print $2}')
FAILURES=$(curl -s http://localhost:26660/metrics 2>/dev/null | grep 'app_oracle_responses{.*status="Failure"}' | awk '{print $2}')

if [ -n "$TOTAL" ] && [ -n "$FAILURES" ]; then
    SUCCESS_RATE=$(awk "BEGIN {print ($TOTAL / ($TOTAL + $FAILURES)) * 100}")

    if (( $(echo "$SUCCESS_RATE >= 98" | bc -l) )); then
        echo "✓ Success rate: ${SUCCESS_RATE}% (Full rewards)"
    elif (( $(echo "$SUCCESS_RATE >= 95" | bc -l) )); then
        echo "⚠ Success rate: ${SUCCESS_RATE}% (Partial rewards - improve performance)"
    else
        echo "✗ Success rate: ${SUCCESS_RATE}% (No rewards - action required)"
    fi
else
    echo "⚠ No metrics data yet"
fi

echo "=== End Health Check ==="

Make it executable:

sudo chmod +x /usr/local/bin/check-oracle.sh
/usr/local/bin/check-oracle.sh

Continuous Monitoring

Run periodic checks via cron:

# Add to crontab
crontab -e

# Check oracle health every 10 minutes
*/10 * * * * /usr/local/bin/check-oracle.sh >> /var/log/oracle-health.log 2>&1

Key Metrics

Oracle Response Metrics

# View success/failure counts
curl -s http://localhost:26660/metrics | grep app_oracle_responses

Output:

app_oracle_responses{chain_id="neutron-1",status="Failure"} 128
app_oracle_responses{chain_id="neutron-1",status="Success"} 841042

Connect Service Metrics

# View Connect-specific metrics
curl -s http://localhost:8002/metrics | grep connect_

For detailed metrics documentation, see Skip’s Metrics Guide.

Common Issues

Low Success Rate

Symptoms: Success rate below 95% Quick Diagnostics:

# 1. Check Connect logs
sudo journalctl -u connect -n 100 | grep error

# 2. Test Connect API responsiveness
time curl -s http://localhost:8080/connect/oracle/v2/prices > /dev/null

# 3. Verify node gRPC
grpcurl -plaintext localhost:9090 list

Common Fixes:

Connect service stopped

sudo systemctl restart connect
sudo systemctl status connect

Slow Connect responses

Increase timeout in app.toml:

client_timeout = "5s"  # Increase from 2s

Then restart neutrond:

sudo systemctl restart neutrond

Network connectivity issues

Test outbound connections:

curl -I https://api.binance.com
curl -I https://api.coinbase.com

Ensure firewall allows outbound HTTPS (port 443).

gRPC connection issues

Verify gRPC is enabled in app.toml:

[grpc]
enable = true
address = "0.0.0.0:9090"

Missing Prices (Normal)

Log message:

{"level":"info","msg":"failed to calculate prices for price feeds","missing_prices":["RNDR/USD","AGIX/USD"]}

This is normal—Connect logs missing prices for feeds temporarily unavailable from providers. This does not affect your success rate unless all major pairs (BTC, ETH, NTRN) are missing.

When to Act:

All prices missing for >5 minutes
Major pairs (BTC/USD, ETH/USD, NTRN/USD) consistently missing

Fix: Restart Connect:

sudo systemctl restart connect

WebSocket Errors (Usually Normal)

Log message:

{"level":"error","provider":"binance_ws","error":"websocket connection handler failed"}

WebSocket disconnections are common and Connect automatically reconnects. Only worry if:

All WebSocket providers fail simultaneously for >30 minutes
Accompanied by low success rate

Alerting Setup

Simple Alert Script

Create an alert script for low success rate:

#!/bin/bash
# /usr/local/bin/alert-oracle.sh

THRESHOLD=95
WEBHOOK_URL="YOUR_DISCORD_OR_SLACK_WEBHOOK"

TOTAL=$(curl -s http://localhost:26660/metrics | grep 'app_oracle_responses{.*status="Success"}' | awk '{print $2}')
FAILURES=$(curl -s http://localhost:26660/metrics | grep 'app_oracle_responses{.*status="Failure"}' | awk '{print $2}')

if [ -n "$TOTAL" ] && [ -n "$FAILURES" ]; then
    SUCCESS_RATE=$(awk "BEGIN {print ($TOTAL / ($TOTAL + $FAILURES)) * 100}")

    if (( $(echo "$SUCCESS_RATE < $THRESHOLD" | bc -l) )); then
        MESSAGE="🚨 Oracle Alert: Success rate is ${SUCCESS_RATE}% (threshold: ${THRESHOLD}%)"

        # Send to webhook (Discord/Slack)
        curl -H "Content-Type: application/json" \
             -d "{\"content\":\"$MESSAGE\"}" \
             "$WEBHOOK_URL"
    fi
fi

Run via cron every 30 minutes:

*/30 * * * * /usr/local/bin/alert-oracle.sh

Prometheus + Grafana

For production monitoring, use Prometheus and Grafana:

Prometheus Configuration

Add to prometheus.yml:

scrape_configs:
  # Connect metrics
  - job_name: 'connect_oracle'
    static_configs:
      - targets: ['localhost:8002']
    scrape_interval: 15s

  # Neutron oracle metrics
  - job_name: 'neutron_oracle'
    static_configs:
      - targets: ['localhost:8001']
    scrape_interval: 15s

  # Neutron node metrics
  - job_name: 'neutron_node'
    static_configs:
      - targets: ['localhost:26660']
    scrape_interval: 15s

Key Prometheus Queries

Oracle success rate:

rate(app_oracle_responses{status="Success"}[5m])
/
(rate(app_oracle_responses{status="Success"}[5m]) + rate(app_oracle_responses{status="Failure"}[5m])) * 100

Connect provider latency (95th percentile):

histogram_quantile(0.95, rate(connect_provider_query_duration_seconds_bucket[5m]))

For complete metrics documentation, see Skip’s Metrics Overview.

Advanced Setups

Remote Signers

If using remote signers (Horcrux, TMKMS), Connect runs on the validator node, not the signer: Compatible versions:

Horcrux: v3.3.0+ (setup guide)
TMKMS: v0.13.1+ (setup guide)

See Skip’s Advanced Setups for configuration details.

Distributed Validators

For distributed validator setups, run Connect on each validator node:

Ensures fast failover
Maintains oracle state on all nodes
No oracle warm-up time during failover

See Skip’s Advanced Setups for detailed configurations.

Additional Resources

Skip Docs: Troubleshooting

Comprehensive troubleshooting and FAQs

Skip Docs: Metrics

Complete metrics reference

Oracle Setup

Review setup and configuration

Validator Incentives

Understand reward calculations

Getting Started

Oracle

Economics

Relayers

Performance Requirements

Reward Tiers

Monitoring Oracle Success Rate

Quick Health Check

Continuous Monitoring

Key Metrics

Oracle Response Metrics

Connect Service Metrics

Common Issues

Low Success Rate

Missing Prices (Normal)

WebSocket Errors (Usually Normal)

Alerting Setup

Simple Alert Script

Prometheus + Grafana

Prometheus Configuration

Key Prometheus Queries

Advanced Setups

Remote Signers

Distributed Validators

Additional Resources

Skip Docs: Troubleshooting

Skip Docs: Metrics

Oracle Setup

Validator Incentives

Getting Started

Oracle

Economics

Relayers

​Performance Requirements

​Reward Tiers

​Monitoring Oracle Success Rate

​Quick Health Check

​Continuous Monitoring

​Key Metrics

​Oracle Response Metrics

​Connect Service Metrics

​Common Issues

​Low Success Rate

​Missing Prices (Normal)

​WebSocket Errors (Usually Normal)

​Alerting Setup

​Simple Alert Script

​Prometheus + Grafana

​Prometheus Configuration

​Key Prometheus Queries

​Advanced Setups

​Remote Signers

​Distributed Validators

​Additional Resources

Skip Docs: Troubleshooting

Skip Docs: Metrics

Oracle Setup

Validator Incentives

Performance Requirements

Reward Tiers

Monitoring Oracle Success Rate

Quick Health Check

Continuous Monitoring

Key Metrics

Oracle Response Metrics

Connect Service Metrics

Common Issues

Low Success Rate

Missing Prices (Normal)

WebSocket Errors (Usually Normal)

Alerting Setup

Simple Alert Script

Prometheus + Grafana

Prometheus Configuration

Key Prometheus Queries

Advanced Setups

Remote Signers

Distributed Validators

Additional Resources