Developer Guide

Extend Hurricane with custom application health checks, business observability metrics, and remote debugging tools.

Custom Health Checks via Django System Check Framework

Django includes a battle-tested System Check Framework designed to inspect your codebase, database schema, environment variables, and third-party integrations for operational anomalies.

Hurricane natively harnesses this framework for Kubernetes container lifecycles:

  • Readiness Probes (tags=["readiness"]): Executed on every GET /ready probe. If any check tagged with "readiness" returns a Django Error or Critical, Hurricane catches the exception, updates the Prometheus readiness metric to 0, and responds with HTTP 500. Kubernetes immediately sheds Ingress traffic from the pod until the check passes.
  • Liveness Probes (tags=["liveness"]): Executed on every GET /alive probe. If an unrecoverable failure occurs, returning an Error causes /alive to fail. Repeated failures trigger Kubernetes to terminate and restart the pod (self-healing).
  • Startup Probes (tags=["startup"]): Executed during initial boot before traffic begins, ensuring critical migrations, encryption keys, and seed data are ready.

Practical Health Check Examples

Register custom checks in any installed app (typically inside a checks.py file imported in your AppConfig.ready()):

1. Database Integrity & Seed Data Check (readiness)

Ensure critical database records exist (e.g. default tenant, active billing currencies, or configuration seed data) and that the database connection pool is healthy:

myapp/checks.py
from django.core.checks import Error, Warning, register
from django.db import connection


@register("readiness")
def check_database_and_seed_data(app_configs, **kwargs):
    errors = []
    try:
        with connection.cursor() as cursor:
            cursor.execute("SELECT 1;")
            row = cursor.fetchone()
            if not row or row[0] != 1:
                errors.append(
                    Error(
                        "Database ping returned unexpected response.",
                        hint="Check database connectivity and connection pool health.",
                        id="myapp.E001",
                    )
                )

        # Verify critical seed data or configuration records
        from myapp.models import SystemConfiguration

        if not SystemConfiguration.objects.filter(is_active=True).exists():
            errors.append(
                Error(
                    "Active SystemConfiguration record missing in database.",
                    hint="Run database seed fixtures before launching traffic.",
                    id="myapp.E002",
                )
            )
    except Exception as exc:
        errors.append(
            Error(
                f"Database query failed during readiness check: {exc}",
                hint="Verify database network access, credentials, and pool limits.",
                id="myapp.E003",
            )
        )
    return errors

2. In-Cluster & Upstream Microservice Checks (readiness)

Verify that upstream dependencies within your Kubernetes cluster (e.g. auth services, payment gateways, search clusters) are responding with acceptable latency:

myapp/checks.py
import urllib.request
from django.core.checks import Error, Warning, register


@register("readiness")
def check_auth_microservice(app_configs, **kwargs):
    errors = []
    # Upstream service running inside the Kubernetes cluster
    auth_service_url = "http://auth-service.default.svc.cluster.local:8080/healthz"

    try:
        req = urllib.request.Request(auth_service_url, headers={"User-Agent": "Hurricane-HealthCheck"})
        with urllib.request.urlopen(req, timeout=1.5) as resp:
            if resp.status != 200:
                errors.append(
                    Error(
                        f"Auth microservice returned non-200 status: {resp.status}",
                        hint="Check auth-service pod logs and Ingress routing.",
                        id="myapp.E010",
                    )
                )
    except Exception as exc:
        # Returning Error sheds traffic until the upstream service is reachable
        errors.append(
            Error(
                f"Unable to reach upstream Auth service: {exc}",
                hint="Verify CoreDNS resolution and auth-service deployment health.",
                id="myapp.E011",
            )
        )
    return errors

3. Clock Synchronization & Solar / Ephemeris State (readiness)

Accurate system clocks are critical for JWT token validation, OAuth flows, and cryptographic signatures. Furthermore, time-sensitive applications (such as solar energy management, dynamic electricity tariffs, agricultural automation, or circadian lighting systems) depend on accurate astronomical calculations and timezone state:

myapp/checks.py
import time
from datetime import datetime, timezone
from django.core.checks import Error, Warning, register


@register("readiness")
def check_clock_and_solar_state(app_configs, **kwargs):
    errors = []

    # 1. System Clock Drift & Monotonicity Check
    # Verify that local container time hasn't drifted wildly from UTC reference
    now_utc = datetime.now(timezone.utc)
    if now_utc.year < 2026:
        errors.append(
            Error(
                f"Container system clock is invalid: {now_utc.isoformat()}",
                hint="Verify host node chrony/NTP synchronization in the Kubernetes cluster.",
                id="myapp.E020",
            )
        )

    # 2. Solar / Astronomical Window Validation
    # In smart energy or IoT deployments, ensure astronomical calculations are operational
    try:
        from myapp.services.astronomy import get_solar_elevation, is_daylight_tariff_active

        # Ensure solar coordinates computation returns valid physical range (-90 to +90 degrees)
        solar_elevation = get_solar_elevation(timestamp=now_utc)
        if not (-90.0 <= solar_elevation <= 90.0):
            errors.append(
                Error(
                    f"Computed solar elevation out of bounds: {solar_elevation}",
                    hint="Check ephemeris ephem/astral library installation and lat/long coordinates.",
                    id="myapp.E021",
                )
            )
    except ImportError:
        # Module not installed; skip solar check
        pass
    except Exception as exc:
        errors.append(
            Warning(
                f"Solar state computation error: {exc}",
                hint="Verify astronomical coordinates in settings.",
                id="myapp.W022",
            )
        )

    return errors

4. Cache & Task Broker Liveness (liveness)

Liveness probes should only fail on unrecoverable process deadlocks (where restarting the container fixes the issue):

myapp/checks.py
from django.core.checks import Error, register
import redis


@register("liveness")
def check_local_worker_deadlock(app_configs, **kwargs):
    errors = []
    # Verify that internal Python worker thread queues and event loop state are responsive
    try:
        from myapp.worker import get_worker_heartbeat

        last_heartbeat_seconds = get_worker_heartbeat()
        if last_heartbeat_seconds > 60:
            errors.append(
                Error(
                    f"Internal worker heartbeat frozen for {last_heartbeat_seconds}s.",
                    hint="Worker thread is deadlocked; container restart required.",
                    id="myapp.E030",
                )
            )
    except Exception as exc:
        errors.append(
            Error(
                f"Worker thread health check failed: {exc}",
                id="myapp.E031",
            )
        )
    return errors

5. Storage Mounts & ML Model Artifacts (startup)

Before marking a pod ready for traffic on boot, confirm that mounted persistent volumes (PVCs) contain required model weights or configuration files:

myapp/checks.py
import os
from django.core.checks import Error, register


@register("startup")
def check_mounted_model_artifacts(app_configs, **kwargs):
    errors = []
    model_path = "/models/classifier_v2.onnx"

    if not os.path.exists(model_path):
        errors.append(
            Error(
                f"Required machine learning model artifact not found at '{model_path}'.",
                hint="Verify that the Kubernetes PersistentVolumeClaim (PVC) is correctly mounted.",
                id="myapp.E040",
            )
        )
    elif os.path.getsize(model_path) < 1024:
        errors.append(
            Error(
                f"Model artifact file at '{model_path}' appears corrupt or empty.",
                hint="Check initContainer model downloader logs.",
                id="myapp.E041",
            )
        )
    return errors

Best Practices for System Checks in Kubernetes

  1. Keep Checks Fast & Non-Blocking: Readiness checks run periodically (typically every 2–5 seconds). Use strict network socket timeouts (e.g. timeout=1.0s) on all external HTTP or database calls.
  2. Differentiate Severity:
    • Error or Critical: Triggers HTTP 500 on the probe, withdrawing or restarting the pod.
    • Warning or Info: Logged for observability without failing the health check.
  3. Use the Right Tag:
    • readiness: Temporary external conditions (database high load, upstream API maintenance). Sheds traffic without killing the pod.
    • liveness: Unrecoverable internal conditions (deadlocked threads, corrupted memory heap). Kills and restarts the pod.
    • startup: One-time initialization prerequisites (schema migrations, seed data, model file downloads). Protects against premature liveness kills during cold starts.

For more details on registering and structuring checks, visit the Official Django System Check Framework Documentation.

Custom Prometheus Metrics

You can register custom application metrics that are exposed directly on /metrics:

myapp/metrics.py
from hurricane.metrics import CalculatedMetric, CounterMetric
from myapp.models import Order


class ActiveOrdersMetric(CalculatedMetric):
    """Current number of active processing orders."""

    code = "active_orders_count"

    @classmethod
    def get_value(cls, *args):
        return Order.objects.filter(status="processing").count()


class UserSignupCounter(CounterMetric):
    """Total user registrations."""

    code = "user_signups_total"

Increment counter in your views:

UserSignupCounter.increment()

Remote Debugging in Kubernetes

Debugging containerized Django applications in a remote Kubernetes cluster is easy with Hurricane's built-in DAP debugger support:

VS Code (Debugpy)

Start Hurricane with the --debugger flag:

python manage.py serve --debugger --debugger-port=5678

Add a configuration to .vscode/launch.json:

{
  "name": "Attach: Remote Hurricane",
  "type": "debugpy",
  "request": "attach",
  "connect": {
    "host": "localhost",
    "port": 5678
  },
  "pathMappings": [
    {
      "localRoot": "${workspaceFolder}",
      "remoteRoot": "/app"
    }
  ]
}

Port-forward the debugger port (kubectl port-forward pod/my-pod 5678:5678) and press F5 in VS Code!

JetBrains PyCharm

Start Hurricane pointing to your PyCharm debug server:

python manage.py serve --pycharm-host=host.docker.internal --pycharm-port=8071
Edit this page on GitHub
Table of Contents