Nexus Ops Hub

Overview

Nexus is a monolithic full-stack application split into three runtime processes: a Fastify 5 API server, a React 19 SPA served by nginx, and a BullMQ worker process for background job execution. All three share the same TypeScript codebase in the nexus/ directory, which is a git subrepo with its own .git (no remote configured).

The API handles all business logic, authentication, and data access through Drizzle ORM against a PostgreSQL 16 database (nexus_crm). Redis 7 backs both the BullMQ job queues and an application cache layer. Socket.io provides real-time push notifications to connected React clients without polling.

Nexus covers six major domains: CRM (contacts, companies, deals, tasks, activities), Marketing (campaigns, sequences, segments, forms, landing pages, lead scoring), Sales (products, quotes, playbooks), Service Desk (tickets, knowledge base, CSAT surveys), Travel Operations (trips, bookings, travelers, suppliers, finance, itinerary builder), and an Operations Hub that aggregates external systems (Slack, n8n, Google Workspace, analytics) into a single view.

Tech Stack

Backend

Frontend

Infrastructure

Key Dependencies

• @fastify/cors — Cross-origin configuration
• @fastify/helmet — Security headers
• @fastify/rate-limit — Request throttling
• @fastify/multipart — File uploads
• @fastify/swagger — API documentation

• pdf-lib — PDF generation for quotes and itineraries
• imapflow + mailparser — Inbound email processing
• dompurify + jsdom — HTML sanitization
• ioredis — Redis client for BullMQ and caching
• date-fns — Date manipulation utilities

Architecture

Nexus runs as 5 Docker containers orchestrated by Docker Compose. All containers have memory limits, JSON log rotation (10 MB / 3 files), and health checks. Traefik handles external TLS termination and path-based routing: /api/*, /socket.io/*, /lp/*, /kb/*, /forms/*, /unsubscribe/*, and /meet/* route to the API container; everything else routes to the web container.

Network Topology

Three Docker networks isolate traffic:

• nexus-net (bridge) — Internal communication between all 5 Nexus containers
• proxy (external) — Shared with Traefik for TLS-terminated inbound traffic
• internal (external) — Shared with the TT-API container for the travel proxy

Named Volumes

• nexus-db-data — PostgreSQL data directory (persistent)
• nexus-redis-data — Redis AOF persistence
• uploads — File uploads shared between API and worker containers

Worker entrypoint: The worker container uses the same Docker image as the API but overrides the command to node dist/workers/index.js. This means a single build produces both the API and worker artifacts.

Database

The nexus_crm database runs in a dedicated PostgreSQL 16 container (nexus-db), separate from the main TT-API database. The schema is managed by Drizzle ORM with TypeScript type generation. All tables use UUIDs for primary keys (except ticket numbers which use SERIAL), soft deletes via isDeleted + deletedAt, and monetary values in cents.

Key Statistics

Schema

54 Tables

• 16 schema module files
• 60 pgEnum type definitions
• 68 columns converted VARCHAR → pgEnum
• All enums use nx_ prefix

Features

Data Patterns

• UUID primary keys (SERIAL for tickets)
• Soft deletes on all domain tables
• JSONB custom fields on contacts, companies, deals
• TEXT[] arrays for flexible tagging

All enums are defined in nexus/src/db/schema/enums.ts. They enforce type safety at the database level and replaced 68 VARCHAR columns through a manual SQL migration (Drizzle cannot handle VARCHAR-to-pgEnum conversions).

RBAC Role System

Nexus uses a 5-tier role-based access control system defined in nexus/src/middleware/rbac.ts. Each role has a numeric rank, and access checks compare the user's rank against the minimum required for the operation. Two guard functions are provided: requireRole(min) for hierarchical checks and requireAnyRole(...roles) for OR-logic checks. Unknown or missing roles receive rank -1 and are always denied.

Protected Route Groups

All 10 main route groups are protected by the RBAC middleware:

• Contacts & Companies
• Deals & Pipelines
• Marketing (campaigns, segments, forms)
• Automation (workflows, sequences)
• Travel Operations (trips, bookings)

• Service Hub (tickets, KB)
• Products & Quotes
• Reports & Analytics
• Integrations (n8n, Slack, Google)
• Settings & Team Management

JWT Authentication Flow

Login

User submits email + password. Server validates with bcryptjs, issues JWT access token (short-lived) and refresh token (long-lived httpOnly cookie).

Bearer Token

React SPA attaches the JWT as Authorization: Bearer <token> on every API call. Fastify preHandler hook decodes and validates.

Token Refresh

When the access token expires, the client calls the refresh endpoint with the httpOnly cookie. A new access token is issued silently.

RBAC Guard

After authentication, requireRole() checks the user's rank against the route minimum. Insufficient rank returns 403 Forbidden.

Rank inheritance: Routes specify a minimum rank (e.g., requireRole('advisor') means rank ≥ 2). Managers and admins automatically inherit all lower-tier permissions without explicit grants.

Auth Mechanisms

Nexus uses three distinct authentication mechanisms, each serving a different type of caller. All three are enforced via Fastify preHandler hooks and return appropriate HTTP status codes on failure.

Internal Service Auth

The serviceAuth.ts middleware provides machine-to-machine authentication for internal services (currently TT-API). It performs a timing-safe comparison of the x-internal-service request header against the TT_INTERNAL_SERVICE_SECRET environment variable. If the secret is not configured, the endpoint returns 503 Service Unavailable rather than silently allowing unauthenticated access.

Internal service routes are registered in server.ts under the /api/internal prefix. These endpoints allow TT-API to sync contacts and deals into the Nexus CRM without requiring JWT authentication.

Secret required: The TT_INTERNAL_SERVICE_SECRET must be set in the production .env file and must match the value configured in TT-API. Without it, all internal service endpoints return 503.

API Routes

The Fastify API exposes 37 route files organized by domain, plus internal service routes. All routes are prefixed with /api/ and require JWT authentication (except auth, public forms, unsubscribe, health, and internal service endpoints). Route handlers delegate to service files for business logic.

Machine-to-machine endpoints registered under /api/internal in server.ts. Protected by serviceAuth.ts middleware (timing-safe x-internal-service header check) instead of JWT auth.

React Frontend

The frontend is a React 19 SPA built with Vite, located in nexus/src/client/ with its own package.json. It uses React Router for client-side navigation and lazy-loads all page components via React.lazy() for code splitting. State management uses Zustand stores for auth and notifications, with component-local state for everything else.

Page Groups (80+ components)

Frontend Architecture

• AppLayout — Wraps all authenticated routes with sidebar navigation, header, and notification panel
• ProtectedRoute — Guards authenticated routes, redirects to /login if no valid session
• Zustand Stores — authStore (JWT, user info, refresh logic) and notificationStore (Socket.io events, unread count)
• API Client — Centralized fetch wrapper in lib/ with automatic token attachment and refresh
• TanStack Table — Used for all data table views with server-side sorting, filtering, and pagination
• @dnd-kit — Drag-and-drop for deal Kanban boards, workflow step ordering, and pipeline stage management

Build output: vite build produces a static bundle in nexus/src/client/dist/ that the nexus-web container serves via nginx. The nginx config includes CSP headers and a catch-all rule that serves index.html for client-side routing.

BullMQ Workers

Five dedicated worker processors run in the nexus-worker container, consuming jobs from Redis-backed BullMQ queues. Each worker has concurrency 5 and provides automatic retries with exponential backoff. A sixth process — the scheduled campaign checker — runs on a 60-second interval to trigger campaigns whose sendAt time has arrived.

Worker Resilience (Round 11)

All four worker processors now include hardened error handling:

• 'error' event handler on every worker — catches connection errors and unexpected failures without crashing the process
• 'stalled' event handler on every worker — logs stalled job IDs for investigation
• maxStalledCount: 2 on all workers — jobs that stall twice are moved to the failed set
• Graceful shutdown with a 10-second timeout using a Promise.race pattern — ensures workers drain cleanly on SIGTERM without hanging indefinitely

Sends individual emails via the configured ESP. Each job represents one recipient for a campaign, sequence step, or ad-hoc email. Handles merge variable interpolation at send time.

• Tracks campaign completion: atomically increments totalSent, flips status to sent when all recipients processed
• Supports A/B test variants via variantId field
• Failed sends retry with exponential backoff (3 attempts)
• Dead-lettered messages trigger alert notifications

Processes inbound webhook events from the ESP — opens, clicks, bounces, spam complaints, and unsubscribes. Updates campaign statistics and contact engagement records.

• Deduplicates events by message ID + event type
• Bounces and complaints trigger automatic contact status updates
• Click events resolve redirect URLs and record the target
• Top-level try/catch with context logging (job ID, event type) for debugging failures

Executes workflow steps asynchronously. Each completed step schedules the next, forming an execution chain. Also handles lead score recalculation via the lead-score-recalculate job name.

• Step types: send_email, update_property, add_to_segment, wait, branch, call_webhook
• Branch steps evaluate conditions and route to different paths
• Wait steps schedule a delayed job (minutes to days)
• Uses WorkflowEnrollmentRow type alias — 10 any casts removed for proper typing

Handles time-based execution for email sequences. Processes sequence-step jobs that fire at configured delay intervals between sequence steps.

• Checks unenroll conditions before each step (replied, converted, manually removed)
• Respects send windows (business hours, timezone-aware)
• Coordinates with the Email Worker for actual delivery
• Uses SequenceEnrollmentRow type alias with typed cnt field for step counting

Refreshes dynamic segment membership. Evaluates segment criteria against all contacts and replaces the current member set with the freshly computed result.

• Atomic replace: deletes all current members, inserts the evaluated set
• Logs member count after each refresh
• Runs on demand when segments are modified or on a configurable schedule

Scheduled Campaign Checker: In addition to the 5 queue workers, an setInterval loop runs every 60 seconds to find campaigns with status='scheduled' and sendAt ≤ now, then triggers their send. This runs in the same worker process.

Travel Proxy

Nexus proxies requests to the main TT-API for travel data that lives in tt_fresh_db. The proxy layer at /api/travel/* forwards requests over the Docker internal network to http://tt-api:3100, adding resilience patterns to protect against downstream failures. The TT-API validates requests using the TT_API_KEY header.

Resilience Patterns

Circuit Breaker States

Closed (Normal)

All requests pass through. Failures within the window are counted. If failures exceed the threshold, circuit opens.

Open (Failing Fast)

All requests immediately fail with a circuit-open error. After the reset timeout, transitions to half-open.

Half-Open (Testing)

Allows requests through. If enough consecutive successes occur, closes the circuit. Any failure reopens it.

Internal network only. The travel proxy communicates over Docker's internal bridge network. It is not exposed to the public internet. Travel data (clients, bookings, trips) in TT-API is the source of truth — Nexus travel routes are deprecated in favor of the proxy.

Slack Integration

Nexus integrates with Slack at two levels: inbound event processing (Slack sends events to Nexus) and outbound messaging (Nexus posts to Slack channels). The signing secret is required — if SLACK_SIGNING_SECRET is not set, the endpoint returns 503.

Inbound: Event Handling

• Webhook verification — All inbound requests are verified using the Slack signing secret with HMAC-SHA256. Invalid signatures are rejected with 401.
• Event processing — Supported events include messages, reactions, app mentions, and interactive actions.
• Slash commands — Custom slash commands for creating tickets, looking up contacts, and checking deal status directly from Slack.
• Request tracking — Slack requests are tracked with type (nx_slack_request_type) and status (nx_slack_request_status) enums.

Outbound: Notifications

• Ticket assignments and SLA breaches posted to support channels
• Deal stage changes posted to sales channels
• Campaign completion summaries posted to marketing channels
• System health alerts posted to the #monitoring channel

Route Files

Email Branding & Compliance

The unsubscribeService.ts provides a getComplianceFooter() function that generates CAN-SPAM compliant email footers for all outbound marketing emails. The footer is styled with the Dark Sanctuary brand identity for a cohesive experience.

Footer Styling

Unsubscribe links are generated with HMAC using the UNSUBSCRIBE_SECRET env var. The unsubscribe endpoint is public (no JWT required) and processes opt-outs immediately. Production startup warns if the default placeholder secret is still in use.

Development

Local development requires two terminal sessions: one for the Fastify backend (tsx watch) and one for the Vite React frontend. The frontend has its own package.json inside nexus/src/client/.

Initial Setup

cd nexus/
npm install                    # Backend dependencies
cd src/client && npm install   # Frontend dependencies
cd ../..                       # Back to nexus root

Backend (Terminal 1)

npm run dev          # tsx watch src/index.ts (hot-reload)
npm run typecheck    # tsc --noEmit
npm run lint         # eslint src/
npm test             # vitest run
npm run test:watch   # vitest (watch mode)

Frontend (Terminal 2)

cd src/client
npx vite dev         # Vite dev server with HMR

Database

npm run migrate      # tsx src/db/migrate.ts

Full Build

npm run build        # tsc (backend) + vite build (frontend)

Environment variables: Copy .env.example to .env and fill in DATABASE_URL, REDIS_URL, JWT_SECRET, and JWT_REFRESH_SECRET at minimum. All external service integrations (Slack, ESP, Claude, Google, Sentry) are optional in development and log to console when keys are absent.

Project Structure

nexus/
  src/
    index.ts              # Fastify entry point (Sentry init, graceful shutdown)
    server.ts             # App builder (plugin + route registration)
    config/               # Env config and validation
    db/                   # Drizzle schema (16 modules), migrations, seed, connection pool
    routes/               # 37 Fastify route files
    routes/integrations/  # 14 integration sub-routes (travel proxy, Slack, n8n, etc.)
    services/             # Business logic (56+ service files)
    middleware/            # authenticate.ts, serviceAuth.ts, rbac.ts, error handling, file upload
    plugins/              # Fastify plugins (CORS, rate limit, Socket.io, Swagger)
    workers/              # BullMQ job processors (4 files + index.ts)
    queues/               # BullMQ queue definitions
    templates/            # Email templates (React Email)
    lib/                  # Redis client, utilities
    client/               # React SPA (separate package.json)
      src/
        pages/            # 80+ route page components
        components/       # Shared + feature components, AppLayout
        stores/           # Zustand stores (auth, notifications)
        lib/              # API client, utilities
        router.tsx        # React Router configuration
  docker/
    Dockerfile.api        # Multi-stage Node.js 20 build
    Dockerfile.web        # Vite build -> nginx:alpine
    docker-compose.yml    # 5-container orchestration
    nginx-frontend.conf   # CSP headers, SPA routing fallback

Deployment

Nexus has no git remote. Deployment is done via tar archive over SSH, followed by a Docker Compose rebuild on the VPS. The source lives at /opt/nexus-crm on the production server.

Deploy Process

Clean Remote

Remove stale source files on VPS to prevent old files from persisting: ssh shane@VPS "sudo rm -rf /opt/nexus-crm/src/"

Tar Deploy

From the nexus/ directory: tar --exclude=node_modules --exclude=.git -cf - . | ssh shane@VPS "sudo tar xf - -C /opt/nexus-crm/"

Rebuild Containers

On the VPS: cd /opt/nexus-crm/docker && sudo docker compose up -d --build

Critical: The docker/.env file must contain POSTGRES_PASSWORD for Docker Compose variable interpolation. This is separate from the main .env file. Without it, the database connection URL will have an empty password and containers will fail to start.

Traefik Routing

Environment Variables (Production)

Required

• DATABASE_URL
• REDIS_URL
• JWT_SECRET
• JWT_REFRESH_SECRET
• POSTGRES_PASSWORD (docker/.env)

Security-Sensitive

• UNSUBSCRIBE_SECRET — Used for email unsubscribe link HMAC. Production startup now warns if the default placeholder value is still in use. Must be changed before go-live.
• N8N_WEBHOOK_SECRET — Shared secret for n8n callback authentication. Added to .env.example and docker-compose.yml. The /api/n8n/webhook/:action route verifies this via timing-safe comparison of the X-Webhook-Secret header.
• TT_INTERNAL_SERVICE_SECRET — Shared secret for internal service auth. Used by serviceAuth.ts middleware on /api/internal/* routes. Must match the value in TT-API's .env. Returns 503 if not set.

Optional (Graceful Degradation)

• SLACK_SIGNING_SECRET, SLACK_BOT_TOKEN
• SENTRY_DSN
• TT_API_URL, TT_API_KEY
• N8N_URL, N8N_API_KEY
• GOOGLE_SERVICE_ACCOUNT_KEY_FILE

Screenshots

Visual reference for the Nexus Ops Hub interface.