BusProxy Queue Processor - Project Overview
Solution Structure
Solution: TELS.BusproxyQueueprocessor.sln
| Project |
Type |
Framework |
Description |
| TELS.BusProxyQueueProcessor |
Console App (Worker Service) |
.NET 8.0 |
Background processor for SQL fallback queue |
| TELS.BusProxyQueueProcessor.Tests |
Unit Tests |
.NET 8.0 |
Service tests |
Purpose
BusProxy Queue Processor is a background worker service that processes messages from the SQL Server fallback queue when BusProxy's primary delivery (RabbitMQ) fails. It ensures guaranteed message delivery by retrying queued messages.
Key Responsibilities
- Poll SQL Queue: Continuously check for messages in fallback queue
- Retry Delivery: Attempt to publish queued messages to RabbitMQ
- Remove on Success: Delete messages from queue after successful delivery
- Retry on Failure: Keep messages in queue for next processing cycle
- Monitoring: Log processing statistics and errors
Architecture
Message Flow
BusProxy API (primary delivery fails)
↓
SQL Server Fallback Queue
↓ (polling every 60s)
BusProxy Queue Processor
↓ (retry delivery)
RabbitMQ / Message Bus
↓ (on success)
Delete from SQL Queue
Deployment Model
- Platform: Nomad scheduler
- Container: Docker
- Environments: Sandbox, Testing, Production
- High Availability: Single instance per environment (queue ensures no duplicate processing)
Technology Stack
| Category |
Technology |
| Runtime |
.NET 8.0 |
| Hosting |
Microsoft.Extensions.Hosting 3.1.11 |
| Configuration |
Microsoft.Extensions.Configuration (JSON, Environment Variables) |
| HTTP Client |
Microsoft.Extensions.Http 5.0.0 |
| Testing |
xUnit 2.4.1, Moq 4.16.1 |
| Deployment |
Nomad, Docker |
| Monitoring |
CloudWatch Logs |
Processing Logic
1. Poll Queue
Every 60 seconds:
Query SQL Server fallback queue
Order by: oldest first
Limit: configurable batch size
2. Process Messages
For each queued message:
Deserialize message payload
Call RabbitMQ/Bus to publish
If success:
Delete from SQL queue
If failure:
Log error, leave in queue for retry
3. Error Handling
- Transient Errors: Keep in queue, retry next cycle
- Permanent Errors: Log and optionally move to dead-letter table
- Processing Timeout: Configurable per-message timeout
Configuration
| Setting |
Source |
Description |
| SQL Connection String |
Environment Variable |
Connection to TELS database |
| Poll Interval |
Config |
How often to check queue (default: 60s) |
| Batch Size |
Config |
Messages per processing cycle |
| RabbitMQ Connection |
Environment Variable |
Target message bus |
| Retry Limit |
Config |
Max retries before dead-lettering |
| Processing Timeout |
Config |
Per-message timeout |
Deployment
Nomad Configuration
- Job Type: Service (long-running)
- Restart Policy: On-failure with delay
- Resource Allocation: 256MB memory, 100 MHz CPU (configurable per env)
- Health Checks: Process liveness
Environments
| Environment |
Instances |
Resources |
| Sandbox |
1 |
Minimal |
| Testing |
1 |
Standard |
| Production |
1 |
Standard + monitoring |
Infrastructure
- Terraform: Infrastructure-as-code in
/terraform/ directories
- envconsul: Configuration management via Consul/Vault
- CloudWatch: Centralized logging
Monitoring & Logging
Logs
- Startup/shutdown events
- Messages processed (count, success, failure)
- Error details (exception stack traces)
- Performance metrics (processing time per batch)
Metrics
- Messages in queue (gauge)
- Messages processed per cycle (counter)
- Success/failure rate (counter)
- Processing latency (histogram)
Integration with BusProxy
| Component |
Role |
| BusProxy API |
Writes messages to SQL queue on primary failure |
| BusProxy Queue Processor |
Reads from SQL queue and retries delivery |
Shared Dependencies:
- SQL Server (TELS database) - fallback queue table
- RabbitMQ connection - target message bus
Key Files
| File |
Purpose |
/src/Program.cs |
Application entry point, host setup |
/src/Services/QueueProcessorService.cs |
Core processing logic |
/src/Data/QueueRepository.cs |
SQL queue access |
/envconsul-config.hcl |
Consul/Vault configuration |
/terraform/ |
Infrastructure definitions (Sandbox, Testing, Prod) |
Notes
- Companion Service: Works with BusProxy API to ensure reliable message delivery
- No Direct API: Background worker only, no HTTP endpoints
- Single Instance: One instance per environment (SQL queue ensures no duplicates)
- Guaranteed Delivery: Together with BusProxy, provides at-least-once delivery semantics
- Fallback for Fallback: If queue processor also fails, messages remain in SQL for manual intervention