TicketExceptions - Project Overview

Solution Structure

The solution (TELS.TicketExceptions.sln) contains 12 projects organized in a layered architecture:

Application Projects

Project	Type	Framework	Description
TELS.TicketExceptions.API	ASP.NET Core Web API	.NET 8.0	REST API for querying/managing ticket exceptions
TELS.TicketExceptions.MQServices	Worker Service	.NET 8.0	Message queue consumer service (27 consumers for exception detection)
TELS.TicketExceptions.DAL	Class Library	.NET 8.0	Data access layer (Dapper + PostgreSQL)
TELS.TicketExceptions.Models	Class Library	.NET 8.0	Shared data models
TELS.TicketExceptions.Contracts	Class Library	.NET Standard 2.0	Published API contracts & MQ events (NuGet package)
TELS.TicketExceptions.DbUp	Console App	.NET 8.0	Database migrations (DbUp)
TELS.TicketExceptions.ServiceProviderEscalationProcessor	Console App	.NET 8.0	Batch job (currently disabled)

Test Projects

Project	Type	Description
TELS.TicketExceptions.API.UnitTesting	Unit Tests	API controller tests
TELS.TicketExceptions.DAL.UnitTesting	Unit Tests	Data access layer tests
TELS.TicketExceptions.MQServices.UnitTesting	Unit Tests	Consumer handler tests
TELS.TicketExceptions.IntegrationTesting	Integration Tests	Full-stack API + DB tests
TELS.TicketExceptions.ServiceProviderEscalationProcessor.UnitTesting	Unit Tests	Escalation processor tests

---

Architecture

Purpose

TicketExceptions is an event-driven microservice that manages Zendesk ticket exceptions for dispatch list tickets within business-defined exception states. It isolates exception handling to decouple the web farm and integrates with the TELS platform's ticketing and operations infrastructure.

Layering Pattern

┌─────────────────────────────────────────────┐
│         API Controllers (3)                  │
│      Controllers/*.cs                        │
├─────────────────────────────────────────────┤
│       MQ Consumers (27 consumers)            │
│      MQServices/Consumers/*.cs               │
├─────────────────────────────────────────────┤
│     Business Logic / Services                │
│      Logic/*, Handlers/*                     │
├─────────────────────────────────────────────┤
│     Data Access Layer (Dapper)               │
│      DAL/PostgresDataContext.cs              │
├─────────────────────────────────────────────┤
│          PostgreSQL Aurora                   │
│  (Shared with CloudAPI - tels_building_     │
│   _services_cloudapi database)               │
└─────────────────────────────────────────────┘

Data Access

• ORM: Dapper (micro-ORM)
• Database: PostgreSQL Aurora Serverless V2 (shared with CloudAPI)
• Database Name: tels_building_services_cloudapi
• Schema: ticket_exceptions + quartz (for job scheduler)
• Migrations: DbUp for schema and stored procedures

Message Queue Integration

• Broker: RabbitMQ via MassTransit
• Pattern: Event-driven with 27 specialized consumers
• Primary Queue: tels.mqservices.ticketexceptions
• Scheduler Queue: ticket-exceptions-message-scheduler (for delayed/scheduled messages)
• Retry Strategy: 30-second delay via IMessageScheduler for failed exception creation
• Synchronization: TicketClaim middleware ensures serialized processing per authorization number

---

Key Features

1. Exception Detection & Automation

27 specialized consumers monitor dispatch list tickets for exception conditions: - ETA-related: Missing ETA, customer notification needed - Bid-related: Bid not received, automation failures, clarification/revalidation missing - Checkout-related: Missed check-in, missed checkout, tech departed - Parts-related: Lead time exceeded, parts not ordered - NTE-related: Extension not approved, approval follow-up needed - D2Q: Invoice/quote not received, quote approval follow-up - Assignment: Service provider not assigned - Compliance: Article notification outcomes, contracted service documents - Status: Completion automation, priority changes, status events

2. Zendesk Integration

• Ticket Creation: Automated exception tickets created in Zendesk
• Status Sync: Bi-directional status updates via integration endpoint
• Call-to-Action Routing: Template-based ticket content (dayside, after-hours, westpark)
• Auto-Solve Logic: Tickets auto-solved when exception conditions resolve

3. Configuration Management

• Exception Rules: Active/inactive toggle per exception type
• Call-to-Actions: Configurable templates by request type and shift
• Feature Toggles: Exception creation enabled/disabled via CloudAPI database
• Ticketing Configuration: Per-organization Zendesk settings

4. Service Provider Escalation

• Batch Notifications: Daily escalation emails to service providers
• Escalation Tracking: History of notifications sent
• Authorization Grouping: Multiple exceptions bundled per service provider

---

API Surface

3 controllers exposing endpoints under ticketExceptions/v1/:

Authentication & Authorization

• Auth: JWT Bearer token validation
• Personas: DirectSupplyPartner (most endpoints)
• Public Integration: Zendesk webhook endpoint for status updates

Key Endpoints

• Exception Rules: CRUD operations for rule management
• Ticket Exceptions: Query exceptions with filtering (authorization, SP, status)
• Call-to-Actions: CRUD for ticket content templates
• Diagnostics: Pending trigger inspection, error testing
• Integrations: Public endpoint for Zendesk status updates

---

Key Technology Stack

Category	Technology
Runtime	.NET 8.0
Web Framework	ASP.NET Core 8.0
Data Access	Dapper 2.1.44 + Npgsql 8.0.5
Database	PostgreSQL Aurora Serverless V2 (shared with CloudAPI)
Messaging	RabbitMQ + MassTransit 8.3.3
Scheduling	Quartz.NET 3.13.1 (PostgreSQL persistence)
Migration	DbUp 5.0.40
HTTP Clients	RestSharp 112.1.0, TELS.WebServiceClients
Mapping	AutoMapper 13.0.1
Resilience	Polly 8.4.2
Logging	NLog 5.3.12
Monitoring	New Relic APM
Auth	JWT Bearer
Testing	xUnit, Moq
Containerization	Docker
CI/CD	GitLab CI
Infrastructure	AWS ECS, Terraform
Secrets	HashiCorp Vault

---

Deployment

Docker Containers

• TELS.TicketExceptions.API - 2 task instances per environment
• TELS.TicketExceptions.MQServices - 1 task instance per environment

Resource Allocation

Environment	API Tasks	API Memory	API CPU	MQ Tasks	MQ Memory	MQ CPU
Sandbox/Testing	2	Variable	Variable	1	Variable	Variable
Prod	2	2GB each	0.5 vCPU each	1	2GB	0.5 vCPU

CI/CD Pipeline (GitLab CI)

• Build → Test → Push → Deploy
• Multi-environment: DEV, QA, SANDBOX, TESTING, PROD
• Container versioning: 1.0.${CI_PIPELINE_ID}
• ARM64 optimized builds

Infrastructure

• AWS ECS (Fargate/EC2)
• Application Load Balancer
• PostgreSQL Aurora Serverless V2 (shared with CloudAPI)
• RabbitMQ broker
• Terraform-managed infrastructure
• Vault for secrets management

Monitoring

• New Relic APM: Transaction tracing, error tracking
• CloudWatch: Metrics and logs
• Synthetic Monitoring: Uptime and SLI/SLO compliance

---

Configuration

Configuration Group	Key Variables
Database	SQLDATABASE__ConnectionString (PostgreSQL via CloudAPI)
RabbitMQ	RMQTELS__Host, RMQTELS__Username, RMQTELS__Password
External Services	APP__AuthorizationServiceEndpoint, APP__CustomersServiceEndpoint, APP__BuildingServicesServiceEndpoint
Auth	APP__SIGNING_KEY (Base64 JWT key), APP__HostAccountRefreshToken
Feature Toggles	Managed in CloudAPI PostgreSQL database (ticketing_configuration table)

---

External Integrations

Integration	Mechanism	Description
Zendesk	REST API via RestSharp	Ticket creation, status updates, comment posting
Auth Service	HTTP	JWT token validation and refresh
Customers Service	HTTP	Customer/facility data queries
Building Services	HTTP	Work item status, dispatch list queries
Bidding Service	MQ Events	Bid conversation, status changes, modifications
Compliance Service	MQ Events	Article notification outcomes
Contracted Services	MQ Events	Document upload tracking

---

Event-Driven Architecture

Consumed Events (27 types)

• Work Items: New work items, priority changes, status events
• Bidding: Conversations, status changes, modifications
• Building Services: Attachments, NTE extensions, project events
• Scheduled Checks: ETA, bid received, check-in/out, parts lead time, automation failures

Published Events

• Internal command messages for scheduled processing
• Retry events for failed exception creation

Message Processing

• Synchronization: TicketClaim middleware ensures one message per authorization number at a time
• Retry: 30-second delay for failed exception creation attempts
• Scheduling: Quartz-based message scheduling for time-delayed checks

---

Business Logic Patterns

Exception Creation Flow

1. Consumer receives event (e.g., bid not received)
2. Check if exception rule is active
3. Check if exception already exists for this condition
4. Check if ticket exception creation is enabled (feature toggle)
5. Generate subject and comment from call-to-action template
6. Create ticket exception record in PostgreSQL
7. Create ticket in Zendesk via REST API
8. Store Zendesk ticket ID as external reference
9. Mark exception as created

Auto-Solve Logic

1. Consumer receives resolution event (e.g., bid received)
2. Find existing ticket exception for authorization number + exception type
3. Mark exception as "solved by TBS"
4. Update Zendesk ticket status to solved

Service Provider Escalation (Batch Job - Currently Disabled)

1. Query all open exceptions grouped by service provider
2. Generate escalation email with list of open exceptions
3. Send notification to SP contacts
4. Track notification in sp_escalation_notification table

---

Key Files & Locations

API Layer: - /TELS.TicketExceptions.API/Controllers/ - /TELS.TicketExceptions.API/Startup.cs

MQ Services: - /TELS.TicketExceptions.MQServices/Consumers/ (27 consumer files) - /TELS.TicketExceptions.MQServices/Program.cs

Data Access: - /TELS.TicketExceptions.DAL/PostgresDataContext.cs - /TELS.TicketExceptions.DAL/Repositories/

Migrations: - /TELS.TicketExceptions.DbUp/Scripts/

Configuration: - /TELS.TicketExceptions.sln - /README.md

---

Testing Strategy

Test Type	Project	Execution
Unit Tests	API.UnitTesting	Controller/service logic
Unit Tests	DAL.UnitTesting	Data access layer
Unit Tests	MQServices.UnitTesting	Consumer handlers
Integration Tests	IntegrationTesting	Full API + DB (Docker)

Local Development Scripts

• LocalSetup.ps1 - Complete local environment setup
• DBSetup.ps1 - Database initialization
• LocalDockerSetup.ps1 - Build Docker images locally
• RunDockerIntegrationTests.ps1 - Run integration tests in Docker

---

Notes

• Database Sharing: TicketExceptions shares the PostgreSQL Aurora database with CloudAPI (tels_building_services_cloudapi)
• Feature Toggle Dependency: Exception creation is controlled by ticketing_configuration in CloudAPI database
• Zendesk Integration: Critical for operations - exceptions create tickets that route to support teams
• 27 Consumers: High fan-in architecture - many event types trigger exception detection logic
• Synchronization: TicketClaim middleware prevents race conditions when multiple events for same work item arrive
• ServiceProviderEscalationProcessor: Separate console app currently disabled, logic may be migrated to MQServices