InfraNotes Financial Analytics Module - Developer Guide

Important

InfraNotes has fully migrated away from NATS in favor of Kafka at the platform level. This module now integrates with InfraNotes via HTTP APIs secured with shared JWT authentication. Any NATS-specific diagrams or references in this guide are historical and should not be used for new integrations; refer to the codebase (HTTP + JWT middleware) for the current behavior.

Overview

This guide is intended for developers who will be maintaining, extending, or integrating with the InfraNotes Financial Analytics Module. It provides technical details about the architecture, key components, and development workflows.

Architecture

The Financial Analytics Module is designed as a decoupled service that integrates with the core InfraNotes platform. It follows a modular structure with clearly defined boundaries between components.

High-Level Architecture

┌───────────────────────────────────────┐     ┌───────────────────────────────────────┐
│                                       │     │                                       │
│         InfraNotes Core               │     │       Financial Analytics Module      │
│                                       │     │                                       │
│  ┌─────────────┐    ┌──────────────┐  │     │  ┌─────────────┐    ┌──────────────┐  │
│  │             │    │              │  │     │  │             │    │              │  │
│  │   Auth &    │    │   Expense    │◄─┼─────┼─►│  Document   │    │  Analytics   │  │
│  │  Security   │◄───┤  Management  │  │     │  │ Processing  │    │   Engine     │  │
│  │             │    │              │  │     │  │             │    │              │  │
│  └─────────────┘    └──────────────┘  │     │  └─────────────┘    └──────────────┘  │
│                                       │     │                                       │
│  ┌─────────────┐    ┌──────────────┐  │     │  ┌─────────────┐    ┌──────────────┐  │
│  │             │    │              │  │     │  │             │    │              │  │
│  │   Project   │    │   Budget/    │  │     │  │ Transaction │    │  Financial   │  │
│  │  Management │    │   Invoice    │  │     │  │ Categorizer │    │   Planner    │  │
│  │             │    │              │  │     │  │             │    │              │  │
│  └─────────────┘    └──────────────┘  │     │  └─────────────┘    └──────────────┘  │
│                                       │     │                                       │
└───────────────────┬─────────────────┬─┘     └───────────────────┬─────────────────┬─┘
                    │                 │                           │                 │
                    │                 │                           │                 │
                    │                 │                           │                 │
                    ▼                 ▼                           ▼                 ▼
         ┌─────────────────┐   ┌─────────────┐          ┌─────────────────┐   ┌─────────────┐
         │                 │   │             │          │                 │   │             │
         │   PostgreSQL    │   │     NATS    │◄─────────│   PostgreSQL    │   │  Document   │
         │   (Core DB)     │   │  Messaging  │          │  (Analytics DB) │   │   Storage   │
         │                 │   │             │          │                 │   │             │
         └─────────────────┘   └─────────────┘          └─────────────────┘   └─────────────┘

Core Design Principles

1. Modularity: Each component is designed with clear boundaries and interfaces.
2. Single Responsibility: Each component focuses on a specific domain concern.
3. Domain-Driven Design: Models and services are organized around business capabilities.
4. Clean Architecture: Separation of concerns with domain models, repositories, services, and handlers.
5. Event-Driven Integration: Kafka-based messaging at the InfraNotes platform level (NATS has been deprecated).
6. Performance Focus: Optimized for handling large transaction datasets.

Project Structure

The Financial Analytics Module follows a standard Go project structure with:

infranotes-module/
├── cmd/                     # Application entry points
│   └── server/              # Main server application
├── internal/                # Private application code
│   ├── analytics/           # Financial analytics components
│   │   ├── analytics/       # Core analytics engine
│   │   ├── budget/          # Budget tracking 
│   │   ├── cashflow/        # Cash flow management
│   │   ├── categorization/  # Transaction categorization
│   │   ├── document/        # Document processing
│   │   ├── goals/           # Financial goal tracking
│   │   └── planning/        # Financial planning
│   └── common/              # Shared utilities and infrastructure
├── migrations/              # Database migration files
├── scripts/                 # Utility scripts
├── docs/                    # Documentation
├── statements/              # Test data (sample financial documents)
└── ...

Core Components

Document Processing

The Document Processing component is responsible for handling uploaded financial documents, extracting transactions, and storing them for analysis.

Key Files:

• internal/analytics/document/models/*.go: Domain models
• internal/analytics/document/services/parser/*.go: Document parsing implementations
• internal/analytics/document/repositories/*.go: Repository interfaces
• internal/analytics/document/repositories/postgres/*.go: PostgreSQL implementations

Extending Parser Support:

To add support for a new document format:

1. Implement a new parser in internal/analytics/document/services/parser/
2. Update the parser factory in internal/analytics/document/services/parser/factory.go
3. Add test cases with sample documents

Example:

// NewXYZParser creates a parser for XYZ format
func NewXYZParser() Parser {
    return &xyzParser{}
}

type xyzParser struct {}

func (p *xyzParser) Parse(ctx context.Context, content []byte, options ParserOptions) ([]models.Transaction, error) {
    // Implementation here
}

Transaction Categorization

The Transaction Categorization component automatically categorizes transactions based on rules and patterns.

Key Files:

• internal/analytics/document/services/categorization/*.go: Categorization services
• internal/analytics/document/repositories/postgres/category_*.go: Category repositories

Adding New Categorization Rules:

To add a new pattern matching strategy:

1. Update the rule matcher in internal/analytics/document/services/categorization/rule_matcher.go
2. Add tests for the new strategy

Analytics Engine

The Analytics Engine analyzes transaction data to provide insights and trends.

Key Files:

• internal/analytics/analytics/services/*.go: Analytics services
• internal/analytics/analytics/repositories/*.go: Analytics repositories
• internal/analytics/analytics/models/*.go: Analytics models

Extending Analytics:

To add a new type of analysis:

1. Create a new service in internal/analytics/analytics/services/
2. Define any necessary models in internal/analytics/analytics/models/
3. Expose endpoints in a handler

Financial Planning

The Financial Planning component manages planned expenses and financial projections.

Key Files:

• internal/analytics/planning/*.go: Planning services and models
• internal/analytics/planning/repositories/*.go: Planning repositories

Budget Tracking

The Budget Tracking component manages budgets and tracks spending against them.

Key Files:

• internal/analytics/planning/budget/*.go: Budget services and models

Goal Tracking

The Goal Tracking component manages financial goals and progress tracking.

Key Files:

• internal/analytics/goals/*.go: Goal services and models
• internal/analytics/goals/ui/*.go: Goal visualization

Integration with InfraNotes Core

Authentication Integration

The Financial Analytics Module uses the same authentication system as InfraNotes core:

1. Every request to the module includes a JWT token in the Authorization header
2. The authentication middleware validates the token
3. User ID and permissions are extracted from the token
4. All repository methods include userID parameters for data isolation

Event Communication

Events are used for cross-service communication:

1. Core events are published to NATS
2. The module subscribes to relevant events
3. The module publishes its own events

Key Events:

• From Core: expense.created, expense.updated, expense.deleted, user.updated, category.updated
• From Analytics: insight.generated, payment.reminder, pattern.detected, document.processed

Development Workflow

Setting Up a Development Environment

1. Clone the repository
2. Run make setup to install dependencies
3. Run make dev to start the development server and PostgreSQL database

Running Tests

• make test: Run all tests
• make test-unit: Run unit tests only
• make test-integration: Run integration tests only

Running Migrations

• make migrate-up: Apply all pending migrations
• make migrate-down: Roll back the last migration
• make migrate-new name=<migration-name>: Create a new migration

Adding a New Feature

1. Define models in the appropriate domain directory
2. Create repository interfaces
3. Implement PostgreSQL repositories
4. Create service implementations
5. Add handler endpoints
6. Write unit tests
7. Add integration tests
8. Document the feature

Performance Optimization

Query Optimization

For handling large transaction datasets, we've implemented:

1. Efficient batch processing: Using PostgreSQL's COPY command for bulk inserts
2. Cursor-based pagination: More efficient than offset-based for large datasets
3. Context timeouts: Prevent long-running queries
4. Optimized query builder: Builds efficient SQL queries with proper conditions
5. Aggregation queries: Use database-level aggregation for analytics

Caching Strategy

Redis caching is implemented for:

1. Frequently accessed aggregated data
2. Projection calculations
3. Category rule sets
4. Visualization data

Common Development Tasks

Adding a New API Endpoint

1. Define the handler function in the appropriate handler
2. Register the endpoint in the router
3. Update OpenAPI documentation

Example:

// Handler
func (h *AnalyticsHandler) GetTransactionSummary(c echo.Context) error {
    // Implementation
}

// Router registration
func registerAnalyticsRoutes(g *echo.Group, h *AnalyticsHandler) {
    g.GET("/summary", h.GetTransactionSummary)
}

Adding a New Repository Method

1. Define the method in the repository interface
2. Implement the method in the PostgreSQL repository
3. Add unit tests

Example:

// Interface
type TransactionRepository interface {
    // Existing methods...
    
    // New method
    GetTransactionSummaryByMonth(ctx context.Context, userID uuid.UUID, year int) ([]models.MonthSummary, error)
}

// Implementation
func (r *TransactionRepository) GetTransactionSummaryByMonth(ctx context.Context, userID uuid.UUID, year int) ([]models.MonthSummary, error) {
    // Implementation
}

Adding a New Database Table

1. Create a new migration file with make migrate-new name=add_table_name
2. Update the migration files with the schema
3. Define the model struct
4. Create repository interfaces and implementations
5. Run the migration with make migrate-up

Troubleshooting

Common Issues

1. Performance problems with large datasets:

   • Check that indexes are properly defined
   • Use EXPLAIN ANALYZE to identify bottlenecks
   • Consider adding caching for expensive calculations

2. Document processing failures:

   • Ensure document format is supported
   • Check error logs for specific parsing failures
   • Test with sample documents

3. Integration issues with InfraNotes Core:

   • Verify NATS connection
   • Check event subscription
   • Validate JWT token configuration

Extending the System

Adding New Analytics Insights

To add a new type of analytical insight:

1. Define the insight model in internal/analytics/analytics/models/
2. Create a service that generates the insight
3. Add event publication for significant insights
4. Add repository methods for persistence
5. Create API endpoints to retrieve insights

Adding New Visualization Types

To add a new visualization:

1. Define the chart model in the appropriate UI package
2. Create generator functions for chart data
3. Add new endpoints or enhance existing ones to include the visualization

Integrating with New External Systems

To add integration with an external system:

1. Create a client package for the external API
2. Define integration services
3. Add configuration for the integration
4. Implement event handlers for synchronization

Best Practices

1. Domain Models: Keep models focused on domain concerns
2. Transactions: Use database transactions for operations that touch multiple tables
3. Context: Always pass context to repository methods for cancellation and timeout
4. User ID: Always filter data by user ID for security
5. Error Handling: Return meaningful errors that can be properly handled
6. Logging: Use structured logging with appropriate levels
7. Testing: Write unit tests for business logic and integration tests for repositories
8. Documentation: Keep API documentation up to date
9. Events: Use events for cross-service communication
10. Performance: Consider the performance impact of changes, especially for large datasets

Contribution Guidelines

1. Follow the Go style guide
2. Write unit tests for all new code
3. Update documentation for new features
4. Run linters before submitting changes
5. Submit detailed pull requests with proper descriptions
6. Keep backward compatibility where possible

Additional Resources

• Go Project Layout
• Domain-Driven Design
• Clean Architecture
• NATS Documentation
• PostgreSQL Performance Tuning