InfraForge Docs

InfraNotes Module · v0

Welcome

Select a document from the sidebar to read it.

InfraNotes Financial Analytics Module - Administrator Guide

Important

InfraNotes has fully migrated away from NATS in favor of Kafka at the platform level, and this module now integrates with InfraNotes via HTTP APIs secured with shared JWT authentication. NATS-specific system requirements and installation steps below are historical; consult the main InfraNotes deployment documentation for the current Kafka-based infrastructure requirements.

Overview

This guide is intended for system administrators responsible for installing, configuring, deploying, monitoring, and maintaining the InfraNotes Financial Analytics Module. It provides comprehensive instructions for all administrative tasks.

System Requirements

Hardware Requirements

  • CPU: 4 cores minimum, 8+ cores recommended for production
  • Memory: 8GB minimum, 16GB+ recommended for production
  • Disk: 100GB minimum, SSD storage recommended
  • Network: 1 Gbps minimum

Software Requirements

  • Operating System: Linux (Ubuntu 20.04+, CentOS 8+) or macOS (12+)
  • Container Runtime: Docker 20.10+ and Docker Compose 2.0+
  • Database: PostgreSQL 13+
  • Message Broker: Kafka (managed as part of the InfraNotes platform)
  • Optional Caching: Redis 6.0+

Installation

  1. Clone the repository

    git clone https://github.com/StackCatalyst/infranotes-module.git
cd infranotes-module

  2. Configure environment variables

    cp .env.example .env
# Edit .env with appropriate values

  3. Start the services

    docker-compose up -d

Manual Installation

  1. Install dependencies

    # Ubuntu/Debian
apt-get update
apt-get install -y postgresql-13 golang-1.17 redis-server

# CentOS/RHEL
dnf install -y postgresql13-server golang redis

  2. Clone the repository

    git clone https://github.com/StackCatalyst/infranotes-module.git
cd infranotes-module

  3. Build the application

    make build

  4. Initialize the database

    make migrate-up

  5. Start the service

    ./bin/server

Configuration

Environment Variables

The Financial Analytics Module is configured primarily through environment variables:

Variable Description Default Required
DB_HOST PostgreSQL host localhost Yes
DB_PORT PostgreSQL port 5432 Yes
DB_NAME PostgreSQL database name infranotes_analytics Yes
DB_USER PostgreSQL username postgres Yes
DB_PASSWORD PostgreSQL password - Yes
DB_SSL_MODE PostgreSQL SSL mode disable No
NATS_URL NATS server URL nats://localhost:4222 Yes
REDIS_HOST Redis host localhost No
REDIS_PORT Redis port 6379 No
REDIS_PASSWORD Redis password - No
AUTH_SECRET JWT authentication secret - Yes
AUTH_PUBLIC_KEY JWT public key for RSA verification - No
PORT Server port 8080 No
LOG_LEVEL Logging level info No
STORAGE_TYPE Document storage type (local, s3) local Yes
STORAGE_PATH Local storage path ./storage If using local storage
AWS_S3_BUCKET S3 bucket name - If using S3 storage
AWS_S3_REGION S3 region - If using S3 storage
AWS_ACCESS_KEY_ID AWS access key - If using S3 storage
AWS_SECRET_ACCESS_KEY AWS secret key - If using S3 storage
CORE_API_URL InfraNotes Core API URL http://localhost:8000 Yes

Configuration Files

In addition to environment variables, the following files can be used for configuration:

  • config/app.yaml: Application configuration
  • config/logging.yaml: Logging configuration

Example app.yaml:

server:
  port: 8080
  timeout: 30s
  cors:
    allowed_origins: ["*"]
    allowed_methods: ["GET", "POST", "PUT", "DELETE", "OPTIONS"]
    
database:
  max_open_conns: 25
  max_idle_conns: 5
  conn_max_lifetime: 5m
  
document_processing:
  max_file_size: 20971520  # 20 MB
  allowed_types:
    - application/pdf
    - text/csv
    - application/vnd.openxmlformats-officedocument.spreadsheetml.sheet
  retention_period: 720h   # 30 days

Deployment

Docker Deployment

For production deployment with Docker:

  1. Create a production docker-compose.yml file:

    version: '3.8'

services:
  app:
    image: stackcatalyst/infranotes-analytics:latest
    restart: always
    ports:
      - "8080:8080"
    environment:
      - DB_HOST=postgres
      - DB_USER=postgres
      - DB_PASSWORD=postgres
      - DB_NAME=infranotes_analytics
      - NATS_URL=nats://nats:4222
      - REDIS_HOST=redis
      - AUTH_SECRET=your-auth-secret
      - STORAGE_TYPE=s3
      - AWS_S3_BUCKET=your-bucket
      - AWS_S3_REGION=your-region
      - AWS_ACCESS_KEY_ID=your-access-key
      - AWS_SECRET_ACCESS_KEY=your-secret-key
      - CORE_API_URL=http://infranotes-core:8000
    depends_on:
      - postgres
      - nats
      - redis
    volumes:
      - ./config:/app/config
    networks:
      - infranotes-network

  postgres:
    image: postgres:13
    restart: always
    environment:
      - POSTGRES_USER=postgres
      - POSTGRES_PASSWORD=postgres
      - POSTGRES_DB=infranotes_analytics
    volumes:
      - postgres-data:/var/lib/postgresql/data
    networks:
      - infranotes-network

  nats:
    image: nats:2.2
    restart: always
    ports:
      - "4222:4222"
    networks:
      - infranotes-network

  redis:
    image: redis:6.0
    restart: always
    networks:
      - infranotes-network

volumes:
  postgres-data:

networks:
  infranotes-network:
    external: true

  2. Deploy the services:

    docker-compose -f docker-compose.yml up -d

Kubernetes Deployment

For Kubernetes deployment:

  1. Create Kubernetes manifests:

    • Deployment
    • Service
    • ConfigMap
    • Secret
    • PersistentVolumeClaim

  2. Apply the manifests:

    kubectl apply -f k8s/

Database Management

Database Migrations

Database migrations are managed using the migrate tool:

  • Apply migrations:

    make migrate-up

  • Rollback migrations:

    make migrate-down

  • Create a new migration:

    make migrate-new name=add_new_feature

Backup and Restore

For PostgreSQL backup and restore:

  • Create a backup:

    pg_dump -h $DB_HOST -U $DB_USER -d $DB_NAME -F c -f backup.dump

  • Restore from backup:

    pg_restore -h $DB_HOST -U $DB_USER -d $DB_NAME -c backup.dump

Database Maintenance

Regular database maintenance tasks:

  • Vacuum the database:

    psql -h $DB_HOST -U $DB_USER -d $DB_NAME -c "VACUUM ANALYZE;"

  • Reindex the database:

    psql -h $DB_HOST -U $DB_USER -d $DB_NAME -c "REINDEX DATABASE $DB_NAME;"

Security

Authentication

The Financial Analytics Module uses JWT-based authentication shared with InfraNotes Core:

  1. Configure the shared JWT secret in AUTH_SECRET environment variable
  2. For RSA-based authentication, provide the public key in AUTH_PUBLIC_KEY

Document Storage Security

For secure document storage:

  1. When using local storage, ensure proper file permissions:

    chmod 700 storage

  2. When using S3, use a dedicated IAM role with minimal permissions:

    • s3:PutObject
    • s3:GetObject
    • s3:DeleteObject
    • s3:ListBucket

  3. Enable bucket encryption in S3

Data Retention

Configure data retention policies:

  1. Set the document retention period in document_processing.retention_period
  2. Schedule the cleanup job in your environment: 
    # Example cron job for daily cleanup
0 2 * * * /path/to/infranotes-module/bin/cleanup

Monitoring and Logging

Logging Configuration

Logs are output in JSON format with configurable levels:

  • Configure log level: Set LOG_LEVEL to debug, info, warn, or error
  • View logs: 
    # For Docker
docker logs -f infranotes-analytics

# For Kubernetes
kubectl logs -f deployment/infranotes-analytics

Monitoring with Prometheus

The application exposes metrics at /metrics endpoint:

  1. Configure Prometheus to scrape this endpoint
  2. Use provided Grafana dashboards for visualization

Example Prometheus config:

scrape_configs:
  - job_name: 'infranotes-analytics'
    scrape_interval: 15s
    static_configs:
      - targets: ['infranotes-analytics:8080']

Health Checks

Health check endpoints:

  • Liveness check: /health/live
  • Readiness check: /health/ready
  • Status check: /health/status

Troubleshooting

Common Issues

Database Connection Issues

Symptoms: Application logs show database connection errors

Solutions:

  1. Check database credentials in environment variables
  2. Verify PostgreSQL is running and accessible
  3. Check network connectivity between application and database
  4. Verify firewall rules allow the connection
  5. Check database max connections setting

Document Processing Failures

Symptoms: Documents show "Processing Failed" status

Solutions:

  1. Check application logs for specific error messages
  2. Verify the document format is supported
  3. Check if the document is corrupted or password-protected
  4. Verify storage configuration and permissions

Out of Memory Errors

Symptoms: Application crashes with OOM errors

Solutions:

  1. Increase container memory limits
  2. Optimize application memory settings: 
    GOMAXPROCS=4 GO_MAX_MEMORY=8G ./bin/server
  3. Add memory monitoring
  4. Consider scaling horizontally

NATS Connection Issues

Symptoms: Application logs show NATS connection errors

Solutions:

  1. Verify NATS server is running
  2. Check NATS URL configuration
  3. Check network connectivity
  4. Verify NATS has enough capacity for the expected message volume

Diagnostic Tools

Application Diagnostics

  • Runtime statistics: /debug/vars (when DEBUG=true)
  • Profiling: /debug/pprof/ (when DEBUG=true)
  • Trace information: /debug/requests (when DEBUG=true)

Log Analysis

Using log analysis tools:

# Find error logs
grep "level=error" /var/log/infranotes/analytics.log

# Find errors related to document processing
grep "document processing" /var/log/infranotes/analytics.log | grep "error"

Database Diagnostics

Query execution analysis:

-- Find slow queries
SELECT query, calls, total_time, mean_time
FROM pg_stat_statements
ORDER BY mean_time DESC
LIMIT 10;

-- Check index usage
SELECT relname, idx_scan, seq_scan
FROM pg_stat_user_tables
ORDER BY seq_scan DESC;

Performance Tuning

Application Tuning

Optimize application performance:

  1. Connection Pooling:

    DB_MAX_OPEN_CONNS=25
DB_MAX_IDLE_CONNS=5
DB_CONN_MAX_LIFETIME=300

  2. Worker Configuration:

    WORKER_POOL_SIZE=10

  3. Caching Configuration:

    REDIS_ENABLE=true
CACHE_TTL=600

Database Tuning

Optimize PostgreSQL performance:

  1. Memory Settings:

    shared_buffers = 2GB
effective_cache_size = 6GB
work_mem = 16MB
maintenance_work_mem = 512MB

  2. Write Performance:

    wal_buffers = 16MB
checkpoint_completion_target = 0.9

  3. Query Planning:

    random_page_cost = 1.1
effective_io_concurrency = 200

Scaling

Horizontal Scaling

To scale the application horizontally:

  1. Deploy multiple instances behind a load balancer
  2. Ensure stateless operation
  3. Configure shared Redis for distributed caching
  4. Configure database connection pooling appropriately

Vertical Scaling

To scale the application vertically:

  1. Increase CPU and memory resources
  2. Optimize database for larger hardware
  3. Increase connection pool sizes
  4. Configure NATS with higher message limits

Backup and Recovery

Application Backup

Back up essential application data:

  1. Configuration files:

    cp -r config/ backup/config/

  2. Environment files:

    cp .env backup/.env

Complete System Backup

For a full system backup:

  1. Back up the database
  2. Back up the document storage
  3. Back up the configuration
  4. Document the environment setup

Disaster Recovery

In case of system failure:

  1. Restore the database from backup
  2. Restore the document storage
  3. Deploy the application with the same configuration
  4. Verify system integrity

Upgrade Procedures

Version Upgrade

To upgrade to a new version:

  1. Back up the current system
  2. Stop the running services
  3. Update the application image or binary
  4. Run database migrations
  5. Start the services
  6. Verify functionality

Rolling Upgrade

For zero-downtime upgrades:

  1. Deploy the new version alongside the old version
  2. Apply database migrations with backward compatibility
  3. Gradually shift traffic to the new version
  4. Monitor for issues
  5. Complete the transition when stable

Integration Management

InfraNotes Core Integration

Managing the integration with InfraNotes Core:

  1. Configure the Core API URL in CORE_API_URL
  2. Set up shared authentication with AUTH_SECRET
  3. Ensure NATS is properly configured for event communication
  4. Verify event subscription status in the health check

External Integrations

The Financial Analytics Module can integrate with external systems:

  1. Configure integration endpoints in the appropriate configuration files
  2. Set up authentication credentials for each integration
  3. Monitor integration health in logs and metrics

Maintenance Procedures

Routine Maintenance

Regular maintenance tasks:

  1. Database optimization (weekly)
  2. Log rotation (daily)
  3. Storage cleanup (monthly)
  4. Certificate renewal (as needed)
  5. Dependency updates (monthly)

Planned Downtime

For planned maintenance:

  1. Notify users in advance
  2. Perform maintenance during low-usage periods
  3. Set up a maintenance page
  4. Test thoroughly before bringing the system back online

Reference

Command-Line Interface

Available CLI commands:

  • Server: bin/server - Start the application server
  • Migrations: bin/migrate - Database migration tool
  • Cleanup: bin/cleanup - Run data cleanup routines
  • Diagnostics: bin/diagnostics - Generate system diagnostics report

Configuration Reference

Complete configuration reference:

Monitoring Reference

Available metrics and their meaning:

  • http_request_duration_seconds: HTTP request duration
  • document_processing_duration_seconds: Document processing time
  • transaction_count: Number of transactions processed
  • categorization_accuracy: Categorization accuracy percentage
  • database_query_duration_seconds: Database query duration

Support and Troubleshooting

For additional support: