Distributed Monolith Pipeline

Overview

This proposal outlines the release pipeline for a distributed monolith system using Bazel as the build system. The architecture consists of component repositories that feed into an integration repository, which maintains the manifest and orchestrates releases.

System Architecture

graph TB
    subgraph "Component Repositories"
        C1[Component A<br/>Independent versioning]
        C2[Component B<br/>Independent versioning]
        C3[Component C<br/>Independent versioning]
        Cn[Component N<br/>Independent versioning]
    end
    
    subgraph "Integration Repository"
        IR[Integration Repo<br/>Root modules & configuration]
        M[Manifest<br/>Pinned versions]
        IT[Integration Testing<br/>Hardware tests]
    end
    
    subgraph "Release Artifacts"
        CD[Consolidated Distribution<br/>Tagged release]
    end
    
    C1 --> IR
    C2 --> IR
    C3 --> IR
    Cn --> IR
    
    IR --> M
    IR --> IT
    IT --> M
    M --> CD

Release Pipeline Stages

The pipeline consists of 5 distinct stages:

Stage Flow Diagram

flowchart LR
    S1[1. COMPONENT PR SUBMITTED] --> S2[2. COMPONENT UNIT TESTS PASS]
    S2 --> S3[3. INTEGRATION PR CREATED]
    S3 --> S4[4. INTEGRATION TESTS PASS]
    S4 --> S5[5. MANIFEST UPDATED & TAGGED]

Detailed Workflow

sequenceDiagram
    participant Dev as Developer
    participant CR as Component Repo
    participant IQ as Integration Queue
    participant IR as Integration Repo
    participant HW as Hardware Test
    participant M as Manifest

    Note over Dev, CR: Component Development Phase
    Dev->>CR: Submit pull request
    CR->>CR: Run component unit tests
    
    alt Unit tests pass
        CR->>CR: Merge PR to main
        CR->>CR: git ref / optional bazel-unstable pkg
        CR->>IQ: Submit PR (request increment)
        
        Note over IQ, IR: Integration Phase
        IQ->>IR: Queue PR (serial processing)
        Note over IR, M: PR uses current manifest
        IR->>HW: Trigger integration tests with PR changes
        
        alt Integration tests pass
            HW->>IR: Test results: PASS
            IR->>IR: Merge PR
            Note over IR, M: Merge updates manifest
            IR->>M: Update manifest with new component version
            IR->>IR: Tag integration release
            Note over M: Next PR will use updated manifest
        else Integration tests fail
            HW->>IR: Test results: FAIL
            IR->>IQ: Reject PR
            IQ->>CR: Notify failure
            Note over M: Manifest remains unchanged
        end
    else Unit tests fail
        CR->>Dev: Notify test failure
    end

Component Repository Workflow

stateDiagram-v2
    [*] --> Development
    Development --> PRSubmission: Submit pull request
    PRSubmission --> UnitTesting: Run component unit tests
    UnitTesting --> MergePR: Tests pass
    UnitTesting --> Development: Tests fail
    MergePR --> DraftRelease: Create draft tag/release (optional)
    DraftRelease --> IntegrationPR: Submit PR to integration repo
    IntegrationPR --> [*]: Awaiting integration results

Integration Repository Workflow

stateDiagram-v2
    [*] --> QueuedPR: PR received from component
    QueuedPR --> LoadCurrentManifest: Load current manifest
    LoadCurrentManifest --> IntegrationTest: Serial queue processing
    IntegrationTest --> ConfigurationTest: Basic integration pass
    ConfigurationTest --> HardwareTest: Config validation pass
    HardwareTest --> MergePR: All tests pass
    HardwareTest --> RejectPR: Tests fail
    MergePR --> UpdateManifest: Merge updates manifest
    UpdateManifest --> TagRelease: Create integration tag
    TagRelease --> ReadyForNext: Ready for next PR
    ReadyForNext --> [*]: Updated manifest available
    RejectPR --> [*]: Notify component repo, manifest unchanged

Manifest Management

graph TD
    subgraph "Manifest Configurations"
        M1[Configuration A<br/>Component versions pinned]
        M2[Configuration B<br/>Component versions pinned]
        Mn[Configuration N<br/>Component versions pinned]
    end
    
    subgraph "Component Versions"
        V1[Component A: v1.2.3]
        V2[Component B: v2.1.0]
        V3[Component C: v1.5.2]
        Vn[Component N: v3.0.1]
    end
    
    M1 --> V1
    M1 --> V2
    M1 --> V3
    M1 --> Vn
    
    subgraph "Consolidated Distribution"
        CD1[Release v2024.10.1<br/>Bazel build artifacts]
    end
    
    M1 --> CD1

Key Features

Serial Queue Processing

• PRs are processed one at a time to ensure stability
• Prevents integration conflicts between simultaneous changes
• Maintains deterministic build states

Independent Component Versioning

• Each component repository maintains its own release cadence
• Components can evolve independently while integration tests ensure compatibility
• Draft releases allow for validation before integration

Hardware Testing Integration

• Integration tests include hardware validation
• Ensures the full system works in realistic deployment scenarios
• Provides confidence in release quality

Iterative Manifest Updates

• Each successful PR merge directly updates the manifest
• Next PR in the queue uses the updated manifest state
• No external dependency management needed - the integration repo maintains state internally
• Ensures timely propagation of successful integrations through the serial queue

Benefits

1. Scalability: Components can be developed and released independently
2. Quality Assurance: Multiple testing stages ensure system reliability
3. Traceability: Clear audit trail from component changes to releases
4. Automation: Minimal manual intervention required
5. Flexibility: Multiple manifest configurations support different deployment scenarios

Implementation Considerations

• Bazel workspace configuration must be maintained in the integration repository
• Hardware test infrastructure needs to be provisioned and maintained
• Serial queue processing may create bottlenecks during high-velocity periods
• Rollback strategies should be defined for failed integrations
• Monitoring and alerting should be implemented for pipeline health