208 lines
5.7 KiB
Plaintext
208 lines
5.7 KiB
Plaintext
JIRA Story - RAG Knowledge Base
|
|
Accounting IA-2698: Reshuffle S3 Import Workflow
|
|
Chunk 1: Overview
|
|
|
|
Metadata:
|
|
storyId=IA-2698, type=overview, domain=fileProcessing, workflow=s3Import
|
|
|
|
Content:
|
|
Purpose: Refactors the S3 file import workflow by moving file reading and processing logic out of the Lambda trigger and into a downstream Step Function.
|
|
|
|
Business Goal:
|
|
Improve traceability, maintainability, and failure recovery by centralizing processing logic within Step Functions instead of embedding it in the S3-triggered Lambda.
|
|
|
|
Core Change:
|
|
|
|
Current: Lambda triggered by S3 both receives AND processes file
|
|
New: Lambda triggered by S3 only initiates workflow, processing occurs downstream
|
|
|
|
Outcome:
|
|
|
|
Better observability of processing steps
|
|
Ability to retry/reprocess failures via Step Function
|
|
Cleaner separation of concerns
|
|
Chunk 2: Preconditions and Dependencies
|
|
|
|
Metadata:
|
|
storyId=IA-2698, type=preconditions, dependencies=S3,StepFunctions
|
|
|
|
Content:
|
|
Required Preconditions:
|
|
|
|
S3 bucket configured to trigger Lambda on file upload
|
|
Existing Step Function capable of handling file processing logic
|
|
|
|
Dependencies:
|
|
|
|
AWS S3 event notification system
|
|
Lambda function (S3 trigger handler)
|
|
AWS Step Functions orchestration
|
|
|
|
Implicit Rule (Made Explicit):
|
|
|
|
File ingestion must still trigger the workflow exactly once per upload
|
|
Existing processing logic must be migrated fully to Step Function
|
|
Chunk 3: Functional Requirement - S3 Trigger Behavior
|
|
|
|
Metadata:
|
|
storyId=IA-2698, type=functional, component=s3TriggerLambda
|
|
|
|
Content:
|
|
New Behavior of S3 Triggered Lambda:
|
|
|
|
Receives S3 event (file upload notification)
|
|
Extracts file metadata (e.g., BucketName, ObjectKey)
|
|
Initiates Step Function execution
|
|
Passes file reference (not file contents) downstream
|
|
|
|
Explicit Rule:
|
|
|
|
Lambda must NOT read or process file contents
|
|
|
|
Implicit Rule (Made Explicit):
|
|
|
|
Lambda becomes a lightweight orchestration trigger only
|
|
Reduces Lambda execution time and complexity
|
|
Chunk 4: Functional Requirement - Step Function Processing
|
|
|
|
Metadata:
|
|
storyId=IA-2698, type=functional, component=stepFunction
|
|
|
|
Content:
|
|
Step Function Responsibilities:
|
|
|
|
Receive file reference from Lambda
|
|
Perform file reading from S3
|
|
Execute all parsing and processing logic
|
|
Handle downstream workflows dependent on file contents
|
|
|
|
Processing Capabilities:
|
|
|
|
Retry failed steps
|
|
Track execution state
|
|
Enable partial or full reprocessing
|
|
|
|
Implicit Rule (Made Explicit):
|
|
|
|
All business logic previously in Lambda must be relocated here
|
|
Step Function becomes the single source of truth for processing flow
|
|
Chunk 5: Workflow and Processing Flow
|
|
|
|
Metadata:
|
|
storyId=IA-2698, type=workflow, orchestration=stepFunctions
|
|
|
|
Content:
|
|
Updated Processing Flow:
|
|
|
|
File uploaded to S3 bucket
|
|
S3 event triggers Lambda
|
|
Lambda extracts file metadata (BucketName, ObjectKey)
|
|
Lambda invokes Step Function execution
|
|
Step Function retrieves file from S3
|
|
Step Function processes file contents
|
|
Downstream processing steps executed within workflow
|
|
|
|
Key Change:
|
|
|
|
File processing is decoupled from trigger event
|
|
Chunk 6: Non-Functional Requirements
|
|
|
|
Metadata:
|
|
storyId=IA-2698, type=nonFunctional, category=architecture
|
|
|
|
Content:
|
|
System Improvements:
|
|
|
|
Traceability: Step Function execution history provides full audit trail
|
|
Maintainability: Business logic centralized and easier to modify
|
|
Resiliency: Failed executions can be retried without re-uploading file
|
|
|
|
Performance Consideration:
|
|
|
|
Slight increase in orchestration overhead
|
|
Reduced Lambda execution burden
|
|
Chunk 7: External System Responsibilities
|
|
|
|
Metadata:
|
|
storyId=IA-2698, type=externalSystems, systems=AWS
|
|
|
|
Content:
|
|
AWS S3:
|
|
|
|
Stores uploaded files
|
|
Triggers Lambda on file creation events
|
|
|
|
AWS Lambda (Trigger Layer):
|
|
|
|
Receives S3 event
|
|
Initiates Step Function execution
|
|
|
|
AWS Step Functions:
|
|
|
|
Orchestrates full file processing workflow
|
|
Handles retries, state tracking, and execution visibility
|
|
|
|
Implicit Rule (Made Explicit):
|
|
|
|
No external system reads file directly except Step Function workflow
|
|
Chunk 8: Business Rules
|
|
|
|
Metadata:
|
|
storyId=IA-2698, type=businessRules, domain=fileProcessing
|
|
|
|
Content:
|
|
Core Rules:
|
|
|
|
File processing must not occur inside S3-triggered Lambda
|
|
Step Function must handle all file reading and processing
|
|
File reference (not content) must be passed between components
|
|
Workflow must remain automatically triggered upon file upload
|
|
|
|
Acceptance Criteria Rule:
|
|
|
|
Step Function execution must still trigger after S3 import (regression requirement)
|
|
|
|
Edge Case Handling:
|
|
|
|
If Step Function fails → workflow can be retried without re-uploading file
|
|
If Lambda fails → file may require re-trigger or retry mechanism
|
|
Chunk 9: Data Quality Assumptions and Risks
|
|
|
|
Metadata:
|
|
storyId=IA-2698, type=dataQuality, riskLevel=low
|
|
|
|
Content:
|
|
Assumptions:
|
|
|
|
S3 event reliably delivers correct file metadata
|
|
Step Function has access to correct S3 permissions
|
|
File format and structure remain unchanged
|
|
|
|
Risks:
|
|
|
|
Misconfigured event payload → Step Function receives invalid file reference
|
|
Migration gaps → logic not fully moved from Lambda
|
|
Increased dependency on Step Function availability
|
|
|
|
Mitigation:
|
|
|
|
Regression testing ensures workflow triggers correctly
|
|
Validation of input payload before Step Function execution
|
|
Chunk 10: Search Queries Supported
|
|
|
|
Metadata:
|
|
storyId=IA-2698, type=queryPatterns, purpose=RAGRetrieval
|
|
|
|
Content:
|
|
This knowledge base supports queries such as:
|
|
|
|
"How is S3 file processing handled after this change?"
|
|
"Why move file processing from Lambda to Step Function?"
|
|
"What does the S3 trigger Lambda do now?"
|
|
"Where is file parsing logic executed?"
|
|
"How do Step Functions improve file processing reliability?"
|
|
"Can file processing be retried without re-uploading?"
|
|
"What data is passed from Lambda to Step Function?"
|
|
"What are the benefits of this architecture change?"
|
|
"What happens when an S3 file is uploaded?"
|
|
"What regression requirement exists for S3 workflow?" |