This guide will walk you through setting up and deploying your first Intelligent Document Processing (IDP) solution using the GenAI IDP Accelerator CDK packages. By the end of this guide, you'll have a fully functional document processing pipeline capable of transforming unstructured documents into structured data.
Prerequisites
Before you begin, ensure you have the following prerequisites installed and configured:
First, let's set up your development environment with the correct tools:
# Install AWS CDK globally
npminstall-gaws-cdk
# Initialize a new CDK project
mkdirmy-idp-project&&cdmy-idp-project
cdkinitapp--language=typescript
# Install AWS CDK globally
npminstall-gaws-cdk
# Initialize a new CDK project
mkdirmy-idp-project&&cdmy-idp-project
cdkinitapp--language=python
# Create and activate a virtual environment
python-mvenv.venv
source.venv/bin/activate
# Install AWS CDK globally
npminstall-gaws-cdk
# Initialize a new CDK project
mkdirmy-idp-project&&cdmy-idp-project
cdkinitapp--language=csharp
# Open the solution in your preferred IDE# For Visual Studio Code:
code.
# For Visual Studio:
startMyIdpProject.sln
Step 2: Install GenAI IDP Packages
Now, let's install the GenAI IDP packages that we'll need for our document processing solution:
# Install core IDP package and the Bedrock LLM processor
npminstall@cdklabs/genai-idp@cdklabs/genai-idp-bedrock-llm-processor
# Install core IDP package and the Bedrock LLM processor
pipinstallcdklabs.genai-idpcdklabs.genai-idp-bedrock-llm-processor
# Add the NuGet packages to your project
dotnetaddsrc/MyIdpProjectpackageCdklabs.GenaiIdp
dotnetaddsrc/MyIdpProjectpackageCdklabs.GenaiIdpBedrockLlmProcessor
Step 3: Bootstrap Your AWS Environment
If you haven't already bootstrapped your AWS environment for CDK, run:
cdkbootstrapaws://ACCOUNT-NUMBER/REGION
Replace ACCOUNT-NUMBER with your AWS account number and REGION with your preferred AWS region.
Step 4: Create Your IDP Stack
Now, let's create the core infrastructure for our document processing solution. We'll modify the main stack file to include the GenAI IDP components:
// lib/my-idp-project-stack.tsimport*ascdkfrom'aws-cdk-lib';import{Construct}from'constructs';import{Bucket,RemovalPolicy}from'aws-cdk-lib/aws-s3';import{Key}from'aws-cdk-lib/aws-kms';import{ServicePrincipal}from'aws-cdk-lib/aws-iam';import{ProcessingEnvironment,ProcessingEnvironmentApi,UserIdentity,WebApplication,ConfigurationTable,TrackingTable}from'@cdklabs/genai-idp';import{BedrockLlmProcessor,BedrockLlmProcessorConfiguration}from'@cdklabs/genai-idp-bedrock-llm-processor';import{AuthorizationType,UserPoolDefaultAction}from'aws-cdk-lib/aws-appsync';exportclassMyIdpProjectStackextendscdk.Stack{constructor(scope:Construct,id:string,props?:cdk.StackProps){super(scope,id,props);constmetricNamespace=this.stackName;// Create KMS key for encryptionconstkey=newKey(this,'CustomerManagedEncryptionKey');key.grantEncryptDecrypt(newServicePrincipal('logs.amazonaws.com'));// Create S3 buckets for document processingconstinputBucket=newBucket(this,'InputBucket',{encryptionKey:key,eventBridgeEnabled:true,// Required for event-driven processingremovalPolicy:RemovalPolicy.DESTROY,autoDeleteObjects:true,});constoutputBucket=newBucket(this,'OutputBucket',{encryptionKey:key,removalPolicy:RemovalPolicy.DESTROY,autoDeleteObjects:true,});constworkingBucket=newBucket(this,'WorkingBucket',{encryptionKey:key,removalPolicy:RemovalPolicy.DESTROY,autoDeleteObjects:true,});// Create user identity for authenticationconstuserIdentity=newUserIdentity(this,'UserIdentity');// Grant bucket access to authenticated usersinputBucket.grantRead(userIdentity.identityPool.authenticatedRole);outputBucket.grantRead(userIdentity.identityPool.authenticatedRole);// Create DynamoDB tables for configuration and trackingconstconfigurationTable=newConfigurationTable(this,'ConfigurationTable',{encryptionKey:key,});consttrackingTable=newTrackingTable(this,'TrackingTable',{encryptionKey:key,});// Create the GraphQL API for document trackingconstapi=newProcessingEnvironmentApi(this,'EnvApi',{inputBucket,outputBucket,encryptionKey:key,configurationTable,trackingTable,authorizationConfig:{defaultAuthorization:{authorizationType:AuthorizationType.USER_POOL,userPoolConfig:{userPool:userIdentity.userPool,defaultAction:UserPoolDefaultAction.ALLOW,},},additionalAuthorizationModes:[{authorizationType:AuthorizationType.IAM,},],},});// Create the processing environmentconstenvironment=newProcessingEnvironment(this,'Environment',{key,inputBucket,outputBucket,workingBucket,configurationTable,trackingTable,api,metricNamespace,});// Create the document processor using Bedrock LLM processor// This uses a sample configuration - for custom configurations, use BedrockLlmProcessorConfiguration.fromFile()constprocessor=newBedrockLlmProcessor(this,'Processor',{environment,configuration:BedrockLlmProcessorConfiguration.lendingPackageSample(),// Optional: Customize the processor with additional settings// classificationMaxWorkers: 10,// ocrMaxWorkers: 10,// evaluationBaselineBucket: new Bucket(this, 'BaselineBucket'),});// Add the processor's state machine to the APIapi.addStateMachine(processor.stateMachine);// Grant API permissions to authenticated usersapi.grantQuery(userIdentity.identityPool.authenticatedRole);api.grantSubscription(userIdentity.identityPool.authenticatedRole);// Create web application for document managementconstwebApplication=newWebApplication(this,'WebApp',{webAppBucket:newBucket(this,'WebAppBucket',{websiteIndexDocument:'index.html',websiteErrorDocument:'index.html',removalPolicy:RemovalPolicy.DESTROY,autoDeleteObjects:true,}),userIdentity,environment,apiUrl:api.graphqlUrl,});// Output the important valuesnewcdk.CfnOutput(this,'InputBucketName',{value:inputBucket.bucketName,description:'Name of the input bucket where documents should be uploaded',});newcdk.CfnOutput(this,'OutputBucketName',{value:outputBucket.bucketName,description:'Name of the output bucket where processed results will be stored',});newcdk.CfnOutput(this,'WebSiteUrl',{value:`https://${webApplication.distribution.distributionDomainName}`,description:'URL of the web application for document management',});}}
# my_idp_project/my_idp_project_stack.pyfromaws_cdkimport(Stack,RemovalPolicy,CfnOutput,)fromaws_cdk.aws_s3importBucketfromaws_cdk.aws_kmsimportKeyfromaws_cdk.aws_iamimportServicePrincipalfromaws_cdk.aws_appsyncimport(AuthorizationType,UserPoolDefaultAction)fromcdklabs.genai_idpimport(ProcessingEnvironment,ProcessingEnvironmentApi,UserIdentity,WebApplication,ConfigurationTable,TrackingTable)fromcdklabs.genai_idp_bedrock_llm_processorimport(BedrockLlmProcessor,BedrockLlmProcessorConfiguration)fromconstructsimportConstructclassMyIdpProjectStack(Stack):def__init__(self,scope:Construct,construct_id:str,**kwargs)->None:super().__init__(scope,construct_id,**kwargs)metric_namespace=self.stack_name# Create KMS key for encryptionkey=Key(self,"CustomerManagedEncryptionKey")key.grant_encrypt_decrypt(ServicePrincipal("logs.amazonaws.com"))# Create S3 buckets for document processinginput_bucket=Bucket(self,"InputBucket",encryption_key=key,event_bridge_enabled=True,# Required for event-driven processingremoval_policy=RemovalPolicy.DESTROY,auto_delete_objects=True)output_bucket=Bucket(self,"OutputBucket",encryption_key=key,removal_policy=RemovalPolicy.DESTROY,auto_delete_objects=True)working_bucket=Bucket(self,"WorkingBucket",encryption_key=key,removal_policy=RemovalPolicy.DESTROY,auto_delete_objects=True)# Create user identity for authenticationuser_identity=UserIdentity(self,"UserIdentity")# Grant bucket access to authenticated usersinput_bucket.grant_read(user_identity.identity_pool.authenticated_role)output_bucket.grant_read(user_identity.identity_pool.authenticated_role)# Create DynamoDB tables for configuration and trackingconfiguration_table=ConfigurationTable(self,"ConfigurationTable",encryption_key=key)tracking_table=TrackingTable(self,"TrackingTable",encryption_key=key)# Create the GraphQL API for document trackingapi=ProcessingEnvironmentApi(self,"EnvApi",input_bucket=input_bucket,output_bucket=output_bucket,encryption_key=key,configuration_table=configuration_table,tracking_table=tracking_table,authorization_config={"default_authorization":{"authorization_type":AuthorizationType.USER_POOL,"user_pool_config":{"user_pool":user_identity.user_pool,"default_action":UserPoolDefaultAction.ALLOW,},},"additional_authorization_modes":[{"authorization_type":AuthorizationType.IAM,},],},)# Create the processing environmentenvironment=ProcessingEnvironment(self,"Environment",key=key,input_bucket=input_bucket,output_bucket=output_bucket,working_bucket=working_bucket,configuration_table=configuration_table,tracking_table=tracking_table,api=api,metric_namespace=metric_namespace)# Create the document processor using Bedrock LLM processor# This uses a sample configuration - for custom configurations, use BedrockLlmProcessorConfiguration.from_file()processor=BedrockLlmProcessor(self,"Processor",environment=environment,configuration=BedrockLlmProcessorConfiguration.lending_package_sample(),# Optional: Customize the processor with additional settings# classification_max_workers=10,# ocr_max_workers=10,# evaluation_baseline_bucket=Bucket(self, "BaselineBucket"),)# Add the processor's state machine to the APIapi.add_state_machine(processor.state_machine)# Grant API permissions to authenticated usersapi.grant_query(user_identity.identity_pool.authenticated_role)api.grant_subscription(user_identity.identity_pool.authenticated_role)# Create web application for document managementweb_application=WebApplication(self,"WebApp",web_app_bucket=Bucket(self,"WebAppBucket",website_index_document="index.html",website_error_document="index.html",removal_policy=RemovalPolicy.DESTROY,auto_delete_objects=True,),user_identity=user_identity,environment=environment,api_url=api.graphql_url,)# Output the important valuesCfnOutput(self,"InputBucketName",value=input_bucket.bucket_name,description="Name of the input bucket where documents should be uploaded")CfnOutput(self,"OutputBucketName",value=output_bucket.bucket_name,description="Name of the output bucket where processed results will be stored")CfnOutput(self,"WebSiteUrl",value=f"https://{web_application.distribution.distribution_domain_name}",description="URL of the web application for document management")
// src/MyIdpProject/MyIdpProjectStack.csusingAmazon.CDK;usingAmazon.CDK.AWS.S3;usingAmazon.CDK.AWS.KMS;usingAmazon.CDK.AWS.IAM;usingAmazon.CDK.AWS.AppSync;usingCdklabs.GenaiIdp;usingCdklabs.GenaiIdpBedrockLlmProcessor;usingConstructs;namespaceMyIdpProject{publicclassMyIdpProjectStack:Stack{publicMyIdpProjectStack(Constructscope,stringid,IStackPropsprops=null):base(scope,id,props){varmetricNamespace=this.StackName;// Create KMS key for encryptionvarkey=newKey(this,"CustomerManagedEncryptionKey");key.GrantEncryptDecrypt(newServicePrincipal("logs.amazonaws.com"));// Create S3 buckets for document processingvarinputBucket=newBucket(this,"InputBucket",newBucketProps{EncryptionKey=key,EventBridgeEnabled=true,// Required for event-driven processingRemovalPolicy=RemovalPolicy.DESTROY,AutoDeleteObjects=true});varoutputBucket=newBucket(this,"OutputBucket",newBucketProps{EncryptionKey=key,RemovalPolicy=RemovalPolicy.DESTROY,AutoDeleteObjects=true});varworkingBucket=newBucket(this,"WorkingBucket",newBucketProps{EncryptionKey=key,RemovalPolicy=RemovalPolicy.DESTROY,AutoDeleteObjects=true});// Create user identity for authenticationvaruserIdentity=newUserIdentity(this,"UserIdentity");// Grant bucket access to authenticated usersinputBucket.GrantRead(userIdentity.IdentityPool.AuthenticatedRole);outputBucket.GrantRead(userIdentity.IdentityPool.AuthenticatedRole);// Create DynamoDB tables for configuration and trackingvarconfigurationTable=newConfigurationTable(this,"ConfigurationTable",newConfigurationTableProps{EncryptionKey=key});vartrackingTable=newTrackingTable(this,"TrackingTable",newTrackingTableProps{EncryptionKey=key});// Create the GraphQL API for document trackingvarapi=newProcessingEnvironmentApi(this,"EnvApi",newProcessingEnvironmentApiProps{InputBucket=inputBucket,OutputBucket=outputBucket,EncryptionKey=key,ConfigurationTable=configurationTable,TrackingTable=trackingTable,AuthorizationConfig=newAuthorizationConfig{DefaultAuthorization=newAuthorizationMode{AuthorizationType=AuthorizationType.USER_POOL,UserPoolConfig=newUserPoolConfig{UserPool=userIdentity.UserPool,DefaultAction=UserPoolDefaultAction.ALLOW,},},AdditionalAuthorizationModes=new[]{newAuthorizationMode{AuthorizationType=AuthorizationType.IAM,},},},});// Create the processing environmentvarenvironment=newProcessingEnvironment(this,"Environment",newProcessingEnvironmentProps{Key=key,InputBucket=inputBucket,OutputBucket=outputBucket,WorkingBucket=workingBucket,ConfigurationTable=configurationTable,TrackingTable=trackingTable,Api=api,MetricNamespace=metricNamespace});// Create the document processor using Bedrock LLM processor// This uses a sample configuration - for custom configurations, use BedrockLlmProcessorConfiguration.FromFile()varprocessor=newBedrockLlmProcessor(this,"Processor",newBedrockLlmProcessorProps{Environment=environment,Configuration=BedrockLlmProcessorConfiguration.LendingPackageSample(),// Optional: Customize the processor with additional settings// ClassificationMaxWorkers = 10,// OcrMaxWorkers = 10,// EvaluationBaselineBucket = new Bucket(this, "BaselineBucket"),});// Add the processor's state machine to the APIapi.AddStateMachine(processor.StateMachine);// Grant API permissions to authenticated usersapi.GrantQuery(userIdentity.IdentityPool.AuthenticatedRole);api.GrantSubscription(userIdentity.IdentityPool.AuthenticatedRole);// Create web application for document managementvarwebApplication=newWebApplication(this,"WebApp",newWebApplicationProps{WebAppBucket=newBucket(this,"WebAppBucket",newBucketProps{WebsiteIndexDocument="index.html",WebsiteErrorDocument="index.html",RemovalPolicy=RemovalPolicy.DESTROY,AutoDeleteObjects=true}),UserIdentity=userIdentity,Environment=environment,ApiUrl=api.GraphqlUrl,});// Output the important valuesnewCfnOutput(this,"InputBucketName",newCfnOutputProps{Value=inputBucket.BucketName,Description="Name of the input bucket where documents should be uploaded"});newCfnOutput(this,"OutputBucketName",newCfnOutputProps{Value=outputBucket.BucketName,Description="Name of the output bucket where processed results will be stored"});newCfnOutput(this,"WebSiteUrl",newCfnOutputProps{Value=$"https://{webApplication.Distribution.DistributionDomainName}",Description="URL of the web application for document management"});}}}
Step 5: Deploy Your IDP Solution
Now that we've defined our IDP stack, let's deploy it to AWS:
# Install dependencies
npminstall
# Build the project
npmrunbuild
# Deploy the stack
cdkdeploy
# Install dependencies
pipinstall-rrequirements.txt
# Deploy the stack
cdkdeploy
# Build the project
dotnetbuild
# Deploy the stack
cdkdeploy
The deployment process will take several minutes as it creates all the necessary AWS resources, including:
S3 buckets for input and output
Lambda functions for document processing
Step Functions workflow for orchestration
DynamoDB tables for tracking and configuration
IAM roles and policies
AppSync GraphQL API for querying document status
Once deployment is complete, note the output values for your input and output bucket names.
Step 6: Configure Bedrock Model Access
Before you can use the IDP solution, you need to ensure you have access to the Bedrock models used in your stack:
The output will include:
- Extracted text from the document (OCR results)
- Structured JSON data with extracted fields based on your configuration
- Document classification results (document type, confidence scores)
- Processing metadata (timestamps, processing duration, etc.)
- Document summary (if summarization is enabled)
GraphQL API Testing
You can also query the processing status using the GraphQL API:
When you're done experimenting with your IDP solution, you can clean up all resources to avoid incurring charges:
cdkdestroy
Next Steps
Now that you have a working IDP solution, consider exploring these advanced topics:
Advanced Configuration
Custom Document Schemas: Define custom extraction schemas for your specific document types
Model Selection: Choose different Bedrock models for classification and extraction
Guardrails: Implement Bedrock guardrails for content filtering and safety
Evaluation: Set up evaluation baselines to measure extraction accuracy
Integration and Automation
API Integration: Connect your IDP solution to existing business systems
Workflow Automation: Integrate with business process management systems
Data Pipeline: Set up automated data pipelines for processed documents
Notifications: Configure alerts and notifications for processing events
Security and Compliance
VPC Isolation: Deploy processing components within a VPC for enhanced security
Encryption: Implement end-to-end encryption for sensitive documents
Access Control: Fine-tune IAM roles and policies for least privilege access
Audit Logging: Enable comprehensive logging for compliance requirements
Monitoring and Optimization
CloudWatch Dashboards: Create custom dashboards for monitoring processing metrics
Cost Optimization: Analyze and optimize costs for your specific workload
Performance Tuning: Adjust concurrency limits and resource allocation
Error Handling: Implement robust error handling and retry mechanisms
Human-in-the-Loop (HITL)
Review Workflows: Set up human review processes for low-confidence extractions
Quality Assurance: Implement quality control checkpoints in your processing pipeline
Feedback Loops: Create mechanisms to improve model performance based on human feedback
Scaling and Production
Multi-Region Deployment: Deploy across multiple AWS regions for high availability
Load Testing: Test your solution with production-level document volumes
Disaster Recovery: Implement backup and recovery procedures
CI/CD Pipeline: Set up automated deployment pipelines for updates
Sample Applications
Explore the provided sample applications to see different implementation patterns:
- BDA Lending Sample: Standard document processing with minimal code
- Bedrock LLM Sample: Custom extraction with VPC isolation
- SageMaker UDOP Sample: Specialized classification with custom models