The S3 bucket that stores the migration accuracy results for the AWS Glue Data Catalog table When the stack is complete, you should see the resources in the following table launched and available in your account. It takes approximately 20вЂ“30 minutes for the deployment to complete. You can view the stack outputsВ on the AWS Management ConsoleВ or by using the following AWS Command Line InterfaceВ (AWS CLI) command:ДЄws cloudformation describe-stacks -stack-name -region us-east-1 -query Stacks.Outputs

On the Review page, select I acknowledge that AWS CloudFormation might create IAM resources with custom names.

On the Specify DetailsВ page, provide values for the parameters that require input (see the following screenshot).

On the Select TemplateВ page, keep the default URL for the CloudFormation template, then choose Next.

The template is launched in the US East (N.

Choose Launch Stack to launch the CloudFormation template.

To deploy the solution, complete the following steps:

Runs the Spark ingestion process from Amazon EMR, connecting to the source database, and extracts data to Amazon S3 in Parquet format.

Launches an EMR cluster, creates and loads the database and tables on the source database, and imports the open-source library for PyDeequ at the EMR primary node.

Launches Secrets Manager for storing the credentials for accessing the source database.

Launches and configures Amazon RDS for MySQL as a source database.

The CloudFormation stack performs the following actions: To make it easy for you to get started, we created an AWS CloudFormation template that automatically configures and deploys the solution for you. In this example, you have four tables in your on-premises database that you want to migrate: tbl_books, tbl_sales, tbl_venue, and tbl_category. The following diagram illustrates the solution architecture. We use this to analyze each column in the given dataset to calculate statistics like completeness, approximate distinct values, and data types. We use profilers, which is one of the metrics computation components of PyDeequ.

Amazon Athena to query the output table to verify the results.AWS Glue to catalog the technical table, which stores the result of the PyDeequ job.We use PyDeequ to validate data between MySQL and the corresponding Parquet files present in the target. Amazon Simple Storage Service (Amazon S3) or Hadoop Distributed File System (HDFS) as the target.Amazon RDS for My SQL as the database engine for the source database.

This solution uses the following services:

IAM roles EMR_DefaultRole and EMR_EC2_DefaultRole available in your account.

A SQL editor to connect to the source database.

An AWS Identity and Access ManagementВ (IAM) policy for AWS Secrets Manager permissions.An Amazon Elastic Compute Cloud (Amazon EC2) key pair.An Amazon Virtual Private Cloud (Amazon VPC) with a public subnet.An AWS account with access to AWS services.PrerequisitesД«efore getting started, make sure you have the following prerequisites: Deequ is written in Scala, whereas PyDeequ allows you to use its data quality and testing capabilities from Python and PySpark. PyDeequ is an open-source Python wrapper over Deequ (an open-source tool developed and used at Amazon). In this post, we walk through a step-by-step process to validate large datasets after migration using PyDeequ. However, developing custom solutions to determine migration accuracy by comparing the data between the source and target can often be time-consuming. This data validation is a critical step and if not done correctly, may result in the failure of the entire project. During data migration, a key requirement is to validate all the data that has been moved from on premises to the cloud. Many enterprises are migrating their on-premises data stores to the AWS Cloud.

Mydatabase home and business