22. Migrating to Aurora
Now let’s talk about homogeneous migration to Aurora databases. So you can perform a homogeneous migration in case you’re using MySQL five, six compliant databases like MySQL, MariaDB or Perkona. So if your source database is one of this, then you can definitely use a homogeneous migration to Aurora. So there are a couple options here. If you can tolerate some downtime, then you can restore from the RDS snapshot, or you can do full load using the native database tools like the MySQL Dumb. Or you can do a full load with DMs. So you can migrate the schema with native tools first, and then run full load with DMs.
And if you require near zero downtime or minimal downtime, then you can consider restoring from an RDS snapshot and then setting up bin log replication like the MySQL bin log replication. Or you can use a full load using native tools like the MySQL dump, and then set up bin log replication. Or of course, you always have the option of using DMs. So you can use full load plus CDC with DMs to get a minimal downtime. But remember that if your source database is running on RDS, then the easiest option to migrate your RDS MySQL database to aura MySQL is by creating an Aurora replica.
So you can create an aura replica from MySQL or PostgreSQL running on RDS and promote it to a standalone database. So essentially what you’re achieving here is a migration from MySQL or PostgreSQL on RDS to Aurora with minimal downtime, right? So that’s about the homogeneous migrations. Let’s look at the heterogeneous ones. So heterogeneous migration to aura. So for heterogeneous migrations, you would use SCT for schema migration or schema conversion, and then you would use DMs for data migration. And for near zero downtime, you should use full load plus CDC when you set up your DMs task, right? So that’s about the heterogeneous migration to Aura.
Now let’s look at migrating to Aurora serverless. So you can migrate between your provisioned Aurora cluster and the aura serverless cluster. So you can migrate from Aura to Aura serverless, you can migrate from aura serverless back to Aurora, or you can also migrate from RDS to Aurora and then to serverless. If you want to migrate from RDS to Aura serverless, then you have to first migrate from RDS to the provisioned Aura cluster, and then you can migrate from that provision cluster to Aura serverless. You cannot migrate directly from RDS to Aura serverless. That’s important thing to remember. All right, now let’s look at some of the strategies for migration to aura.
So for example, if you’re migrating from MySQL or PostgreSQL on RDS to aura, so that’s a homogeneous migration. So you can simply restore from a snapshot to aura, and you can use manual or automated snapshot. In this case, another option is to replicate to an aura read replica. You simply create an aura read replica and promote that replica to a standalone database, and this typically results in minimal to no downtime. And of course you can also use DMs, and if you use it with full load and CDC, that’s going to give you minimal downtime. Then if you have your MySQL running on EC Two or on premise, and if you want to migrate it to Aura, which again is a homogeneous migration, you can restore from a backup file stored on S Three.
So you can take a backup and move that file to S Three, and then you can restore from the backup to your new Aurora cluster. You can also restore from text files like CSV or XML files stored in S Three, or you can restore using the MySQL dump utility as well. And as always, you can definitely use DMs for this. If you have postgres SQL running on EC Two or on premise. And if you want to migrate to Aura, which again is homogeneous migration, you can first migrate to RDS and then migrate from RDS to Aurora. Simply migrate your postgres SQL database to RDS and then create an Aura reader replica and promote it to a standalone database.
o that’s how you would migrate from PostgreSQL running on your on premise or EC Two to Aurora cluster. And definitely you can use DMs as well. If you’re migrating from MariaDB Oracle SQL Server to Aurora, which in this case is a heterogeneous migration, then you would have to use DMs and SCT. So you’d use SCT for schema conversion first and then use DMs for data migration. So these were a couple of strategies for migration to Aurora. If you want to explore more options, then I would suggest reviewing this white paper about migrating your databases to Amazon Aurora. All right, so that’s about it. Let’s continue to the next lecture.
23. Migrating Redis workloads to ElastiCache
In this lecture, let’s look at the approaches for migrating redis workloads to ElastiCache. So you basically have two options offline migration and online migration. So with offline migration, you migrate using a backup. And for online migration, you migrate from an endpoint. So you do this from the AWS console for your ElastiCache console. So let’s look at the offline migration first. So for offline migration to ElastiCache for redis, we’d create a redis backup file first. So this is a RDB file. Then you create your S Three bucket and upload this backup to S Three. And once you have this backup in S Three, you can see this RDB file to your ElastiCache for redis cluster. So you simply create an Elastic cache for redis cluster and choose the option to see the RDB file from S Three when creating the cluster.
And we have seen in the Redis hands on demo that when we create a redis cluster, we have the option to seed a file from S Three. So you can use that option to seed the RDB file or the backup file from S Three to your new ElastiCache cluster. So that’s how you perform an offline migration. And remember that ElastiCache should have read access to this file sitting in S Three. Now let’s look at the online migration approach. So this is a real time data migration approach. You migrate your self hosted redis sitting on EC Two to ElastiCache. So you have your source redis cluster running on an EC Two instance with your client application connected to it. So to migrate from this redis cluster sitting on EC Two, what you do is you create your target redis cluster, or you can also choose an existing one if you like.
And then from the Actions menu, you simply choose Migrate data from Endpoint. So you can perform this migration right from within your ElastiCache console. And you can monitor the progress using the Event section on the ElastiCache Console. And once the migration is complete, you can failover your application to the new database. And remember, you have the option to decide when to perform the failure. Now, some of the good things to know about online redis migration. First, let’s talk about the source cluster. So your source cluster must have redis off disabled. So you have to disable redis Auth before you can perform the online redis migration.
Also, the protected mode should be disabled. And if you have any bind configuration on the source cluster, then that configuration should allow request from Elastic cache. All right, now let’s talk about the target cluster. The target cluster must have cluster mode disabled. Okay, you cannot do online redis migration with cluster mode enabled clusters. Also, you must have multiaz enabled and you should keep the encryption disabled. And needless to say, the target cluster should have sufficient memory. All right? And if you find that your use case does not fit these requirements, then you can always perform an offline migration by seeding the RDB file. All right, so that’s all about migrating to redis on ElastiCache. Let’s continue to the next lecture.
24. Migrating to DocumentDB
Now let’s talk about migrating to document DB. So, you can migrate your MongoDB workloads to document DB, and you have four approaches for migrating to document DB. So you can perform an offline migration, online migration, and you can also use a combination of online and offline migrations, and that, that’s called as hybrid approach. And in addition to this, you can also consider using a dual right approach. So, let’s look at these one by one. Where’s the offline migration, this is the simplest and fastest, but results in the longest downtime. And you can use this for non critical, non production workloads. So you have your client application connected to the source database.
The first thing you do is stop rights on the source, and then you create an EC two instance and dump the database data along with the indexes on this instance. And for this purpose, you can use mongodump or Mongo export utilities. So mongodump Mongo Restore works with Bison files and typically used with homogeneous migrations and Mongo export. Mongo import uses JSON or CSV files, and it supports both homogeneous as well as heterogeneous migrations. All right? And Document DB provides what’s called as a document DB index tool that you can use to export and import your indexes from the source database to the target database. So you dump your data and indexes using one of these tools, and then restore those indexes and data into document DB.
And once this process is complete, you can point your database to document DB. So that’s how an offline migration works in document DB. Now let’s look at the online migration approach. Now, this option involves medium level of complexity. It’s the slowest of all, but it provides us with minimal downtime. So you can use this for production workloads, and it uses DMs. So you have your client application connected to the source database, and during the process of migration, your application continues to write to the source database. So you migrate your indexes using the document DB index tool. So you dump your indexes into an EC two instance and restore them to documentb.
And then you use DMs with full load and CDC to move your data from source database to document. And remember that DMs does not migrate indexes. So once you have completed your full load and the CDC is in progress, and once you find that your document DB database is in sync with the source, you can go ahead and point your application to document DB and that completes your online migration. Then you can also consider using a mix of these two approaches, and that’s called as a hybrid approach. So it’s a mix of offline and online migration. And because it’s a combination of two, it’s the most complex approach, but faster than the online approach and provides minimal downtime. So you can definitely consider using this for production workloads.
And typically, you choose the hybrid approach when your database size is large or if you don’t have enough network bandwidth. So this approach has two phases or two parts. The way it works is you have your client application connected to your source database, and your application will continue writing to the source database. Then in phase one, what you do is you export your data using mongodump. So you dump your data and indexes to an EC to instance. And if your data is on premise, then you can transfer the data to AWS using either Direct, Connect or Snowball. And once your data is on the EC to instance, then you can restore that data along with indexes to document DB.
So you can use the document DB index tool to migrate indexes and you restore to document DB using the Mongol Restore utility. Then in phase two, what you do is you simply use DMs in CDC mode. So this is a CDC only mode, and you use it to replicate the ongoing changes, all right? And once the source and the target database or the source database and document DB are in sync, then you can go ahead and point your application to document DB. So that’s how you would use hybrid approach. Now, let’s quickly look at the dual right approach. So in dual right approach, you have your client application connected to source database, application continues to write to the database, and then you dump your data to an easy to instance, and also create indexes and restore data to document DB.
And once this process is complete, you simply start writing to document DB. So in this approach, you write to both source and the target database simultaneously. And once you’re ready for cut over, you can point your application to the target document DB database. So this approach is typically used for heterogeneous migrations. For example, if you want to migrate from a relational database to document DB in case of heterogeneous migration, you’ll have to create the indexes manually. You cannot use the document DB index tool in case of heterogeneous migrations. All right, so that’s about migrating to document DB. Let’s continue to the next lecture.
25. Streaming use cases for DMS
In this lecture, let’s talk about the streaming use cases for DMs. And as I mentioned early on in this section, the use cases for DMs are growing beyond just migrating your legacy databases to AWS. And streaming is one such use case. You can use DMs to stream data from a source database to an appropriate target. So that’s like an ongoing replication. So you can have DMs read from a source database and stream that data, let’s say to red shift. You can stream to S Three data lakes, you can stream to Kinesis streams, you can stream to Elasti Cache Service, and so on.
So, with Red shift, you can stream from any OLTP database to Red shift for analytics, you can stream to data lakes like S Three data lakes to hydrate them, you can stream to Kinesis data streams, you can stream to Elastic search service to run full text searches on the data stored on your source database. The use cases are numerous, and you can use a fan out architecture like the one you see here. Or you can also use a fan in architecture. So let’s look at the fan in architecture example as well. So, in fan in architecture, you can have your DMs instance read from different databases, like you see on the screen here. And the DMs instance can then push the data from all these different databases to, let’s say, redshift.
And you can use this for analytics purpose. For example, you can create different analytics dashboards, let’s say using QuickSite. And then your business users can use those QuickSite dashboards to take different business decisions. So this is an example of a fan in architecture. Let’s look at one more streaming example. This is an example of streaming replication and analytics. So you have Aurora database, and DMs is reading from Aurora and pushing that data to a Kinesis data stream. And further, this data is pushed to kinesis firehose or kinesis data firehose. And from Kinesis firehose, you move that data to S three. So in effect, you are streaming from Aura and replicating your data into an S Three bucket.
And then you can use Athena to run queries on the data sitting in S Three. And you can also create Quick side dashboards using the data returned by Athena. And these dashboards can be used by your business users for their decision making, right? So these were some of the streaming use cases of DMs. And with that, we come to the end of this section. And this old server brings us to the end of my part of this course. Stefan will be taking you through the rest of the course. I hope you found all this information useful, and I truly hope that it helps you in your preparation for your certification exam. So make the most of it, and I wish you all the very best for your exams. Thank you so very much.
