Migrating your Parse database
The first step is to migrate the data from your Parse hosted app to a self-hosted MongoDB. After this is done, your clients will continue talking to
api.parse.com, and you may manage your app using the hosted Parse Dashboard.
Once your data is migrated to your self-hosted MongoDB, you will be able to adjust your app's request limit all the way up to 600 requests/second, at no extra charge. You will not be charged by Parse for any database storage fees as the data is no longer hosted on Parse's servers.
Once your data is migrated, your app's traffic to
api.parse.com will be given a higher priority over apps that have not yet migrated their data. If your app has not migrated its data to a self-hosted MongoDB by April 28, 2016, a small number of database requests may fail with a
428 error code. If you run into this, migrate your data to a self-hosted MongoDB without delay.
1. Migrate Parse DB to Self-Hosted MongoDB
Set up a MongoDB instance that conforms to our database specifications. If this is your first time setting up a production MongoDB instance, we recommend using either mLab or ObjectRocket. These are database-as-a-service companies which provide fully managed MongoDB instances, and can help you scale up as needed.
Once you have Mongo set up, take note of the Mongo connection URL. Use the database migration tool to transfer your data. The database migration tool can be found in the Dashboard under App Settings → General → Migrate to external database.
Make sure to size your MongoDB at least 3x. Due to data being compressed in the hosted Parse database, make sure to size your Mongo at least 3X the current amount of data storage you are using (you can find this information in your hosted Parse app's Analytics overview page).
Ensure the user in the connection string has admin privileges. If the user in the connection string does not have admin privileges, you will need to set the
falseyourself prior to the migration. You can configure it back to
trueafter the migration is completed. (Why?)
Latency between Parse and your self-hosted MongoDB should not exceed 20ms. We recommend choosing either mLab or ObjectRocket for your hosted MongoDB as they both have datacenters in the US East geographic region.
If you plan on hosting your production database in a different geographic region, you can do so after first migrating your data out of Parse and into the self-hosted MongoDB in US East. You should also plan to host your Parse Server in the same geographic region to minimize latency.
Over-provision your database during the migration. Make sure you're running on a good sized instance or plan. This usually means working with someone like mLab or Object Rocket and paying more for a high performance line. If your migration job takes to long to achieve sync, you're probably using insufficient hardware. Once you finalize the migration, you can trim down your data, drop duplicated indexes, and finally downsize your instance.
Finding the right long term balance between price and performance will be left to you, but for migrating aim on the higher priced side for best performance. (This advice also applies to folks hosting on AWS, Digital Ocean, Heroku, or our other partners.)
Connections from Parse's servers will use IP addresses in the following range:
What happens next?
- Copy Snapshot The database migration tool first takes a snapshot of your existing data and transfers it to your Mongo database.
- Sync Next, it will pause to allow manual verification, while continuing to keep things in sync with writes that are coming in from your live app. While you are in this state, your app continues to read and write from your Parse hosted database.
- Verify Connect to your Mongo instance and browse through the collections in the newly created database. Check the collection counts and do a few spot checks to ensure that your data was migrated successfully.
You may stop the migration and try again as many times as you need (until you click on Finalize). The tool will keep things in sync for up to 24 hours after the migration started. You can go to App Settings → General → Migrate to external database and click on "Migrate" to start over if needed. You will first need to clean up your target database as the tool will only migrate data to an empty database.
Verify and back up your data
Once the migration job completes the Sync phase, take some time to Verify your new database. You are responsible for the backup and recovery of your data, so go ahead and back up your new database before proceeding. Both mLab and ObjectRocket have step-by-step guides to perform your first backup.
Finalize the database migration
Once you're satisfied with the database migration, you can finalize the transfer in the migration UI.
It is important to note that the Finalize action is irreversible. You will not be able to switch back to the Parse managed database. You will be responsible for any backups and recovery of your data. Only proceed once you have backed up your new database.
I have finalized the migration. Now what?
At this point, your app is still hitting
api.parse.com, but is using your MongoDB instance. You will need to administrate your database instance yourself, including maintaining indexes and scaling up. Parse cannot assist with the recovery of data that is no longer hosted on its servers, so it is very important that you back up your self-hosted database regularly.
Troubleshooting common errors
- The destination database was not empty. The database you are migrating to is not empty. Please clean up and retry.
- The destination database was too small. The database you are migrating to is too small to hold all of your data. Add more space to your host or buy more space if you are using a service such as mLab or ObjectRocket.
- This migration was cancelled. You can try again from the app settings page. The job was cancelled manually by the user.
- This migration was not finalized in time. When a migration job is ready to be finalized, Parse will keep the database in sync for 24 hours. If you do not take any action to finalize the job within 24 hours, the job will be cancelled automatically.
- The mongo user provided is not authorized to do migration. The user in the connection string doesn't have the necessary access to complete the migration.
You must set failIndexKeyTooLong option. You need to set the
failIndexKeyTooLongsetting of your mongo server to false.(Why?)
- The job failed too many times and reached the max retry limit. The job failed several times and we gave up trying. One possible reason can be that your database hardware cannot handle the migration load. If you have large collections (containing more than 1 million objects), please consider upgrading your database hardware. During the migration the system throughput is much higher than normal, so it requires better hardware. You can resize your host after the migration is done.
- The job failed and we cannot clean up the destination and retry.The job failed, likely due to a timeout, and it could not try again as it was unable to drop the collections in the destination database. One possible reason can be that your database hardware is not sufficient. Please consider upgrading your database hardware (mainly, increase the RAM) and try again. If the problem remains, open a support ticket.
If your database migration job does not reach the final verification step after several attempts, take note of your migration job id and file a bug report with our team. Make sure to include as much detail upfront as possible:
- Migration job id
- MongoDB version
- Storage engine
- Hosting provider
- Hardware details (CPU, RAM, available disk space)