NOTE: Most users should now follow the documentation here to migrate existing data into Crypteron. The older post below documents a longer, cautious process suitable for certain high-risk scenarios (e.g. nuclear plant).
Today we’ll cover how to migrating existing data from a live application so that Crypteron now protects it. We’ll assume a SQL database here, so we’ll talk in terms of columns but if you’re using an object store, just think “property” when you read “column” below. The general idea is as follows:
- Your authorized application always has access to it’s data (encrypted or not)
- “Migration” merely involved your app writing data to the encrypted column. Behind the scenes, Crypteron handles the encryption and key management as necessary.
A simple example
Lets look at an example where a user’s credit card number must now to be encrypted. The logic is as simple as
- Create a new column for the encrypted credit card number. If you had
CreditCardpreviously, add another column
Secure_CreditCard. This new column will have all
NULLs to begin with.
- When writing, write the credit card number to the encrypted column and set the clear-text column to
NULL. You’ve just migrated this particular record.
- When reading, read from the encrypted column. If the encrypted column is still
NULL, your data hasn’t migrated yet from step #2 above, so use the clear-text column value.
That’s it! This is visually depicted as follows
This ensures that your application continues to stay online and work while seamlessly migrating over your most frequently used data first. There is no need to take your database or application offline just for migrating. The best part is that your database’s natural access patterns are preserved i.e. you won’t suddenly overload your database with a data-intensive migration job.
Bulk migration as a background job
You can accelerate the above “just-in-time” migration above by running a small job that explicitly migrates data. In C# with CipherDB for Entity Framework, it looks something like below. In Java, CipherDB for Hibernate is equivalently similar.
// Read => Write sample code
using (var secDbCtx = new CipherDbContext())
int count = 0;
foreach (var user in secDbCtx.Users.Where(u => u.Secure_CreditCard == null))
// migrate user table, sensitive fields
user.Secure_CreditCard = user.CreditCard;
user.CreditCard = null;
// save in chunks for performance
if ((++count % 100) == 0)
secDbCtx.SaveChanges(); // leftovers from last loop
Crypteron CipherDB automatically encrypts-on-writes and decrypts-on-reads, so it’s really that simple. If you’re using CipherObject, the only addition is that you’ll have to invoke
Seal() before the writes and
Unseal() after the reads – but the overall concept is the same.
If you’re migrating a very large number of records, it’s wise to limit work to batches by using something like
secDbCtx.Users.Where(u => u.Secure_CreditCard == null).Take(5000) and adding an outer loop to march through your data set in such batches. For large data sets, we suggest performing the migration over the weekend or adding some sleep or throttling above to limit the impact of the migration job on your live production workload.
If you’ve followed the above pattern, all clear-text values in that column will be
NULL at the end. After you’ve confirmed this, you can safely remove the older clear-text column to clean up your database schema.
If you have any questions, please don’t hesitate to contact us through support.
One thought on “Migrating existing live data into Crypteron”
Thanks, this really helped me.