Migrating existing live data into Crypteron

We've had several customers ask us about migrating existing data from a live application such that it is now protected by the Crypteron data security platform. We'll assume a SQL database here, so we'll talk in terms of columns but if you're using an object store, just think "property" when you read "column" below. The guiding principles are

  1. Your authorized application always has access to data whether it's still clear-text or it's encrypted
  2. "Migration" is merely writing the clear-text value to an encrypted column

A simple example

Lets look at an example where a user's credit card number must now to be encrypted. The logic is as simple as

  1. Create a new column for the encrypted credit card number. If you had CreditCard previously, add another column Secure_CreditCard. This new column will have all NULLs to begin with.
  2. When writing, write the credit card number to the encrypted column and set the clear-text column to NULL. You've just migrated this particular record.
  3. When reading, read from the encrypted column. If the encrypted column is still NULL, your data hasn't migrated yet from step #2 above, so use the clear-text column value.

That's it! This is visually depicted as follows

 

This ensures that your application continues to stay online and work while seamlessly migrating over your most frequently used data first. There is no need to take your database or application offline just for migrating. The best part is that your database's natural access patterns are preserved i.e. you won't suddenly overload your database with a data-intensive migration job.

Bulk migration as a background job

You can accelerate the above "just-in-time" migration above by running a small job that explicitly migrates data. In C# with CipherDB for Entity Framework, it looks something like below. In Java, CipherDB for Hibernate is equivalently similar.

// Read => Write sample code
using (var secDbCtx = new CipherDbContext())
{
    int count = 0;
    foreach (var user in secDbCtx.Users.Where(u => u.Secure_CreditCard == null))
    {
        // migrate user table, sensitive fields
        user.Secure_CreditCard = user.CreditCard;
        user.CreditCard = null;

        // save in chunks for performance
        if ((++count % 100) == 0)
            secDbCtx.SaveChanges(); 
    }

    secDbCtx.SaveChanges(); // leftovers from last loop
}

Crypteron CipherDB automatically encrypts-on-writes and decrypts-on-reads, so it's really that simple. If you're using CipherObject, the only addition is that you'll have to invoke Seal() before the writes and Unseal() after the reads - but the overall concept is the same.

If you're migrating a very large number of records, it's wise to limit work to batches by using something like secDbCtx.Users.Where(u => u.Secure_CreditCard == null).Take(5000) and adding an outer loop to march through your data set in such batches. For large data sets, we suggest performing the migration over the weekend or adding some sleep or throttling above to limit the impact of the migration job on your live production workload.

Post migration

If you've followed the above pattern, all clear-text values in that column will be NULL at the end. After you've confirmed this, you can safely remove the older clear-text column to clean up your database schema.

If you have any questions, please don't hesitate to contact us through support.

Leave a Reply

Your email address will not be published. Required fields are marked *

You may use these HTML tags and attributes:

<a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <s> <strike> <strong>

This site uses Akismet to reduce spam. Learn how your comment data is processed.

Recent blog posts

Migrating existing live data into Crypteron

You’re already live in production. And you have sensitive in the clear. Read this article to see how Crypteron can help.

Encryption, Entity Framework and Projections

Projections in Entity Framework live outside the entity lifecycle. Read more to learn how your can use Crypteron to secure such data.

PCI DSS and key rotations simplified

PCI compliance requires data encryption keys to be changed frequently. Here is how you can do it easily.

Your data-center is not secure and what you can do about it

There is no secure perimeter anymore. Neither in your corporate network nor in your data center. Fight a winning battle armed with self-protecting data rather than a losing one trying to protecting the infrastructure.

Introducing the Crypteron Startup Innovators Program

Qualifying startups get up to 50% off all plans. Tell us how you’re changing the world and the our Startup Innovators Program will support your journey.

6 encryption mistakes that lead to data breaches

If encryption is so unbreakable, why do businesses and governments keep getting hacked? Six common encryption mistakes that lead to data breaches.

Announcing the new Crypteron Community Edition

Starting today you can now sign up for the Crypteron Community Edition for free with no performance limitations.

Data breach response – One click to save your business

Get breathing room – when you need it the most. Respond to a data breach with a single click.

Why We Need Proper Data-At-Rest Encryption: 191M U.S. Voters’ Data Exposed

Adding security at the application level is a large step forward in protecting data from the constant threat of data breaches

How to encrypt large files

CipherStor is blazingly fast! Here we show how to use it within your data-flow pipeline to maintain high performance when encrypting large files.

Migrating existing live data into Crypteron

by Sid Shetye time to read: 2 min
0