2 minutes on Migrating Data

I had a great talk with my friend Dave today. He’s a Data Scientist. He knows his stuff, for sure.

We talked about a number of things, but one that really stuck out was data migration. He says never to migrate via code, use a tool. You are reinventing the wheel. You are locked into your solution. All the risk is in your court. And the solution is not flexible. With that said. He went on to say the most efficient way to move data is with a primary key and a hash.

The destination side will request all the primary key and row hash. Taking the primary key it will check if the row exists. If it does exist it will compare the hash of the source to the hash of the destination row. If they match then the process is repeated for the next row. If they don’t match, then the primary key is added to a list of rows to request from the source. If the primary key does not exist then the primary key is added to the list of rows to be retrieved from the source. When the row comparison is completed all the rows that are stale or do not exist are requested from the source and persisted to the destination.


If you enjoy grunt work you’ll do the above. If you are a developer who enjoys building robust applications you’ll leave the grunt work to the tools.

Chronic Contractor

This developer is always looking for a gig. There is always something better. Chronic Contractors are expensive. Mileage per dollar varies.

They have battled many and because of this they see problems. Something about every gig frustrates them.

Chronic Contractor tend to work in cutting edge technologies and loathe older “inferior” technologies. They have no allegiance, they don’t care if the ship sinks, another is waiting in the docks.

Insecurinator Developer

This developer refuses to find a better job. They have had only one job, and they will not leave it — It’s the only breast they have fed from.

They are treated poorly, paid next to nothing yet they refuse to look for greener pastures. They have an unlimited supply of excuses; they may even in principle agree to find a new job, but they never will.

Mini-Me Developer

This developer follows the King of the Hill Developer like fly’s to shit. This person is the voice of the King of the Hill Developer in their absence. They developer similar mannerisms and will spout similar phrases used by the King of the Hill Developer. They will some day grow up to be a King of the Hill Developer.

King of the Hill Developer

Typically this developer has been in few organizations. They tend to be the smart frog in the small pond. When another smarter developer joins them, they try to subjugate them. Tactics include withholding information, passive put downs and excessive explaining.

King of the Hill Developers have all the answers, similar to the Go To developer, they have the grand scheme in their minds and only give information as needed. When idea’s arise that are better or will make them look bad they vigorously deny the validity of the idea.

Since they work in a royal vacuum, their designs and code tend to be flawed.

The Ego Interviewer

You’ll begin the interview and the questions won’t feel right. They’ll be extremely technical border lining on absurd or they’ll be edge cases, in either case, specialized knowledge is required to answer correctly. The questions are a setup for failure and an opportunity for the interviewer to stroke their ego. In fact, the entire interview has little or nothing to do with the position. It’s all about the interviewer taking a hit off the validation crack-pipe.

Similar needy behavior will occur on the job. The Ego Interviewer will not hire someone who is smarter than them. They can’t have someone proving them wrong. They desire to be the smartest guy, the “go to guy” (an anti-pattern) of the group. Once they’ve decided they are smart and you are not, The Ego Interviewer will insist on explaining everything to you in the minutest detail. Again this is another strategy to display their intelligence and superiority.

The Ego Interviewer does not create a cohesive work environment, most developers in these environments will be on edge. The developers are secretly hoping the Ego Interviewer will be sick today.

The Ego Interviewer will continue to passively step on you to stroke their ego. If you challenge them they’ll stop at nothing to quell your defiance. They only way out is get them replaced or move to a different team.

Going Faster Tip #6

Use tools for other purposes than they are intended for. For example operating on large number of files can be a challenge is C#. Tools like Ant, NAnt, Rake are designed for this purpose.

Going Faster Tip #3

In Windows, type the first part of the previous command and then hit F8. The shell will perform a backward search through previous commands that match the first part of what you’ve just typed.