No item should ever go un-processed

I have recently been fixing a support incident with a system of ours which processes data in the form of CSV files emailed to an exchange mailbox. Occasionally the producer of these CSVs doesn’t form them correctly and the whole thing packs up; the item stays in the mailbox, and somebody has to go digging to fix it, and get the whole thing up and running before the backlog of data to be processed gets rather large. This has brought to light two important points about data-processing in my mind:

  1. No item should ever go “un-processed” – if it fails to go down the normal handling path, it should go down an error handling path – either way it should leave the main queue.
  2. There must always be a mechanism for a human to intervene, and correct (or choose to ignore) malformed data; this makes for nice error-handling path for point 1.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s