I have recently been fixing a support incident with a system of ours which processes data in the form of CSV files emailed to an exchange mailbox. Occasionally the producer of these CSVs doesn’t form them correctly and the whole thing packs up; the item stays in the mailbox, and somebody has to go digging to fix it, and get the whole thing up and running before the backlog of data to be processed gets rather large. This has brought to light two important points about data-processing in my mind:
- No item should ever go “un-processed” – if it fails to go down the normal handling path, it should go down an error handling path – either way it should leave the main queue.
- There must always be a mechanism for a human to intervene, and correct (or choose to ignore) malformed data; this makes for nice error-handling path for point 1.