Just a normal conversation between researchers:
A: I ran the test, the effect is negative and not significant.
B: That must be wrong. I saw the graph, the effect clearly exists and is positive.
A: I'll double check the code.
Maybe person B is right: Examining raw data is a useful way to get an initial intuition for what is going on. However, would person A have checked the code if the result was in line with this initial intuition?
When do you check your code?
It is awfully tempting to only check when "the results make no sense". However, many errors can sneak in unnoticed because they do not necessarily overturn results: a forgotten control variable, incorrect clustering of standard errors, a loop that runs but executes something other than intended.
And this is exactly where the danger lies: It is very human to allow this asymmetry of checks because we mistake "reasonable" for "correct. So how to fix it?
Check? Check.
White no method is completely foolproof, with my co-authors we have converged on a pretty solid model:
Step 1: Document what each piece of code does. So, include explanations directly inside the code, such as "in this analysis we only include people who graduated college, i.e., for whom college_grad==1".
Step 2: Verify that the code does what you think it does. Continuing the above example, verify that the code keeps (rather than drops) people for whom college_grad==1. And, obviously, do not just blindly run code produced by somebody else (including AI): make sure you understand the code!
Step 3: Ask a co-author to verify. If your co-author does not program in your chosen programming language, ask them to re-code the analysis in their favorite software. Or at least walk them through your code, explain, and show that your code does what it is supposed to.
Step 4: For situations when you are doing something very complex, consider testing your code on a very small subsample of the data such that you can manually verify the output. This is great for example when you are reshaping or matching data.
Step 5: Compare your code to your paper. Are you actually doing what you are saying you are doing?
The most dangerous errors
The most dangerous coding errors are not the ones that give you absurd results - you will spot those quickly. The most dangerous errors are those that give you results that you expected. So, please, apply the same level of scrutiny to the results than spark joy as to those that do not; the future you (and science) will thank you :)