Replying to
My process of reverse engineering this was quite methodical.
Since it is hand-written assembly, it employs numerous clever tricks, global program knowledge, and ad hoc calling conventions for every function.
Many functions have straightforward dataflow and a clear calling convention. Some require examining the demanded registers across all callers, to determine whether a register is written or clobbered.
But a nightmarish trio of functions are mutually recursive. To correctly analyze these, I annotated points with the register and stack state. Loops merge states on backedges, branches merge states on join. So far so good; this logic remains relatively local.
At recursive calls, I'd initially assume the call would update all state and I'd represent this with fresh values for all registers. This lets me do a dataflow analysis as usual. But some of these fresh values were observed, so I couldn't stop yet, as a full fixpoint was required. I repeatedly propagated constraints between the functions until the calling conventions were fully resolved.
Yes, I did this by hand. Yes, I'm crazy.
I'm working on a decompiler for PDP-11 assembly to period-appropriate C, but I haven't finished it yet, so I instead played as the static analyzer.
5/6
#programminglanguages #reverseengineering
