Most upvoted comment

Most popular programming book on Reddit. rank no. 54

A modest proposal (Absolutely no babies involved)(r/ReverseEngineering)

For those wanting to delve into the more formal side of things, I’d say you need:

• Compilers. Program analysis and formal verification grew out of compiler theory, and therefore you really can not get enough of compilers (principles and practice). It’s hard to recommend a single source, but Compilers: Principles, Techniques, and Tools (2nd Edition) is often recommended. I personally enjoy Modern Compiler Implementation in ML, and Advanced Compiler Design & Implementation (do not buy this one used due to the crazy amount of errata).
• Programming language theory and formal semantics. Semantics with Applications: An Appetizer is a nice textbook that makes a dry subject entertaining, and is written by the same authors of Principles of Program Analysis (the benefit here being the consistent notation across the two textbooks). I also found the University of Washington CSE P505 online video lectures and materials to be very helpful.
• Lattice theory for abstract interpretation. This one requires a mathematical background; there is no escaping it. Introduction to Lattices and Order is the classic text in this field.
• Program analysis itself. Start by reading the BitBlaze publications from the beginning until present, and follow with the BAP publications. You might find this book on decision procedures helpful. BitBlaze and BAP have also released their source code, which is extremely helpful in studying program analysis. The static analysis components are written in OCaml, for which I have released an interpreter that runs inside of IDA. After you’ve done this, dig through the archives of the reverse engineering reddit, looking at the PDFs with formal-sounding titles. Program analysis is huge, and there are a multitude of rabbit holes to venture down; you might end up somewhere completely different from everyone else.

Permalink

More details about a book.