Rust Debugging & Profiling Notes

Video Guide

https://youtu.be/gXbNs0dhvB0

What's Broken

• generate_value(): generates dupes
• find_record(): Off by one causes panic
• fibonacci(): slow recursive
• ORM validates against "admin" keyword which causes save to fail

Level 1: Print Debugging

Print / Formatting / dbg

cargo run

Notes

• https://doc.rust-lang.org/std/fmt/index.html
  • https://doc.rust-lang.org/std/fmt/#formatting-traits
  • https://doc.rust-lang.org/std/fmt/#fmtdisplay-vs-fmtdebug
  • https://doc.rust-lang.org/std/fmt/trait.Display.html
  • https://doc.rust-lang.org/std/fmt/trait.Debug.html
  • https://doc.rust-lang.org/std/fmt/derive.Debug.html
• https://doc.rust-lang.org/std/macro.eprintln.html
• https://doc.rust-lang.org/std/macro.dbg.html
• https://en.wikipedia.org/wiki/Standard_streams
  • https://stackoverflow.com/questions/3385201/confused-about-stdin-stdout-and-stderr

Logging / Tracing

# With logging
cargo run --features do_logging

# With tracing
cargo run --features do_tracing

Notes

• https://github.com/rust-lang/log
• https://github.com/tokio-rs/tracing
  • https://docs.rs/tracing-subscriber/latest/tracing_subscriber/

Backtraces

# Enable backtrace demo (will panic!)
# Edit config.rs: RUN_PANIC_CODE = true
RUST_BACKTRACE=1 cargo run

Notes

• https://doc.rust-lang.org/std/panic/enum.BacktraceStyle.html
  • RUST_BACKTRACE=1 cargo run
  • RUST_BACKTRACE=full cargo run

Level 2: Step Debugging

1. Install CodeLLDB extension in VSCode
   1. https://code.visualstudio.com/docs/languages/rust#_debugging
      1. https://marketplace.visualstudio.com/items?itemName=vadimcn.vscode-lldb
   2. https://lldb.llvm.org/
2. Try the following
   • Try the BREAKPOINT: Rust: on panic setting to automatically break during level_1::do_backtrace
   • Set a breakpoint in record::generate_value() with the config::USE_BROKEN_RECORD_VALUE_CALCULATION = true
   • Step through level_2::do_step_debug to inspect macro generated code and validation

Notes

1. Click the "Run and Debug" icon (Play button with a bug) on the left
   1. If asked, select "LLDB"
   2. Create launch.json to not have to keep selecting options
   3. In the "Breakpoints" section (bottom left), check "All Exceptions"
2. Panics:
   1. The debugger catches the signal at the moment the CPU stops
   2. Often deep inside the Rust standard library's panic handler (rust_panic assembly), rather than the line of code that caused the panic
      1. https://users.rust-lang.org/t/vscode-set-up-rust-std-lib-source-code-navigation-and-debugging/69064
      2. rustup component add rust-src
   3. Looking at the "top" of the stack (the crash location). Need to walk down the stack to find your code
   4. Look at the Call Stack pane (usually in the bottom left of VS Code)
   5. Should see a list of function names
      1. The top ones will be greyed out or confusing (e.g., rust_panic, etc.)
   6. Scan the list until you see the first function that belongs to your project
      1. It should look like: find_record (or main::find_record)

Level 3: Profiling

• Try with config::STABLE_SORT set to true (more allocations) and false (in place)

flamegraph vs flamechart

	Flamegraph (default)	Flamechart (--flamechart)
X-axis	Represents Population (frequency)	Represents Time
Behavior	It sorts stack traces alphabetically and merges identical stacks	It preserves the chronological order of samples and does not merge distinct calls.
Visual	If function A calls function B at the beginning of the program, and again at the end, a standard flamegraph merges them into one single, wide bar.	If function A calls function B at the beginning, and again at the end, you will see two distinct bars for B separated by a gap (or other functions) on the graph.
Goal	To answer "Where is the most CPU time being spent in total?"	To answer "When did this specific spike occur, and what was the sequence of events?"

flamegraph-rs

• https://github.com/flamegraph-rs/flamegraph

# cargo install flamegraph
cargo flamegraph --dev
cargo flamegraph --dev --flamechart

samply

• https://github.com/mstange/samply

# Samply (install: cargo install samply)
samply record cargo run

cargo-instruments

• https://github.com/cmyr/cargo-instruments
  • MacOS only,
  • Linux: perf + hotspot for CPU and heaptrack for memory

cargo instruments -t alloc
cargo instruments -t time

Instruments Notes

• Memory profiling -t alloc
  • Probably have to select a section on the timeline to inspect, otherwise the callstack will be too large
• CPU profiling -t time
  • Select flamegraph display
  • Apply the Filters (at the bottom):
    • Separate by Thread (optional, but usually helpful)
    • Invert Call Tree
      • Puts the function doing the work (the "leaf" node) at the top of the list
    • Hide System Libraries
      • Removes OS noise (kernel calls)

Bonus Level

• Interactive debugging with ML model can fail
  • Likely due to trying to load large tensors (model weights) with debugger
  • Use print debugging