LD_PRELOAD to intercept readlink of rustc compiler hangs

It isn't enough in this case, because you'd get a stack like dlsym -> calloc -> jemalloc initialization -> intercepted readlink -> ???. It doesn't yet have a function pointer for the original readlink.

Hence the suggestions to either do the readlink syscall by hand during initialization, or intercept the allocator.

Yay!! Got i working!!

Question: I am seeing my constructor function called twice and all the static variables being reinitialized only with rustc. works fine for ls -al symlink In hindsight, I always noticed this. you can see it the logs above.

Before I went down the path of intercepting malloc and company. I wanted to give "intercept readlink and call syscall while initialization" another try and figure out the hang issue i had before. Sounded like a simpler solution and I like simple :slight_smile:

First I tried "intercept readlink and call syscall always" And it does seem works fine.

But I noticed by constructor function was being called twice only for rustc. wierd. Not sure why. works fine for ls -al symlink

Then I went on to try "intercept readlink and call syscall while initialization" But the wierd behaviour caused more problems, even the variables I was using to track initialization, got uninitialized.

Testing on ls -al works fine. Notice the constructor is called only once and everything looks fine as expected.

bash-4.4$ LD_PRELOAD=/ws/sarvi-sjc/redhook/examples/readlink/libreadlink.so ls -al test.link 
Constructor: begin, initialized=false, pid=31934, thread_id=ThreadId(1)
readlink_get: REAL: 0x0
readlink_get: done dlsym_next: REAL: 0x7efd0615a8d0
Constructor: end, initialized=true, pid=31934, thread_id=ThreadId(1)
readlink_get: REAL: 0x7efd0615a8d0
readlink: get
lrwxrwxrwx. 1 sarvi eng 9 Sep  7 16:50 test.link -> test.file
bash-4.4$ 

Running the same of rustc -vV. The constructor gets called twice. And the pid is the same both times, so I am guessing another processes is not being spun out or something. I suspect something basic is happening that I might be missing. Any idea what is going on here?

bash-4.4$ LD_PRELOAD=/ws/sarvi-sjc/redhook/examples/readlink/libreadlink.so rustc -vV
Constructor: begin, initialized=false, pid=31369, thread_id=ThreadId(1)
readlink_get: REAL: 0x0
readlink_get: done dlsym_next: REAL: 0x7fef61e468d0
Constructor: end, initialized=true, pid=31369, thread_id=ThreadId(1)
readlink_get: REAL: 0x7fef61e468d0
readlink: get
readlink: syscall because not initialized
Constructor: begin, initialized=false, pid=31369, thread_id=ThreadId(1)
readlink_get: REAL: 0x0
readlink_get: done dlsym_next: REAL: 0x7f5a437758d0
Constructor: end, initialized=true, pid=31369, thread_id=ThreadId(1)
rustc 1.47.0-nightly (7e6d6e5f5 2020-08-16)
binary: rustc
commit-hash: 7e6d6e5f535321c2223f044caba16f97b825009c
commit-date: 2020-08-16
host: x86_64-unknown-linux-gnu
release: 1.47.0-nightly
LLVM version: 10.0
bash-4.4$ 

Here is the test code

extern crate core;
extern crate libc;
#[macro_use]
extern crate ctor; 


use libc::{c_void,c_char,c_int,size_t,ssize_t};
use std::io::Write;
use std::sync::atomic;
use std::process;
use std::thread;

#[cfg(any(target_os = "macos", target_os = "ios"))]
pub mod dyld_insert_libraries;

/* Some Rust library functionality (e.g., jemalloc) initializes
 * lazily, after the hooking library has inserted itself into the call
 * path. If the initialization uses any hooked functions, this will lead
 * to an infinite loop. Work around this by running some initialization
 * code in a static constructor, and bypassing all hooks until it has
 * completed. */

static INIT_STATE: atomic::AtomicBool = atomic::AtomicBool::new(false);

pub fn initialized() -> bool {
    INIT_STATE.load(atomic::Ordering::SeqCst)
}

// This fn works fine when used like this:
// print(format_args!("hello {}", 1));
fn print(args: std::fmt::Arguments<'_>) {
    std::io::stderr().write_fmt(args).unwrap()
}

#[ctor]
fn initialize() {
    print(format_args!("Constructor: begin, initialized={}, pid={}, thread_id={:?}\n", initialized(), process::id(),thread::current().id()));
    Box::new(0u8);
    readlink_get();
    INIT_STATE.store(true, atomic::Ordering::SeqCst);
    print(format_args!("Constructor: end, initialized={}, pid={}, thread_id={:?}\n", initialized(), process::id(),thread::current().id()));
}


#[link(name = "dl")]
extern "C" {
    fn dlsym(handle: *const c_void, symbol: *const c_char) -> *const c_void;
}

const RTLD_NEXT: *const c_void = -1isize as *const c_void;

pub unsafe fn dlsym_next(symbol: &'static str) -> *const u8 {
    let ptr = dlsym(RTLD_NEXT, symbol.as_ptr() as *const c_char);
    if ptr.is_null() {
        panic!("redhook: Unable to find underlying function for {}", symbol);
    }
    ptr as *const u8
}

type Readlink = unsafe extern "C" fn (path: *const c_char, buf: *mut c_char, bufsiz: size_t) -> ssize_t;

#[no_mangle]
pub unsafe extern "C" fn readlink(path: *const c_char, buf: *mut c_char, bufsiz: size_t) -> ssize_t {
    let readl:Readlink;
    if !initialized() {
        print(format_args!("readlink: syscall because not initialized\n"));
        libc::syscall(libc::SYS_readlink, path, buf, bufsiz) as ssize_t
    } else {
        readl = readlink_get();
        if (readl as *const u8) == (0 as *const u8) {
            print(format_args!("readlink: syscall\n"));
            libc::syscall(libc::SYS_readlink, path, buf, bufsiz) as ssize_t
        } else {
            print(format_args!("readlink: get\n"));
            readl(path, buf, bufsiz)
        }
    }
}

fn readlink_get() -> unsafe extern fn (path: *const c_char, buf: *mut c_char, bufsiz: size_t) -> ssize_t  {
    static mut REAL: *const u8 = 0 as *const u8;
    unsafe {
        print(format_args!("readlink_get: REAL: {:?}\n", REAL));
        let x=REAL;
        if (REAL as *const u8) == (0 as *const u8) {
            REAL = dlsym_next(concat!("readlink", "\0"));
            print(format_args!("readlink_get: done dlsym_next: REAL: {:?}\n", REAL));
        }
        ::std::mem::transmute(x)
    }    
}

Probably rustc calls exec* without forking, which will keep the same pid but create a new address space, load the new executable and obviously call the global ctors again. This is perfectly normal behavior on *nix systems.

Normal rustc doesn't exec, but the ~/.cargo/bin/rustc shim from rustup does -- it figures out which toolchain is active, sets a few environment variables, then execs to the real rustc.

I was investigating whether LD_AUDIT might work better for your use case, since audit libraries are opened in a separate namespace. You could manually intercept readlink in la_symbind* hooks, but hopefully still resolve to the real readlink yourself.

However, I ran into bad interactions with jemalloc when I was just getting started:

Thanks for investigating this. Will need to read up on LD_AUDIT.

Thanks to your and @comex adivce I have taken the approach of direct syscalls just during the initialization period and it seems to have gotten me past the initialization recursion/loop/hang issue.

I have updated my fork of redhook library with those test cases and updated macro for other folks wanting to build LD_preload libraries.

That has also gotten my wisktrack past both jemalloc intialization as well tracing infra structure initialization.

So far so good. I still need to cover the statically linked program cases, which I havent scratched the surface on.