c++ - x64 function detouring without inline assmebly -

good evening, fellow coders , hackers.

i'm experimenting binary patching, more precise: detouring functions not called via vftable.

what doing in detail

i'm injecting dll running process , determine start address of function (original function) scanning signature.

once i've found it, rewrite first 13 bytes own shell code redirect every call function function of dll (hook function):

mov rax, <dll_function_address> jmp rax ret 

the hook function calls address of original function again, after removing shellcode in order prevent endless recursion. once original function returns, hook function meant return value has returned preserve regular code flow.

the issue

you might have noticed shellcode no means preserve registers (not rax, might not manipulated hook function, manipulated shellcode.

therefore, original function fails when called hook function. wanted add inline assembly push registers stack first action in hook function popping them before passing control original function, visual studio not have x64 support inline assembly.

i supplement shellcode opcodes push registers stack. can't add opcodes pop them because restore original code before calling original function

the solution i'm trying circumvent

i know clean approach problem relocate whole function. way, wouldn't forced remove hook before calling original function. add opcodes push registers jmp shellcode , opcodes pop them infront of relocated function. i'm trying circumvent because don't know how dynamically determine end of original function, cant figure out how many bytes move.

questions

1. is there kind of calling convention or else i'm not yet aware of, allow me declare hook function in way force compiler not use registers, stack?
2. does know how can tell whether 0xc3 (ret) byte function end?
3. could configure visual studio 2013 use alternative compilers? x64 inline assembly support?

one solution be, instead of restoring original function hook call same instructions overwritten hook, example:

original function:

<do_stuff>   push rbp   mov rsp, rbp   sub 8, rsp   add rbx, rcx   .... 

hooked function:

<do_stuff>   mov rax, <dll_function_address>   jmp rax   nop   add rbx, rcx   ... 

your dll function:

  ... // hooked stuff   push rbp // repeat code replaced hooked function   mov rsp, rbp   sub 8, rsp   jmp <do_stuff + sizeof(shellcode) + nopsled> // call , add offset of hook shellcode 

if shell code doesn't align instruction can nop sled shown in example.

as rax problem, think can this:

<do_stuff>   push <dll_function_address>   ret   nop   add rbx, rcx   ... 

it pushes hook address on stack , call ret, ret take first address on stack , jump there.

or maybe can use call instruction:

<do_stuff>   call <dll_function_address>   nop   add rbx, rcx   ... 

once hook called, replace hooked function do. remaining issue ret dll function continue on nop instruction, need remove 5 (the size of call instruction) adress pushed in stack, that, found this: https://stackoverflow.com/a/12631122/2838914.