Sunday, August 26, 2007

How To Create 32-bit Import Libraries Without .OBJs or Source

This article is intentionally titled the same as Microsoft KB article 131313 (was Q131313) because that KB article does not give enough information for you to create a .LIB file that works with Windows system DLLs.

There are some functions in the Win32 API that have no import library. One example is the function named RemoveControlByName() in OCCACHE.DLL. According to the documentation, the only way to use the function is to use LoadLibrary() and GetProcAddress(). This is error-prone, requires a lot of code to maintain, and isn't usable with the Delay-Load feature of VC++.

Obviously it would be preferable to have a .LIB to link against, but creating such a library is challenging. If you use Dependency Walker to look at OCCACHE.DLL, you'll see that the function is named simply RemoveControlByName. While that seems obvious, it shouldn't be possible to have this name because it doesn't include any notation for the calling convention.

If the function were __cdecl, then the function name should have started with an underline, such as _RemoveControlByName. If the function were __stdcall, then the underline should have been added as well as suffix to indicate the number of bytes in its call stack. RemoveControlByName has five 4-byte paramters, so a __stdcall signature should have looked like _RemoveControlByName@20. However, the function name has no decorations at all, which should be impossible according to Microsoft's discussion of name decoration.

The Q131313 article discusses the general case of manually creating a .LIB file for a .DLL file. The discussion under Creating a .DEF File says "The reason for this limitation is based on an assumption made by the LIB utility that all names are automatically exported without a leading underscore." This seems promising because we don't have a leading underscore. However, functions exported from Windows DLLs(almost) always using __stdcall, and the discussion under Creating a .DEF File is only applicable to __cdecl.

In spite of those warnings, I spent quite a bit of time trying to craft a .DEF file that described what I was trying to do. (You can use LIB.EXE to compile a .DEF file to a .LIB without any .OBJ files) Although most .DEF files simply list the undecorated function names under the EXPORTS section, the syntax of the .DEF file allows for creating aliases and other strange constructs. Some of the things I tried included:

; Raw function name
RemoveControlByName

; Alias the undecorated name to __stdcall
RemoveControlByName=RemoveControlByName@20

; Explicit reference to the necessary DLL
RemoveControlByName@20 = OCCACHE.RemoveControlByName

However, none of these generated a .LIB that worked. The first two I couldn't link against and the third would link but fail to run.

I thought I'd learn something by looking at the symbol that the parent program is trying to link against, but that was even worse:

__imp_?RemoveControlByName@20

I knew my header file was correct, so that was definitely the correct name. However, you'll notice the leading "__imp", which indicates that the symbol is being imported from a DLL. This was something else that apparently needed to be included in my hand-crafted .LIB file, and I hadn't seen any discussion anywhere on how to do that.

I've tried to solve this problem two other times in the last several years, and this was the point I gave up in both of those cases. However, this time, failure was not an option. I needed the solution.

I tried the second option in the KB article, described under Stubbing Out Functions. I painstakingly created a dozen functions that mimicked the signatures of the functions listed in the header file. If you are doing this for yourself, here's a tip for creating a Visual Studio project. The final file you need is a .LIB file, so it's tempting to use one of the Visual Studio projects for creating a .LIB. However, that's wrong. If you created a .LIB, then you'll link with the stub functions you created, which is useless. What you really want is a DLL project, which happens to create a .LIB as a by-product.

Anyway, I created the functions as described in the KB article. Since my project used .cpp files, all of the calls in my header file had to be declared extern "C". For example:

extern "C" {
#include "occache.h"
};

If you look in any of the standard Windows include files, such as winbase.h, you'll see this same declaration. Note that this declaration has no relation to _cdecl and therefore has no impact on the calling convention. In other words, it doesn't force all of the functions in occache.h to be called with _cdecl. What this declaration does is to modify all of the linker symbols so that they won't include the C++ name decorations, which encode all of the function's parameters into the function name.

I also updated all of the function signatures in the header file to include __declspec(dllexport).

I compiled it, linked my main application to the new .LIB file, and it linked! I thought I was done, but when I ran the application, I received the error "Entry Point Not Found: The procedure entry point _RemoveControlByName@20 could not be located in the dynamic link library OCCACHE.DLL"

I examined the .LIB with DUMPBIN. Under Public Symbols, I now see two definitions for RemoveControlByName:

_RemoveControlByName@20
__imp__RemoveControlByName@20

It appears that the "__imp" definition was a result of adding __declspec(dllexport), so that explained why the application linked successfully. One problem solved.

Continuing my examination of the DUMPBIN information, I saw that RemoveControlByName was defined as:

Archive member name at FE0: OCCACHE.DLL/
46D12949 time/date Sun Aug 26 00:18:33 2007
uid
gid
0 mode
38 size
correct header end

Version : 0
Machine : 14C (x86)
TimeDateStamp: 46D12949 Sun Aug 26 00:18:33 2007
SizeOfData : 00000024
DLL name : OCCACHE.DLL
Symbol name : _RemoveControlByName@20
Type : code
Name type : name
Hint : 9
Name : _RemoveControlByName@20

To determine whether or not this was correct, I used DUMPBIN to compare against known-good definitions in USER32.LIB, where I found that the "Name type" in the USER32 records was defined as "undecorate" instead of "name". Obviously, there was a magic incantation to set this flag, presumably in the .DEF file.

I added a .DEF file and spent several hours trying various alias combinations, none of which worked. Finally, in desperation, I created a .DEF file that contained just the raw function names. For example:

EXPORTS
RemoveControlByName

I built my application, it linked, and it ran. What happened?

I ran DUMPBIN again on my library. The record describing RemoveControlByName now contained the "undecorate" attribute:

Archive member name at FFC: OCCACHE.DLL/
46D12740 time/date Sun Aug 26 00:09:52 2007
uid
gid
0 mode
38 size
correct header end

Version : 0
Machine : 14C (x86)
TimeDateStamp: 46D12740 Sun Aug 26 00:09:52 2007
SizeOfData : 00000024
DLL name : OCCACHE.DLL
Symbol name : _RemoveControlByName@20
Type : code
Name type : undecorate
Hint : 6
Name : RemoveControlByName

Also, the very last line in the record showed that the "Name" was RemoveControlByName, with no decoration. Exactly what I needed.

It's clear that there's quite a bit of undocumented behavior here. Adding the entry to the .DEF file had a rather dramatic effect on the generated library. I couldn't find any mention of this behavior in the documentation on .DEF files. The only relevant reference I could find was in Microsoft's documentation under __cdecl, where it says "Underscore character (_) is prefixed to names, except when exporting __cdecl functions that use C linkage." This statement is true, but it's not the whole truth. To create a .LIB file that can link against such a construct, you also must have the function declared in a .DEF file.

In summary, to create a .LIB file that will let you link with Windows system DLLs, you need to:
  1. Follow the instructions in Q131313 under Stubbing Out Functions. Make sure you name the project the same as the Windows DLL.
  2. Make sure your dummy functions are defined with __declspec(dllexport) as well as __stdcall.
  3. For the header file used by the parent application, make sure that your function declarations are surrounded with extern "C".
  4. Add a .DEF file to your project that includes the function names with no decoration.

7 comments:

  1. Thank you VERY much! :)

    I had the same problem with a non-Windows DLL... your post was a great source of information to solve that problem!

    It seems that some of the declarations you used are redundant. I managed to include my header file without the extern "C" thing. I was also able to leave out __declspec(dllexport)... but as long as it works, everything should be fine. ;)

    Hauke P.

    ReplyDelete
  2. You don't want to leave out the extern "C" declaration. It must be present or C++ code won't work. It doesn't matter if you are only writing C code, but I'm not sure why you'd leave out one line of code at the risk of breaking future C++ development.

    The __declspec(dllexport) declaration is related to the "imp" declarations I mentioned in my blog post. Again, if you leave it out, some code configurations will break.

    ReplyDelete
  3. Awesome,I was about to give up at the point of trying different combinations of keywords,internal names,symbols in the .DEF file.
    I happened to find your blog and the article by just searching for "Name type:undecorate" in Google.
    and I tried as you said,it worked!
    Thanks a lot!

    ReplyDelete
  4. I am using lib.exe of MS VC++ 2010 . I tried to do "EXPORTS myfunc" in the DEF file, but I get "name type: no prefix" instead of "name type: undecorate". The symbol name is "_myfunc" instead of "_myfunc@0" as expected. The name "myfunc" is correct. Do you have an idea how to tell lib.exe to use the undecorated attribute as well as the correct symbol name?

    ReplyDelete
  5. Sorry for the years-later comment, Jim, but this is related to what I am working on lately. Anonymous's comment of February 16, 2008, makes sense in the context of the undocumented behavior you (and I) have observed. When given the undecorated symbol in the DEF file, and linking a DLL that defines the symbol and does not use 'extern "C",' I am seeing that the decorated (not the undecorated) symbol is included in the generated import library. Dumpbin does show that the symbol's name type is "undecorate," and the name is, indeed, the undecorated name, while the symbol itself is, indeed. the decorated name.

    What this means, practically, is that you can write a DLL that defines a function called, say, "showMsgBox," export that function's name by including it, undecorated, in a DEF file, compile a client that will link to the DLL, and it will find the decorated version of showMsgBox in the import library. MS clearly says you have to put the decorated version of the name in the DEF file. If you do, that works too, with dumpbin showing the name type as "name," but the symbol is the same as when you include the undecorated name in the DEF file. In both cases, it exports the decorated name as the symbol. If you never knew that the C++ compiler was decorating the symbol in both the DLL and client code, you'd think this was all working because the symbols match in both places (DLL and client) and you are exporting that exact symbol in your DEF file, just what you'd expect to have to do. The fact that they all sort of secretly use the decorated versions makes it all work in a way that hides the existence of the decorated versions from the programmer, and you don't need to use 'extern "C"' at any point.

    Here's the relevant portion from dumpbin when I exported
    "?showMsgBox@@YAHPEA_W0@Z":

    DLL name : SimpleDLL.dll
    Symbol name : ?showMsgBox@@YAHPEA_W0@Z (int __cdecl showMsgBox(wchar_t *,wchar_t *))
    Type : code
    Name type : name
    Hint : 0
    Name : ?showMsgBox@@YAHPEA_W0@Z

    And here's the relevant portion from dumpbin when I exported
    "showMsgBox":

    DLL name : SimpleDLL.dll
    Symbol name : ?showMsgBox@@YAHPEA_W0@Z (int __cdecl showMsgBox(wchar_t *,wchar_t *))
    Type : code
    Name type : undecorate
    Hint : 0
    Name : showMsgBox

    I've spent the last two days trying to find something on this. Your blog post is the only acknowledgment I can find that anyone else has ever observed this.

    For a bit more on it, see my post at StackOverflow: http://stackoverflow.com/questions/36295272/how-do-decorated-names-get-into-import-libraries-when-i-only-provide-the-undecor

    ReplyDelete
  6. For last 4 to 5 days I was trying create lib. I tried many ways to solve it. The error which you got, i was getting similar errors. I searched online, but couldn't get the solution. I was totally annoyed.
    And finally I got the solution from your blog.
    It's really good information in it. Salute for your patience and work.
    Thanks a ton :)

    ReplyDelete
  7. cool, now i can use ntdll functions without having to install the driver kit, thanks

    ReplyDelete