Let's talk about the standard salesforce.com "lookup" popup window for a few minutes. You know what I'm talking about.. this button right here
It's a handy little button that pops up whenever you need to search for related records. It does a pretty good job but it has some serious drawbacks:
It's virtually impossible to modify your search criteria. What if you want your users to do some crazy search based upon custom logic or search a field that is not typically available? Sorry... you are out of luck. Not possible.
It's terrible for creating new records. Let's say that a user searches for a specific related record and it (absolutely) doesn't exist. To create the new record they need to close the lookup window, navigate to the tab to create the new related record, create the new record, then go back to the original record they were either editing or creating, pop up the lookup window again and find their newly created record. Wow! That's a lot of work.3. "Quick Create" is evil! You can enable the "Quick Create" option for an entire org but don't do it! It displays, by default, on the tab home pages for leads, accounts, contacts, forecasts, and opportunities! The major problems are that you can only create new records for these 5 objects (what about the other ones!?), you can't customize the fields on the page and validation rules don't fire (can you say, "bad data").
Here's the Solution!
I have some good news and some bad news. For standard page layouts I can't help you. Go vote for this idea and this idea.However, for Visualforce page I have a solution to all of these problems with code!
Here's the code you need to accomplish this and it's also available on this gist. You need two Visualforce pages (the record you are editing and the popup window) and two Apex controllers (a simple one for the record you are editing and the controller for the search and new record popup).
MyCustomLookupController
Here's the Apex controller for the record you are either creating or editing. This is an extremely simple controller that just creates a new contact so you can use the lookup for the related account field.
MyCustomLookup
This is the "magical" Visualforce page that uses jQuery to intercept the popup and instead of showing the standard salesforce.com pop, shows our custom popup instead. The user experience is seamless.
CustomAccountLookupController
The Apex controller for the custom popup window is yours to customize. I know what you are thinking, "Free at last! Free at last! Thank God Almighty, we are free at last!" This class has all of your custom search functionality plus the method to create a new account. This is demo code so the search is very limited and does not prevent soql injections.
CustomAccountLookup
Any finally the Visualforce page for the popup itself. It contains a tabbed interface easily allowing a user to search for records and create new ones. Make sure you look at the code for the second tab for creating a new record. I have better things to do than change the fields on the input form every time a new field is created or something is made required. The solution is to usefield sets! So when an administrator makes a change, they can simply update the field set and the popup reflects the change accordingly. Life is good.