A synomym for "busy". Taken from a phone reference; when you leave a phone off the hook, you get a "busy" signal. The phrase is used to convey that there were a lot of people attending a particular event.
Off the hook is actually a modernization of a series of slang words. Closely related to off the chain, there refering to something being so "fresh" and "new" that its literally right off the store shelf. (started in reference to clothes, the hangar being the hook)