30 years ago, Santa (Ed Asner) was delivering presents to an orphanage when one of the babies crawls into his sack. Without realizing it, he takes the baby back to the North Pole and upon the discovery turns the baby over to a workaholic elf who always wanted children and is now willing to slow down to raise the orphan. And thus Papa Elf (Bob Newhart) raises the human child Buddy (Will Ferrell) in the elf world.
This is fine except for the fact that Buddy grows up to be about 6'3" - this is gigantic in the elf world. Everyone loves and accepts Buddy even though he doesn't quite fit in. A slip of the tongue reveals that he is adopted, this coupled with his need to fit in results in a trek from the North Pole to New York to meet his birth father, Walter Hobbs (James Caan).
There is a small problem - Walter didn't know that Buddy's mother was pregnant let alone had a child so it takes some convincing to get Walter to believe that Buddy is his. Buddy also discovers why Walter is on Santa's Bad List. Walter has no time for his family, including Buddy's half-brother Michael and Walter is rather unpleasant.
With his father not wanting much to do with him, Buddy finds a job as an Elf at Gimble's where he meets Jovie. Buddy begins to fall in love with Jovie, and with her help he sets out to save Christmas spirit and help his father find happiness.