Japanese Roots

I don't know how I missed it but I came across Jared Diamond's 1998 Discover article, Japanese Roots, today. Jared's findings are pretty much in line with my understanding of the role Korea played in Japan's history which is that several  mass exodus of Koreans, each wave caused by demise of a Korean kingdom (Buyeo, Gaya, Baekje, Goguryeo, Balhae), migrated to Japan and eventually conquered it. I doubt it was much of a conquering though. Most likely it was just a long population expansion, drowning the natives (Jomons) by sheer numbers, followed by a long fight amongst themselves until everyone got tired enough to choose a king. And Japanese language is probably a cocktail of exiles' languages and dialects, not that of just one like Goguryeo as the author suggested.
This doesn't mean that I think Japanese are Koreans though. That kind of reasoning would make me an African. Their ancestors were Koreans but now they are Japanese. What puzzles me is why Japanese average height is noticeably shorter than that of Korean? Curse of the Ainu?