Actually, it doesn't fit cleanly into any specific category. But it has elements of a religion, an ethnicity, a culture and so on... basically, it is a nationality. The religion is a principal component of the identity, sure, but it's certainly not a required one. It differs from other religions and this way. And it differs from other ethnicities in that it accepts converts. (Conversion is not merely into the religion, but is an acceptance of the individual into the clan.)
Think of it like a clan or a tribe. What is the modern equivalent? A nation.
Israel is the religious and historic name for the ancient nation of the Jewish people. That is, the Jewish people in the national sense... NOT the religious sense. All Jews are accepted, even Christian or non-practicing ones. You are judging Israel on the grounds that it is unlike other nations but the truth is that it isn't Israel which is different, but Judaism. Your mistake stems from an incomplete or incorrect interpretation of what Judaism actually is.
The fact of the matter is that terms like ethnicity or nationality are fluid, flexible terms that take on different forms depending on the subject at hand. I'm not so sure that your belief that Italian and Japanese are not ethnicities is correct. Drawing the line between ethnicities is as problematic as drawing the line between races. There is no easy definition for these terms, they are wholly subjective creations whose definitions morph over time.
They're both nationalities and if Israel identified itself as the Israeli state rather than the Jewish state, then you would be right.They are also identities that transcend mere nationality in the legal sense of the word. I have many Irish friends who resolutely consider themselves "Irish" despite being proud Americans. If "Irish" were merely a nationality in the way you are using the word then the term "Irish American" would be contradictory. But it isn't. Rather, Irish Americans enjoy a nationality that is tied to their identity; a borderless nationality based on a shared heritage. One which may be personified in the state of Ireland but is by no means exclusively defined by it.
One thing I can say for certain though. Judaism can be, (and often is) defined as a nation of people. Personally, I've struggled with trying to find an accurate definition for my identity over the years and have settled on "nationality" as the most comprehensive, (yet imperfect) definition available.
-----
edit:
Nationality can also mean membership in a cultural/historical group related to political or national identity, even if it currently lacks a formal state. This meaning is said by some authorities to cover many groups, including Kurds, Basques, Catalans, English, Welsh, Scots, Palestinians, Tamils, Quebecers and many others.
In several non-English speaking areas of the world, the cognate word for nationality in local language may be understood as a synonym of ethnicity, as nation can be defined as a grouping based on cultural self-determination rather than on relations with a state. For example, many people would say they are Kurds, i.e., of Kurdish nationality, even though no such Kurdistan state exists (the postulated homeland is divided among five countries). In the context of former Soviet Union and former Yugoslavia, nationality is often used as translation of the Russian and Serbo-Croatian terms (национальность/ natsionalnost, народност/narodnost) used for ethnic groups and local affiliations within those (former) states. Similarly, the term "nationalities of China" refers to cultural groups in China. Spain is one Nation, made out by nationalities, which are not nations, or can be considered smaller nations within the Spanish Nation.http://en.wikipedia.org/wiki/NationalityHow is Judaism any different?