Easily the most recognizable and dramatic visual associated with Japanese organized crime is the irezumi (Japanese traditional tattoo) many yakuza adorn their bodies with. These colorful, elaborate tattoos should be impressive to anyone who has ever been poked with a sharp stick, because that's what happens thousands of times to make the tattoos materialize. In Peter B.E. Hill's The Japanese Mafia, it is stated that in the 1970s and 1980s, 70% of yakuza had extensive tattooing, and that statistic held true regardless of rank within the group.
Irezumi is the traditional Japanese style, which is done by a trained tattoo artist (who previously apprenticed with a master), by hand. That means no buzzing needle machine-gun to speed things up, just manual tools. It's quite painful compared to regular tattooing. While my life-goal of avoiding intentional contact with sharp things that leave permanent marks prevents me from getting a tattoo, I'd be curious to feel for myself just what it is so many men and women, yakuza or not, have taken part in. Large tattoos can take years of weekly visits and cost tens of thousands of dollars.
But why are they such a big deal for yakuza?
Much of it comes from the way Japanese culture views tattooing overall. For a long time, tattooing was a form of punishment in Japan. Convicted criminals were tattooed with rings on their arms to permanently mark them as such. Tattooing has even been outlawed at times in Japanese history, associating them directly with criminality.
Tattoos, for yakuza, serve a number of purposes. Such extensive tattooing shows a long-term commitment to the gang. If you're willing to cover your body in ink, you're in the yakuza life for the long-haul. It's a pain-tolerance thing, too. Only a Real Man can tolerate the pain of traditional tattooing. Here's a picture of a woman with extensive tattooing to balance out that silly statement (Shoko Tendo, author of autobiography Yakuza Moon).
The reasons for yakuza tattoos really aren't that different from the tattoos associated with American gangs. Branding yourself as an outcast, tolerating the pain and spending the money to show commitment are part of the package in both cases.
Irezumi tattooing, though, does have the benefit of being a lot more artistic. While many American tattoos seem to inspire a bit of disgust, irezumi tattoos are awe-inspiring. Instead of gothic lettering or a photo-realistic face or something like that, Japanese culture, history, and myth provide the source material for irezumi tattoos. Imagery of samurai, geisha, dragons and more populate the backs of yakuza (and tattoo enthusiasts brave enough to take on the pain and ostracism should their secret come to light). The traditional imagery has its own meanings for the wearers. Power, good luck, wisdom, endurance and other lofty ideas can factor in. It matches up well with the traditional yakuza political stance of emperor-worship and reverence for their own (self-professed) history as noble outlaws descended from the men who protected townspeople from abusive samurai.
Despite the artistry and tradition of both the tattoo artists and the tattoos themselves, they still have that criminal association. As such, they aren't something the "common folk" like seeing around their establishments. Japanese public baths have signs up specifically prohibiting customers with tattoos from entry. With the way Japanese police are cracking down on yakuza lately, yakuza might just heed those signs now.
As a side note, I'd recommend that if you're an American in Japan with heavy tattooing, consider covering it up. Wear long sleeves, turtlenecks, etc. where possible. It'll make interactions with businesses and general people a little less tense.
There are also some interesting side effects to these massive tattoos. In Jake Adelstein's Tokyo Vice, he writes about some of them. A woman who'd slept next to yakuza characterized the skin as being reptilian; pleasant to sleep next to in the summer, but chilling in the winter. Additionally, according to Adelstein, it seems that having such a huge portion of skin covered in ink, which affects sweating, also affects the liver. Because the body can't get rid of toxins as efficiently, the liver has to work harder. Liver failure can apparently be linked to these tattoos. Peter Hill corroborates Adelstein's claim; however, the liver damage is attributed to the pigments used by traditional irezumi artists, rather than difficulty sweating. I suspect both play their own roles.
For more information about irezumi, here are a few good resources worth checking out:
An interview with Horiyoshi III, one of the most well-known irezumi artists. Interestingly, he differentiates between tattoos and irezumi, which literally means tattoo. Important distinction, or old-fart grumping about kids?
Horiyoshi's personal homepage, with an extensive gallery of his work.
The wikipedia page on irezumi, for some of the basics.
A basic guide on the meanings of some of the commonly-found symbols used in irezumi. I can't vouch for these being 100% accurate but it's pretty good overall for a topic that's pretty hard to research in English.
If I missed anything notable, let me know. I'm always looking to expand articles like this one.