DENVER (AP) — Pot may be legal in some states — but the neighbors don’t have to like it.

Marijuana and hemp have joined wacky paint colors and unsightly fences as common neighborhood disputes facing homeowners’ associations. Though a few HOAs have willingly changed their rules to accommodate for legal marijuana use or home-growing, many more are banning home pot smoking.

Homeowners’ associations can’t ban members from using marijuana in their homes when it’s legal. But if neighbors can see or smell weed, the law is clear — HOAs have every right to regulate the drug as a nuisance, or a threat to children along the lines of a swimming pool with no fence.

“The fact that people may be legally entitled to smoke doesn’t mean they can do it wherever they want, any more than they could walk into a restaurant and light up a cigarette,” said Richard Thompson, who owns a management consulting company that specializes in condominium and homeowner associations.

Thompson said his home condo development in Portland, Oregon, is a prime example of how marijuana’s growing acceptance has sparked neighbor conflicts.

“As soon as spring and summer come around, we hear complaints about marijuana smoke because people are out on their patios and they have the windows down,” he said.

It’s not clear how many homeowners’ associations have confronted marijuana conflicts in the 23 states with some form of legal marijuana. But lawyers who specialize in HOA disputes, as well as a Colorado regulatory agency that advises HOAs, say there are growing conflicts among neighbors who want to smoke pot and others who don’t want to see it or smell it.

“What we’re really seeing more now is regulating the associations’ common areas,” such as smoke wafting onto playgrounds or others’ porches, said Erin McManis, an attorney in Phoenix whose firm represents hundreds of Arizona HOAs.

The Carrillo Ranch homeowners association in Chandler, Arizona, earlier this year took the rare step of withdrawing a proposed ban on residents smoking medical marijuana in their front and backyards and on their patios.

The HOA planned a meeting on the topic in March, but withdrew the proposal after many residents opposed the ban as too harsh.

“This is a personal-freedom issue where people were going to dictate how other people should live,” Carrillo Ranch resident Tom LaBonte told The Arizona Republic in February, when the HOA dropped its proposal.

HOA lawyers say the Carrillo Ranch case illustrates the value of HOAs when the law changes, as with marijuana.

“Coming together and working on issues is something associations have been doing for a long time,” McManis said. “We’re hopeful that’s how it’s going to go forward now with medical marijuana.”

Smoke isn’t the only neighbor complaint posed by loosening marijuana laws. Growing pot and hemp is prompting neighbor disputes, too.

A suburban Denver retiree learned the hard way this spring that he needed neighbors’ permission before growing hemp. Jim Denny, of Brighton, Colorado, learned about marijuana’s non-intoxicating cousin and decided to try the crop on a 75-by-100-foot plot in his yard.

But Denny’s hemp plot ran afoul of his homeowners’ association, which ruled the hemp experiment unacceptable.

“As soon as they heard about it, they said, ‘We’re not going to let anyone grow marijuana here,'” Denny said. “I explained to them that hemp is not marijuana, but they were dead-set against it.”

So with his hemp plants about 2 feet tall, Denny invited hemp activists to come transplant them to somewhere without opposition from a homeowner association. Denny sold the plants for about $3 each, a good price for a plant whose seeds can cost up to $10 each because it can’t be imported.

Hemp activists volunteered to pay Denny’s fines for flouting the HOA, which could have run to $600 a day. But Denny decided that living peacefully with his neighbors trumped making a political point.

“I had people calling up and saying, ‘It’s just a shame; we’ll pay your fines all the way through to the end.’ But I decided in the end not to fight it,” said Denny, a technical writer and former software engineer. “At the end of the day, I live here.”

Copyright 2014 The Associated Press. All rights reserved. This material may not be published, broadcast, rewritten or redistributed.