Madison was born Holly Cullen in Astoria, Oregon. Her father traveled around due to work in the timber industry, and from age 2-11, she grew up on Prince of Wales Island, Alaska, near Ketchikan. Her family then moved back home to St. Helens, Oregon. In 1999, Madison moved to Los Angeles to attend Loyola Marymount University, where to make money she competed to become a Hawaiian Tropic model and worked at Hooters. This led to invitations to the Playboy Mansion, at a time when Hefner had seven other girlfriends. Madison moved into the mansion and became one of Hefner's official girlfriends in August 2001, after more than a year of visiting the mansion. In February 2002, after all but two of his girlfriends left, Madison became Hef's "#1 Girl" and moved into his bedroom at the mansion. Hefner's two other girlfriends, Bridget Marquardt and Kendra Wilkinson, have their own bedrooms down the hallway.

Holly has spoken openly about plastic surgery. She has revealed in magazine interviews that she had to talk "Hef" into letting her get a rhinoplasty (nose job). She ultimately succeeded by removing the bump from her nose. She has also had breast augmentation, going from an A cup breast to a D cup.

In 2003, Madison shot a pictorial for Cyber Girl of the Week, a feature of the Playboy website. She also appeared on the cover and inside of the November 2005 issue of Playboy magazine, along with Marquardt and Wilkinson as part of a special pictorial promoting The Girls Next Door. The girls were featured once again on the September 2006 issue of Playboy.

In season three of the show during the episode "My Bare Lady," Holly stated that she wants to have a career producing Playboy spreads. She also helped design and put together the girls' 2008 The Girls Next Door calendar.

A self-described animal lover, in 2006 Madison started writing articles about pets for a friend's quarterly magazine, The Coolest Little Pet Magazine. She describes one of her jobs as "mistress of the Mansion pooches". In April 2007, Madison was featured nude in a PETA ad, alongside the tagline "I'd Rather Go Naked Than Wear Fur".

Madison has stated that she wants children with Hefner and often gives him hints. In February 2007, rumors circulated that she and Hefner were engaged, despite the fact he is still legally married to Kimberley Conrad. However, in March 2007, Hefner stated he has no plans to marry, even though Holly is the "love of his life".