Not necessarily, there are a whole world of neutral colors out there, heck just picking a white could take hours upon hours there are thousands of shades of white. What your Realtor means by neutralizing your home is to make it appealing to the largest number of buyers. But keep in mind that paint colors in particular are affected by the lighting in the room, both in daylight and evening hours and the furnishings and accessories that are in the room. House Beautiful's March 2016 edition is entiled "where will COLOR take you?" and it had a great article "Shift Into Neutral" which showcased 12 colors, some of which will challenge the color-phobes amongst us, but are all considered neutral.
Here are the colors:
Revere Pewter HC-172 Is a combination of grey and beige often referred to as greige. It is a warm grey that changes with the light.
Pale Powder 204 This aqua is considered by Jonathan Rosen as the new white for homes surrounded by greenery or with ocean views.
Stony Ground 211 I call colors like this a non-color, it is essential beige or taupe but it can look gray, taupe or grey depending on the light and what is in the room.
Wallflower SW 6281 I have been looking for the perfect lilac for years, I had a bathroom that turned out purple and and office that was only slightly less so. I didn't want a girly frou-frou lilac but I love the soft feminity of the color, turns out I should have been pairing it with navy, charcoal or black and choosing this lilac.
Borrowed Light 235 David Mitchell says this "atmospheric blue...is a great backdrop for art--or whatever you want to put in front of it. Unlike white, which is always in your face, this sort of recedes. And that's what a good neutral does. You want to know it's there, but you don't necessarily want to know why."
Doeskin SW 6044 This is a taupe that has a lot of mauve in it, honestly it's more pink than beige as many taupes are but if you need to warm up a space this could be your color.
Hale Navy HC-154 This blue is really, really dark, but remember dark colors recede making small rooms feel larger than they are. Also, like black, navy is a true neutral, is there anything you don't wear with your dark rinse jeans?
Hush AF-95 This beige has been described as "nearly foolproof" like many beiges this one will change with the light, but you will need to make sure your furniture stands out to avoid the look of "builder's beige." If you choose to use beige make sure it looks like a design decision not like you're opting out of making a color choice.
Huntington White DC-02 Choosing a great white is tricky, too cool and the room will feel icy and cold to warm and you get yellow overtones and a murky feel, this color is supposed to be smack dab in the middle which should be perfect. White was the color forever, then we moved into beige and grays and it has taken a back seat but the right white can be fashion forward. It works well in either modern or traditional settings and is the go to color in Scandinavian or Modern Country styles.
DKC-17 This is a brown red shade that makes for a surprising neutral, but have you ever seen anything that wasn't made better by a traditional Turkish rug?
Barista AF-175 I admit it, I have a huge affinity for chocolate brown, I mean chocolate right? I once painted my kitchen this dark brown color, with white cabinets and appliances and beige floors and counters. I also hung this huge abstract painting on the wall in shades of orange. This kitchen was striking, it drew comments and was a blast to cook in. It also helped me sell my house in 1 week for $25,000 over asking price so I'm not going to argue.
Gray 2121-10 This color is described as "off-black", another really dark color and I love it. This is the color I'm going to paint my kitchen. As you can tell from my description of how I used Barista, I love dark dramatic colors in the kitchen with white cabinets and light counters, this time I have wood floors and an open plan, so the grey will serve as an accent wall, with lighter gray wrapping into my living space.
As you can tell from this list, there are a world of neutrals out there so don't feel like you are stuck with beige or white when you are selling your home. The best advice I can give you is to pick a neutral that fits the mood of your home and be consistent from room to room. Your rooms don't have to be the same color but they need to feel like they are in the same house.