We've been doing more in the garden this year, as we make it more fully our own. So far we haven't taken anything out (with the exception of an ivy hell-bent on ripping off our siding), but have pushed back in some areas.
The house came with a well-established perennial garden so last year we mostly tried to maintain things and watch how it unfolded over the year. Most of our new planting was done in containers - flower boxes, vegetables, herbs. The one place where we went ahead and ripped stuff out was the area directly next to the house in back. It doesn't get a lot of light and was a tangle of grasses, groundcovers that had migrated from the area below the deck, a small shrub, and one small sorrowful iris patch that bloomed much later than the irises around the neighborhood. Oh, and a lot of weeds. We cleared out a space in the center so we could have a pumpkin patch, which we then did not carefully maintain and ended up with vines growing over everything else, an equally unattractive tangle. We hope to do better with it this year.
What I've done a lot of this year is pruning! When we moved in the shrubs hadn't been thoroughly pruned in some time. I cut things back but didn't get into the undergrowth. Over the winter I did some reading on pruning and saw that you can cut back up to 1/3 of the plant. So as soon as we had some warm days I went in to cut back and to get rid of the dead wood underneath. Since at current count we have 12 shrubs and two very vigorous vines needing the same kind of attention that's a lot of pruning. Plus the two sage bushes that would like to take over that part of the garden.
Everything looks a lot better now - the space is opened up and I can see what's going on. Part of what I see are what look like weeds, or at least from their behavior they're acting that way... but I can't remember what they did last year. I'm pretty sure a couple of these turn out nicely later in late summer - and I'm just as certain that one ends up being ugly and a nuisance to get out. But which is which? I'll just have to keep a careful eye on them all.