Absolutely. Not only is it annoying to listen to, but it’s also a waste of water and money. Leaky toilets can add up in your water bills. According to a Penn State Article, “10 Easy Ways You Can Conserve Energy” (October 27, 2009, Volume 56, No. 09), “A ‘running’ toilet can waste two gallons of water per minute. A silent leak in a toilet can waste up to 7,000 gallons of water per month”. Here is a good example; say you had a running toilet that was leaking 2 gallons of water a minute for two weeks; 2 gallons x 14 days x 24 hours x 60 minutes = 40,320 gallons of wastewater. At a cost of .002 cents a gallon, (average cost of a gallon of water in the U.S., this varies per state, and you can look up what you actually pay on your water bill) you will have spent $80.64 on this leaky toilet in additional water costs in just two weeks' time.
If your toilet continues to run after you flush it, it indicates that some part of the internal mechanism is out of order or broken. Sometimes a little jiggling of the toilet handle solves the problem. Sometimes a simple part needs to be replaced. The worst-case scenario is that the entire mechanism needs to be replaced. No matter what the cause make sure to diagnose the problem and get it fixed.