Log everything and make portion control a priority. If I ate only bananas, but consumed 25 a day, I wouldn't lose weight either. While bananas are healthy, 25 of them would be about 2,750 calories.
Plan your success and you'll succeed with your plan.