Frey says dismal economy has hit minorities the hardest in terms of poverty rates