My grandparents used the same term for Brazil nuts -- I think it was a generational thing because they had African American friends.

I grew up in the northern burbs of Chicago.