My understanding was that they were buying silks and spices from abroad with silver and gold at higher and higher prices, so that whatever silver and gold they were able to obtain, ultimately flowed out towards the East, causing inflation. I don't think it was a core issue, but it was something that contemporary Romans considered very alarming. They felt that Rome was losing it's Roman-ness, and pouring money into other countries. How important that was in the grand scheme of things is debatable. If you have good resources to share, I'd like to read them!