Fed's Hammack: The main worry is that public will lose confidence inflation will return to 2%.