Those of us who dare to question authority thought it was quite fishy that the FBI would be given powers to demand information without a warrant and to suppress discussion of the inquiry - it made the headlines relating to library records more than once. Quite a few libraries posted signs saying something like, "The FBI has not been here to search through our records," so that when the FBI did come, they could take down the sign as an indication that the FBI had come and thereby skirt the law forbidding their telling that they had been asked for information.  Much hoopla has come from the provisions. 
Now, shocker of shockers, the FBI has been found to have improperly used the Patriot Act in a number of cases, but the report suggests agent error and shoddy documentation rather than criminal intent. Yes, but if we're not allowed to keep tabs on the Bureau, then who will?