The focus should be on providing basic broadband for all and allowing markets to deliver higher speeds, it said. The committee argued: "We believe that a 50p levy placed on fixed telecommunication lines is an ill-directed charge.
"It will place a disproportionate cost on a majority who will not, or are unable to, reap the benefits of that charge."
The government's broadband plans - outlined in its Digital Economy Bill - have two main strands. It wants to ensure a minimum speed of 2Mbps to all parts of Britain by 2012, and then deliver ultra-fast broadband to most of the country by 2017.
The committee agreed that the government should help deliver 2Mbps to all by 2012 but said that it was "concerned" that the government had not defined what it guaranteed.
Broadband speeds can vary at different times of the day and can also vary with distance from an exchange. The committee believes that the government should commit to delivering a minimum of 2Mbps "under normal circumstances, to all users at all times".
It also said that it was concerned that government intervention in delivering ultra-fast speeds - known as Next Generation Access - would distort the market and that there was little demand for such services.
"We think the market can be trusted to deliver faster speeds," committee chairman Peter Luff told BBC News. "The real priority should be the universal service obligation and the whole effort to increase digital inclusion."
But the Department for Business, Innovation and Skills said that the government wanted everyone to "access the huge social, economic and health benefits" that high speeds offer.
"Our analysis shows that without intervention, the market will only reach up to 70% of the country, so it's vital we act now to ensure no area is left behind," said a spokesman.
"The 50p duty we have proposed is modest, fair and affordable and is the best way to drive further investment in our networks."
Bis believes the tax will allow a £1 billion upgrade of the UK's digital networks and will be of particular benefit to rural areas.
"We're currently consulting on the most effective way to deploy this investment with public and commercial benefits in mind, and will consider the Committee's report in our final response," said the spokesman.
The government published the Digital Economy Bill in November 2009. It is currently being scrutinised in the House of Lords.
One of the most hotly-debated elements is the so-called "three strikes rule" that would give regulator Ofcom new powers to disconnect or slow down the connections of persistent net pirates.
