Jonathan Haskel, an external member of the Bank's monetary policy committee, said a wave of investment "stopped in its tracks" in 2016 following the vote.
He said the UK had "suffered much more" of a productivity slowdown than other large economies because of Brexit.
The Treasury said it did not recognise Mr Haskel's figures.
The Bank of England declined to comment.
Mr Haskel, who was interviewed by website newsletter The Overshoot, was asked why he thought the UK was an "extreme outlier" when it came to facing a slowdown in productivity.
He said: "Yes, we suffered much more. A bit of that is that we have this larger financial sector. But I think it really goes back to Brexit.
"If you look in the period up to 2016, it's true that we had a bigger slowdown in productivity up to 2016, but we had a lot of investment. We had a big boom between 2012-ish to 2016.
"But then investment just plateaued from 2016, and we dropped to the bottom of G7 countries."
Since the Brexit referendum in 2016, there has been a global pandemic, swiftly followed by an energy crisis, which has made estimating any financial impact directly related to Brexit difficult.
But investment has stalled since the referendum.
Some economists, including the International Monetary Fund, have suggested that uncertainty surrounding Brexit, including the unsettled issue of the Northern Ireland Protocol, has deterred at least some spending.
Mr Haskel said that the Brexit referendum had an impact on economic growth as a result of the reduction in trade, with the UK opting to leave the EU and its single market and secure trade deals elsewhere.
He referred to a calculation to show what the UK economy could have looked like if investment had carried on growing at the rate it had been before the referendum, compared to what it is currently growing at.
Mr Haskel described the hit to the economy as the "productivity penalty", which amounted to about 1.3% of gross domestic product (GDP). GDP is an important tool for looking at how well, or badly, an economy is doing.
"That 1.3% of GDP is about £29bn, or roughly £1,000 per household," he added.
Sir Richard Branson is among the business bosses who have suggested the cost of Brexit red tape would put them off investing in the UK.
The pro-Brexit group Briefings for Business claims there is no evidence of a Brexit-related hit to investment.
A Treasury spokeswoman added that since leaving the EU single market, the UK had grown faster than France and Germany.
"The government is making the most of our Brexit freedoms to grow the economy, including ambitious financial services sector reforms which will unlock over £100bn of investment, and we are reviewing EU-derived rules in other critical growth sectors this year," she said.
Figures released last week showed that the UK narrowly avoided falling into recession in 2022, but the Bank expects it to enter one this year.
A recession is typically defined as when the economy shrinks for two consecutive three-month periods. It means the economy is generally performing badly and companies may make less money and cut jobs, leaving the government with less tax revenue.
The Bank of England is the UK's central bank and has a major influence on managing the country's economy. It is responsible for setting interest rates, which are at a 14-year high in an attempt to reduce inflation - the rate at which prices rise.
Mr Haskel, who has consistently voted in favour of hiking interest rates, also told The Overshoot that inactivity in the labour market was a distinctly "British thing".
"Inactivity behaviour here looks very different to other countries: we've had a rise while other countries have had a fall," he said.
Nearly nine million people in the UK are currently "economically inactive", which is where people are not actively looking for work.
The government has been considering plans to coax retired middle-aged workers back into jobs in an attempt to boost the economy.