Amongst the difficulties of the new premier and the Whig party were the position and attitude of Brougham. He was not the leader of any party, and had no personal following in the House of Commons. Moreover, he himself had repeatedly declared that nothing would induce him to exchange his position as an independent member of parliament for any office, however great. On the day following the resignation of the Tory government, he reluctantly consented to postpone for one week his motion on parliamentary reform. The attorney-generalship was offered to him and indignantly refused. He himself affirms that he desired to be master of the rolls, which would have left him free to sit in the House of Commons. But this was positively interdicted by the king, and objected to by Lord Althorp, who declared that he could not undertake to lead the House with so insubordinate a follower behind him. But as it was impossible to leave Brougham out of the ministry, it was determined to offer him the chancellorship. Brougham himself hesitated, or affected to hesitate, but finally yielded to the representations of Lord Grey and Lord Althorp. On the 22nd of November the great seal was delivered to him by the king, and he was raised to the peerage as Baron Brougham and Vaux. His chancellorship lasted exactly four years.

Lord Brougham took a most active and prominent part in all the great measures promoted by Grey's government, and the passing of the Reform Bill was due in a great measure to the vigour with which he defended it. But success developed traits which had hitherto been kept in the background. His manner became dictatorial and he exhibited a restless eccentricity, and a passion for interfering with every department of state, which alarmed the king. By his insatiable activity he had contrived to monopolize the authority and popularity of the government, and notwithstanding the immense majority by which it was supported in the reformed parliament, a crisis was not long in arriving. Lord Grey resigned, but very much by Brougham's exertions the cabinet was reconstructed under Lord Melbourne, and he appeared to think that his own influence in it would be increased. But the irritability of his temper and the egotism of his character made it impossible for his colleagues to work with him, and the extreme mental excitement under which he laboured at this time culminated during a journey to Scotland in a behaviour so extravagant, that it gave the final stroke to the confidence of the king. At Lancaster he joined the bar-mess, and spent the night in an orgy.

In a country house he lost the great seal, and found it again in a game of blindman's-buff. At Edinburgh, in spite of the coldness which had sprung up between himself and the Grey family, he was present at a banquet given to the late premier, and delivered a harangue on his own services and his public virtue. All this time he continued to correspond with the king in a strain which created the utmost irritation and amazement at Windsor.

Shortly after the meeting of parliament in November the king dismissed his ministers. The chancellor, who had dined at Holland House, called on Lord Melbourne on his way home, and learned the intelligence. Melbourne made him promise that he would keep it a secret until the morrow, but the moment he quitted the ex-premier he sent a paragraph to The Times relating the occurrence, and adding that "the queen had done it all." That statement, which was totally unfounded, was the last act of his official life. The Peel ministry, prematurely and rashly summoned to power, was of no long duration, and Brougham naturally took an active part in overthrowing it. Lord Melbourne was called upon in April 1835 to reconstruct the Whig government with his former colleagues. But, formidable as he might be as an opponent, the Whigs had learned by experience that Brougham was even more dangerous to them as an ally, and with one accord they resolved that he should not hold the great seal or any other office. The great seal was put in commission, to divert for a time his resentment, and leave him, if he chose, to entertain hopes of recovering it.

These hopes, however, were soon dissipated; and although the late chancellor assumed an independent position in the House of Lords, and even affected to protect the government, his resentment against his "noble friends" soon broke out with uncontrolled vehemence. Throughout the session of 1835 his activity was undiminished. Bills for every imaginable purpose were thrown by him on the table of the House, and it stands recorded in Hansard that he made no less than 221 reported speeches in parliament in that year. But in the course of the vacation a heavier blow was struck: Lord Cottenham was made lord chancellor. Brougham's daring and arrogant spirit sank for a time under the shock, and during the year 1836 he never spoke in parliament. Among the numerous expedients resorted to in order to keep his name before the public, was a false report of his death by a carriage accident, sent up from Westmorland in 1839. He was accused, with great probability, of being himself the author of the report. Such credence did it obtain that all the newspapers of October 22, excepting The Times, had obituary notices.

However, for more than thirty years after his fall he continued to take an active part in the judicial business of the House of Lords, and in its debates; but it would have been better for his reputation if he had died earlier. His reappearance in parliament on the accession of Queen Victoria was marked by sneers at the court, and violent attacks on the Whigs for their loyal and enthusiastic attachment to their young sovereign; and upon the outbreak of the insurrection in Canada, and the miscarriage of Lord Durham's mission, he overwhelmed his former colleagues, and especially Lord Glenelg, with a torrent of invective and sarcasm, equal in point of oratory to the greatest of his earlier speeches. Indeed, without avowedly relinquishing his political principles, Brougham estranged himself from the whole party by which those principles were defended; and his conduct in general during the years following his loss of office revealed his character in a very unfavourable light. He continued, however, to render judicial services in the privy council, and the House of Lords. The privy council, especially when hearing appeals from the colonies, India, and the courts maritime and ecclesiastical was his favourite tribunal; its vast range of jurisdiction, varied by questions of foreign and international law, suited his discursive genius.