The next year they began to build a few houses for the accommodation of the company at Southborough and at Rusthall

In this space the assembly room (called Mount-Ephraim-House) was brought home from Rusthall to Mount Ephraim, on which a bowling green was inclosed, a tavern (now [in 1766] a lodging house but still retains its original name of the castle) was opened and many lodging houses were erected for the use of the company; but the triumph of the hill was short, Mount Sion became a formidable rival, and quickly eclipsed its growing splendor; for when the ball-room, the bowling green, and the lodging houses arose so near the spring, a less convenient distance was generally avoided

Thus in the course of a few years we find Tunbridge forsaken; Southborough and Rusthall raised and ruined; Mount Ephraim drooping; and Mount Sion in the full bloom of prosperity; this last indeed not only rivalled, but despoiled her predecessors, and triumphantly transferred their ornaments to herself; for many houses were brought from Southborough, Rusthall, and Mount Ephraim, to be rebuilt on Mount Sion; and some, whole and entire as they were, were wheeled on sledges to be fixed in this new seat of favour.

The cold-bath at Rusthall was built by Mr. James Long at a very considerable expense. This bath is esteemed equal to any in the kingdom, being most plentifully supplied with the finest rock water from the neighbouring hills. The bath was at first adorned with amusing waterworks, and had a handsome and convenient house over it, in every room of which was something curious, calculated to divert and surprise the company. The ground and gardens belonging to the bath were elegantly laid out, and embellished with fountains, and other ornaments suitable to the place; in short, the whole was most completely disposed for a scene of amusement. But all this is now [1766] far gone in decay through the want of management.

… it [Rusthall Manor] was alienated to Mr. O'Connor, whose son John sold it to George Kelly, Esq., who resided at Tunbridge Wells, and served the office of high sheriff of Kent in 1762, in which year he was knighted. He died possessed of this manor in 1772, leaving his three sisters his co-heiresses, viz. Anne Shorey, widow, Hannah Tanner, widow, and Martha, wife of James Spagg, Esq. Mrs. Tanner died in 1780, since whose death and that of her two sisters, this manor has successively devolved on Miss Elizabeth Shorey and Thomas Christopher Gardner, Esq., the present lord.