On the death of William Gordon in 1847, the estate of Fyvie reverted to the descendants of Lord Rockville and Lady Dumfries. Lord Rockville was a brother of General Gordon, born in 1739, and married in 1769 to Anne, the young widow of Lord Dumfries. This couple had had four sons and four daughters.

Charles, the eldest son, went to school in Holland to learn modern languages, and was afterwards sent to Spain to stay with his mother's brother, Sir James Duff, who for many years held the office of British Consul at Cadiz. After that he returned home and entered the Army, eventually marrying and having three sons and a daughter. Two of the sons became Lairds of Fyvie.

The second eldest son, William, took his eldest brother's place in Spain with his uncle, who on his death left him the baronetcy with the obligation of assuming the name of Duff before that of Gordon.

He was, by all accounts, an exceedingly handsome, distinguished looking man, a good musician, and highly cultured. He was Member of Parliament for Worcester for some years, married well, and had two sons and two daughters. He died in 1823.

Alexander, the third son, was very tall, strong and athletic. It is said that he once walked from Edinburgh to Fyvie in two days – seventy miles the first day and sixty the next. He entered the Army where he proved extremely popular and became aide-de-camp to Lord Hardwicke when he was Lord-Lieutenant of Ireland. He was killed at the Battle of Talavera in Spain in 1806.

Cosmo, the fourth and youngest son, was also in the Army. He served in India with his uncle's regiment, and rose to the rank of General. After his return to England, he married and left one son, who died in 1876.

At the death of William Gordon, therefore, it was Charles, the eldest son of Lord Rockville, who inherited Fyvie; but he was then seventy one, so he surrendered the property to his eldest son, merely taking an annuity from the estate for himself. He visited Fyvie subsequently only once, for a few months in 1848, and he died in 1851.

Charles' eldest son Cosmo Gordon, the fifth 'Gordon of Fyvie', was born in 1810. He rose to the rank of Colonel in the Madras Artillery, married, but had no children. His second son, Captain Alexander Henry Gordon (Alick) – the sixth 'Gordon of Fyvie' likewise married but had no children.

Although Alick Gordon made his home at Rockville in Hampshire, he was a frequent visitor to his elder brother at Fyvie, and there comes down to us today some weird but interesting tales which are worth noting here:

Remember, at that date, the bare rambling rooms and passageways of the Castle would have been poorly lit by lamps or flickering candles, creating a shadowy elusive atmosphere. The great stone stairway with its haunting legends, the long stone corridors where echo multiplied each footstep, were, after dark, shrouded in a mysterious gloom full of eerie suggestion. The creaking chimney- vanes on a stormy night made noises that would fill the stoutest heart with fear. For a long time, ghostly groans heard between the Gordon and Meldrum Towers were attributed to supernatural origin – until the discovery that the pumping ram for water caused a peculiar vibration along the pipes, which accounted for the noises.

But there were other tales less easy to explain away – of the Laird being shaken out of bed at night by unseen visitors; of a wind that arose indoors on windless nights and blew the coverings off sleeping guests; of a lady staying at the Castle, in the 'ghost room', being startled by a heavy heart-broken sigh and who ran to look into the corridor to see who it was, only to find no one there. Again it was said that another lady, going upstairs to bed, in a room near the 'murder room', heard stealthy footsteps behind her and suddenly her candle went out. When she turned to remonstrate with the supposed perpetrator of a silly joke, she found herself completely alone on the great staircase.

Another story from this time is that Alexander (Alick) Gordon and a friend, when they were lads, decided to hang towels out of every window in the Castle to see if there were any hidden rooms. They waited until everyone else in the Castle had gone on a picnic and then got the housekeeper to go though all the rooms with them, hanging a towel from each window; but when they went outside to check, they found five windows with no towel. Three of these were later accounted for, but two could never be traced, leading to the conclusion that these rooms had no access from within the Castle.

The belief prevailed at the time (why, it is impossible to say), that the legendary Ladye of Fyvie was visible only to one of Gordon blood, and a tale exists illustrating this:

A wife of one of the Deputy-Sheriffs of Aberdeen came to stay at the Castle, and brought with her a maid called Thompson. This maid innocently reported that she had seen a lady in a lovely green dress going up the principal staircase at night. When this was repeated at the breakfast table, everyone present disclaimed being the owner of a green dress, till somebody tactlessly remarked that it must have been the Green Ladye of Fyvie – but that she never appeared except to a Gordon. The lady from Aberdeen looked startled, and replied that, although she called always called her maid Thompson whoever she might be, her current maid's real name was Gordon!

Other stories abound about the Green Ladye, who seems to be a pretty solid apparition – people have stepped aside to let her pass on the stairway, never thinking that what they were seeing was anything other than a guest.

It was a favourite superstition that the Ladye invariably appeared before the death of the reigning Laird, and Alick Gordon himself seems to have experienced this. In 1879, when his brother Cosmo lay ill, Alick was coming along the passage leading from the Gordon bedroom to the great staircase, when suddenly in the dim December afternoon he saw before him, barring his way, the Green Ladye, faintly iridescent in the waning light. As she looked at him she slowly dropped a curtsey, then drifted away into the darkness of the corridor. And Alick Gordon, white faced and unnerved, staggered on to the room where his wife sat sewing, and told her what he had just seen – that the Green Ladye had hailed him as the head of the house, and that meant that Cosmo would die.

On the 8^th December Cosmo did die, and Alick Gordon came into possession of Fyvie. He was Laird for four or five years only, and another weird story tells of Alick's death:

It appears that the next heir to Fyvie (after Alick), Sir Maurice Duff-Gordon, was well known to be a man of wild life, who was said to be heavily in debt. Alick was convinced that if his cousin Maurice came into the property he would squander it rapidly, and was therefore determined to cut him out of the inheritance. He consulted his solicitor, papers were drawn up, and he took them with him to Aberdeen to sign them in the presence of his banker; but on the threshold of the bank he fell dead with the papers unsigned!

Thus, had he lived some ten minutes longer, the fate of Fyvie would have been very different. As it was, Sir Maurice inherited the estate, and was summoned to the Castle to attend the funeral.

There had always been curiosity about the 'secret chamber', and in 1885 Sir Maurice removed the safe which stood in a recess in the wall in the Charter Room, and began to open up the stairway that his sister had discovered (she had visited the Castle in 1884 and had spent a lot of time in that room going through the Castle charters). He had great difficulty getting any workers to undertake the job (tales of evil, of the 'black vomit' (plague) abounded), and the ones that he succeeded in obtaining had come from a distance away. One of the masons who was present when the opening was made, told that in the thickness of the wall through which they bored with difficulty, they found a human skeleton bricked up. However, the excavation work had only been going on a few days when Sir Maurice had a fall in the drawing room and broke his leg. His wife then insisted that he stop the investigation – and they were.

Subsequently, she suffered with her eyesight, and the old prediction that foretold that the wife of any Laird that tried to break into the secret chamber would go blind, alarmed her. She actually consulted a famous clairvoyant of the day in what she thought would be a safe way to ascertain the contents of the secret chamber, but her ideas about priceless treasure were destined to be dashed – the clairvoyant told her that there was nothing in the chamber but dust and a few scraps of paper!

Now this was interesting for, some years prior to any excavation taking place, a discovery was made in the little passage near the Meldrum Tower dressing room, of a stone in the wall which worked on a pivot, and which was exactly over the secret chamber. This stone, set in motion, disclosed a bottomless hole, and after some pennies were thrown down, a carpenter's tool tied to a piece of string was lowered until it touched what was thought to be the floor of the chamber. Nothing further was recorded as being done at that time, but it is interesting that the clairvoyant did not see the pennies and the carpenter's tool down there.

At a later date, when an attempt was being made to locate a chimney in the morning room, in a recess where tradition indicated there had once stood the alter of a chapel, the remains of a child was discovered. They stirred speculation of a long ago tragedy – was some crime committed against the innocent little victim – an unwanted daughter, an inconvenient heir? We will never know.

Alick Gordon's predictions were correct, and Sir Maurice's reign as Laird of Fyvie was brief. By 1885 the estate was on the market, and in 1889 it was purchased for the sum of £175,000 from the trustees of Sir Maurice Duff-Gordon by Alexander Forbes-Leith.