The castle was built in the first half of the 19th century. The building of the new castle for Chancellor Jerzy Ossoliński was probably the Italian architect Wawrzyniec Senes.
In the 14th century, there was a brick Gothic castle in this place, built by an unknown by name representative of the Toporczyk knightly family. Notes on the existence of the castle, which in the past could have been more called a knightly stronghold, appeared in the book of emoluments of the property of the Archbishopric of Kraków, Liber Beneficiorum, whose author was Jan Długosz himself.
In 1621, Jerzy Ossoliński of the Topór coat of arms received from his father Jan Zbigniew as an inheritance, an estate. Jerzy in the first half of the 17th century, the heir transformed the building into a castle. In 1650 Jerzy Ossoliński died and the property was taken over by his daughter Anna Teresa and her husband Zygmunt Denhoff of his own coat of arms, the starost of Bydgoszcz. In 1655, the castle was conquered and partly looted by the Swedes, later it passed from hand to hand to successive owners and fell into disrepair.
In the eighteenth century, the castle fell into the hands of Antoni Bartłomiej Ledóchowski, a courtier of Stanisław August Poniatowski. Antoni, after the unsuccessful "Kościuszkowski" Uprising decided to sell his goods in the Mazovian and Volhynian lands. He decided to settle in Ossolin at the castle, where in 1816 he decided to blow him up. To this day, the motivation behind this decision is not fully known. Unofficial versions say that he did it to get to the alleged treasure hidden in the walls of the castles, or as an act of concern for his sons, fearing that in the future they might incur many drunken or card debts against the family fortune. After those events, the bridge, the entrance gate and the tower have been preserved. In 1831, the castle was handed over to Ignacy Ledóchowski, general and commander of the defense of Modlin during the "Listopadowe" Uprising . Using building material from the ruins of the remaining castle, he rebuilt the Ledóchowski residences in Górki Klimontowskie.
Until 1915 there was a bridge, a tower and walls of the entrance gate. In the 1920s, Michał Karski became the owner, who used what was left of the castle and created a distillery. It existed together with the tower until 1944, when retreating German soldiers, afraid of being used as a high vantage point by the approaching Soviets, blew up the tower. Only the arcade of the bridge that used to lead to farm buildings and some foundations have survived to our times.