At a young men's meeting held at the court-house on Friday evening, November 28th, 1834, to make arrangements for a course of public lectures in the following winter, Thomas Coleman, Martin I. Townsend, Thaddeus B. Bigelow, Ralph Hawley and Giles B. Kellogg were appointed "to take into consideration the expediency and practicability of forming a young men's association" in the city. On Friday evening, December 12th, they made a report favoring the organization of one, and submitted a draft of a constitution for it. The latter was adopted for the basis of The Troy Young Men's Association, and a committee of five persons from each of the first four wards was appointed to obtain signers to it. On the following Friday evening, the names of four hundred and twenty-six persons were reported as signed to the constitution. John T. McCoun was then elected president of the association, and on the following Wednesday evening the other officers were chosen.

Two rooms were rented on the second floor of the building No. 197 River Street, in one of which the books of the Troy Library, by an agreement with that organization, were placed, and in the other, newspapers and periodicals. The library and reading-room were opened to the use of the members in February, 1835. On April 20th, that year, the act incorporating the Troy Young Men's Association, was passed by the Legislature. On May 1st, 1846, the association occupied the front rooms on the second floor of the Athenaeum. On the extension of the building to the alley in 1851, the library and reading room were moved to the two rooms now used by the association. The first art exhibition for the benefit of the association was opened in the lecture hall on the third floor on February 18th, 1858. Two other art-exhibitions were made in the following winters for the same purpose. One-half of the receipts of the Loan Exhibition of the Troy Society of Decorative Art, held in 1878 in the basement of the Troy Savings Bank building, was given the association. By an act of the Legislature, passed May 8th, 1880, the control, disposal and management of the real and personal property of the association were vested in a board of twenty-three trustees. They were privileged to organize and add to the departments of the association a free library and a free reading-room whenever the condition of its affairs and trust funds warranted the action. By contributions of the citizens, the Athenaeum was purchased on January 21st, 1882, for $24,500, and given to the association. Subsequent contributions increased the subscriptions to $35,892; the surplus being used to pay for the renovation of the building.

On Friday evening, December 12th, 1884, the semi-centennial anniversary of the organization of the Troy Young Men's Association was celebrated in Music Hall. Benjamin H. Hall, Esq., read an historical sketch of the institution, and speeches were made by some of the officers and members. Lewis E. Gurley, president of the Free Reading Association (founded as the Holly Tree Inn, May 4th, 1874; incorporated March 13th, 1877), formally transferred its property to the trustees of the Troy Young Men's Association, who then took the entire control of the latter's affairs and property. In 1885, the rooms of the association were renovated and refitted, and the books in the library classified and placed in new cases conveniently arranged in the back-rooms on the second and third floors, and in a side room, near the newspaper room, on the first story. On August 7th, that year, the use of the books in the library and the newspapers in the reading room were made free to the people of Troy, under certain rules and restrictions. In January, 1889, the books in the room on the first floor were placed in the general library on the second floor.

In 1862, William R Yourt willed $5,000 to the association, and the same year, George M. Selden gave it railroad stock valued at $2,000. In 1870, Clarence Willard bequeathed it $10,000; in 1879, Roxanna A. Loomis, $1,000; in 1881, F. O. Mather, about $12,000; and on December 12th, 1884, Mrs. Betsey A. Hart, placed to its use $10,000. In 1887, by the will of E. Thompson Gale, it received $2,000, and in 1889, by that of Joseph W. Fuller, $1,000.

The association's library contains about 28,000 bound volumes, not including 835 bound files of newspapers. Many of the books are rare and valuable and few unimportant. DeWitt Clinton has held the office of librarian of the association since December 22d, 1874, and William H. Henderson, of assistant, since May 1st, 1885.