Ornstein was born in Kremenchuk, a large town in the Ukrainian province of Poltava, then under Imperial Russian rule. He grew up in a musical environment his father was a Jewish cantor, while a violinist uncle encouraged the young boy's studies. Ornstein was recognized early on as a prodigy on the piano; in 1902, when the celebrated Polish pianist Josef Hofmann visited Kremenchug, he heard the eight-year old Ornstein perform. Hofmann gave him a letter of recommendation to the highly regarded St. Petersburg Conservatory. Soon after, Ornstein was accepted as a pupil at the Imperial School of Music in Kiev, then headed by Vladimir Puchalsky. A death in the family forced Ornstein's return home. In 1903, Osip Gabrilovich heard him play and recommended him to the Moscow Conservatory. In 1904, the ten-year-old Ornstein auditioned for and was accepted by the St. Petersburg school. There he studied composition with Alexander Glazunov and piano with Anna Yesipova. By the age of eleven, Ornstein was earning his way by coaching opera singers. To escape the pogroms incited by the nationalist and antisemitic organisation Union of the Russian People, the family emigrated to the United States in February 1906. They settled in New York's Lower East Side, and Ornstein enrolled in the Institute of Musical Art predecessor to the Juilliard School where he studied piano with Bertha Feiring Tapper. In 1911, he made a well-received New York debut with pieces by Bach, Beethoven, Chopin, and Schumann. Recordings two years later of works by Chopin, Grieg, and Poldini demonstrate, according to music historian Michael Broyles, "a pianist of sensitivity, prodigious technical ability, and artistic maturity."
Ornstein soon moved in a very different direction. He began imagining and then writing works with new sounds, dissonant and startling. Ornstein himself was unsettled by the earliest of these compositions: "I really doubted my sanity at first. I simply said, what is that? It was so completely removed from any experience I ever had." On March 27, 1914, in London, he gave his first public performance of works then called "futurist", now known as modernist. In addition to a Busoni arrangement of three Bach choral preludes and several pieces by Schoenberg, Ornstein played a number of his own compositions. The concert caused a major stir. One newspaper described Ornstein's work as "the sum of Schoenberg and Scriabine [sic
] squared." Others were less analytical: "We have never suffered from such insufferable hideousness, expressed in terms of so-called music."
Ornstein's follow-up performance provoked a near-riot: "At my second concert, devoted to my own compositions, I might have played anything. I couldn't hear the piano myself. The crowd whistled and howled and even threw handy missiles on the stage.The reaction, however, was by no means universally negative the Musical Standard called him "one of the most remarkable composers of the day...[with] that germ of realism and humanity which is indicative of genius." By the next year, he was the talk of the American music scene for his performances of cutting-edge works by Schoenberg, Scriabin, Bartok, Debussy, Kodaly, Ravel, and Stravinsky (many of them U.S. premieres), as well as his own, even more radical compositions.