Born in Switzerland in 1936, Gregory grew up in a family where cultural stars like Arthur Rubinstein and Salvador Dalí were regular guests. He began collecting art as a teenager and went on to establish the well-known art publisher Vendome Press, while writing books on travel and history.