The first appearance of an 'orchestra' in the Western World was in approximately the 6th Century B.C. in Ancient Greece, but no musical instruments were involved. The word 'orchestra' denoted a large semicircular space for dancing in Attic, or Greek, theatre. The chorus in Greek theatre was vital to both plot and dramatic action, providing a commentary and dialogue with the main characters in the drama. They stood in the orchestra area, and both sang and danced.
In Roman times, the orchestra area became the location for stage effects, and was even flooded for theatrical sea battles. The notion of a 'pit' area in front of a stage seems to be the defining characteristic of the orchestra at this time, and of course orchestra pits still exist in opera houses and theatres today.
There were many ensembles of musical instruments around the world, some of which pre-date the Greek theatre. In China, for example, the outrageous Emperor Hu (the Tiger) is recorded to have had a 500-piece, all-girl orchestra in 200 A.D. By this date the Chinese traditions which had begun in 1500 B.C. had become so fully established that such extravagance was considered acceptable.
The Indonesian gamelan orchestra seems to have originated around the 2nd Century B.C. This all-percussion orchestra was also linked closely with theatrical performance. There are many other examples of early instrumental musical groupings that might today qualify for the name "orchestra", but of course this was not the name they chose for themselves. The history of the orchestra as we know it today is largely bound up with the history of Western Greco-Roman civilisation.
The Middle Ages were not a prosperous time for groups of instruments, since most music was sung and the instruments would be used mainly for accompaniment. The theatrical ideas of Greece and Rome also disappeared during these centuries, and it is only with the emergence of the Renaissance in around the 15th Century that instrumental music began to come into its own. Even then, it was a matter of two hundred years or so before what might be called an 'orchestra' appeared.
The courts of the Renaissance frequently housed small instrumental ensembles. Perhaps the most celebrated example in music is the one used for Monteverdi's Orfeo of 1607, which called for approximately 40 instruments. These include: harpsichord and organ, trumpets and drums for ceremonial passages, recorders, viola da gamba, lute, and guitar. The most significant inclusion, however, are doubled bowed string parts. It should be pointed out that there is still a great deal of discussion about precisely which instruments played which parts, because the surviving score is unclear in many respects, but the presence of a string section is the first distinguishing feature of an orchestra (although Monteverdi would not have used the word).
By 1637, many such instrumental groupings centred on a string section had sprung up around the courts of Europe. Marin Mersenne, a monk and mathematician, wrote a thesis in that year entitled Traité d'harmonie universelle in which he theorizes about acoustics and describes the properties of various instruments. This includes an extensive discussion of strings, during which he lists the instruments of the court ensemble of Louis XIII, known as 'Les 24 Violons du Roy'.
6 Dessus (Violins)
4 Haute-Contre or Haute-Contre Taille (Alto Violas)
4 Taille (perhaps Tenor Violas)
4 Quinte or Cinquiesme (more Violas)
6 Basse (Violoncellos)
You can see from this list that half the ensemble are violas of various sizes, and Mersenne devotes a great deal of his thesis to discussing the viola. This kind of ensemble seems to have predominated throughout the 17th Century. Instrument makers in Paris developed a number of wind instruments during the 1670s and 80s, and the new oboes, bassoons, flutes, and recorders were soon added to the string groups. Brass instruments and drums would also be included when the occasion demanded (usually for military events). Keyboard instruments were used as continuo. The term 'orchestra' started to be used during this period, to replace the more conventional 'chapel' (or Kapelle). These groupings really form the basis of the Baroque orchestra.