It began back in 1996-1997 when one of the companies hit on an idea that would take the hobby to a whole new level. There was a huge market for game-used items, and a market for cards. Maybe by adding small pieces of game-used items into cards, they could expand the popularity of cards and offer some cards that were exceedingly valuable. One thing is for certain, the idea took off.
In 1997 Upper Deck was the first company to put any game-used material into a baseball card making GU cards of Ken Griffey Jr., Tony Gwynn, and the legendary Rey Ordonez. Their popularity lead to an explosion in game-used cards, and it has been an interesting development to follow.
In 1999 Upper Deck pushed the hobby forward with one of the classic themed sets: 1999 Upper Deck a Piece of History 500 Home Run Club Bat cards. These cards included the first game-used cards for many Hall of Famers and included all players who were members of the 500 home run club at the time. The cards were all /350 except for Ruth who was /50 (there had been an earlier version of Ruth that was /350 that had a different image, and did not have the words "500 Home Run." Upper Deck and Fleer followed up the success of the 500 Home Run club by releasing Game-Used sets revolving around the 3000 hit club.
One thing that was interesting about this time was that the concept of "patch" cards or "prime" cards had not really come into their own. At that point the companies were non-discriminate about the specific piece they were putting into cards, not realizing that collectors would find the certain pieces of game-used items cards much more appealing. This held true for bat pieces as well and Fleer even included a bat barrel in their 2000 Fleer set as seen below.
In 2001 for the first time (I believe) the companies started to distinguish between patches and other jerseys. One classic example is the 2001 Leaf Certified Fabric of the Game set. The set had many jersey cards, but the "Century" edition specifically had patch cards.
One thing that was still not clearly defined was what exactly constituted a patch, or a "prime" piece. Some examples of cards were ultimately considered prime, some not, and some the companies had not even really considered before a few more years.
In the "not prime" category, the cards that lead off are the ones that have stitches. One might think that stitching is unique enough to warrant being considered a patch, and in fact, stitching cards made it into the 2001 Leaf Certified Century set (indicating that the company felt that way) but over time the market has not paid almost any premium for these.
But there were many more options that had yet to have been decided. Button and air holes were to be found in packs, though it does not seem like the companies ever assumed that they were worthy of a prime designation (either labeled as such on the cards, or like the 2001 Leaf Certified Century set, separated out and included in a specific set which would indicate that there was an expectation that collectors would distinguish between it and other cards.) Today there are some collectors who search out these pieces (button holes more often than air holes) but for most collectors it is interesting, but not particularly desirable.
There were a whole number of items that were different enough that it seems the companies did not even consider them collectors items at all. These include buttons, laundry tags, and some labels. For roughly the first decade of game-used items, none of these (to the best my knowledge) found their way into packs. Considering how many game-used items were cut up in that time period, it is hard not to wonder whether the companies considered that unusable "junk" and simply disposed of them. (If someone worked at one of the companies then and can comment, it would be great to know whether this might be the case.) Over the last few years the companies have figured out that pretty much any component of a jersey, pair of pants etc, is worth something and they have found ways to highlight these interesting pieces.
The companies have also found ways to improve their utilization finding ways to improve on concepts which had been used. AN example is the stitched name in the lining of the pants. While it was featured as far back as 2005 (and maybe even earlier), Panini has recently included the full name, making a really unique game-used item that is as specific to the player as a bat barrel. (This is not to even consider that a Carlos Gomez patch would ever be more appealing than a Babe Ruth patch, but for a second imagine a full Ruth name tag patch...)
To this day, there are still some items that are a bit less clear. This card was actually labeled by Donruss as "prime", but there are probably people on both sides (as the owner of the card I hope people do view it as prime but I would not consider it as nice as a piece of blue patch or even the letter shown above. Thankfully, a home/away jersey 1/1 is not too shabby even if the prime piece is not the best.)
There are also cards such as the Walter Johnson GU Ball card, which people have been split on (although given that it's his only GU cards, many people's attitude is that they will take what they can get...)
Ultimately collecting is in the eye's of the beholder and I'm sure the card companies were thrilled to discover that there is a different collectors for so many different parts of the jerseys. We are still somewhat early in the era of game-used cards (considering that cards have been around for well over 100 years). For collectors though, the introduction of game-used material has added a fascinating dimension to the hobby, and one that continues to evolve.