Though you obviously can't expect a brand new puppy to hold much of a stay, you can absolutely begin to lay the foundations of solid self-restraint and attention that make up the heart of a strong stay. As with the other "Foundations" articles I've been writing, this one focuses on what you can do with a puppy in the early stages to help give him rock-solid behaviors later in life.
The core of the stay is self-control, and I think that's where a lot of stays break down. The dog understands the behavior, but the handler hasn't built up a rewarded habit in which the dog knows how to look at distractions without having to get up and engage with them.
The other key component of a successful, real world stay is something called proofing. A dog can understand a behavior really well, and you can build a deep habit through repetition and rewards, but when something brand new happens, it can confuse the dog's sense of what's going on and thus disrupt the behavior. For example, you can have an amazing stay in the house, in the backyard, at dog class, and at the coffeeshop, and the dog will hold beautiful stays in those locations and in other locations that are similar. However, if you bring him to a soccer game and a kid kicks the ball near the dog, me may break his stay because you never proofed him against that situation. He's never practiced holding his stay while watching a ball fly by.
So with puppies, the name of the self-control game is to practice both the behavior and to do some low-difficulty proofing.
Even a young pup can learn to hold a stay if you are gentle, patient, and reasonable in your expectations. Once your puppy can sit fairly reliably, get him into a sit next to you. Put your flat palm in front of his face like a policeman, and say "stay." If he holds his sit (and he should, since you taught him to wait for a release word when sitting), mark it and reward him. Then release him. Always use your release word at the end of a stay so the dog learns to wait for that word. You decide when stays end, not the dog.
Reset your puppy into a sit next to you, give the hand signal and say "stay," and then wait a couple of seconds before marking the success and rewarding the pup. If you wait too long and the puppy breaks the stay, simply withhold the reward and reset him. Resist the urge to tell your puppy "no." He's a puppy, and he has no idea what you're asking for. A "no" is neither fair nor clear at this stage. Instead, go back and build forward from the last point where you had success.
Always, always, always set your puppy up to succeed and reward him for it. Don't set him up to fail and then punish him. Aside from being kinder and having fewer potential side-effects, rewarding success is simply faster than punishing failure because it teaches the dog what you want.
When your puppy can reliably sit next to you, receive the stay hand signal and command, and actually hold it for a few seconds (literally 3-5 seconds), start to work in multiple rewards. Don't forget that the marker, whether it's a "yes!" or a click, is not a release word. You want your pup to be able to receive a mark and reward for holding a stay and then keep holding the stay until you release him. That's how you're going to extend the duration of the stay and also proof it.
Holding a stay at all is probably enough of a foundation for most puppies, though if your puppy likes the game, you can certainly progress from here by having him stay for longer intervals between rewards or by having him stay in front of you instead of the side, using the same hand signal and word. Remember, if you're a puppy, staying next to a person and staying in front of a person aren't the same thing, so your pup may have to relearn from scratch. Dogs aren't good generalizers in the first place, and puppies even more so.
As always, if your pup is breaking the stay or you find yourself repeating yourself, go back to something easier. All you're looking for is for your pup to learn that when you say "stay" and give the hand signal, he needs to wait expectantly for rewards and then the release word.
When dogs get older, we practice the "stay" in all kinds of contexts, like waiting at the door to be let out, waiting politely for you to make his dinner, and even holding a stay when company comes through door before he's allowed to greet. We also have adult dogs do a "daily stay" when we're out hiking so they get the hang of staying in all kinds of contexts. It makes for great photos, since you can pose the dogs someplace nice for a pretty landscape shot, and then you can release them for a fun action photo.