There are many vertical and horizontal cuts needed to fit around all the individual stairs, making the process somewhat complex. Plan the stair molding project completely for all the stairs before making any cuts. Attractive, high-quality, solid-wood molding is expensive, so you don’t want to make improper cuts and end up with scrap.
Consider using finger-jointed material instead of using solid-wood molding and trim stock, if the trim is going to be painted. Using this material instead of solid wood will reduce the material costs considerably. Other than the cost savings, I recommend using painted finger-jointed materials instead of natural solid wood, because it is easy to repair the painted surface.
Even when using a good-quality miter saw, the fitting can be difficult because you will not be cutting standard 45-degree miter angles. Since you have already tried and failed, you should have several pieces of scrap molding you can use for the fitting measurements. Just several short pieces will be adequate.
The simplest method to determine the proper angles is to just use a projector or carpenter’s angle finder to measure the angle of the stairwell. Divide this angle in half, and cut the mating ends on two test pieces at this angle on the miter saw.
Place the two pieces on the landing and the stairs to see how they fit. Make any needed minor adjustments to the cut angles, and try again until they fit perfectly. Keep in mind that you can always use a little wood filler in the gap, if you are planning to paint the finished trim.
If you do not have a protractor or angle finder, place a short scrap piece on the landing, and draw a line along the top of it on the wall. Place the scrap piece on the sloping stringer, and draw a line along it until it intersects the first line.
Now that you have the proper intersection point, place a piece of the scrap-trim piece on the sloped stringer, and slide until the tip meets the intersection point of the lines. Mark the bottom of the trim piece where it meets the landing. This will be an approximate angle for the miter cut. Cut some additional test pieces to get a perfect fit.
Once you have determined the proper miter angles, instead of trying to determine the exact length of the trim piece(s) needed, purposely cut them slightly on the long side. Fit them in place as well as possible and to determine how much has to be trimmed off. If it does require two pieces of trim down the stairs, make the cut on the square ends, not on the mitered ends.