A subject line test is only useful if you can explain what won and why you will reuse it. Testing a witty line against a completely different offer, preview text, and From name at the same time teaches you nothing. Isolate one element and wait for a real sample.
Test one thing
Hold the body, From name, send time, and audience constant. Change only the subject, or only the preview text. Common splits: curiosity vs clarity, question vs statement, specific number vs vague benefit, and branded vs personal From names on a separate test.
If product wants to change the offer in the same send, that is a different experiment. Do not call it a subject test.
Sample size and timing
Do not call a winner after 40 opens. For most lists you need enough unique clicks or replies that a few random events cannot flip the result. Send both variants into the same window so a news cycle does not favor one side. Leave a holdout if you also care about unsubscribes, not just opens.
Stop peeking every ten minutes. Early noise looks like a story. It is not.
Opens are a noisy metric
Apple Mail Privacy Protection and image-blocking inflate or hide opens. Pair the test with clicks and replies when the mail asks for a click. If the goal is a reply, measure replies. Optimizing only for opens trains you to write clickbait that tanks trust.
What to do with a winner
Roll the winner to the remainder of the send, log the hypothesis, and retire the loser. Do not keep running the same curiosity-gap trick until the list learns to ignore you. Rotate patterns the way you rotate offers.
Write the result down in one sentence: “Clarity beat mystery on invoice reminders, n=…” Next quarter’s intern should be able to reuse it without rerunning folklore.
From name is a different test
People argue about subject lines when the From name is the thing they actually scan. Test From name on its own week. Do not bundle it with a new subject pattern. A personal From on a blast can lift opens and also lift complaints if the body is clearly a campaign.
