P-Value and Statistical Significance: A Plain-Language Guide with Real Data

Home › Research Methodology › P-Value and Statistical Significance

Quick answerA p-value tells you how surprising your result would be if there were really no effect or difference. A small p-value (usually below 0.05) means the result would be very unlikely by chance alone, so it is called statistically significant. A large p-value means chance could easily explain it.

Below, we explain p-values and significance step by step with two real tests on ScholarsColab’s data: one that is significant and one that is not.

P-value essentials at a glance

Term Plain meaning
Null hypothesis (H0) The “nothing is going on” assumption: no difference, no effect
P-value How likely a result at least this extreme would be if H0 were true
Significance level (alpha) The cut-off you choose in advance, usually 0.05 (5%)
Statistically significant p is below alpha, so you reject H0
Not significant p is above alpha: not enough evidence to reject H0 (this does not prove H0)

Example: two real tests on ScholarsColab’s data

To explain p-values, we run two tests on ScholarsColab’s own data: 41,792 papers, mostly published between 1998 and 2026, by 967 ScholarsColab mentors (data updated 15 October 2026).

Step 1
Ask a question and state the null hypothesis. Do review articles get more citations than original research papers? H0: there is no difference in citations between the two.

Step 2
Choose the cut-off before testing. We use the usual significance level of 0.05.

Step 3
Look at the data.

Median citations by type of paperOriginal research620,930 papersReview articles13980 papers
Median citations per paper, by type of paper. Papers published 2010–2022 by ScholarsColab mentors; type identified from the title. Source: ScholarsColab Research Methods Index.

Step 4
Run the test and read the p-value. A Mann-Whitney U test comparing 980 review articles with 20,930 original research papers (published 2010–2022) gives a p-value of less than 0.001. That is below 0.05, so the difference is statistically significant: it is very unlikely to be due to chance alone.

Put simply: if you pick one review and one original paper at random, the review has more citations about 59.3% of the time.

Step 5
Now a test that is not significant. In one research field, reviews and original papers look much more alike. Now the same test only for Management, economics & finance: 33 reviews and 998 original papers, with median citations of 2 and 4. The p-value is 0.989, so here the difference is not statistically significant: the data do not give enough evidence that reviews and original papers differ in this field.

p below 0.05reject H0: significant
vs
p above 0.05not enough evidence

Step 6
Notice why the two results differ. The first test had a clear difference and thousands of papers. The second had a small difference and few reviews. P-values depend on both the size of the difference and the amount of data.

What a p-value is not

  • Not the probability that your hypothesis is true. It assumes H0 is true and asks how surprising the data would be.
  • Not a measure of how big or important an effect is. With huge samples, tiny and unimportant differences can be significant. Always report the size of the difference too.
  • Not proof of “no effect” when it is large. A non-significant result may simply mean there was too little data.
  • Not a magic line. p = 0.049 and p = 0.051 are practically the same evidence.

How to report a p-value

  1. Name the test and why it suits your data (for example, Mann-Whitney U for skewed values).
  2. Give the sample sizes of each group.
  3. Report the effect size or the difference itself (medians, means, odds ratio).
  4. Report the exact p-value (for example p = 0.032), or p < 0.001 for very small values.
  5. Interpret it in words, without overstating it.

Common mistakes reviewers point out

  • Reporting only “significant” or “p < 0.05” without the actual value or effect size.
  • Running many tests and reporting only the significant ones (sometimes called p-hacking).
  • Treating “not significant” as “no difference”.
  • Using a test whose assumptions the data do not meet.

Frequently asked questions

What does a p-value of 0.05 mean?

It means that, if there were truly no effect, a result at least as extreme as yours would occur about 5% of the time. Results below this level are conventionally called statistically significant.

What does statistically significant mean?

It means the observed result would be unlikely if there were no real effect, so the null hypothesis is rejected at the chosen significance level. It does not mean the effect is large or important.

Is a smaller p-value better?

A smaller p-value means stronger evidence against the null hypothesis, but it does not mean a bigger or more important effect. Effect size answers that question.

What if my p-value is above 0.05?

The result is not statistically significant: there is not enough evidence to reject the null hypothesis. It does not prove there is no effect, especially with small samples.

Why is 0.05 used as the cut-off?

It is a widely used convention, not a law. Some fields use stricter levels such as 0.01, and many statisticians recommend reporting exact p-values and effect sizes rather than relying on a single cut-off.

Use this data in your own work

Use this data in your own research. Download the aggregated figures behind these charts (counts, medians and shares by type of paper, period, method and discipline) as a CSV file. Please cite it as:ScholarsColab (2026). ScholarsColab Research Methods Index [Data set]. Vidhilekha Soft Solutions Pvt Ltd. Retrieved 15 October 2026, from https://scholarscolab.in/research-methodology/. Licensed under CC BY 4.0.Download the data (CSV) →

Need help applying this to your research?

Discuss your research problem with a PhD mentor, hand the whole project to a research project manager, or post your project and let matched experts apply.

Hire a Research Project Manager Hire Experts Browse Mentor Profiles

Mentors whose published work uses this approach:

More research methodology guides

About the data on this page. The ScholarsColab Research Methods Index is built from the Google Scholar publication lists of PhD researchers who mentor on ScholarsColab, and is updated daily as mentors and publications are added. Methods, designs and paper types are identified from titles, so papers whose titles do not mention them are not counted in those groups. The mentors are not a random sample of all researchers (engineering and computer science are strongly represented), so the figures describe this group of researchers and may not apply to research as a whole.