The fruit of Alpinia oxyphylla is often used in traditional Chinese medicine to treat dementia and memory defects. In this study, we isolated a novel acidic polysaccharide (AOP70-2-1) from A. oxyphylla fruit. Structural analysis showed that AOP70-2-1 consists of β-D-GlcAp-(1→, →2,3,6)-α-D-Galp-(1→, α-L-Araf-(1→, →2,5)-α-L-Araf-(1→, →4)-α-D-Glcp-(1→, →3,4)-α-D-Xylp-(1→, →3,6)-β-D-Manp-(1→, and α-L-Rhap-(1→. Morphological analysis indicated that AOP70-2-1 had an irregular sheet structure. The crude polysaccharide (AOP70) from A. oxyphylla significantly improved learning and memory ability of Alzheimer’s disease (AD) mice, and AOP70 exhibited comparable or even better effects than huperzine A. Most important, AOP70 reduced NO, IL-1β, TNF-α, and PGE-2 concentrations to normal levels. AOP70-2-1 significantly inhibited NO production in lipopolysaccharide stimulated BV2 microglial cells. Note that the effect of 2.6 μM AOP70-2-1 was superior to indomethacin. AOP70-2-1 also remarkably decreased the values of IL-6 and TNF-α. AOP70-2-1 may be a bioactive component of AOP70 and has the potential for the treatment of AD.