The other thing that can invalidate noise measurements is having spur reject mode on. And, of course any averaging done in log units (e.g. averaging points as dBm instead of mW).
Can you confirm that when you “averaged” the sweeps it was in power units, not log units?