in the context non-believers = people who were once believers but fell away, forsaking the gathering of ourselves together.
(IIRC in the biblical passage Paul was talking specifically about people who were fucking their mothers, and he said that this sort of thing is not even practiced among...