Does there exist a set $A \subseteq \mathbb{N}$ such that $|A \cap \{1, \ldots, N\}| = o((\log N)^2)$ and every sufficiently large integer can be written as $p + a$ for some prime $p$ and $a \in A$?

Ruzsa proved that any additive complement $A$ to the primes must satisfy $\liminf_{N \to \infty} \frac{|A \cap \{1, \ldots, N\}|}{\log N} \geq e^\gamma$, where $\gamma$ is the Euler-Mascheroni constant.