We calculate the spin-independent scattering cross section for direct detection that results from the electromagnetic polarizability of a composite scalar baryon dark matter candidate – "Stealth Dark Matter", that is based on a dark SU(4) confining gauge theory. In the nonrelativistic limit, electromagnetic polarizability proceeds through a dimension-7 interaction leading to a very small scattering cross section for dark matter with weak scale masses. This represents a lower bound on the scattering cross section for composite dark matter theories with electromagnetically charged constituents. We carry out lattice calculations of the polarizability for the lightest baryons in SU(3) and SU(4) gauge theories using the background field method on quenched configurations. We find the polarizabilities of SU(3) and SU(4) to be comparable (within about 50%) normalized to the baryon mass, which is suggestive for extensions to larger SU(N) groups. The resulting scattering cross sections with a xenon target are shown to be potentially detectable in the dark matter mass range of about 200-700 GeV, where the lower bound is from the existing LUX constraint while the upper bound is the coherent neutrino background. Significant uncertainties in the cross section remain due to the more complicated interaction of the polarizablity operator with nuclear structure, however the steep dependence on the dark matter mass, $1/m_B^6$, suggests the observable dark matter mass range is not appreciably modified. We briefly highlight collider searches for the mesons in the theory as well as the indirect astrophysical effects that may also provide excellent probes of stealth dark matter.