I think the "chastity" talk concerned devotion to the deity. A deity was seen as a husband so you had to be 'faithful' and the language was easily misread as concerning real-life intimacies. In the Letter to Diognetus- very early 2nd century- we see early Christianity as not marked out by any overt lifestyle differences. Quoting from that text:

"For the Christians are distinguished from other men neither by country, nor language, nor the customs which they observe. For they neither inhabit cities of their own, nor employ a peculiar form of speech, nor lead a life which is marked out by any singularity. "

https://www.newadvent.org/fathers/0101.htm