In order to enable Apple MDM enrollment, the DME server must be configured as an Apple MDM server. To do this, go to Server > Certificates > MDM > DME MDM (only available if Enable external MDM is OFF).
In this window, you need to create an Apple Push Notification Service (APNS) Certificate, which is required for using DME as an Apple MDM server.
You also need to create a self-signed certificate for Apple MDM, which is used to validate your DME installation against the APNS Certificate. This is also done in the DME MDM window.
For information about how to create and install the APNS and self-signed certificates, see DME MDM.