Revert "Extend LE Audio allow list candidate device"

This reverts commit 6d85c6b9972f4473e6fea7b2d730600fb3909e61.

Reason for revert: no in plan for 24Q2

Bug: 327554841
Test: build

Change-Id: I090aba7e994ffb31738a2e330b9c387c7bb52421
1 file changed