Color Gallery - Delete Button
The Delete button in the Color Gallery deletes all the named colors currently selected in the gallery. If any of the selected colors are currently in use a warning will appear giving you the option to delete the entire set of selected colors, just delete the colors which are not being used or cancel the operation.
If you delete any colors which are in use in your document, any objects using those colors will not change in appearance, but the colors applied to them will become local colors.
Alternatively, you can right-click on the color in the gallery to open the pop-up menu and choose Delete which you may find easier.