To customize the category thumbnail you'll need to edit a specific category first. You can do that from the Categories page by clicking on the edit icon like it is displayed on the image below.
When you are on the Edit Category page, click on the category thumbnail to open the modal pop-up for further thumbnail customization.
Inside the pop-up modal, you are able to either upload a custom thumbnail or completely remove the custom image and use a letter instead, which is the default for each category.
Once you select the image to upload you will be able to crop the image directly from the application interface so you don't have to worry about the image size and dimensions.