Turns out we protected only the PluginServlet, which is only used for Plugins and not the main Admin console itself.