我目前正在开发一个项目，但无法将站点地图上传到 GSC，因为它找不到 Robots.txt 文件并给了我错误代码:(503 服务不可用)
我还没有深入研究.htaccess。有人也可以向我解释一下它的用法吗？
现在回答问题。当我查看它时，第一行是:
#<mittwald 503 for bots>
###########503-for-Bots##########################
#################################################
SetEnvIfNoCase User-Agent "Googlebot" bots
#SetEnvIfNoCase User-Agent "Googlebot-Image" bots
#SetEnvIfNoCase User-Agent "Googlebot-Mobile" bots
#SetEnvIfNoCase User-Agent "Java" bots
#SetEnvIfNoCase User-Agent "UptimeRobot" bots
SetEnvIfNoCase User-Agent "bingbot" bots
#SetEnvIfNoCase User-Agent "msnbot" bots
#SetEnvIfNoCase User-Agent "msnbot-media" bots
SetEnvIfNoCase User-Agent "AACrawler" bots
SetEnvIfNoCase User-Agent "AcoonBot" bots
SetEnvIfNoCase User-Agent "AhrefsBot" bots
SetEnvIfNoCase User-Agent "BLEXBot" bots
SetEnvIfNoCase User-Agent "BUbiNG" bots
SetEnvIfNoCase User-Agent "Baiduspider" bots
SetEnvIfNoCase User-Agent "Baiduspider-image" bots
SetEnvIfNoCase User-Agent "CCBot" bots
SetEnvIfNoCase User-Agent "CaBot" bots
SetEnvIfNoCase User-Agent "CompSpyBot" bots
SetEnvIfNoCase User-Agent "Daum" bots
SetEnvIfNoCase User-Agent "DoCoMo" bots
SetEnvIfNoCase User-Agent "DotBot" bots
SetEnvIfNoCase User-Agent "Exabot" bots
SetEnvIfNoCase User-Agent "Ezooms" bots
SetEnvIfNoCase User-Agent "Faraday" bots
SetEnvIfNoCase User-Agent "FeedBurner" bots
SetEnvIfNoCase User-Agent "HuaweiSymantecSpider" bots
SetEnvIfNoCase User-Agent "Infohelfer" bots
SetEnvIfNoCase User-Agent "JobboerseBot" bots
SetEnvIfNoCase User-Agent "Jooblebot" bots
SetEnvIfNoCase User-Agent "Landau-Media-Spider" bots
SetEnvIfNoCase User-Agent "MJ12bot" bots
SetEnvIfNoCase User-Agent "Mail.RU_Bot" bots
SetEnvIfNoCase User-Agent "MegaIndex" bots
SetEnvIfNoCase User-Agent "MegaIndex.ru" bots
SetEnvIfNoCase User-Agent "MetaJobBot" bots
SetEnvIfNoCase User-Agent "MetricsBot" bots
SetEnvIfNoCase User-Agent "MojeekBot" bots
SetEnvIfNoCase User-Agent "Nutch-1.10" bots
SetEnvIfNoCase User-Agent "OnPageBot" bots
SetEnvIfNoCase User-Agent "OpenWebSpider" bots
SetEnvIfNoCase User-Agent "PRTGCloudBot" bots
SetEnvIfNoCase User-Agent "PiplBot" bots
SetEnvIfNoCase User-Agent "Pixray-Seeker" bots
SetEnvIfNoCase User-Agent "Qwantify" bots
SetEnvIfNoCase User-Agent "Ralocobot" bots
SetEnvIfNoCase User-Agent "SEOkicks" bots
SetEnvIfNoCase User-Agent "SEOkicks-Robot" bots
SetEnvIfNoCase User-Agent "Screaming" bots
SetEnvIfNoCase User-Agent "Seekport" bots
SetEnvIfNoCase User-Agent "SemrushBot" bots
SetEnvIfNoCase User-Agent "SeznamBot" bots
SetEnvIfNoCase User-Agent "Site24x7" bots
SetEnvIfNoCase User-Agent "Slurp" bots
SetEnvIfNoCase User-Agent "Sosospider" bots
SetEnvIfNoCase User-Agent "Speedy" bots
SetEnvIfNoCase User-Agent "Spider" bots
SetEnvIfNoCase User-Agent "Spiderlytics" bots
SetEnvIfNoCase User-Agent "Superfeedr" bots
SetEnvIfNoCase User-Agent "Toweya.com" bots
SetEnvIfNoCase User-Agent "TurnitinBot" bots
SetEnvIfNoCase User-Agent "TwengaBot" bots
SetEnvIfNoCase User-Agent "Twiceler" bots
SetEnvIfNoCase User-Agent "Twitterbot" bots
SetEnvIfNoCase User-Agent "UnisterBot" bots
SetEnvIfNoCase User-Agent "WebCapture" bots
SetEnvIfNoCase User-Agent "Yandex" bots
SetEnvIfNoCase User-Agent "ZumBot" bots
SetEnvIfNoCase User-Agent "adscanner" bots
SetEnvIfNoCase User-Agent "aggregator:Spinn3r" bots
SetEnvIfNoCase User-Agent "crawler" bots
SetEnvIfNoCase User-Agent "<a href="https://stackoverflow.com/cdn-cgi/l/email-protection" class="__cf_email__" data-cfemail="dfbcadbea8b3baad9fbeb3baa7bef1bcb0b2" rel="noreferrer noopener nofollow">[email protected]</a>" bots
SetEnvIfNoCase User-Agent "domaincrawler.com" bots
SetEnvIfNoCase User-Agent "echobot" bots
SetEnvIfNoCase User-Agent "heritrix" bots
SetEnvIfNoCase User-Agent "htdig" bots
SetEnvIfNoCase User-Agent "ia_archiver" bots
SetEnvIfNoCase User-Agent "imaBot" bots
SetEnvIfNoCase User-Agent "infometrics-bot" bots
SetEnvIfNoCase User-Agent "jobs.de-Robot" bots
SetEnvIfNoCase User-Agent "linkdex.com" bots
SetEnvIfNoCase User-Agent "magpie-crawler" bots
SetEnvIfNoCase User-Agent "meanpathbot" bots
SetEnvIfNoCase User-Agent "pricebot" bots
SetEnvIfNoCase User-Agent "psbot" bots
SetEnvIfNoCase User-Agent "rogerbot" bots
SetEnvIfNoCase User-Agent "seoscanners" bots
SetEnvIfNoCase User-Agent "smtbot" bots
SetEnvIfNoCase User-Agent "spbot" bots
SetEnvIfNoCase User-Agent "spider" bots
SetEnvIfNoCase User-Agent "ssearch_bot" bots
SetEnvIfNoCase User-Agent "tb-webde" bots
SetEnvIfNoCase User-Agent "textlink_status_crawler" bots
SetEnvIfNoCase User-Agent "tracemyfile" bots
SetEnvIfNoCase User-Agent "trendictionbot" bots
SetEnvIfNoCase User-Agent "vebidoobot-Robot" bots
#### Retry 12h
Header always set Retry-After "43200" env=bots
RewriteEngine on
RewriteCond %{ENV:bots} 1
RewriteRule .* - [R=503,L]
################################################
################################################
#</mittwald 503 for bots>
# Disallow access to important files
<FilesMatch "(^\.|wp-config\.php|xmlrpc\.php|(?<!robots)\.txt|(liesmich|readme)\.*)">
Require all denied
</FilesMatch>
# ----------------------------------------------------------------------
# | Compression |
# ----------------------------------------------------------------------
<IfModule mod_deflate.c>
# Force compression for mangled `Accept-Encoding` request headers
# https://developer.yahoo.com/blogs/ydn/pushing-beyond-gzipping-25601.html
<IfModule mod_setenvif.c>
<IfModule mod_headers.c>
SetEnvIfNoCase ^(Accept-EncodXng|X-cept-Encoding|X{15}|~{15}|-{15})$ ^((gzip|deflate)\s*,?\s*)+|[X~-]{4,13}$ HAVE_Accept-Encoding
RequestHeader append Accept-Encoding "gzip,deflate" env=HAVE_Accept-Encoding
</IfModule>
</IfModule>
# - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
# Compress all output labeled with one of the following media types.
#
# (!) For Apache versions below version 2.3.7 you don't need to
# enable `mod_filter` and can remove the `<IfModule mod_filter.c>`
# and `</IfModule>` lines as `AddOutputFilterByType` is still in
# the core directives.
#
# https://httpd.apache.org/docs/current/mod/mod_filter.html#addoutputfilterbytype
<IfModule mod_filter.c>
AddOutputFilterByType DEFLATE "application/atom+xml" \
"application/javascript" \
"application/json" \
"application/ld+json" \
"application/manifest+json" \
"application/rdf+xml" \
"application/rss+xml" \
"application/schema+json" \
"application/vnd.geo+json" \
"application/vnd.ms-fontobject" \
"application/x-font-ttf" \
"application/x-javascript" \
"application/x-web-app-manifest+json" \
"application/xhtml+xml" \
"application/xml" \
"font/collection" \
"font/eot" \
"font/opentype" \
"font/otf" \
"font/ttf" \
"image/bmp" \
"image/svg+xml" \
"image/vnd.microsoft.icon" \
"image/x-icon" \
"text/cache-manifest" \
"text/calendar" \
"text/css" \
"text/html" \
"text/javascript" \
"text/plain" \
"text/markdown" \
"text/vcard" \
"text/vnd.rim.location.xloc" \
"text/vtt" \
"text/x-component" \
"text/x-cross-domain-policy" \
"text/xml"
</IfModule>
# - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
# Map the following filename extensions to the specified
# encoding type in order to make Apache serve the file types
# with the appropriate `Content-Encoding` response header
# (do note that this will NOT make Apache compress them!).
#
# If these files types would be served without an appropriate
# `Content-Enable` response header, client applications (e.g.:
# browsers) wouldn't know that they first need to uncompress
# the response, and thus, wouldn't be able to understand the
# content.
#
# https://httpd.apache.org/docs/current/mod/mod_mime.html#addencoding
<IfModule mod_mime.c>
AddEncoding gzip svgz
</IfModule>
</IfModule>
# ----------------------------------------------------------------------
# | Expires headers |
# ----------------------------------------------------------------------
# Serve resources with far-future expires headers.
#
# (!) If you don't control versioning with filename-based
# cache busting, you should consider lowering the cache times
# to something like one week.
#
# https://httpd.apache.org/docs/current/mod/mod_expires.html
<IfModule mod_expires.c>
ExpiresActive on
ExpiresDefault "access plus 1 month"
# CSS
ExpiresByType text/css "access plus 1 year"
# Data interchange
ExpiresByType application/atom+xml "access plus 1 hour"
ExpiresByType application/rdf+xml "access plus 1 hour"
ExpiresByType application/rss+xml "access plus 1 hour"
ExpiresByType application/json "access plus 0 seconds"
ExpiresByType application/ld+json "access plus 0 seconds"
ExpiresByType application/schema+json "access plus 0 seconds"
ExpiresByType application/vnd.geo+json "access plus 0 seconds"
ExpiresByType application/xml "access plus 0 seconds"
ExpiresByType text/calendar "access plus 0 seconds"
ExpiresByType text/xml "access plus 0 seconds"
# Favicon (cannot be renamed!) and cursor images
ExpiresByType image/vnd.microsoft.icon "access plus 1 week"
ExpiresByType image/x-icon "access plus 1 week"
# HTML
ExpiresByType text/html "access plus 0 seconds"
# JavaScript
ExpiresByType application/javascript "access plus 1 year"
ExpiresByType application/x-javascript "access plus 1 year"
ExpiresByType text/javascript "access plus 1 year"
# Manifest files
ExpiresByType application/manifest+json "access plus 1 week"
ExpiresByType application/x-web-app-manifest+json "access plus 0 seconds"
ExpiresByType text/cache-manifest "access plus 0 seconds"
# Markdown
ExpiresByType text/markdown "access plus 0 seconds"
# Media files
ExpiresByType audio/ogg "access plus 1 month"
ExpiresByType image/bmp "access plus 1 month"
ExpiresByType image/gif "access plus 1 month"
ExpiresByType image/jpeg "access plus 1 month"
ExpiresByType image/png "access plus 1 month"
ExpiresByType image/svg+xml "access plus 1 month"
ExpiresByType image/webp "access plus 1 month"
ExpiresByType video/mp4 "access plus 1 month"
ExpiresByType video/ogg "access plus 1 month"
ExpiresByType video/webm "access plus 1 month"
# Web fonts
# Collection
ExpiresByType font/collection "access plus 1 month"
# Embedded OpenType (EOT)
ExpiresByType application/vnd.ms-fontobject "access plus 1 month"
ExpiresByType font/eot "access plus 1 month"
# OpenType
ExpiresByType font/opentype "access plus 1 month"
ExpiresByType font/otf "access plus 1 month"
# TrueType
ExpiresByType application/x-font-ttf "access plus 1 month"
ExpiresByType font/ttf "access plus 1 month"
# Web Open Font Format (WOFF) 1.0
ExpiresByType application/font-woff "access plus 1 month"
ExpiresByType application/x-font-woff "access plus 1 month"
ExpiresByType font/woff "access plus 1 month"
# Web Open Font Format (WOFF) 2.0
ExpiresByType application/font-woff2 "access plus 1 month"
ExpiresByType font/woff2 "access plus 1 month"
# Other
ExpiresByType text/x-cross-domain-policy "access plus 1 week"
</IfModule>
# BEGIN W3TC Browser Cache
<IfModule mod_mime.c>
AddType text/css .css
AddType text/x-component .htc
AddType application/x-javascript .js
AddType application/javascript .js2
AddType text/javascript .js3
AddType text/x-js .js4
AddType text/html .html .htm
AddType text/richtext .rtf .rtx
AddType image/svg+xml .svg
AddType text/plain .txt
AddType text/xsd .xsd
AddType text/xsl .xsl
AddType text/xml .xml
AddType video/asf .asf .asx .wax .wmv .wmx
AddType video/avi .avi
AddType image/bmp .bmp
AddType application/java .class
AddType video/divx .divx
AddType application/msword .doc .docx
AddType application/vnd.ms-fontobject .eot
AddType application/x-msdownload .exe
AddType image/gif .gif
AddType application/x-gzip .gz .gzip
AddType image/x-icon .ico
AddType image/jpeg .jpg .jpeg .jpe
AddType image/webp .webp
AddType application/json .json
AddType application/vnd.ms-access .mdb
AddType audio/midi .mid .midi
AddType video/quicktime .mov .qt
AddType audio/mpeg .mp3 .m4a
AddType video/mp4 .mp4 .m4v
AddType video/mpeg .mpeg .mpg .mpe
AddType application/vnd.ms-project .mpp
AddType application/x-font-otf .otf
AddType application/vnd.ms-opentype ._otf
AddType application/vnd.oasis.opendocument.database .odb
AddType application/vnd.oasis.opendocument.chart .odc
AddType application/vnd.oasis.opendocument.formula .odf
AddType application/vnd.oasis.opendocument.graphics .odg
AddType application/vnd.oasis.opendocument.presentation .odp
AddType application/vnd.oasis.opendocument.spreadsheet .ods
AddType application/vnd.oasis.opendocument.text .odt
AddType audio/ogg .ogg
AddType application/pdf .pdf
AddType image/png .png
AddType application/vnd.ms-powerpoint .pot .pps .ppt .pptx
AddType audio/x-realaudio .ra .ram
AddType image/svg+xml .svg .svgz
AddType application/x-shockwave-flash .swf
AddType application/x-tar .tar
AddType image/tiff .tif .tiff
AddType application/x-font-ttf .ttf .ttc
AddType application/vnd.ms-opentype ._ttf
AddType audio/wav .wav
AddType audio/wma .wma
AddType application/vnd.ms-write .wri
AddType application/font-woff .woff
AddType application/font-woff2 .woff2
AddType application/vnd.ms-excel .xla .xls .xlsx .xlt .xlw
AddType application/zip .zip
</IfModule>
<IfModule mod_expires.c>
ExpiresActive On
ExpiresByType text/css A31536000
ExpiresByType text/x-component A31536000
ExpiresByType application/x-javascript A31536000
ExpiresByType application/javascript A31536000
ExpiresByType text/javascript A31536000
ExpiresByType text/x-js A31536000
ExpiresByType text/html A3600
ExpiresByType text/richtext A3600
ExpiresByType image/svg+xml A3600
ExpiresByType text/plain A3600
ExpiresByType text/xsd A3600
ExpiresByType text/xsl A3600
ExpiresByType text/xml A3600
ExpiresByType video/asf A31536000
ExpiresByType video/avi A31536000
ExpiresByType image/bmp A31536000
ExpiresByType application/java A31536000
ExpiresByType video/divx A31536000
ExpiresByType application/msword A31536000
ExpiresByType application/vnd.ms-fontobject A31536000
ExpiresByType application/x-msdownload A31536000
ExpiresByType image/gif A31536000
ExpiresByType application/x-gzip A31536000
ExpiresByType image/x-icon A31536000
ExpiresByType image/jpeg A31536000
ExpiresByType image/webp A31536000
ExpiresByType application/json A31536000
ExpiresByType application/vnd.ms-access A31536000
ExpiresByType audio/midi A31536000
ExpiresByType video/quicktime A31536000
ExpiresByType audio/mpeg A31536000
ExpiresByType video/mp4 A31536000
ExpiresByType video/mpeg A31536000
ExpiresByType application/vnd.ms-project A31536000
ExpiresByType application/x-font-otf A31536000
ExpiresByType application/vnd.ms-opentype A31536000
ExpiresByType application/vnd.oasis.opendocument.database A31536000
ExpiresByType application/vnd.oasis.opendocument.chart A31536000
ExpiresByType application/vnd.oasis.opendocument.formula A31536000
ExpiresByType application/vnd.oasis.opendocument.graphics A31536000
ExpiresByType application/vnd.oasis.opendocument.presentation A31536000
ExpiresByType application/vnd.oasis.opendocument.spreadsheet A31536000
ExpiresByType application/vnd.oasis.opendocument.text A31536000
ExpiresByType audio/ogg A31536000
ExpiresByType application/pdf A31536000
ExpiresByType image/png A31536000
ExpiresByType application/vnd.ms-powerpoint A31536000
ExpiresByType audio/x-realaudio A31536000
ExpiresByType image/svg+xml A31536000
ExpiresByType application/x-shockwave-flash A31536000
ExpiresByType application/x-tar A31536000
ExpiresByType image/tiff A31536000
ExpiresByType application/x-font-ttf A31536000
ExpiresByType application/vnd.ms-opentype A31536000
ExpiresByType audio/wav A31536000
ExpiresByType audio/wma A31536000
ExpiresByType application/vnd.ms-write A31536000
ExpiresByType application/font-woff A31536000
ExpiresByType application/font-woff2 A31536000
ExpiresByType application/vnd.ms-excel A31536000
ExpiresByType application/zip A31536000
</IfModule>
<IfModule mod_deflate.c>
AddOutputFilterByType DEFLATE text/css text/x-component application/x-javascript application/javascript text/javascript text/x-js text/html text/richtext image/svg+xml text/plain text/xsd text/xsl text/xml image/bmp application/java application/msword application/vnd.ms-fontobject application/x-msdownload image/x-icon image/webp application/json application/vnd.ms-access application/vnd.ms-project application/x-font-otf application/vnd.ms-opentype application/vnd.oasis.opendocument.database application/vnd.oasis.opendocument.chart application/vnd.oasis.opendocument.formula application/vnd.oasis.opendocument.graphics application/vnd.oasis.opendocument.presentation application/vnd.oasis.opendocument.spreadsheet application/vnd.oasis.opendocument.text audio/ogg application/pdf application/vnd.ms-powerpoint image/svg+xml application/x-shockwave-flash image/tiff application/x-font-ttf application/vnd.ms-opentype audio/wav application/vnd.ms-write application/font-woff application/font-woff2 application/vnd.ms-excel
<IfModule mod_mime.c>
# DEFLATE by extension
AddOutputFilter DEFLATE js css htm html xml
</IfModule>
</IfModule>
<FilesMatch "\.(css|htc|less|js|js2|js3|js4|CSS|HTC|LESS|JS|JS2|JS3|JS4)$">
FileETag MTime Size
<IfModule mod_headers.c>
Header set Pragma "public"
Header append Cache-Control "public"
Header unset Set-Cookie
Header set X-Powered-By "W3 Total Cache/0.9.7"
</IfModule>
</FilesMatch>
<FilesMatch "\.(html|htm|rtf|rtx|svg|txt|xsd|xsl|xml|HTML|HTM|RTF|RTX|SVG|TXT|XSD|XSL|XML)$">
FileETag MTime Size
<IfModule mod_headers.c>
Header append Vary User-Agent env=!dont-vary
Header set Pragma "public"
Header append Cache-Control "public"
Header set X-Powered-By "W3 Total Cache/0.9.7"
</IfModule>
</FilesMatch>
<FilesMatch "\.(asf|asx|wax|wmv|wmx|avi|bmp|class|divx|doc|docx|eot|exe|gif|gz|gzip|ico|jpg|jpeg|jpe|webp|json|mdb|mid|midi|mov|qt|mp3|m4a|mp4|m4v|mpeg|mpg|mpe|mpp|otf|_otf|odb|odc|odf|odg|odp|ods|odt|ogg|pdf|png|pot|pps|ppt|pptx|ra|ram|svg|svgz|swf|tar|tif|tiff|ttf|ttc|_ttf|wav|wma|wri|woff|woff2|xla|xls|xlsx|xlt|xlw|zip|ASF|ASX|WAX|WMV|WMX|AVI|BMP|CLASS|DIVX|DOC|DOCX|EOT|EXE|GIF|GZ|GZIP|ICO|JPG|JPEG|JPE|WEBP|JSON|MDB|MID|MIDI|MOV|QT|MP3|M4A|MP4|M4V|MPEG|MPG|MPE|MPP|OTF|_OTF|ODB|ODC|ODF|ODG|ODP|ODS|ODT|OGG|PDF|PNG|POT|PPS|PPT|PPTX|RA|RAM|SVG|SVGZ|SWF|TAR|TIF|TIFF|TTF|TTC|_TTF|WAV|WMA|WRI|WOFF|WOFF2|XLA|XLS|XLSX|XLT|XLW|ZIP)$">
FileETag MTime Size
<IfModule mod_headers.c>
Header set Pragma "public"
Header append Cache-Control "public"
Header unset Set-Cookie
Header set X-Powered-By "W3 Total Cache/0.9.7"
</IfModule>
</FilesMatch>
<FilesMatch "\.(bmp|class|doc|docx|eot|exe|ico|webp|json|mdb|mpp|otf|_otf|odb|odc|odf|odg|odp|ods|odt|ogg|pdf|pot|pps|ppt|pptx|svg|svgz|swf|tif|tiff|ttf|ttc|_ttf|wav|wri|woff|woff2|xla|xls|xlsx|xlt|xlw|BMP|CLASS|DOC|DOCX|EOT|EXE|ICO|WEBP|JSON|MDB|MPP|OTF|_OTF|ODB|ODC|ODF|ODG|ODP|ODS|ODT|OGG|PDF|POT|PPS|PPT|PPTX|SVG|SVGZ|SWF|TIF|TIFF|TTF|TTC|_TTF|WAV|WRI|WOFF|WOFF2|XLA|XLS|XLSX|XLT|XLW)$">
<IfModule mod_headers.c>
Header unset Last-Modified
</IfModule>
</FilesMatch>
<IfModule mod_headers.c>
Header set Referrer-Policy ""
</IfModule>
# END W3TC Browser Cache
# BEGIN WordPress
<IfModule mod_rewrite.c>
RewriteEngine On
RewriteBase /
RewriteRule ^index\.php$ - [L]
RewriteCond %{REQUEST_FILENAME} !-f
RewriteCond %{REQUEST_FILENAME} !-d
RewriteRule . /index.php [L]
</IfModule>
# END WordPress
# Wordfence WAF
<Files ".user.ini">
<IfModule mod_authz_core.c>
Require all denied
</IfModule>
<IfModule !mod_authz_core.c>
Order deny,allow
Deny from all
</IfModule>
</Files>
# END Wordfence WAF
这是否意味着:“我们注册了 503 错误” 或者 “我们故意为以下内容造成 503 错误...”
如果是这样，我该如何更改它，它会解决问题吗？
当我输入:https://example.de/robots.txt时，它也会显示它，但是当我使用测试工具(google robots.txt Checker或Merkle)检查它时，它会显示它没有出现。
最佳答案
当 Googlebot 访问时，您将名为 bots 的环境变量设置为值 1:
SetEnvIfNoCase User-Agent "Googlebot" bots
它会查看 User-Agent header ，如果它包含“Googlebot”，则将 bots 变量设置为默认值(即 1)。
然后，当 bots 变量为 1 时，您故意提供 503 Service Unavailable 状态:
RewriteEngine on
RewriteCond %{ENV:bots} 1
RewriteRule .* - [R=503,L]
您还添加了一个 header ，告诉机器人在 12 小时内返回:
#### Retry 12h
Header always set Retry-After "43200" env=bots
这让我认为该规则是作为处理某些高流量事件的临时措施而制定的，然后从未被删除。
要解决此问题，您应该从 .htaccess 文件中删除此代码，并允许机器人抓取您的网站。至少您应该从禁止列表中删除 googlebot 和 bingbot，因为它们是搜索引擎蜘蛛。当他们无法抓取时，它就会完全杀死您的搜索引擎优化。
关于wordpress - .htaccess 规则阻止机器人爬行？，我们在Stack Overflow上找到一个类似的问题： https://stackoverflow.com/questions/71881747/