crawlerbase.de trace
Bot-Datenbank →
www.google.com 2 Hops · 168 ms gesamt · final 200 · HTTP/2
+ Neu ↻ Re-Run

Übersicht

final URL: https://www.google.com/
2
Hops
168 ms
Total Time
60
Bots erlaubt
0
Bots blockiert

Redirect-Chain

2 Hops
  1. 00
    301 HTTP/2 https://google.com
    TTFB 69 ms Total 69 ms Size 220 B IP 142.251.14.139 Server gws
    → https://www.google.com/
    TLS Issuer C = US, O = Google Trust Services, CN = WE2 Gültig noch 64 Tage
    Alle Header (11)
    locationhttps://www.google.com/
    content-typetext/html; charset=UTF-8
    content-security-policy-report-onlyobject-src 'none';base-uri 'self';script-src 'nonce-jelu5ID54SJf7LsWFeP7-w' 'strict-dynamic' 'report-sample' 'unsafe-eval' 'unsafe-inline' https: http:;report-uri https://csp.withgoogle.com/csp/gws/other-hp
    dateSat, 09 May 2026 21:11:37 GMT
    expiresMon, 08 Jun 2026 21:11:37 GMT
    cache-controlpublic, max-age=2592000
    servergws
    content-length220
    x-xss-protection0
    x-frame-optionsSAMEORIGIN
    alt-svch3=":443"; ma=2592000,h3-29=":443"; ma=2592000
  2. 01
    200 HTTP/2 https://www.google.com/
    TTFB 93 ms Total 99 ms Size 26,3 KB IP 142.251.156.119 Server gws
    TLS Issuer C = US, O = Google Trust Services, CN = WE2 Gültig noch 64 Tage
    Alle Header (14)
    content-typetext/html; charset=ISO-8859-1
    dateSat, 09 May 2026 21:11:37 GMT
    expires-1
    cache-controlprivate, max-age=0
    content-security-policy-report-onlyobject-src 'none';base-uri 'self';script-src 'nonce-VkmZ8Cg_jWETJdvidPd2xQ' 'strict-dynamic' 'report-sample' 'unsafe-eval' 'unsafe-inline' https: http:;report-uri https://csp.withgoogle.com/csp/gws/other-hp
    accept-chSec-CH-Prefers-Color-Scheme
    p3pCP="This is not a P3P policy! See g.co/p3phelp for more info."
    content-encodinggzip
    servergws
    content-length26900
    x-xss-protection0
    x-frame-optionsSAMEORIGIN
    set-cookieSOCS=CAAaBgiA8vnPBg; expires=Tue, 08-Jun-2027 21:11:37 GMT; path=/; domain=.google.com; Secure; SameSite=lax
    set-cookieAEC=AaJma5tOwQmZD5Ar-X7UuhmWIcEcFmxOTUXrDZfQbKTOtokOdOsO58TwJA; expires=Thu, 05-Nov-2026 21:11:37 GMT; path=/; domain=.google.com; Secure; HttpOnly; SameSite=lax
    set-cookie__Secure-ENID=33.SE=qmxwzuY7Tp-b21EeUVuoyOks5bqHnMrgbKcP5ESqkBnO3dejLohSr0mn8UimQTMA0_XkqhbrNvKsZR6QDc4NcKDBGrzaPcritiSTg5lBK_be9KxEdpa9o4TvV78CONEqbOZhYxK_NFTMPPakY3z6JHtnT6jybk-Frz4YFlugPoiLrIml2H8WQSIVokBr_VKRVmzPxlquwEDNrZrw2XHmyou66jLsAg; expires=Wed, 09-Jun-2027 13:29:55 GMT; path=/; domain=.google.com; Secure; HttpOnly; SameSite=lax
    set-cookie__Secure-BUCKET=CKYB; expires=Thu, 05-Nov-2026 21:11:37 GMT; path=/; domain=.google.com; Secure; HttpOnly
    alt-svch3=":443"; ma=2592000,h3-29=":443"; ma=2592000

robots.txt für google.com

vorhanden 4 Gruppen 1 Sitemap
Sitemaps:
robots.txt Quelltext (6,4 KB) 
User-agent: *
User-agent: Yandex
Disallow: /search
Allow: /search/about
Allow: /search/howsearchworks
Disallow: /sdch
Disallow: /groups
Disallow: /index.html?
Disallow: /?
Allow: /?hl=
Disallow: /?hl=*&
Allow: /?hl=*&gws_rd=ssl$
Disallow: /?hl=*&*&gws_rd=ssl
Allow: /?gws_rd=ssl$
Allow: /?pt1=true$
Disallow: /imgres
Disallow: /u/
Disallow: /setprefs
Disallow: /m?
Disallow: /m/
Allow:    /m/finance
Disallow: /wml?
Disallow: /wml/?
Disallow: /wml/search?
Disallow: /xhtml?
Disallow: /xhtml/?
Disallow: /xhtml/search?
Disallow: /xml?
Disallow: /imode?
Disallow: /imode/?
Disallow: /imode/search?
Disallow: /jsky?
Disallow: /jsky/?
Disallow: /jsky/search?
Disallow: /pda?
Disallow: /pda/?
Disallow: /pda/search?
Disallow: /local?
Disallow: /local_url
Disallow: /products?
Disallow: /product_
Disallow: /products_
Disallow: /products;
Disallow: /print
Disallow: /books/
Disallow: /bkshp?*dq=
Disallow: /bkshp?*q=
Disallow: /books?*dq=
Disallow: /books?*q=
Disallow: /books?*qtid=
Disallow: /books?*output=
Disallow: /books?*pg=
Disallow: /books?*jtp=
Disallow: /books?*jscmd=
Disallow: /books?*buy=
Disallow: /books?*zoom=
Allow: /books/about
Allow: /books?*zoom=1
Allow: /books?*zoom=5
Allow: /books/content?*zoom=1
Allow: /books/content?*zoom=5
Disallow: /patents?
Disallow: /patents/download/
Disallow: /patents/pdf/
Disallow: /patents/related/
Disallow: /scholar
Disallow: /citations?
Allow: /citations?user=
Allow: /citations?view_op=new_profile
Allow: /citations?view_op=top_venues
Allow: /scholar_share
Disallow: /s?
Disallow: /maps?
Allow: /maps?daddr=
Allow: /maps?entry=wc
Allow: /maps?f=
Allow: /maps?hl=
Allow: /maps?q=
Allow: /maps?saddr=
Allow: /maps?sid=
Allow: /maps?*output=classic
Allow: /maps?*file=
Disallow: /mapslt?
Disallow: /maphp?
Disallow: /maps/
Allow: /maps/$
Allow: /maps/@
Allow: /maps/?daddr=
Allow: /maps/?entry=wc
Allow: /maps/?f=
Allow: /maps/?hl=
Allow: /maps/?q=
Allow: /maps/?saddr=
Allow: /maps/?sid=
Allow: /maps/search/
Allow: /maps/sitemap.xml
Allow: /maps/sitemaps/
Allow: /maps/dir/
Allow: /maps/d/
Allow: /maps/reserve
Allow: /maps/about
Allow: /maps/contrib/
Allow: /maps/match
Allow: /maps/ms?
Allow: /maps/place/
Allow: /maps/_/
Allow: /search?*tbm=map
Allow: /maps/vt?
Allow: /maps/preview
Disallow: /maps/api/js/
Allow: /maps/api/js
Disallow: /mld?
Disallow: /staticmap?
Disallow: /help/maps/streetview/partners/welcome/
Disallow: /help/maps/indoormaps/partners/
Disallow: /lochp?
Disallow: /ie?
Disallow: /uds/
Disallow: /transit?
Disallow: /trends?
Disallow: /trends/music?
Disallow: /trends/hottrends?
Disallow: /trends/viz?
Disallow: /trends/embed.js?
Disallow: /trends/fetchComponent?
Disallow: /trends/beta
Disallow: /trends/topics
Disallow: /trends/explore?
Disallow: /trends/api
Disallow: /musica
Disallow: /musicl
Disallow: /musics
Disallow: /urchin_test/
Disallow: /movies?
Disallow: /wapsearch?
Disallow: /reviews/search?
Disallow: /cbk
Disallow: /profiles/me
Disallow: /s2/profiles/me
Allow: /s2/profiles
Allow: /s2/oz
Allow: /s2/photos
Allow: /s2/search/social
Allow: /s2/static
Disallow: /s2
Disallow: /transconsole/portal/
Disallow: /aclk
Disallow: /tbproxy/
Disallow: /support/forum/search?
Disallow: /reviews/polls/
Disallow: /hosted/images/
Disallow: /accounts/ClientLogin
Disallow: /accounts/ClientAuth
Disallow: /accounts/o8
Allow: /accounts/o8/id
Disallow: /quality_form?
Disallow: /labs/popgadget/search
Disallow: /compressiontest/
Disallow: /analytics/feeds/
Disallow: /analytics/partners/comments/
Disallow: /analytics/portal/
Disallow: /analytics/uploads/
Allow: /alerts/manage
Allow: /alerts/remove
Disallow: /alerts/
Allow: /alerts/$
Disallow: /phone/compare/?
Disallow: /travel/clk
Disallow: /travel/entity
Disallow: /travel/search
Disallow: /travel/flights/booking
Disallow: /travel/flights/s/
Disallow: /travel/flights/search
Disallow: /travel/hotels/stories
Disallow: /travel/hotels/*/stories
Disallow: /travel/story
Disallow: /hotelfinder/rpc
Disallow: /hotels/rpc
Disallow: /evaluation/
Disallow: /forms/perks/
Disallow: /shopping/suppliers/search
Disallow: /edu/cs4hs/
Disallow: /trustedstores/s/
Disallow: /trustedstores/tm2
Disallow: /trustedstores/verify
Disallow: /shopping?
Disallow: /shopping/product/
Disallow: /shopping/seller
Disallow: /shopping/ratings/account/metrics
Disallow: /shopping/ratings/merchant/immersivedetails
Disallow: /shopping/reviewer
Disallow: /shopping/search
Disallow: /shopping/deals
Allow: /shopping?udm=28$
Disallow: /storefront
Disallow: /storepicker
Disallow: /about/careers/applications/candidate-prep
Disallow: /about/careers/applications/connect-with-a-googler
Disallow: /about/careers/applications/jobs/results?page=
Disallow: /about/careers/applications/jobs/results/?page=
Disallow: /about/careers/applications/jobs/results?*&page=
Disallow: /about/careers/applications/jobs/results/?*&page=
Disallow: /landing/signout.html
Disallow: /gallery/
Disallow: /landing/now/ontap/
Allow: /maps/reserve
Allow: /maps/reserve/partners
Disallow: /maps/reserve/api/
Disallow: /maps/reserve/search
Disallow: /maps/reserve/bookings
Disallow: /maps/reserve/settings
Disallow: /maps/reserve/manage
Disallow: /maps/reserve/payment
Disallow: /maps/reserve/receipt
Disallow: /maps/reserve/sellersignup
Disallow: /maps/reserve/feedback
Disallow: /maps/reserve/terms
Disallow: /maps/reserve/m/
Disallow: /maps/reserve/b/
Disallow: /maps/reserve/partner-dashboard
Disallow: /local/cars
Disallow: /local/dealership/
Disallow: /local/dining/
Disallow: /local/place/products/
Disallow: /local/place/reviews/
Disallow: /local/place/rap/
Disallow: /local/tab/
Disallow: /localservices/
Disallow: /nonprofits/account/
Disallow: /uviewer
Disallow: /landing/cmsnext-root/

# AdsBot
User-agent: AdsBot-Google
Disallow: /maps/api/js/
Allow: /maps/api/js
Disallow: /maps/api/place/js/
Disallow: /maps/api/staticmap
Disallow: /maps/api/streetview

# New user agent groups must also have a user agent reference in the global (*)
# group. See "Order of precedence" section in
# https://goo.gle/rep#order-of-precedence-for-user-agents
User-agent: Yandex
Disallow: /about/careers/applications/jobs/results
Disallow: /about/careers/applications-a/jobs/results

# Crawlers of certain social media sites are allowed to access page markup when
# google.com/imgres* links are shared. To learn more, please contact
# images-robots-allowlist@google.com.
User-agent: facebookexternalhit
User-agent: Twitterbot
Allow: /imgres
Allow: /search
Disallow: /groups
Disallow: /hosted/images/
Disallow: /m/

Sitemap: https://www.google.com/sitemap.xml

robots.txt für www.google.com

vorhanden 4 Gruppen 1 Sitemap
Sitemaps:
robots.txt Quelltext (6,4 KB) 
User-agent: *
User-agent: Yandex
Disallow: /search
Allow: /search/about
Allow: /search/howsearchworks
Disallow: /sdch
Disallow: /groups
Disallow: /index.html?
Disallow: /?
Allow: /?hl=
Disallow: /?hl=*&
Allow: /?hl=*&gws_rd=ssl$
Disallow: /?hl=*&*&gws_rd=ssl
Allow: /?gws_rd=ssl$
Allow: /?pt1=true$
Disallow: /imgres
Disallow: /u/
Disallow: /setprefs
Disallow: /m?
Disallow: /m/
Allow:    /m/finance
Disallow: /wml?
Disallow: /wml/?
Disallow: /wml/search?
Disallow: /xhtml?
Disallow: /xhtml/?
Disallow: /xhtml/search?
Disallow: /xml?
Disallow: /imode?
Disallow: /imode/?
Disallow: /imode/search?
Disallow: /jsky?
Disallow: /jsky/?
Disallow: /jsky/search?
Disallow: /pda?
Disallow: /pda/?
Disallow: /pda/search?
Disallow: /local?
Disallow: /local_url
Disallow: /products?
Disallow: /product_
Disallow: /products_
Disallow: /products;
Disallow: /print
Disallow: /books/
Disallow: /bkshp?*dq=
Disallow: /bkshp?*q=
Disallow: /books?*dq=
Disallow: /books?*q=
Disallow: /books?*qtid=
Disallow: /books?*output=
Disallow: /books?*pg=
Disallow: /books?*jtp=
Disallow: /books?*jscmd=
Disallow: /books?*buy=
Disallow: /books?*zoom=
Allow: /books/about
Allow: /books?*zoom=1
Allow: /books?*zoom=5
Allow: /books/content?*zoom=1
Allow: /books/content?*zoom=5
Disallow: /patents?
Disallow: /patents/download/
Disallow: /patents/pdf/
Disallow: /patents/related/
Disallow: /scholar
Disallow: /citations?
Allow: /citations?user=
Allow: /citations?view_op=new_profile
Allow: /citations?view_op=top_venues
Allow: /scholar_share
Disallow: /s?
Disallow: /maps?
Allow: /maps?daddr=
Allow: /maps?entry=wc
Allow: /maps?f=
Allow: /maps?hl=
Allow: /maps?q=
Allow: /maps?saddr=
Allow: /maps?sid=
Allow: /maps?*output=classic
Allow: /maps?*file=
Disallow: /mapslt?
Disallow: /maphp?
Disallow: /maps/
Allow: /maps/$
Allow: /maps/@
Allow: /maps/?daddr=
Allow: /maps/?entry=wc
Allow: /maps/?f=
Allow: /maps/?hl=
Allow: /maps/?q=
Allow: /maps/?saddr=
Allow: /maps/?sid=
Allow: /maps/search/
Allow: /maps/sitemap.xml
Allow: /maps/sitemaps/
Allow: /maps/dir/
Allow: /maps/d/
Allow: /maps/reserve
Allow: /maps/about
Allow: /maps/contrib/
Allow: /maps/match
Allow: /maps/ms?
Allow: /maps/place/
Allow: /maps/_/
Allow: /search?*tbm=map
Allow: /maps/vt?
Allow: /maps/preview
Disallow: /maps/api/js/
Allow: /maps/api/js
Disallow: /mld?
Disallow: /staticmap?
Disallow: /help/maps/streetview/partners/welcome/
Disallow: /help/maps/indoormaps/partners/
Disallow: /lochp?
Disallow: /ie?
Disallow: /uds/
Disallow: /transit?
Disallow: /trends?
Disallow: /trends/music?
Disallow: /trends/hottrends?
Disallow: /trends/viz?
Disallow: /trends/embed.js?
Disallow: /trends/fetchComponent?
Disallow: /trends/beta
Disallow: /trends/topics
Disallow: /trends/explore?
Disallow: /trends/api
Disallow: /musica
Disallow: /musicl
Disallow: /musics
Disallow: /urchin_test/
Disallow: /movies?
Disallow: /wapsearch?
Disallow: /reviews/search?
Disallow: /cbk
Disallow: /profiles/me
Disallow: /s2/profiles/me
Allow: /s2/profiles
Allow: /s2/oz
Allow: /s2/photos
Allow: /s2/search/social
Allow: /s2/static
Disallow: /s2
Disallow: /transconsole/portal/
Disallow: /aclk
Disallow: /tbproxy/
Disallow: /support/forum/search?
Disallow: /reviews/polls/
Disallow: /hosted/images/
Disallow: /accounts/ClientLogin
Disallow: /accounts/ClientAuth
Disallow: /accounts/o8
Allow: /accounts/o8/id
Disallow: /quality_form?
Disallow: /labs/popgadget/search
Disallow: /compressiontest/
Disallow: /analytics/feeds/
Disallow: /analytics/partners/comments/
Disallow: /analytics/portal/
Disallow: /analytics/uploads/
Allow: /alerts/manage
Allow: /alerts/remove
Disallow: /alerts/
Allow: /alerts/$
Disallow: /phone/compare/?
Disallow: /travel/clk
Disallow: /travel/entity
Disallow: /travel/search
Disallow: /travel/flights/booking
Disallow: /travel/flights/s/
Disallow: /travel/flights/search
Disallow: /travel/hotels/stories
Disallow: /travel/hotels/*/stories
Disallow: /travel/story
Disallow: /hotelfinder/rpc
Disallow: /hotels/rpc
Disallow: /evaluation/
Disallow: /forms/perks/
Disallow: /shopping/suppliers/search
Disallow: /edu/cs4hs/
Disallow: /trustedstores/s/
Disallow: /trustedstores/tm2
Disallow: /trustedstores/verify
Disallow: /shopping?
Disallow: /shopping/product/
Disallow: /shopping/seller
Disallow: /shopping/ratings/account/metrics
Disallow: /shopping/ratings/merchant/immersivedetails
Disallow: /shopping/reviewer
Disallow: /shopping/search
Disallow: /shopping/deals
Allow: /shopping?udm=28$
Disallow: /storefront
Disallow: /storepicker
Disallow: /about/careers/applications/candidate-prep
Disallow: /about/careers/applications/connect-with-a-googler
Disallow: /about/careers/applications/jobs/results?page=
Disallow: /about/careers/applications/jobs/results/?page=
Disallow: /about/careers/applications/jobs/results?*&page=
Disallow: /about/careers/applications/jobs/results/?*&page=
Disallow: /landing/signout.html
Disallow: /gallery/
Disallow: /landing/now/ontap/
Allow: /maps/reserve
Allow: /maps/reserve/partners
Disallow: /maps/reserve/api/
Disallow: /maps/reserve/search
Disallow: /maps/reserve/bookings
Disallow: /maps/reserve/settings
Disallow: /maps/reserve/manage
Disallow: /maps/reserve/payment
Disallow: /maps/reserve/receipt
Disallow: /maps/reserve/sellersignup
Disallow: /maps/reserve/feedback
Disallow: /maps/reserve/terms
Disallow: /maps/reserve/m/
Disallow: /maps/reserve/b/
Disallow: /maps/reserve/partner-dashboard
Disallow: /local/cars
Disallow: /local/dealership/
Disallow: /local/dining/
Disallow: /local/place/products/
Disallow: /local/place/reviews/
Disallow: /local/place/rap/
Disallow: /local/tab/
Disallow: /localservices/
Disallow: /nonprofits/account/
Disallow: /uviewer
Disallow: /landing/cmsnext-root/

# AdsBot
User-agent: AdsBot-Google
Disallow: /maps/api/js/
Allow: /maps/api/js
Disallow: /maps/api/place/js/
Disallow: /maps/api/staticmap
Disallow: /maps/api/streetview

# New user agent groups must also have a user agent reference in the global (*)
# group. See "Order of precedence" section in
# https://goo.gle/rep#order-of-precedence-for-user-agents
User-agent: Yandex
Disallow: /about/careers/applications/jobs/results
Disallow: /about/careers/applications-a/jobs/results

# Crawlers of certain social media sites are allowed to access page markup when
# google.com/imgres* links are shared. To learn more, please contact
# images-robots-allowlist@google.com.
User-agent: facebookexternalhit
User-agent: Twitterbot
Allow: /imgres
Allow: /search
Disallow: /groups
Disallow: /hosted/images/
Disallow: /m/

Sitemap: https://www.google.com/sitemap.xml

Bot-Matrix

Pfad / 60 erlaubt 0 blockiert
Suchmaschinen (13)
BotUA-TokenmatchedRegelVerdict
Googlebot
Google
 Googlebot * implizit erlaubt
Googlebot-Image
Google
 Googlebot-Image * implizit erlaubt
Googlebot-News
Google
 Googlebot-News * implizit erlaubt
AdsBot-Google
Google
 AdsBot-Google * implizit erlaubt
Mediapartners-Google
Google
 Mediapartners-Google * implizit erlaubt
Bingbot
Microsoft
 bingbot * implizit erlaubt
DuckDuckBot
DuckDuckGo
 DuckDuckBot * implizit erlaubt
YandexBot
Yandex
 YandexBot * implizit erlaubt
Baiduspider
Baidu
 Baiduspider * implizit erlaubt
Applebot
Apple
 Applebot * implizit erlaubt
Sogou web spider
Sogou
 Sogou web spider * implizit erlaubt
Yeti (Naver)
Naver
 Yeti * implizit erlaubt
SeznamBot
Seznam.cz
 SeznamBot * implizit erlaubt
AI / LLM (17)
BotUA-TokenmatchedRegelVerdict
Google-Extended
Google
 Google-Extended * implizit erlaubt
Applebot-Extended
Apple
 Applebot-Extended * implizit erlaubt
GPTBot
OpenAI
 GPTBot * implizit erlaubt
ChatGPT-User
OpenAI
 ChatGPT-User * implizit erlaubt
OAI-SearchBot
OpenAI
 OAI-SearchBot * implizit erlaubt
ClaudeBot
Anthropic
 ClaudeBot * implizit erlaubt
Claude-User
Anthropic
 Claude-User * implizit erlaubt
Claude-SearchBot
Anthropic
 Claude-SearchBot * implizit erlaubt
PerplexityBot
Perplexity
 PerplexityBot * implizit erlaubt
Perplexity-User
Perplexity
 Perplexity-User * implizit erlaubt
CCBot
Common Crawl
 CCBot * implizit erlaubt
Bytespider
ByteDance
 Bytespider * implizit erlaubt
meta-externalagent
Meta
 meta-externalagent * implizit erlaubt
Amazonbot
Amazon
 Amazonbot * implizit erlaubt
cohere-ai
Cohere
 cohere-ai * implizit erlaubt
Diffbot
Diffbot
 Diffbot * implizit erlaubt
MistralAI-User
Mistral
 MistralAI-User * implizit erlaubt
SEO-Tools (10)
BotUA-TokenmatchedRegelVerdict
AhrefsBot
Ahrefs
 AhrefsBot * implizit erlaubt
AhrefsSiteAudit
Ahrefs
 AhrefsSiteAudit * implizit erlaubt
SemrushBot
Semrush
 SemrushBot * implizit erlaubt
MJ12bot (Majestic)
Majestic
 MJ12bot * implizit erlaubt
DotBot (Moz)
Moz
 DotBot * implizit erlaubt
rogerbot (Moz)
Moz
 rogerbot * implizit erlaubt
BLEXBot (WebMeUp)
WebMeUp
 BLEXBot * implizit erlaubt
DataForSeoBot
DataForSEO
 DataForSeoBot * implizit erlaubt
Screaming Frog SEO Spider
Screaming Frog
 Screaming Frog SEO Spider * implizit erlaubt
Sitebulb
Sitebulb
 Sitebulb * implizit erlaubt
Social / Messenger (11)
BotUA-TokenmatchedRegelVerdict
facebookexternalhit
Meta
 facebookexternalhit facebookexternalhit, Twitterbot implizit erlaubt
meta-externalfetcher
Meta
 meta-externalfetcher * implizit erlaubt
Twitterbot
X (Twitter)
 Twitterbot facebookexternalhit, Twitterbot implizit erlaubt
LinkedInBot
LinkedIn
 LinkedInBot * implizit erlaubt
Slackbot-LinkExpanding
Slack
 Slackbot-LinkExpanding * implizit erlaubt
Discordbot
Discord
 Discordbot * implizit erlaubt
TelegramBot
Telegram
 TelegramBot * implizit erlaubt
WhatsApp
Meta
 WhatsApp * implizit erlaubt
Pinterestbot
Pinterest
 Pinterestbot * implizit erlaubt
Reddit Bot
Reddit
 Reddit * implizit erlaubt
TikTokSpider
ByteDance
 TikTokSpider * implizit erlaubt
Archive (1)
BotUA-TokenmatchedRegelVerdict
archive.org_bot
Internet Archive
 archive.org_bot * implizit erlaubt
Monitoring (3)
BotUA-TokenmatchedRegelVerdict
UptimeRobot
UptimeRobot
 UptimeRobot * implizit erlaubt
Pingdom
SolarWinds
 Pingdom * implizit erlaubt
StatusCake
StatusCake
 StatusCake * implizit erlaubt
Security-Scanner (5)
BotUA-TokenmatchedRegelVerdict
CensysInspect
Censys
 CensysInspect * implizit erlaubt
Shodan
Shodan
 Shodan * implizit erlaubt
Palo Alto Expanse
Palo Alto
 expanse * implizit erlaubt
l9scan / leakix
LeakIX
 l9scan * implizit erlaubt
NetcraftSurveyAgent
Netcraft
 NetcraftSurveyAgent * implizit erlaubt

Mehr Details, Verifikations-Hinweise und ein robots.txt-Generator findest du in der Bot-Datenbank.