crawlerbase.de trace
Bot-Datenbank →
github.com 1 Hop · 67 ms gesamt · final 200 · HTTP/2
+ Neu ↻ Re-Run

Übersicht

final URL: https://github.com/
1
Hops
67 ms
Total Time
60
Bots erlaubt
0
Bots blockiert

Redirect-Chain

1 Hop
  1. 00
    200 HTTP/2 https://github.com
    TTFB 33 ms Total 67 ms Size 123,7 KB IP 140.82.121.3 Server github.com
    TLS Issuer C = GB, O = Sectigo Limited, CN = Sectigo Public Server Authentication CA DV E36 Gültig noch 85 Tage
    Alle Header (17)
    dateSat, 09 May 2026 21:05:40 GMT
    content-typetext/html; charset=utf-8
    varyX-PJAX, X-PJAX-Container, Turbo-Visit, Turbo-Frame, X-Requested-With, Accept-Language, Sec-Fetch-Site,Accept-Encoding, Accept, X-Requested-With
    content-languageen-US
    etagW/"1f13abce10d1dbd07c613f515a3cfdcf"
    cache-controlmax-age=0, private, must-revalidate
    strict-transport-securitymax-age=31536000; includeSubdomains; preload
    x-frame-optionsdeny
    x-content-type-optionsnosniff
    x-xss-protection0
    referrer-policyorigin-when-cross-origin, strict-origin-when-cross-origin
    content-security-policydefault-src 'none'; base-uri 'self'; child-src github.githubassets.com github.com/assets-cdn/worker/ github.com/assets/ gist.github.com/assets-cdn/worker/; connect-src 'self' uploads.github.com www.githubstatus.com collector.github.com raw.githubusercontent.com api.github.com github-cloud.s3.amazonaws.com github-production-repository-file-5c1aeb.s3.amazonaws.com github-production-upload-manifest-file-7fdce7.s3.amazonaws.com github-production-user-asset-6210df.s3.amazonaws.com *.rel.tunnels.api.visualstudio.com wss://*.rel.tunnels.api.visualstudio.com github.githubassets.com objects-origin.githubusercontent.com copilot-proxy.githubusercontent.com proxy.individual.githubcopilot.com proxy.business.githubcopilot.com proxy.enterprise.githubcopilot.com *.actions.githubusercontent.com wss://*.actions.githubusercontent.com productionresultssa0.blob.core.windows.net productionresultssa1.blob.core.windows.net productionresultssa2.blob.core.windows.net productionresultssa3.blob.core.windows.net productionresultssa4.blob.core.windows.net productionresultssa5.blob.core.windows.net productionresultssa6.blob.core.windows.net productionresultssa7.blob.core.windows.net productionresultssa8.blob.core.windows.net productionresultssa9.blob.core.windows.net productionresultssa10.blob.core.windows.net productionresultssa11.blob.core.windows.net productionresultssa12.blob.core.windows.net productionresultssa13.blob.core.windows.net productionresultssa14.blob.core.windows.net productionresultssa15.blob.core.windows.net productionresultssa16.blob.core.windows.net productionresultssa17.blob.core.windows.net productionresultssa18.blob.core.windows.net productionresultssa19.blob.core.windows.net github-production-repository-image-32fea6.s3.amazonaws.com github-production-release-asset-2e65be.s3.amazonaws.com insights.github.com wss://alive.github.com wss://alive-staging.github.com api.githubcopilot.com api.individual.githubcopilot.com api.business.githubcopilot.com api.enterprise.githubcopilot.com edge.fullstory.com rs.fullstory.com; font-src github.githubassets.com; form-action 'self' github.com gist.github.com copilot-workspace.githubnext.com objects-origin.githubusercontent.com; frame-ancestors 'none'; frame-src viewscreen.githubusercontent.com notebooks.githubusercontent.com www.youtube-nocookie.com; img-src 'self' data: blob: github.githubassets.com media.githubusercontent.com camo.githubusercontent.com identicons.github.com avatars.githubusercontent.com private-avatars.githubusercontent.com github-cloud.s3.amazonaws.com objects.githubusercontent.com release-assets.githubusercontent.com secured-user-images.githubusercontent.com user-images.githubusercontent.com private-user-images.githubusercontent.com opengraph.githubassets.com marketplace-screenshots.githubusercontent.com copilotprodattachments.blob.core.windows.net/github-production-copilot-attachments/ github-production-user-asset-6210df.s3.amazonaws.com customer-stories-feed.github.com spotlights-feed.github.com objects-origin.githubusercontent.com *.githubusercontent.com images.ctfassets.net/8aevphvgewt8/; manifest-src 'self'; media-src github.com user-images.githubusercontent.com secured-user-images.githubusercontent.com private-user-images.githubusercontent.com github-production-user-asset-6210df.s3.amazonaws.com gist.github.com github.githubassets.com assets.ctfassets.net/8aevphvgewt8/ videos.ctfassets.net/8aevphvgewt8/; script-src github.githubassets.com; style-src 'unsafe-inline' github.githubassets.com; upgrade-insecure-requests; worker-src github.githubassets.com github.com/assets-cdn/worker/ github.com/assets/ gist.github.com/assets-cdn/worker/
    servergithub.com
    content-encodinggzip
    accept-rangesbytes
    set-cookie_gh_sess=e%2FnzH47GBjN4fqJunXbQ5Qfnp7iHtq%2FfvVIjDtvS%2FrICAisqcgG9mBZM9F3em4tfJymo4qVVS4uQ07Gw6B5pxtyZ6TUJKyt669YiijjH5FdEXLZtpU%2FxXKrjApWyTtwoVU%2BJ2dvPvUf6wKUXi2dJrk%2FvcWsH513tw61RI7XJgX2Z9vCA8Z55zY57YjlEOc81KwkgDUjpsISPXyeTZ3LyjUN1g1BSBM170TcjC8WsB7%2BYq831rjpO9IgZjzPQpXxbAGC9M5i9cWZNTAVbtkTZ3Q%3D%3D--7KeGDjECXF7JVi1y--PykXxxbj%2BSf34M4JDrFS8w%3D%3D; path=/; HttpOnly; secure; SameSite=Lax
    set-cookie_octo=GH1.1.549949316.1778360758; expires=Sun, 09 May 2027 21:05:58 GMT; domain=.github.com; path=/; secure; SameSite=Lax
    set-cookielogged_in=no; expires=Sun, 09 May 2027 21:05:58 GMT; domain=.github.com; path=/; HttpOnly; secure; SameSite=Lax
    x-github-request-idA40E:68A63:E0D9B11:A8F48DA:69FFA1B6

robots.txt für github.com

vorhanden 5 Gruppen 0 Sitemaps
robots.txt Quelltext (2,2 KB) 
# If you would like to crawl GitHub contact us via https://support.github.com?tags=dotcom-robots
# We also provide an extensive API: https://docs.github.com
User-agent: bingbot
Disallow: /ekansa/Open-Context-Data
Disallow: /ekansa/opencontext-*
Disallow: /account-login
Disallow: */tarball/
Disallow: */zipball/
Disallow: /Explodingstuff/
Disallow: /copilot/
Disallow: /copilot/c/

User-agent: adidxbot
Disallow: /ekansa/Open-Context-Data
Disallow: /ekansa/opencontext-*
Disallow: /account-login
Disallow: */tarball/
Disallow: */zipball/
Disallow: /Explodingstuff/
Disallow: /copilot/
Disallow: /copilot/c/

User-agent: BingPreview
Disallow: /ekansa/Open-Context-Data
Disallow: /ekansa/opencontext-*
Disallow: /account-login
Disallow: */tarball/
Disallow: */zipball/
Disallow: /Explodingstuff/
Disallow: /copilot/
Disallow: /copilot/c/

User-agent: baidu
crawl-delay: 1


User-agent: *

Disallow: /*/*/pulse
Disallow: /*/*/projects
Disallow: /*/*/forks
Disallow: /*/*/issues/new
Disallow: /*/*/milestones/new
Disallow: /*/*/issues/search
Disallow: /*/*/commits/
Disallow: /*/*/branches
Disallow: /*/*/contributors
Disallow: /*/*/tags
Disallow: /*/*/stargazers
Disallow: /*/*/watchers
Disallow: /*/*/network
Disallow: /*/*/graphs
Disallow: /*/*/compare

Disallow: /*/tree/
Disallow: /gist/
Disallow: /*/download
Disallow: /*/revisions
Disallow: /*/commits/*?author
Disallow: /*/commits/*?path
Disallow: /*/comments
Disallow: /*/archive/
Disallow: /*/blame/
Disallow: /*/raw/
Disallow: /*/cache/
Disallow: /.git/
Disallow: */.git/
Disallow: /*.git$
Disallow: /search/advanced
Disallow: /search$
Disallow: /*q=
Disallow: /*.atom$

Disallow: /ekansa/Open-Context-Data
Disallow: /ekansa/opencontext-*
Disallow: */tarball/
Disallow: */zipball/

Disallow: /*source=*
Disallow: /*ref_cta=*
Disallow: /*plan=*
Disallow: /*return_to=*
Disallow: /*ref_loc=*
Disallow: /*setup_organization=*
Disallow: /*source_repo=*
Disallow: /*ref_page=*
Disallow: /*source=*
Disallow: /*referrer=*
Disallow: /*report=*
Disallow: /*author=*
Disallow: /*since=*
Disallow: /*until=*
Disallow: /*commits?author=*
Disallow: /*report-abuse?report=*
Disallow: /*tab=*
Allow: /*?tab=achievements&achievement=*

Disallow: /account-login
Disallow: /Explodingstuff/

Disallow: /copilot/
Disallow: /copilot/c/

Bot-Matrix

Pfad / 60 erlaubt 0 blockiert
Suchmaschinen (13)
BotUA-TokenmatchedRegelVerdict
Googlebot
Google
 Googlebot * implizit erlaubt
Googlebot-Image
Google
 Googlebot-Image * implizit erlaubt
Googlebot-News
Google
 Googlebot-News * implizit erlaubt
AdsBot-Google
Google
 AdsBot-Google * implizit erlaubt
Mediapartners-Google
Google
 Mediapartners-Google * implizit erlaubt
Bingbot
Microsoft
 bingbot * implizit erlaubt
DuckDuckBot
DuckDuckGo
 DuckDuckBot * implizit erlaubt
YandexBot
Yandex
 YandexBot * implizit erlaubt
Baiduspider
Baidu
 Baiduspider * implizit erlaubt
Applebot
Apple
 Applebot * implizit erlaubt
Sogou web spider
Sogou
 Sogou web spider * implizit erlaubt
Yeti (Naver)
Naver
 Yeti * implizit erlaubt
SeznamBot
Seznam.cz
 SeznamBot * implizit erlaubt
AI / LLM (17)
BotUA-TokenmatchedRegelVerdict
Google-Extended
Google
 Google-Extended * implizit erlaubt
Applebot-Extended
Apple
 Applebot-Extended * implizit erlaubt
GPTBot
OpenAI
 GPTBot * implizit erlaubt
ChatGPT-User
OpenAI
 ChatGPT-User * implizit erlaubt
OAI-SearchBot
OpenAI
 OAI-SearchBot * implizit erlaubt
ClaudeBot
Anthropic
 ClaudeBot * implizit erlaubt
Claude-User
Anthropic
 Claude-User * implizit erlaubt
Claude-SearchBot
Anthropic
 Claude-SearchBot * implizit erlaubt
PerplexityBot
Perplexity
 PerplexityBot * implizit erlaubt
Perplexity-User
Perplexity
 Perplexity-User * implizit erlaubt
CCBot
Common Crawl
 CCBot * implizit erlaubt
Bytespider
ByteDance
 Bytespider * implizit erlaubt
meta-externalagent
Meta
 meta-externalagent * implizit erlaubt
Amazonbot
Amazon
 Amazonbot * implizit erlaubt
cohere-ai
Cohere
 cohere-ai * implizit erlaubt
Diffbot
Diffbot
 Diffbot * implizit erlaubt
MistralAI-User
Mistral
 MistralAI-User * implizit erlaubt
SEO-Tools (10)
BotUA-TokenmatchedRegelVerdict
AhrefsBot
Ahrefs
 AhrefsBot * implizit erlaubt
AhrefsSiteAudit
Ahrefs
 AhrefsSiteAudit * implizit erlaubt
SemrushBot
Semrush
 SemrushBot * implizit erlaubt
MJ12bot (Majestic)
Majestic
 MJ12bot * implizit erlaubt
DotBot (Moz)
Moz
 DotBot * implizit erlaubt
rogerbot (Moz)
Moz
 rogerbot * implizit erlaubt
BLEXBot (WebMeUp)
WebMeUp
 BLEXBot * implizit erlaubt
DataForSeoBot
DataForSEO
 DataForSeoBot * implizit erlaubt
Screaming Frog SEO Spider
Screaming Frog
 Screaming Frog SEO Spider * implizit erlaubt
Sitebulb
Sitebulb
 Sitebulb * implizit erlaubt
Social / Messenger (11)
BotUA-TokenmatchedRegelVerdict
facebookexternalhit
Meta
 facebookexternalhit * implizit erlaubt
meta-externalfetcher
Meta
 meta-externalfetcher * implizit erlaubt
Twitterbot
X (Twitter)
 Twitterbot * implizit erlaubt
LinkedInBot
LinkedIn
 LinkedInBot * implizit erlaubt
Slackbot-LinkExpanding
Slack
 Slackbot-LinkExpanding * implizit erlaubt
Discordbot
Discord
 Discordbot * implizit erlaubt
TelegramBot
Telegram
 TelegramBot * implizit erlaubt
WhatsApp
Meta
 WhatsApp * implizit erlaubt
Pinterestbot
Pinterest
 Pinterestbot * implizit erlaubt
Reddit Bot
Reddit
 Reddit * implizit erlaubt
TikTokSpider
ByteDance
 TikTokSpider * implizit erlaubt
Archive (1)
BotUA-TokenmatchedRegelVerdict
archive.org_bot
Internet Archive
 archive.org_bot * implizit erlaubt
Monitoring (3)
BotUA-TokenmatchedRegelVerdict
UptimeRobot
UptimeRobot
 UptimeRobot * implizit erlaubt
Pingdom
SolarWinds
 Pingdom * implizit erlaubt
StatusCake
StatusCake
 StatusCake * implizit erlaubt
Security-Scanner (5)
BotUA-TokenmatchedRegelVerdict
CensysInspect
Censys
 CensysInspect * implizit erlaubt
Shodan
Shodan
 Shodan * implizit erlaubt
Palo Alto Expanse
Palo Alto
 expanse * implizit erlaubt
l9scan / leakix
LeakIX
 l9scan * implizit erlaubt
NetcraftSurveyAgent
Netcraft
 NetcraftSurveyAgent * implizit erlaubt

Mehr Details, Verifikations-Hinweise und ein robots.txt-Generator findest du in der Bot-Datenbank.