From patchwork Fri Dec 22 20:52:30 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Mark Wielaard X-Patchwork-Id: 182787 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a05:7300:2483:b0:fb:cd0c:d3e with SMTP id q3csp1321608dyi; Fri, 22 Dec 2023 12:53:05 -0800 (PST) X-Google-Smtp-Source: AGHT+IF/FlZeZlC/AdMjSP8UniKBvAqDGgM38HqMzoseSU1U2pdq1tpZ22YHyWSw7VR1ijrfjqz/ X-Received: by 2002:a05:6214:21c8:b0:67a:2493:fdf with SMTP id d8-20020a05621421c800b0067a24930fdfmr3101661qvh.17.1703278385474; Fri, 22 Dec 2023 12:53:05 -0800 (PST) ARC-Seal: i=2; a=rsa-sha256; t=1703278385; cv=pass; d=google.com; s=arc-20160816; b=t8Z0u+bDwoX/xdDbvJE6kttkdvGWSyRZwpZPFA1JQOCvcI9CzMWP+24vUDimzLMqpn N7tOO0Ga8TUo+Zf3+mQucF27a3eZaj5/ZMigyhzy52NcneHOWEj67o3c3PhMxzH39kQY +W+wyRVRVQbEYYWB+TmjD2OORd1ebKgoeToPOVWXmIDs80iNz+X18th3jvVSDHB5JCTU sx2yqvfQ733R9jBoMCBjKgo+FCQvwo3741UPk7sv7x3fs0AU7ZBujO3Db7KkTxOELm7R WMsbNHXA3gos6Teok1XtqBnxYtmdbstVils039rwWBBujyMHYIKb8ef8+NapYSl3Rir2 CyZA== ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=errors-to:list-subscribe:list-help:list-post:list-archive :list-unsubscribe:list-id:precedence:content-transfer-encoding :mime-version:message-id:date:subject:cc:to:from:arc-filter :dmarc-filter:delivered-to; bh=yIMyt1MDIZpTFjWZfaxepyvSIqi4uV4mmT6asHm0kyE=; fh=zX7wLOO+EB1ovSOEbQzD6PAkD5gSAiAaroIJ3ku0PnA=; b=HxpV4PHq6MnsxI3wh94VptIijXPo9+wqv+qXNyKfI215AnjfVtGRs6Zep2pK3jcQIX TmwtCUqhGrzLcunpgvb8ybdQCgrfojLGt8tD/joDsT1Kg0GLAi0qi5+3zTlV7xM4GJC4 rdXz7w0HeNSr5N9hLcBCrpOfcvAKqgXsN8C0dMALOAOOSgQdcyZEli+HJzX8DUrA5Ur/ PqjyaiMi4ICw1lihdc2gbqgrUyl+Os/JQcW3DBRlTmWxVY69vZUX8NJqxZBVqELBCpkD kHEUbvslbfgxIH0BILXTVJpm4HH1CcGxYR1uyX+Ej+tmbQfa913fzSA4tNmEvZyR8suR pkWw== ARC-Authentication-Results: i=2; mx.google.com; arc=pass (i=1); spf=pass (google.com: domain of gcc-patches-bounces+ouuuleilei=gmail.com@gcc.gnu.org designates 8.43.85.97 as permitted sender) smtp.mailfrom="gcc-patches-bounces+ouuuleilei=gmail.com@gcc.gnu.org" Received: from server2.sourceware.org (server2.sourceware.org. [8.43.85.97]) by mx.google.com with ESMTPS id b7-20020a0cb3c7000000b0067f756eb9f9si4953251qvf.226.2023.12.22.12.53.05 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 22 Dec 2023 12:53:05 -0800 (PST) Received-SPF: pass (google.com: domain of gcc-patches-bounces+ouuuleilei=gmail.com@gcc.gnu.org designates 8.43.85.97 as permitted sender) client-ip=8.43.85.97; Authentication-Results: mx.google.com; arc=pass (i=1); spf=pass (google.com: domain of gcc-patches-bounces+ouuuleilei=gmail.com@gcc.gnu.org designates 8.43.85.97 as permitted sender) smtp.mailfrom="gcc-patches-bounces+ouuuleilei=gmail.com@gcc.gnu.org" Received: from server2.sourceware.org (localhost [IPv6:::1]) by sourceware.org (Postfix) with ESMTP id 35E3F3858433 for ; Fri, 22 Dec 2023 20:53:05 +0000 (GMT) X-Original-To: gcc-patches@gcc.gnu.org Delivered-To: gcc-patches@gcc.gnu.org Received: from gnu.wildebeest.org (gnu.wildebeest.org [45.83.234.184]) by sourceware.org (Postfix) with ESMTPS id A87AB3858D33 for ; Fri, 22 Dec 2023 20:52:40 +0000 (GMT) DMARC-Filter: OpenDMARC Filter v1.4.2 sourceware.org A87AB3858D33 Authentication-Results: sourceware.org; dmarc=none (p=none dis=none) header.from=klomp.org Authentication-Results: sourceware.org; spf=pass smtp.mailfrom=klomp.org ARC-Filter: OpenARC Filter v1.0.0 sourceware.org A87AB3858D33 Authentication-Results: server2.sourceware.org; arc=none smtp.remote-ip=45.83.234.184 ARC-Seal: i=1; a=rsa-sha256; d=sourceware.org; s=key; t=1703278362; cv=none; b=CUybtzeQD88pMaxyzJCTVXxI/i+4L11Q4HbahWWaTRj9VAT/bm/xqshBmhPJiDbFXEE7qhA0zj//P5UXqVeCsEZKNyUDTAAQZEdpAjuWR1YWcco3O5oaVlYwKs19POHLpm/t99V0l/r3EqYR5SwbuyDYIDzw9UUA9Pb3pQxBKkY= ARC-Message-Signature: i=1; a=rsa-sha256; d=sourceware.org; s=key; t=1703278362; c=relaxed/simple; bh=7PC+xI8CG00lrOKZR9h2/VV9yEWxqbsLdv9GbkmpA5s=; h=From:To:Subject:Date:Message-Id:MIME-Version; b=kkiAE4Mx4Y3QKe9GbfPa07MUJqhoL1QlVTwrXY9hwLx20uJd1vkL8igiuvWYynUNi/nwF83/dLS9YFajK2a5EcdWlHQW5EUEllxYMGj/KErfnuJ2tdQFmhM9+l1v2iYFEraE7XJCSgTW9wx0gq3osNf4kzwDU1EzAJiLRvRwcyQ= ARC-Authentication-Results: i=1; server2.sourceware.org Received: from csb.redhat.com (deer0x03.wildebeest.org [172.31.17.133]) (using TLSv1.2 with cipher ADH-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by gnu.wildebeest.org (Postfix) with ESMTPSA id 20D2E302BB42; Fri, 22 Dec 2023 21:52:39 +0100 (CET) Received: by csb.redhat.com (Postfix, from userid 10916) id E57ABD0A87; Fri, 22 Dec 2023 21:52:38 +0100 (CET) From: Mark Wielaard To: gcc-patches@gcc.gnu.org Cc: Gerald Pfeifer , Mark Wielaard Subject: [COMMITTED] robots.txt: Disallow a few more bugzilla queries Date: Fri, 22 Dec 2023 21:52:30 +0100 Message-Id: <20231222205230.182419-1-mark@klomp.org> X-Mailer: git-send-email 2.39.3 MIME-Version: 1.0 X-Spam-Status: No, score=-8.7 required=5.0 tests=BAYES_00, GIT_PATCH_0, JMQ_SPF_NEUTRAL, KAM_DMARC_STATUS, SPF_HELO_NONE, SPF_PASS, TXREP, T_SCC_BODY_TEXT_LINE autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on server2.sourceware.org X-BeenThere: gcc-patches@gcc.gnu.org X-Mailman-Version: 2.1.30 Precedence: list List-Id: Gcc-patches mailing list List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: gcc-patches-bounces+ouuuleilei=gmail.com@gcc.gnu.org X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1786016836090087858 X-GMAIL-MSGID: 1786016836090087858 Some spiders are hitting bugzilla hard generating dependency trees or graphs, downloading large attachements or requesting all bugs in xml format. Disallow all that. --- htdocs/robots.txt | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/htdocs/robots.txt b/htdocs/robots.txt index b9fc830d..057c5899 100644 --- a/htdocs/robots.txt +++ b/htdocs/robots.txt @@ -10,4 +10,8 @@ Disallow: /cgit/ Disallow: /svn Disallow: /cgi-bin/ Disallow: /bugzilla/buglist.cgi +Disallow: /bugzilla/show_bug.cgi*ctype=xml* +Disallow: /bugzilla/attachment.cgi +Disallow: /bugzilla/showdependencygraph.cgi +Disallow: /bugzilla/showdependencytree.cgi Crawl-Delay: 60