Linguee Bot information

What is a bot? Is this page relevant to me?

A bot (also: crawler, spider) is a computer program that automatically browses the World Wide Web to gather information. (See the Wikipedia article on web crawlers for general information.) Most search engines, like Google or Yahoo, make use of crawlers for indexing the web to provide a fast search.
This page is relevant to you if you are a webmaster who wants to control which site is visited by the Linguee bot in which manner.

What does the Linguee bot do?

Since Linguee is a search engine, our web crawler is a fundamental piece of our technology. Most of the multilingual text content you find on Linguee is gathered by an automated indexing process involving the web crawler. The Linguee bot will scan the content of any website it encounters to search for multilingual text. It does not harvest e-mail addresses, and it won't index content that isn't multilingual.
We want our crawler to be as polite as possible. In case it causes you any inconvenience, please let us know [bot@linguee.com] and make sure you provide all necessary information.

I found the user agent "Linguee Bot" in my web server access log. How can I verify that this was a genuine Linguee bot and not some malevolent spider?

The best way to find out is to determine if the access originated from our officially listed IP range. Currently, our crawler operates from the following addresses:
188.138.1.140
188.138.1.146
188.138.1.148
188.138.1.150
188.138.1.152
188.138.1.168
188.138.1.169
188.138.1.174
188.138.1.40
188.138.1.42
188.138.102.243
188.138.102.50
188.138.117.207
188.138.16.101
188.138.17.215
188.138.17.227
188.138.17.240
188.138.24.19
188.138.32.26
188.138.33.158
188.138.33.163
188.138.33.168
188.138.33.198
188.138.33.236
188.138.40.156
188.138.40.207
188.138.41.132
188.138.41.22
188.138.41.231
188.138.41.90
188.138.48.137
188.138.48.143
188.138.48.30
188.138.57.145
188.138.72.173
188.138.75.203
188.138.75.243
188.138.75.89
188.138.9.197
188.138.9.205
188.138.9.210
188.138.9.217
188.138.9.42
217.172.189.149
217.172.189.151
217.172.189.159
217.172.189.26
217.172.189.72
217.172.190.5
62.138.14.107
62.138.14.108
62.138.14.109
62.138.14.110
62.138.14.111
62.138.14.112
62.138.14.113
62.138.14.114
62.138.14.115
62.138.14.116
62.138.14.117
62.138.14.118
62.138.14.119
62.138.14.120
62.138.14.121
62.138.14.122
62.138.14.123
62.138.14.124
62.138.14.125
62.138.14.126
62.138.14.163
62.138.14.164
62.138.14.165
62.138.14.166
62.138.14.176
62.138.14.177
62.138.14.178
62.138.14.179
62.138.14.180
62.138.14.197
62.138.14.208
62.138.14.209
62.138.14.210
62.138.14.211
62.138.14.212
62.138.14.213
62.138.14.214
62.138.14.215
62.138.14.216
62.138.14.217
62.138.14.218
62.138.14.219
62.138.14.220
62.138.14.221
62.138.14.222
62.138.14.223
62.138.14.224
62.138.14.225
62.138.14.226
62.138.14.227
62.138.14.228
62.138.14.229
62.138.14.235
62.138.14.236
62.138.14.237
62.138.14.238
62.138.14.239
62.138.14.240
62.138.14.241
62.138.14.242
62.138.14.243
62.138.14.244
62.138.14.245
62.138.14.246
62.138.14.247
62.138.14.248
62.138.14.249
62.138.14.250
62.138.14.251
62.138.14.252
62.138.14.253
62.138.14.254
62.138.14.33
62.138.14.36
62.138.14.37
62.138.14.38
62.138.14.39
62.138.14.40
62.138.14.41
62.138.14.42
62.138.14.43
62.138.14.44
62.138.14.45
62.138.14.47
62.138.14.48
62.138.14.49
62.138.14.50
62.138.14.51
62.138.14.52
62.138.14.53
62.138.14.54
62.138.14.55
62.138.14.57
62.138.14.58
62.138.14.59
62.138.14.60
62.138.14.61
62.138.14.62
62.138.14.63
62.138.14.64
62.138.14.67
62.138.16.10
62.138.16.11
62.138.16.18
62.138.16.19
62.138.16.20
62.138.16.21
62.138.16.22
62.138.16.23
62.138.16.4
62.138.16.5
62.138.16.6
62.138.16.7
62.138.16.8
62.138.16.9
85.25.103.133
85.25.103.134
85.25.103.135
85.25.103.137
85.25.103.138
85.25.103.18
85.25.103.36
85.25.103.38
85.25.103.56
85.25.103.58
85.25.103.68
85.25.207.134
85.25.207.13
85.25.207.14
85.25.207.168
85.25.207.16
85.25.207.172
85.25.207.174
85.25.207.26
85.25.207.5
85.25.217.150
85.25.217.197
85.25.217.238
85.25.217.53
85.25.218.120
85.25.218.5
85.25.218.69
85.25.237.106
85.25.237.110
85.25.237.116
85.25.237.200
85.25.237.72
85.25.43.161
85.25.43.164
85.25.43.199
85.25.43.244
85.25.43.28
85.25.43.32
85.25.43.75
85.25.74.42
85.25.74.73
85.25.74.78
85.25.74.80
85.25.74.92
85.93.93.142
85.93.93.144
85.93.93.145
If you find a user agent string containing "Linguee" in your access logs that seems to originate from some other address, we would appreciate if you could provide us [bot@linguee.com] with a snippet of your access log. The authentic Linguee bot will identify itself as
Linguee Bot (http://www.linguee.com/bot; bot@linguee.com)

How can I prevent the Linguee bot from visiting my site?

There are basically two options to control our crawler's behaviour:

1. The crawler will comply with any standard-conforming rule you provide in your robots.txt file. (See the Wikipedia article on the Robots Exclusion Standard for general information on this file.) To lock out the Linguee bot, you can put the following lines into your robots.txt file:
User-agent: Linguee
Disallow: /
This will result in our crawler visiting your site only once, accessing just this one file to check your robots policy. It won't return anytime soon. The crawler will also comply with a crawl rate setting and with exclusion policies given by specific HTML meta tags. Please see the linked article above for more information.

Please note that if the robots.txt file does not mention Linguee, the Linguee bot will follow Googlebot instructions. This is the same behavior that the Applebot exhibits.
2. You can also submit [bot@linguee.com] your internet domain name to request an exclusion from our indexing process. This will prevent our web crawler from visiting your site at all. If your site is a multi-domain site with a large number of domains, we kindly ask you to choose option 1, if possible.

I have found a new site which contains valuable multilingual content that doesn't seem to be included in Linguee's search index. What can I do?

We gladly accept hints and content donations. Just send a mail [bot@linguee.com] with the relevant URL and we will evaluate the site as a candidate. Please be patient with the results, though – it may take a few weeks until you find the text on Linguee.