Linguee Bot information

What is a bot? Is this page relevant to me?

A bot (also: crawler, spider) is a computer program that automatically browses the World Wide Web to gather information. (See the Wikipedia article on web crawlers for general information.) Most search engines, like Google or Yahoo, make use of crawlers for indexing the web to provide a fast search.
This page is relevant to you if you are a webmaster who wants to control which site is visited by the Linguee bot in which manner.

What does the Linguee bot do?

Since Linguee is a search engine, our web crawler is a fundamental piece of our technology. Most of the multilingual text content you find on Linguee is gathered by an automated indexing process involving the web crawler. The Linguee bot will scan the content of any website it encounters to search for multilingual text. It does not harvest e-mail addresses, and it won't index content that isn't multilingual.
We want our crawler to be as polite as possible. In case it causes you any inconvenience, please let us know [bot@linguee.com] and make sure you provide all necessary information.

I found the user agent "Linguee Bot" in my web server access log. How can I verify that this was a genuine Linguee bot and not some malevolent spider?

The best way to find out is to determine if the access originated from our officially listed IP range. Currently, our crawler operates from the following addresses:

62.138.14.127
62.138.14.135
62.138.14.136
62.138.14.143
62.138.14.154
62.138.14.208
62.138.14.210
62.138.14.211
62.138.14.213
62.138.14.220
62.138.14.245
62.138.14.37
62.138.14.39
62.138.14.41
62.138.14.60
62.138.14.63
62.138.16.11
62.138.16.12
62.138.16.14
62.138.16.17
62.138.16.175
62.138.16.176
62.138.16.188
62.138.16.191
62.138.16.192
62.138.16.199
62.138.16.200
62.138.16.201
62.138.16.203
62.138.16.206
62.138.16.215
62.138.16.216
62.138.16.217
62.138.16.223
62.138.16.224
62.138.16.237
62.138.16.239
62.138.16.240
62.138.16.251
62.138.16.8
62.138.16.93
62.138.18.16
62.138.18.208
62.138.18.218
62.138.18.33
62.138.18.35
62.138.18.39
62.138.18.40
62.138.18.42
62.138.18.43
62.138.18.66
62.138.18.68
62.138.24.46
62.138.24.79
85.25.207.10
85.25.207.142
85.25.207.206
85.25.207.214
85.25.207.228
85.25.207.234
85.25.207.41
85.25.207.42
85.25.207.44
85.25.207.74
85.25.217.102
85.25.217.105
85.25.217.111
85.25.217.133
85.25.217.142
85.25.217.143
85.25.217.144
85.25.217.145
85.25.217.149
85.25.217.155
85.25.217.156
85.25.217.159
85.25.217.164
85.25.217.169
85.25.217.196
85.25.217.212
85.25.217.213
85.25.217.224
85.25.217.226
85.25.217.229
85.25.217.234
85.25.217.238
85.25.217.245
85.25.217.26
85.25.217.27
85.25.217.42
85.25.217.47
85.25.217.78
85.25.218.11
85.25.218.111
85.25.218.120
85.25.218.147
85.25.218.152
85.25.218.169
85.25.218.203
85.25.218.69
85.25.218.74
85.25.218.76
85.25.218.87
85.25.218.90
85.25.218.92
85.25.237.101
85.25.237.105
85.25.237.106
85.25.237.110
85.25.237.176
85.25.237.198
85.25.237.20
85.25.237.227
85.25.237.229
85.25.237.71
85.25.237.72
85.25.237.73
85.25.237.74
85.25.237.75
85.25.237.76
85.25.237.80
85.25.237.89
85.25.237.99
85.25.43.111
85.25.43.118
85.25.43.140
85.25.43.144
85.25.43.159
85.25.43.166
85.25.43.177
85.25.43.201
85.25.43.215
85.25.43.219
85.25.43.223
85.25.43.226
85.25.43.233
85.25.43.245
85.25.43.53
85.25.43.91
85.25.74.134
85.25.74.154
85.25.74.68
85.25.74.79
85.93.93.100
85.93.93.102
85.93.93.118
85.93.93.119
85.93.93.121
85.93.93.122
85.93.93.142
85.93.93.144
85.93.93.145
85.93.93.147
85.93.93.148
85.93.93.173
85.93.93.96
188.138.102.144
188.138.102.157
188.138.102.159
188.138.102.174
188.138.102.18
188.138.102.199
188.138.102.200
188.138.102.204
188.138.102.220
188.138.102.231
188.138.102.242
188.138.102.243
188.138.102.26
188.138.102.31
188.138.102.8
188.138.102.80
188.138.33.73
188.138.33.75
188.138.41.102
188.138.41.169
188.138.41.224
188.138.41.24
188.138.41.30
188.138.41.95
188.138.41.97
188.138.57.105
188.138.57.141
188.138.57.203
188.138.57.21
188.138.57.5
188.138.57.6
188.138.75.112
188.138.75.217
188.138.75.71
188.138.75.72
188.138.75.84
188.138.75.86
188.138.75.92
217.172.189.144
217.172.189.35
217.172.189.44
217.172.189.87

If you find a user agent string containing "Linguee" in your access logs that seems to originate from some other address, we would appreciate if you could provide us [bot@linguee.com] with a snippet of your access log. The authentic Linguee bot will identify itself as
Linguee Bot (http://www.linguee.com/bot; bot@linguee.com)

How can I prevent the Linguee bot from visiting my site?

There are basically two options to control our crawler's behaviour:

1. The crawler will comply with any standard-conforming rule you provide in your robots.txt file. (See the Wikipedia article on the Robots Exclusion Standard for general information on this file.) To lock out the Linguee bot, you can put the following lines into your robots.txt file:
User-agent: Linguee
Disallow: /
This will result in our crawler visiting your site only once, accessing just this one file to check your robots policy. It won't return anytime soon. The crawler will also comply with a crawl rate setting and with exclusion policies given by specific HTML meta tags. Please see the linked article above for more information.

Please note that if the robots.txt file does not mention Linguee, the Linguee bot will follow Googlebot instructions. This is the same behavior that the Applebot exhibits.
2. You can also submit [bot@linguee.com] your internet domain name to request an exclusion from our indexing process. This will prevent our web crawler from visiting your site at all. If your site is a multi-domain site with a large number of domains, we kindly ask you to choose option 1, if possible.

I have found a new site which contains valuable multilingual content that doesn't seem to be included in Linguee's search index. What can I do?

We gladly accept hints and content donations. Just send a mail [bot@linguee.com] with the relevant URL and we will evaluate the site as a candidate. Please be patient with the results, though – it may take a few weeks until you find the text on Linguee.