AdvancedCat-eLog search

YouTube · Post by **Jools** » 22 Nov 2006, 06:48

Mats has been working on this feature and I have uploaded the beta which is now available for use:

http://www.planetcatfish.com/catelog/search.php

At this stage we're looking for feedback and comments from anyone.

Jools

WhitePine · Post by **WhitePine** » 22 Nov 2006, 06:56

One of my favorite plant sites has a similar search feature... and you can search by region... such as africa, central america, asia, se asia... etc...

Here's a link

http://www.aquaticplantcentral.com/foru ... /index.php

MatsP · Post by **MatsP** » 22 Nov 2006, 11:00

I'm well aware that there is no region search, and the basic reason for that is that there is no sensible region data in the database as it is.

The two fields that contain region data are:
Type locality
Distribution

Both of those have no standard format for the content, nor are they necessarily very precise, but for a computer search to work, there's a bigger problem: you need something that the computer can easily classify into one or more regions.

To achieve this, we'd need to add some sort of sensible data (preferrably in numeric form, as computers are much better at "doing stuff" with numbers than they are with text).

I have been thinking about it a little bit, and a "distribution range" would be a field of two coordinates, indicating a rectangle of the geographic distribution as the north/east and south/west corners of the distribution.

From that we could give large and small regions of distritbution, or even a fully-free selection of coordinates to search within.

The main problem is of course for someone to enter this data for 1600+ fishes...

[I've done just under 800 data submissions for the Cat-eLog, and that's just filling in data that is already in existance somewhere else (fishbase, CLOFFSCA, Baensch atlas, etc). To look up the fish's distribution and add the coordinates would take looking at a map and determining which region(s) the fish belongs to. It's going to take a while to do the whole Cat-eLog that way...

Of course, if we go a little bit coarser and have a set number of regions (say europe [all two fishes], north america, central america, northern south america, souther south america, africa, indochina, SE asia, australia/pacific), it would probably be fairly easy to just add a region per genus for most species. But then someone will come back and say "that's not good enough, I want fish from Orinoco basin only" or some such - for most purposes, just selecting a genus (or even family) will indicate which coarse region the fish is from in the first place - there are of course a handfull of famlies that are more spread out than that, but most families are one continent only.

I'm happy to discuss other suggestions...

--
Mats

MatsP · Post by **MatsP** » 22 Nov 2006, 16:47

I just had a play with the plant finder, and whilst it's got some "nice" features (like classifying whether something is easy or difficult to grow, etc), it doesn't allow any combination searches - it allows you to search for a plant from a region, but you'll get ALL plants from that region. Or you can search for those that are difficult to grow, but you get them from ALL regions.

--
Mats

YouTube · Post by **Jools** » 22 Nov 2006, 21:39

I totally think region is a viable option and strongly agree with Mats however that it is a LOT of work. I also like Mats idea of defining a basic continent high level (+ a category for marines). FWIW, I propose it would work like this:

I define a list of drainage basins, each species belongs to one or more basins - think of a basin like a genus. However, we can also add a lower level (species) for a fish that comes from a particular river.

Examples,

Lake Tanganyika
Orinoco
Brahmaputra
Amazon(Upper)
Xingu
Stanley Pool etc.

Having said that, I think fishbase has a (standard?) list of basins.

A nice side effect of this is that each basin can also belong to one our more countries.

I guess this would remove the need for distribution but certainly not type locality which is an instance within distribution.

Perhaps I could find a few days over Christmas to plug this in for 1000+ species.

Jools

Post by **Dinyar** » 23 Nov 2006, 00:41

MatsP wrote:<snip>I have been thinking about it a little bit, and a "distribution range" would be a field of two coordinates, indicating a rectangle of the geographic distribution as the north/east and south/west corners of the distribution.<snip>

One word: mashup

There are whole communities of sites mashing Google, Amazon, eBay, etc. Why reinvent the wheel when you can mash it instead?

Not sure exactly how PFK's Fish Mapper works, but presumably it's pulling "Occurrence" coordinates from FishBase. So assuming PC could get permission from FB to use its data (as PFK already has), the data has already been entered. Don't know how you would "source the data" (APIs, etc), but presumably it's doable.

We all know FB data is not perfect, but no data source is, and I would think lots of interesting mashups could be done by integrating FB and PC data in multi-field database queries.

To my mind the hard question is what kind of geographic data would be most useful to a typical PC user's complex database query. Continent? Country? River basin? Biotope? Biogeographic region? All of the above?

Post by **Shane** » 23 Nov 2006, 08:33

I agree with Jools and believe that drainage (i.e. water shed or basin) would be the most useful and most accurate. Country is meaningless. Is an Orinoco fish Colombian or Venezuelan? Is an Amazonian fish Colombian, Brasilian, or Peruvian. Fish do not understand political borders and rivers are often borders between nations. A subset field would also be useful, e.g. Amazon: Xingu, or Orinoco: Meta, or Incomati: Crocodile.
-Shane

YouTube · Post by **Jools** » 23 Nov 2006, 09:45

Dinyar wrote:Not sure exactly how PFK's Fish Mapper works, but presumably it's pulling "Occurrence" coordinates from FishBase. So assuming PC could get permission from FB to use its data (as PFK already has), the data has already been entered. Don't know how you would "source the data" (APIs, etc), but presumably it's doable.

That's exactly what Rusty was going to do a few months back, I know how to do it maybe I'll get the time in January or thereabouts... I think Mats was also gearing up to look into this.

I know in detail how fish mapper works but it has a lot of limitations. The main one is undescribed species. I've not worked this out, but I strongly feel that if you took the catelog traffic on this site, species by species, and then worked out a percentage of species that were actually described and had fishbase map data, it wouldn't cover a lot.

Maybe I should work that out...

Anyway, the point remains that lots of fishes appear on this site long before fishbase and science more broadly, catches up.

Overall I'd see the mashup augmenting hand cranked data, not replacing it.

Jools

MatsP · Post by **MatsP** » 23 Nov 2006, 10:56

Jools,

How about I work a little bit on a [major drainage][minor drainage] combination data-base to go into the species table?

That would have to be a list tho [or a separate table], so that a fish that occurs in multiple places (say it's occuring both in Orinoco and Rio Negro, for example) can be listed.

I'm not sure what the best way to do this is, but I think it's not too difficult to do...

I'll play with it a bit for a few days, and let you know.

Obviously, meanwhile, keep the thougths and ideas coming.

By the way, I took a quick snapshot of the top100 list of those 58% of the hits are for described species, 42% for undescribed species. But I would also like to add that the top two species are undescribed...

--
Mats

MatsP · Post by **MatsP** » 23 Nov 2006, 13:28

I've played a little with my test-database (that is based on Jools' version), and come up with a system of:

Continent:Drainage:River

For example, we take Hypancistrus Zebra:

Continent: South America
Drainage: Amazon basin
River: Xingu

For another fish, I take a broader example... Clarias batriachus:
Continent: Asia
Drainage: Mekong, Chao Phraya

[Those are just examples, and I'm not SURE about this concept, but it seems like a reasonable solution].

Note that a single species can "occur" in several places, as the table is not restricted to a single occurrence per fish.

--
Mats

YouTube · Post by **Jools** » 24 Nov 2006, 18:13

MatsP wrote:By the way, I took a quick snapshot of the top100 list of those 58% of the hits are for described species, 42% for undescribed species. But I would also like to add that the top two species are undescribed

As they say in politics, there's lies, damn lies and statistics. The thing I was getting as was this: take the top 100 and then count the hits on each species. What % of hits are for undescribed Vs described. Anyway, it's kind of academic as I'm really beginning to think this is the kind of thing we should to to keep ourselves at the top of the game.

Jools

YouTube · Post by **Jools** » 24 Nov 2006, 18:20

MatsP wrote:I've played a little with my test-database (that is based on Jools' version), and come up with a system of:

Continent:Drainage:River

For example, we take Hypancistrus Zebra:

Continent: South America
Drainage: Amazon basin
River: Xingu

For another fish, I take a broader example... Clarias batriachus:
Continent: Asia
Drainage: Mekong, Chao Phraya

[Those are just examples, and I'm not SURE about this concept, but it seems like a reasonable solution].

Note that a single species can "occur" in several places, as the table is not restricted to a single occurrence per fish.

--
Mats

For data presentation, this is fine. Suggest the following:

New table: Continent
New table: Drainage linked to continent (bit worried drainages might span continents in some cases but it's a start).
New table: WaterBody (River, stream, brook etc.) lined to drainage

Bit worried now as how do we define a species that lives in a tributary of the Xingu? Is the Xingu the drainage in that case?

However, if you're thinking in terms of data structure, you're looking at it from the wrong end of the gun.

In terms of representing the fact that a species exists in multiple drainages, what you need is a distribution table which holds just species ID and the WaterBody ID.

So, we're actually say what species live in this bit of water rather than the other way around. This needs a bit more pondering!

Jools

Psy · Post by **Psy** » 25 Nov 2006, 01:33

I'm still getting some odd results. If the genus is unset, species from other families seem to leak in. It happens atleast for searching Pims and Plecos, possibly others.

Example:
If I select loricaides between 5 and 8 inches, since I want a medium size pleco, but do not set genus, since I don't have a preference. No sorting.

50 returns come back, led by Callichthys callichthys, another Calllichthys, and Anapsidoglanis, and a few other non family members.

WhitePine · Post by **WhitePine** » 25 Nov 2006, 02:34

You guys rock

Where you are heading was what I was hinting at when I posted the link from apc. I can't wait to see the results.. and If I can help in any way, please let me know.

YouTube · Post by **Jools** » 26 Nov 2006, 09:25

Psy wrote:I'm still getting some odd results. If the genus is unset, species from other families seem to leak in. It happens atleast for searching Pims and pl*cos, possibly others.

I think this is fixed in the latest version - try it now?

Jools

MatsP · Post by **MatsP** » 27 Nov 2006, 10:35

There was a bug in the previous version where I'd forgotten a "increment condition count" so the family didn't get included in the search if there was any other condition (and in fact, if there wasn't another condition, it would complain that there wasn't ANY).

I just tried with the new version, and it works.

If anyone is seeing problems with the advanced search, please copy and paste the link to the results page [yes, it's usually very long!].

--
Mats

MatsP · Post by **MatsP** » 27 Nov 2006, 15:30

Jools wrote:
MatsP wrote:By the way, I took a quick snapshot of the top100 list of those 58% of the hits are for described species, 42% for undescribed species. But I would also like to add that the top two species are undescribed
As they say in politics, there's lies, damn lies and statistics. The thing I was getting as was this: take the top 100 and then count the hits on each species. What % of hits are for undescribed Vs described. Anyway, it's kind of academic as I'm really beginning to think this is the kind of thing we should to to keep ourselves at the top of the game.

Jools

The thing I was getting as was this: take the top 100 and then count the hits on each species. What % of hits are for undescribed Vs described.
42% undescribed, 58% described, as per my calculations. I can e-mail you the spreadsheet if you want.

Bear in mind, of course, that Planet Catfish's cat-elog by far beats ANY other fish-database I've seen. Ok, so it's not got all the content of Fishbase, but it's got more (for the aquarist at least) useful data. It's well-organized and easily accessible from many perspectives, and it's got a good set of images too...

The lack right now is in searchability on geographic data, and we're missing the actual geographic data in a form that is at least somewhat computer friendly.

--
Mats

MatsP · Post by **MatsP** » 27 Nov 2006, 15:48

Jools wrote:
MatsP wrote:I've played a little with my test-database (that is based on Jools' version), and come up with a system of:

Continent:Drainage:River

For example, we take Hypancistrus Zebra:

Continent: South America
Drainage: Amazon basin
River: Xingu

For another fish, I take a broader example... Clarias batriachus:
Continent: Asia
Drainage: Mekong, Chao Phraya

[Those are just examples, and I'm not SURE about this concept, but it seems like a reasonable solution].

Note that a single species can "occur" in several places, as the table is not restricted to a single occurrence per fish.

--
Mats
For data presentation, this is fine. Suggest the following:

New table: Continent
New table: Drainage linked to continent (bit worried drainages might span continents in some cases but it's a start).
New table: WaterBody (River, stream, brook etc.) lined to drainage

Bit worried now as how do we define a species that lives in a tributary of the Xingu? Is the Xingu the drainage in that case?

However, if you're thinking in terms of data structure, you're looking at it from the wrong end of the gun.

In terms of representing the fact that a species exists in multiple drainages, what you need is a distribution table which holds just species ID and the WaterBody ID.

So, we're actually say what species live in this bit of water rather than the other way around. This needs a bit more pondering!

Jools

I'll mail you my accomplishment so far - it's not using exactly those names, but the concept is the same. Ranaming tables isn't very hard

A tributary to river Xingu is a separate entity in the water-body entries, but it's still falling into (via Xingu) the Amazon, so it would be under the Amazon entry in the drainage.

I'm still open to suggestions on how to solve the actual connection of species -> water body, particularly for those species that are relatively widespread, say most of the Amazon basin.

--
Mats

YouTube · Post by **Jools** » 27 Nov 2006, 21:07

This will need more time than I usually have during the week. Leave it with me for a few days...

Jools

YouTube · Post by **Jools** » 05 Feb 2007, 20:14

Can I move this to "resolved" as we're making some progress with it?

Jools

MatsP · Post by **MatsP** » 06 Feb 2007, 09:28

Yes, I think any "new bugs or suggestions" can go in a new thread. I've moved it (as you can see).

--
Mats