[#CODEBASE-2] Vernacular scripts (utf8) must display properly - Project Blacklight Jira

Details

Type: Bug
Status: Closed
Priority: Blocker
Resolution: Duplicate
Affects Version/s: 2.1
Fix Version/s: 2.1
Component/s: None

Description:

Hide
concerned about display of UTF8 due to Ruby being less UTF8 friendly.

Latest Ruby is supposed to make this easier in some way ...

We know Bob got the searching working (yay bob!) with solrmarc.

It's a deal breaker for us if we don't get the non-latin stuff displaying well. Our librarians have already tested our vufind searchworks for at least the following:

chinese
hebrew
arabic
russian
cyrillic
all sorts of diacritics

right to left issues for hebrew arabic (and whatever else)
- we have "display" fields in our index and can hammer this stuff into submission, but does ruby have any cool built in stuff to do it (or gems or something)

There is already some example data in the solrmarc svn project. It's currently in the 2.0 branch, under test/data

  unicornWHoldings
  hebrew
  diacriticTests
  non-latin <-- I didn't put that there, so I don't know what's in it.

If we need more examples, I can certainly get them. I'm only confident there's already hebrew and chinese in there, and lots of diacritics. I was trying to write tests for the diacritics, but I couldn't get them to pass and fail at the right times.

- Naomi

Show
concerned about display of UTF8 due to Ruby being less UTF8 friendly. Latest Ruby is supposed to make this easier in some way ... We know Bob got the searching working (yay bob!) with solrmarc. It's a deal breaker for us if we don't get the non-latin stuff displaying well. Our librarians have already tested our vufind searchworks for at least the following: chinese hebrew arabic russian cyrillic all sorts of diacritics right to left issues for hebrew arabic (and whatever else) - we have "display" fields in our index and can hammer this stuff into submission, but does ruby have any cool built in stuff to do it (or gems or something) There is already some example data in the solrmarc svn project. It's currently in the 2.0 branch, under test/data   unicornWHoldings   hebrew   diacriticTests   non-latin <-- I didn't put that there, so I don't know what's in it. If we need more examples, I can certainly get them. I'm only confident there's already hebrew and chinese in there, and lots of diacritics. I was trying to write tests for the diacritics, but I couldn't get them to pass and fail at the right times. - Naomi

Issue Links

Duplicate

This issue duplicates:
CODEBASE-45 non-latin script display - Right to Left languages

pre-requisite

This issue has to wait on :
~~CODEBASE-62~~ switch to using solrmarc for out-of-the-box indexing

Activity

Ascending order - Click to sort in descending order

Hide

Permalink

Naomi Dushay added a comment - 13/Feb/09 3:43 PM

Bess is also going to send Stanford some specific examples of vernacular scripts (chinese at least) displaying in a blacklight UI.

Show

Naomi Dushay added a comment - 13/Feb/09 3:43 PM Bess is also going to send Stanford some specific examples of vernacular scripts (chinese at least) displaying in a blacklight UI.

Hide

Permalink

Naomi Dushay added a comment - 13/Feb/09 3:44 PM

careful with "composed" vs "decomposed" chars (Bob will know - I probably have terminology wrong)

Show

Naomi Dushay added a comment - 13/Feb/09 3:44 PM careful with "composed" vs "decomposed" chars (Bob will know - I probably have terminology wrong)

Hide

Permalink

Bess Sadler added a comment - 13/Mar/09 1:17 PM

Can you give us some specific records that it appears are not displaying correctly right now? Via either search works, or Blacklight?

Show

Bess Sadler added a comment - 13/Mar/09 1:17 PM Can you give us some specific records that it appears are not displaying correctly right now? Via either search works, or Blacklight?

Hide

Permalink

Naomi Dushay added a comment - 17/Mar/09 5:00 PM

This should probably be split into a couple of issues:

A. sorting of non-latin scripts. Within some languages (chinese? see below) and interfiled with latin scripts.

B. Right-to-Left vernacular content display.

On Mar 13, 2009, at 11:54 AM, Tom Cramer wrote:

Naomi,

I'm not sure what your citing with "broken UTF-8 display", but wrt supporting vernacular scripts in Unicode in the BL display, there are three separate classes of issues:

1. do the scripts display properly in search results and detailed record pages, especially for entries in right-to-left scripts? It is tricky to display fields that have entries in both roman script and Hebrew/Arabic in an intelligible fashion. Socrates does this now, though it took some work.

This may be an indexing issue. The index needs to have the appropriate fields as "display" fields. In our vufind implementation, I relied on java's BIDI and the ordering of MARC field concatenation. I got "mostly" there, as I recall. Bob Haschart may be able to improve this.

2. Indexing; is the vernacular script data indexed, and accessible for vernacular script search arguments? (I think this is already in place for BL, no?)

UVa needs to included fielded searching (as a pulldown "everything" "title" "author" "subject" "ISBN" ... in the plugin). Naomi needs to index vernacular fields properly and set up the fielded searches properly. Bob already did the work for solrmarc and it's wonderful.

3. Sorting: do the results of UTF-8 searches sort in the appropriate alphanumeric order (whatever that may mean on a language by language basis--in Chinese, e.g., I believe sort order is determined by number of strokes in the first pictographic character, not by the alphabet).

I don't know who should tackle this. Bob? We have a title_sort field ... that doesn't sort correctly. We have a way to map Latin diacritics to plain chars (borrowed from your code), but I'm not sure that's working ... and not sure what to do about non-latin chars.

Having just converted Symphony to Unicode and partnered with Sirsi to get iLink to display and sort results appropriately, we have a solid set of test cases and internal expertise to help get BL up to snuff on these. Lauren would be the key resource in coordinating this testing and feedback from the appropriate language experts.

There is at least one Arabic record with vernacular script in the blacklight demo data. Basically, use the language facet to find Hebrew and Arabic. If you don't have someone to look at Hebrew or Arabic results, we might be able to help.

Show

Naomi Dushay added a comment - 17/Mar/09 5:00 PM This should probably be split into a couple of issues: A. sorting of non-latin scripts. Within some languages (chinese? see below) and interfiled with latin scripts. B. Right-to-Left vernacular content display. On Mar 13, 2009, at 11:54 AM, Tom Cramer wrote: Naomi, I'm not sure what your citing with "broken UTF-8 display", but wrt supporting vernacular scripts in Unicode in the BL display, there are three separate classes of issues: 1. do the scripts display properly in search results and detailed record pages, especially for entries in right-to-left scripts? It is tricky to display fields that have entries in both roman script and Hebrew/Arabic in an intelligible fashion. Socrates does this now, though it took some work. This may be an indexing issue. The index needs to have the appropriate fields as "display" fields. In our vufind implementation, I relied on java's BIDI and the ordering of MARC field concatenation. I got "mostly" there, as I recall. Bob Haschart may be able to improve this. 2. Indexing; is the vernacular script data indexed, and accessible for vernacular script search arguments? (I think this is already in place for BL, no?) UVa needs to included fielded searching (as a pulldown "everything" "title" "author" "subject" "ISBN" ... in the plugin). Naomi needs to index vernacular fields properly and set up the fielded searches properly. Bob already did the work for solrmarc and it's wonderful. 3. Sorting: do the results of UTF-8 searches sort in the appropriate alphanumeric order (whatever that may mean on a language by language basis--in Chinese, e.g., I believe sort order is determined by number of strokes in the first pictographic character, not by the alphabet). I don't know who should tackle this. Bob? We have a title_sort field ... that doesn't sort correctly. We have a way to map Latin diacritics to plain chars (borrowed from your code), but I'm not sure that's working ... and not sure what to do about non-latin chars. Having just converted Symphony to Unicode and partnered with Sirsi to get iLink to display and sort results appropriately, we have a solid set of test cases and internal expertise to help get BL up to snuff on these. Lauren would be the key resource in coordinating this testing and feedback from the appropriate language experts. There is at least one Arabic record with vernacular script in the blacklight demo data. Basically, use the language facet to find Hebrew and Arabic. If you don't have someone to look at Hebrew or Arabic results, we might be able to help.

Hide

Permalink

Naomi Dushay added a comment - 19/Mar/09 8:03 PM

This has been split into two separate issues, so this one is now closed.

Show

Naomi Dushay added a comment - 19/Mar/09 8:03 PM This has been split into two separate issues, so this one is now closed.

Hide

Permalink

Bess Sadler added a comment - 02/Apr/09 8:24 AM

I won't really be able to test this until we can do some more complex indexing, which will be enabled by using solrmarc.

Show

Bess Sadler added a comment - 02/Apr/09 8:24 AM I won't really be able to test this until we can do some more complex indexing, which will be enabled by using solrmarc.

Hide

Permalink

Bess Sadler added a comment - 03/Apr/09 12:28 PM

The problems expressed here are covered separately in other tickets.

Show

Bess Sadler added a comment - 03/Apr/09 12:28 PM The problems expressed here are covered separately in other tickets.

Hide

Permalink

Razan Abbass added a comment - 19/Feb/10 11:08 AM

Show

People

Assignee:

Bess Sadler

Reporter:

Naomi Dushay
Votes:

0

Watchers:

0

Dates

Created:

05/Feb/09 3:08 PM

Updated:

08/May/10 5:40 PM

Resolved:

03/Apr/09 12:28 PM

Blacklight Plugin

Vernacular scripts (utf8) must display properly

Details

Issue Links

Activity

People

Dates