Category: Search

Searching in spreadsheets is important :-)

I want to be able to filter on field 1 having one condition, OR field 2 having another condition.
For example, in the LIST() below I want to be able to display tracker items whose beginning date are in the next N days, OR the ending date being in the next M days:
{CODE()}
{LIST()}
{filter content="7" field="tracker_id"}
{filter field="tracker_status" content="o"}
{filter range="tracker_field_TaskStartDate" from="now" to="5 days"} <-- But here I'd like to be able to filter not on one field but rather on two fields, something like: tracker_field_TaskStartDate from="now" to="5 days" OR tracker_field_TaskEndDate from="now" to="10 days"
(...)
{LIST}
{CODE}

On a Tiki24.1 I use many trackers

On one of those trackers all the fields permanent name have been modified and an underscore with 4 numbers was added.

{CODE()}
tracker_field_sessionsSelector_7947 => "tokenarfryzdxxetctpfqwtwbfaqfpbufxwdb"
tracker_field_sessionsSelector_7947_text => "68"
tracker_field_paymentsDate_3798 => "2022-07-28"
tracker_field_paymentsFamilyName_7190 => "ספז"
tracker_field_paymentsFamilyName_7190_text => "ספז"
tracker_field_paymentsFamilyName_7190_unstemmed => "ספז"
tracker_field_paymentsStage_4531 => "Banked"
tracker_field_paymentsStage_4531_text => "Banked"
...
{CODE}

However in the tracker the fields displayed the normal permanent name. (tracker_field_sessionsSelector)
I re indexed several times and the problem was still there.

I deleted the table :
index_62e2377698171
index_pref_en
index_pref_he

And reindexed.

Surprise, now in the tracker => fields the permanent name has been changed to "tracker_field_sessionsSelector_7947"

I don't see anything in the tracker settings that could have done this

I added two (2) 'CustomSearch' to a single page in separate sections and separate div id's and different data, the search data and result keeps defaulting to the last CustomSearch on the page. below is the scenario:


{CUSTOMSEARCH}
{CODE(caption="Custom Search 1 & 2:")}{CUSTOMSEARCH( tpl="firstCustomSearch.tpl" id="first_search")}
{filter field="tracker_id" content="2"}
 {output template="templates/first-template.tpl" pagination="y"}   
//output data 1

{CUSTOMSEARCH}

//Some texts and logics seprating them here

//In another section

{CUSTOMSEARCH( tpl="secondCustomSearch.tpl" id="second_search")}
{filter field="tracker_id" content="4"}
 {output template="templates/second-template.tpl" pagination="y"}   
//output data 2

{CUSTOMSEARCH}{CODE}

//Output

Searches in first_search form field defaults to data on 'output data 2' and displays result also in it's custom_search_result div. With this, it skips the data and search on 'output data 1' entirely.

Perhaps, the fact that they share same js 'customsearch' variable picks the last one, not sure.

Also, the inclusion of 'destdiv' parameter with different id's (of course) resulting to respective output data section enters an infinite JQuery loading blocking the data from being populated on the page. While a rename of 'destdiv' parameter to say 'destinationdiv' (undeclared parameter name, yes) populates the display without the blocking but doesn't affect the behavior of Customsearch (of course).

I hope I've been clear enough with this succinct explanation.

Let me know if you would want an elaborate one.

When ''Automatic indexing of file content'' is active, trying to upload pdf files fails.

The pdf file is still uploaded but the end user is told that upload failed.

Try uploading pdfs in [http://nycdwight-11908-6600.show.tikiwiki.org/tiki-upload_file.php?galleryId=4] ! (beware of the 8MB size limit) (verify that ''Automatic indexing of file content'' is still active)

They display an error « File upload failed: SyntaxError: JSON.parse: unexpected character at line 1 column 1 of the JSON data ».
Only for pdfs.

This is an example return answer to /tiki-ajax_services.php?controller=file&action=upload_multiple :

<feff>{"files":[{"size":15295,"name":"document_BBB_02_argentine.pdf","type":"application\/pdf","fileId":"5788","galleryId":"137","md5sum":"7a035db1d92779f8dcdefe083b292d94","info":{"fileId":"5788","galleryId":"137","name":"Document BBB 02 Argentine","description":"","created":"1554159196","filename":"document_BBB_02_argentine.pdf","filesize":"15295","filetype":"application\/pdf","data":null,"user":"Jyhem","author":"","hits":"0","maxhits":null,"lastDownload":null,"votes":null,"points":null,"path":"90792de52961c34118f976ebe4af3a75","reference_url":null,"is_reference":null,"hash":"7a035db1d92779f8dcdefe083b292d94","search_data":"\n","metadata":"","lastModif":"1554159196","lastModifUser":"Jyhem","lockedby":"","comment":"","archiveId":"0","deleteAfter":null},"syntax":"{img fileId=\"5788\" thumb=\"box\"}"},{"size":15741,"name":"document_BBB_01_bresil.pdf","type":"application\/pdf","fileId":"5789","galleryId":"137","md5sum":"50d71753a6487af8c023f28d8df58d24","info":{"fileId":"5789","galleryId":"137","name":"Document BBB 01 Bresil","description":"","created":"1554159196","filename":"document_BBB_01_bresil.pdf","filesize":"15741","filetype":"application\/pdf","data":null,"user":"Jyhem","author":"","hits":"0","maxhits":null,"lastDownload":null,"votes":null,"points":null,"path":"c84d5b2eb5e278b26a0d60ab7bc4c606","reference_url":null,"is_reference":null,"hash":"50d71753a6487af8c023f28d8df58d24","search_data":"\n","metadata":"","lastModif":"1554159196","lastModifUser":"Jyhem","lockedby":"","comment":"","archiveId":"0","deleteAfter":null},"syntax":"{img fileId=\"5789\" thumb=\"box\"}"}]}


The issue seems to be one or more invisible BOM (zero-length nonbreaking spaces) character before the json string.

My guess is, this regression is related to our changing the way pdfs are indexed. Since Tiki18, we use a php library bundled with Tiki. Before, we relied on a command line executable. Sadly, reverting to the old way of putting the following in tiki-admin.php?page=fgal#contentfgal_admin-5 does not work:
|| application/pdf: | pdftotext -enc UTF-8 %1 - ||

I have my wiki set to use MySQL full-text search, but since Friday it seems to be using PHP (Lucene?) search instead -- and failing.

It *was* giving an out-of-memory error. I switched Lucene from incremental indexing to bulk indexing -- which should not affect MySQL search at all. I then tried to build an index, which timed out after 60 seconds. (Not surprising: my wiki has 5000+ pages.)

Now my "full-text" search (using module="search_box") gives 0 results.

{syntax type="tiki" editor="plain"}
On my multilingual Tikis when I access the control panel I often see;

{CODE()}
You have the following unused indexes:

index_pref_en
index_pref_he

If you don't need them (for debugging), run the following command:

php console.php index:cleanup
{CODE}

False warning or... ?

To reproduce:

1) Log in as admin.
2) Go to Groups Admin
3) Assign tiki_p_search permissions to a group with no administrative ("tiki" type) permissions.
4) When page refreshes, you will see that all Admin permissions have been assigned to the group, and the Administrative links have been added to the Application Menu module.
---
Confirmed by dthacker

{THUMB(id=44, url="tiki-browse_image.php?imageId=44")}{THUMB}

A user belonging to this group also gets admin menu after logging in.

At https://doc.tiki.org/Wiki-Argument-Variables we have a list of the available wiki Argument Variables.

We have a permanent variable for pageid and itemId (wiki pages and trackers) and we can use it to dynamically populate plugins for exemple (like the plugin list)

However we have other object types like __files__ for which we don't have a permanent variable.
While I wonder if it is not hidden somewhere, it will be nice to have a fielId variable.

By extension wouldn't be possible to have simply objectId ?

On Tiki25 I can set on a page a List plugin filtering "this" object title - the page title.
Then the results can be used in a wiki template (with other wikiplugins) and if I use ~np~{~/np~{page}} inside the template (or other wikiplugins).
The code will know what page (title) I'm talking about.

On Tiki26 this is not enough and the page (title) is "unkown" or picked randomly based on last list result of all the wiki pages even if there is only ONE results (obviously you can have only one page with a title - duplication is not allowed).

See my solutions.

---

It is on a customized Tiki with a List => Smarty template => Wikiplugin Include AND it work in multilingual.
This to allow a user without page editing permission (test typing only) to edit/translate content of blocks in the page without modifying it.

So I set the importance to low however it may be something we have to look at in the future and other cases.

{mediaplayer src="display2088"}

{syntax type="tiki" editor="plain"}
When attaching a document to a wiki-page, for example an MS-Word file, this attachment seems not to be indexed. There's also no option to indicate which handlers for different mime-types to use. This option is available when using file-galery, but not for page attachments.
As a result the file will never show up in any search results.
Furthermore: it would be convenient to be able to indicate that the indexing of attachments should be logged. This would ease system debugging in the case mentioned.

!!Problems

__Pretty Much Resolved by the wiki ALIAS function__ - so closed.

Redirect plugin
*is not included by default in TikiWiki because it could be used for bad things
*creates redundancy (in search results, page listings, etc) - maybe redirected pages should not appear in list?

Renamed pages
*do not automatically redirect - creates broken links.


!!Proposed solution:

!!!renamed pages
*When a page is renamed the user must choose "hard or soft redirect" perhaps better known as "redirect or refer from old page?"
**hard redirect places redirect plugin on pagename-old to pagename-new.
**soft redirect puts something at top of page like
^This page has been renamed: pagename^


!!!Add synonyms/aliases to a page

suggestion (mlp): adding aliases to a page should automatically create pages with hard or soft redirects to pagename. Note that the adding of alias pages must not destroy data if page already exists.

Ex.:
dev.tikiwiki.org/Tracker
dev.tikiwiki.org/Trackers
dev.tikiwiki.org/Bug tracker

would be aliases. It would avoid the pollution we have here: http://dev.tikiwiki.org/tiki-orphan_pages.php


doc.tikiwiki.org/Install
doc.tikiwiki.org/Installation
doc.tikiwiki.org/Installer

Each wiki page should be able to put one or many aliases. These aliases would work in search. All the aliases should have an important weight in the internal search engine.

We could use this instead of renaming pages. Also, when we do rename a page, we could have an option to have the old page name to be an alias of the new one. Thus, better for external search engines.

Theses aliases could even be used as meta tag for this page.

Putting aliases to non wiki pages (ex.: tiki-forums.php) would make ((doc:structures)) more useful. Now, using structures for site navigation only makes sense if you only have wiki pages. And who ''only'' wants to have wiki pages with all the great features offered by TikiWiki? :-)

It also help to use cleaner page links in sentences. If my Wiki page is called Install, I have to do the following now:

{img src=images/code.png}%%% {CODE()}
For more information about ((Install|Installation))
{CODE}

With aliases, I could do:
{img src=images/code.png}%%% {CODE()}
For more information about ((Installation))
{CODE}

Page aliases - and hard redirects - should permit to set status "Moved Permanently" for robots to send traffic to main page.

What would we do with page renames? (which correct links in wiki pages). Needs some thought. We don't want some unwanted changing of text in existing wiki pages.


Do we need?
Redirect to internal or external http is ok because a special permission is needed to use. -> tiki_p_wiki_alias . In security admin, warm that giving tiki_p_wiki_alias to untrusted people is a security risk.


Related:
[wish1119|Better handling of page renaming]
[wish1610|Redirect plugin : should permit to set status "Moved Permanently"]
[wish1292|Plural WikiWords when using ((WikiWord))]

I have a Tiki22 with a smarty template that use an embedded wikiplugin list.

{CODE()}
<td>{wikiplugin _name=list}
{literal}
{filter type=trackeritem}
{filter content=3 field=tracker_id}
{filter field=tracker_field_parentsIdNumber content={/literal}{$row.parentId}{literal}}
{OUTPUT()}<a href="tel:{display name=tracker_field_parentsMobile}">{display name=tracker_field_parentsMobile}</a> <a href="tel:{display name=tracker_field_parentsMobile}"><i class="fas fa-mobile-alt text-dark"></i></a> <a href="https://wa.me/972{display name=tracker_field_parentsMobile}"><i class="fab fa-whatsapp text-success"></i></a>{OUTPUT}
{/literal}
{/wikiplugin}</td>
{CODE}

If I use mysql search engine I have the expected results.
If I switch to elasticSearch I have no result for the query.

When I rebuilt the index using the console (--log) there are no errors or warning.
{CODE()}
tsaharoniki@server001:~/public_html$ php console.php i:r -p --log
[18-10-2020 13:54] Started rebuilding index... logging to file: /tmp/Search_Indexer_elastic_tsaharoniki_console.log

Unified search
--------------

Engine: Elastic, version 7.9
59 secs/59 secs [============================] -- Rebuilding preferences index
Indexed
trackeritem: 2362
tracker: 23
trackerfield: 192
file: 1363
file gallery: 18
comment: 0
user: 43
group: 6
calendaritem: 26
calendar: 1
wiki page: 46
category: 43
Rebuilding index done
Index: tsaharoniki_main_5f8c1f0180f47

Fallback unified search
------------------------

Engine: MySQL, version 10.3.23-MariaDB-0+deb10u1
Index: index_5f8c1f20f3b21

Execution Statistics
---------------------

Execution time: 58 secs
Current Memory usage: 77.9 MiB
Memory peak usage before indexing: 19.7 MiB
Memory peak usage after indexing: 79.8 MiB
Number of queries: 232317
{CODE}

If I use the html front-end I have a 500.
{img fileId="1455" thumb="box"}
{img fileId="1456" thumb="box"}
{img fileId="1457" thumb="box"}

On a Tiki24 I set a tracker with "Restrict non admins to wiki page access only".

Doing so, the item content is visible through plugins (like list and customSearch) for users with corresponding permissions.

However if I use the Tiki search (tiki-searchindex.php) in the results and in the Search control panel, search Items to display in search results, Default where, Entire site is enable anywone can see parts of the trackeritem content for this tracker.

Tiki Search is not a Wiki page, therefore it shouldn't display the items in a tracker where Restrict non admins to wiki page access only" is enable

Module execution failed if the visitor is anonymous.
Anonymous can't see listpages.php even if they have Wiki admin permissions

This filter is no longer work if the user is anonymous

{CODE()}
{filter field="title" content="~np~{~/np~{page}}"}
{CODE}
Instance coming... (not reproducible with default settings... more tomorrow)

I would like to be able to filter searches by Category ''names''. This would improve usability across a Tiki site.

I'm aware that when search filtering by Category, there is a popup on the search page that allows a user to select a Category (or Categories) which will then appear in the Category filter box in a way the search filters can understand. This is handy, but it would be handier to be able to directly type a Category name (or part of a Category name) into the filter box.

The Category selection box also does not help when composing {LIST()} blocks from PluginList with {filter} blocks. At the moment, {filter} blocks must be composed with prior knowledge of the Category ID number. One must use {filter categories="12 AND 38"} instead of, for example, {filter category-names="Czech Novels of the 1900s AND Polish Novels of the 1900s"}. This makes it harder for people to approach writing wiki pages when they want to include lists of objects in Categories.

I imagine a version of this feature would be easy to write for someone familiar with the codebase, because methods for searching by Category ID have already been built and there is already a function get_category_id($name) in lib/categories/categlib.php.

My users would like a button to turn search highlighting off. Implementation could be as simple as submitting the current URL with the highlighting part stripped off. They would prefer an Ajax implementation but they'll settle for a full page refresh.

Right now, if you have a multitiki installation with 10 sites, and you want to rebuild the unified search index for them all, you have to run 10 times the command with the "--site=" para adapted to each case.

However, when you run setup.sh, by default setup.sh runs for all sites in a multitiki installation.

It would be nice if console.php had the chance (by default or not) to be run once for all sites in that multitiki installation at once


with some param like "--site=all":
{CODE(colors="shell")}
php console.php d:u --site=all
{CODE}

I am using Tiki for educational scenarios, and frequentñy I (or my students) want to change misspelled words (for instance) on long wiki pages.

Nowadays I'm taking the content out of the wiki-edit form, search and replace on in external program, anb back.
I could be nice to have this feature integrated into the edit page form.

In Tiki search results, there should be a "also do the same search with the following search engines:"

http://www.google.com/sitesearch/
http://wordpress.org/extend/plugins/bing-404/


{CODE(caption="This code worked in 6.x in templates")}{if $mid eq 'tiki-searchresults.tpl'}<div class="simplebox" align="center" style="padding:5px;border:2px solid #000000">
<img src="pics/icons/information.png" alt="Information" style="vertical-align:middle;" />
For additional results, try searching for <strong>{$words}</strong> in:
<span class="button"><a href="http://doc.tiki.org/tiki-searchresults.php?words={$words}&where=wiki">Tiki Documentation</a></span> or
<span class="button"><a href="http://www.google.com/search?q=tikiwiki+{$words}">Search all over the Web with Google</a></span> or
<span class="button"><a href="http://www.google.com/search?q=site:tiki.org {$words}">Search *.tiki.org with Google</a></span>
</div>{/if}
{CODE}


Since this will be configurable, we could also encourage people to send the site manager feedback, and thus tell us: what they were looking for and what they were expecting to find

Going to Admin Categories > Bug on dev.tiki.org returns alerts on top of the page:
{CODE()}
ErrorErrorError
Tracker list_items ran out of memory after 0 items.Malformed search query: Parsing search query failed: "org.elasticsearch.common.ParsingException: [_na] query malformed, must start with start_object"Notice: invalid variable value: $_GET["maxRecords"] = undefined
{CODE}

Since Tiki24 we have a new search interface to find a plugin in our LONG list of plugin at : tiki-admin.php?page=textarea#contentadmin_textarea-2

This is a real nice improvement saving resources and space however it is not working at Dev.t.o

{file type="gallery" fileId="1747" showicon="y"}

In the Admin, if you search for "socnets" you will have results but nowhere to go as they are wrongly "not found in an admin panel".

{img fileId="1948" thumb="box"}

{img fileId="1949" thumb="box"}

May be temporary ?

the "advanced" search function (which is the default search these days BTW) does not search through anything but Dutch Roman characters (eg the English American character set)

I have 4 pages with the Alón name on them as an example, searching for Alón or Alon shows no results, BTW the non advanced search works correctly

This is on Tiki 4.2

On a multilingual project this makes the search function totally redundant as far as I am concerned

Tested in v3.1 on XP/IEv6, Vista/IEv7, Vista/Firefox. The help button (blue circle w/ question mark) next to "Advanced Search" in the module search_box does not bring up any help info. Instead, it points to the anchor #advanced_search_help on whatever page the user is viewing, but the anchor doesn't exist. IE shows an "Error on page" in the browser status bar.

Tested in XP/IEv6 with "Floating Help" UI feature on - same result. Unknown if "Floating Help" changes results in IEv7 or FF.