Cron job for search

Currently disabled

The search stopped working today after some modifications I made and I can't get it working again :( I have disabled it, and removed the cron job for now.

I added a search form to the site at lower left. This required me to hook up the 'cron job' which basically is a command which runs at a set time every hour or so to update some things, like the search database. You don't want the server to be indexing the whole site every time somebody does a search for performance reasons, so this does it hourly and then when you search you see the index from that time. To do it, I had to edit the cron-lynx.sh file in the scripts folder and then edit my crontab (crontab -e) to add:

43 * * * * /path/tdb/scripts/cron-lynx.sh

The cron-lynx.sh file actually runs lynx (a text browser) and hits a certain page on the site (this way it's hitting it from the server itself, the only way it will work) which re-indexes the site.

The scripts clean up temp files and the like as well.

Minutes Hell

So I grabbed all the minutes that were available on both the AOL site and the Yahoo site. I ran them through some filters to clean up the code as best as i could, and come to find that I just can't get these pages to display decently at all. Drupal offers two methods of displaying text - the standard 'Filtered HTML' mode (the system is expecting just plain old text, and allows some minimal set of markup:


* Allowed HTML tags: a em strong cite code ul ol li dl dt dd
* Lines and paragraphs break automatically.
* Images can be added to this post.


The trouble is, the pages were never formatted the same way more than about four times, and there is no consistency across them at all for markup. I can't easily fit the various layouts into one single format without rewriting them all. At work, for instance, we have developed a document specification (much like MIL spec has) and all docs are written to meet that spec, right down to fonts used and spacing, etc. The easiest way to enforce this would be to create a template form for entering minutes here on the site, but I wouldn't want to force anyone to type the crap in online, that seems onerous. However the desire to impose some kind of order on the minutes / newsletter chaos is strong in me.

Most of the old minutes are filled with tables, breaks, paragraph marks, Microsoft word classes, ...the list goes on. It seems that the minutes are almost more trouble than they're worth, expect for the fact that they contain vital motions and votes that may someday be relevant to an as-yet unforeseen situation. I wondered briefly if they best idea might not be to create PDFs of all meeting minutes and attach them to a book entry for each year. That way the formatting is preserved and the minutes would not be editable. Problem with this approach is that the consistency of presentation (and subsequent searching of the docs for some piece of info) becomes a pain, though the Sencha can attach a file to a page if given the permissions to do so.

I also tried enabling full HTML to just use the existing HTML docs (which were mostly saved out of Word) with no good result. The minutes from AOL are all inside tables which means they can't go up as is. I manually removed all outer tables (using a progam which works on multiple docs at a time), but still results were lackluster for another reason (below):

Full HTML
* Lines and paragraphs break automatically.

but the presentation is not as expected because while Drupal says it's full HTML, it seems that it inserts line breaks and paragraph marks automatically which means if you have these elements sprinkled throughout your HTML coded, Drupal will add more, making the whole page look too spaced out and crappy.

Another option they give is PHP code (which can be dangerous to allow if you don't know what you're doing). However, this might provide another solution: If the minutes were stored as simple text files on the server some place, I could write a simple PHP include to well, include them. That's probably not an option since the Sencha wouldn't be able to submit them to the site directly.

Unfortunately, preserving some of the more atrocious formatting (like words colored green and red, and various hues of backgroundc colors) is hard to justify. The minutes would need to be reformatted, by hand anyway to get them into a readable state unless I just print them to PDF right off wherever they are living right now (AOL, Word, HTML, etc.). I guess this may be how I proceed since cleaning up the minutes has become a nightmare.

However, given more time, the book outline feature offers a compelling means of accomplishing this goal as well: If the minutes for each meeting were broken down into their components (What News, New Business, etc.) each as a page in the book, then formatting each becomes less of a problem. You would always do a page of a given type the same way, and you could always click 'printer ready version' for a quick printout of all the minutes or some subsection thereof. The Book feature is a very powerful tool and I'd hate to give it up for relatively useless PDFs.

Arrrgh - Maybe I'll try both and see which one is more friendly.

Persona Stories

I moved all of the persona stories from the AOL site over to a section of the library. I removed the persona story from the user profile because most of them were just too damn long for that area, and I wanted to allow an easy way to print them all out if you wanted to. I might add a place to put a link TO your persona story in your profile. The other thing to mention is that the page with your persona story can be AUTHORED by you (or set that way) so your 'avatar' image (user picture) shows up there, but I believe it is also possible to add images to your persona story to spice it up a bit.

We need to talk about permissions on all these pages. Who can edit / add / delete / etc.

Site layout guide

A quick reference quide for the site describing the elements and what they do. This image was created using Photoshop layers, and can be updated if the layout of the site changes.

Another good thing about this guide is that as we move stuff around, we have something to remember what it used to look like :)

Click 'original' for full size image.

Tags

I added the ability to "tag" forum posts (this has always been available on pages but nobody uses that but me). This means you will see a new field called "Topic tags" when you start a new forum post (or edit an old one). You can type freely into that field, and as you do a list should appear with matching tags (if there are any existing ones in the database). If not, you can create new tags on the fly (so be careful that you type in what you want). Separate tags with commas. Assigning a tag will make it appear as a link at the upper right of your post, and any user clicking that link will get ALL posts that have that tag. I went through and tagged all old Newsletters as "news" for instance. That way on any Newsletter you'll see a "news" tag (there may be others assigned to it as well) and clicking it will list ALL Newsletters so tagged. Just a way to make it easier to pull out those things you want. You should be able to tag images in the same way.

This post has been tagged "features" so you can see what I mean (though there aren't any others tagged that way yet)

Upgrades Dec, 2010

Core Upgrade

Took the time today to update the core files to the latest version, 6.20 from 6.15 (yes, there were several releases in-between I never got around to doing). The upgrade went pretty smooth, but if you notice anything not working as expected please let me know. I also upgraded several of the modules we use:

• blocks404 - when you click something that doesn't exist, this takes care of the error and makes it look pretty
• captcha - makes it harder on spammers
• event - provides the calendar and events
• image - provides the image gallery functionality
• img_assist - makes it easier to add a picture to a story
• simple_access - makes it possible to restrict pages to just members
• subscriptions - sends emails when stuff gets updated
• views - customize lists of things (like the whats new page and user listings, etc.)

updated Dec 30:

Feeds

Added a new link along the top called "feeds" which is an RSS feed
aggregator for appropriate Celtic and ancient history news items.
Right now there's only a few feeds (and two from the same place), but
I may add more if I find good ones.

User List

the old /profile list is now gone and redirects to a new /users page so if you are using a bookmark to access that page it will now take you to the updated page. The new users page offers a bit more profile info to logged in users and can be sorted a lot of ways. How it works is pretty well explained on that page so I won't repeat it all here. Check it out by clicking "users" at the top of any page.

OpenID login

You can log in with OpenID now, but this may confuse members who don't know what it is. OpenID providers include Google, Yahoo, etc. and
once you know what your OpenID is you can use it to log in to lots of
sites and not have to remember lots of different usernames and passwords. On our site if you already have an account, you have to "claim" your OpenID (or "pair it" to your existing account) by logging in and going to the "my account" link and adding your OpenID url on the OpenID tab. That way you can log in using your Yahoo or Google credentials and don't have to remember a separate login. When you use OpenID you're not actually logging in to *our* site (and so you don't have to worry about this site getting your credentials or anything), you're redirected back to Yahoo or Google to log in there and our site says, "hey, if they say you're who you say you are thats good enough for us too."

updated Jan 4:
The logs constantly get a foreach() error when running the scheduled "cron" job and finally found a workaround for it here which involves a hack on the taxonomy module. Simple fix, but every upgrade will blow it away.